Category Archives: Backup

The volume shadow copy provider is not registered in the system

1 Reply

On a Windows Server 2012 R2 VM protected by Veeam Backup & Replication, Veeam’s application aware processing was failing with the error below.

Retrying snapshot creation attempt (Could not create backup checkpoint for virtual machine ‘%VMNAME%’: The volume shadow copy provider is not registered in the system. (0x80042304). (Virtual machine ID 464FEFAA-0EF5-4E9F-98BF-D76EBE72717B))
Failed to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing with failover) Details: Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.
Retrying snapshot creation attempt (Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.)
Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

The VM is also protected by Azure Site Recovery/Hyper-V Replica and the virtual host running the VM logged a “The volume shadow copy provider is not registered in the system” error each time an application consistent recovery point was attempted.

I checked the registry key HKLM\SYSTEM\CurrentControlSet\Services\VSS\Providers and noticed one of the providers ({74600e39-7dc5-4567-a03b-f091d6c7b092}) was missing a CLSID key.

Reg1

I found another VM running Windows Server 2012 R2 and navigated to the same registry key.  In this case the CLSID key was populated.  I copied the CLSID key from the working system and applied to the VM with the issue.

Reg2

Next I restarted the Hyper-V Volume Shadow Copy Requestor on the VM and found that the VSS error was resolved.

Hyper-V VM Stuck In Backing Up… State

24 Replies

In this post I’ll show how to resolve the issue of a VM that’s stuck in the “backing up…” state as shown by Hyper-V Manager, without having to reboot the virtual host.

BackingUp0

 

If a VM is stuck in the backing up… state it’s probably due to an error with the Microsoft Hyper-V VSS Writer.  Open an elevated command prompt and run “vssadmin list writers”.  The output should look like it does below, with no errors listed.

BackingUp1

 

The Microsoft Hyper-V VSS Writer runs within the Hyper-V Virtual Machine Management service, so in order to restart the VSS writer and clear the error, you have to restart the Hyper-V Virtual Machine Management service.  I’ve restarted this service without any issues, but please test this on a test server first.

BackingUp2

You can restart the service from the Services mmc, but if the Hyper-V VSS Writer is in an error state the service may hang on shutdown, in this case you’ll have to kill the vmms.exe process from Task Manager.

BackingUp3

When you do this VMs will disappear from Hyper-V Manager, but will reappear when you restart the Hyper-V Virtual Machine Management service.  Following the service restart the VM should no longer by in a backing up… state.

 

Resolving SharePoint 2013 VSS Errors With Veeam And SCDPM

2 Replies

Backing up a virtual SharePoint 2013 environment using Veeam and MS System Center Data Protection Manager 2012 SP1 (SCDPM) failed with VSS errors.  Veeam is being used to create VM image based backups and SCDPM is being used to provide SharePoint backups with item level recovery.

The Veeam backup failed with:

Unable to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing). Details: Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to create VSS snapshot. –tr:Failed to perform pre-backup tasks.
Make sure VM does not have ‘iSCSI Software Target Storage Provider’ feature installed.

The SCDPM backup ran, but logged the warning:

Backup metadata enumeration failed

DPM could not obtain backup metadata information for SharePoint Farm Sharepoint Farm\%SQL_SERVER%\SharePoint_Config on %SHAREPOINT_SERVER%. If the data source is a SharePoint farm then a valid recovery point has been created. However, content databases from this recovery point can be recovered using the alternate location option only. (ID 3134)

The SharePoint web front end server had two VSS errors in the Windows Application Event Log:

Event ID 8194

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
PrepareForSnapshot Event

Context:
Execution Context: Writer
Writer Class Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Name: SharePoint Services Writer
Writer Instance ID: {e3f3a187-6600-4750-b96d-b5579dc09e9d}

Event ID 8193

Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,…). hr = 0x80070005, Access is denied.
.

Operation:
Initializing Writer

Context:
Writer Class Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Name: SharePoint Services Writer
Writer Instance ID: {4d6e7cca-8810-41f2-a56f-ac87156c426f}

One of two SharePoint servers running SharePoint Search had one VSS error in the Windows Application Event Log:

Event ID 8213

Volume Shadow Copy Service error: The process that hosts the writer with name OSearch15 VSS Writer and ID {0ff1ce15-0201-0000-0000-000000000000} does not run under a user with sufficient access rights. Consider running this process under a local account which is either Local System, Administrator, Network Service, or Local Service.

Operation:
Initializing Writer

Context:
Writer Class Id: {0ff1ce15-0201-0000-0000-000000000000}
Writer Name: OSearch15 VSS Writer
Writer Instance Name: OSearch15 Replication Service

When configuring SCDPM SharePoint protection you have to run the ConfigureSharePoint.exe command, see here for more details.  You provide this command with the SharePoint Farm Administrator account and this account is not normally an Windows Administrator on the server.  Event ID 8193 indicated that permissions were an issue.

In order to resolve the VSS errors, open registry editor on the web front end and search servers.  Navigate to HKLM\SYSTEM\CurrentControlSet\Services\VSS\Diag right-click on the key and choose permissions.

On the web front end server I granted the SharePoint Farm Administrator account full control to the key.

On both SharePoint Search servers I granted the SharePoint Search and the SharePoint Farm Administrator accounts full control to the key.  On one of the search servers under HKLM\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl, both the SharePoint Farm Administrator account and the the SharePoint Search were listed, but on the other only the SharePoint Search account was listed.  I added a REG_DWORD key under HKLM\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl and set the value to 1 for the SharePoint Farm Administrator account.

Following a reboot of the servers the backups worked correctly in both Veeam and SCDPM and the Event Log errors disappeared.

Veeam Tape Backup Fails With Unable to start tape backup session: StartSessionTape failed

Leave a reply

Following an upgrade to Veeam Backup & Replication 7.0 Patch 4, tape backups failed with the error “Unable to start tape backup session: StartSessionTape failed”.

In order to get the backups working again I upgraded the firmware on the tape library, which simultaneously upgraded the firmware on the tape drives.  I then upgraded the drivers for the tape library and the tape drives.  Following these steps the tape jobs ran successfully again.

Windows Server 2012 R2 Update – Hyper-V Backup Issue

1 Reply

UPDATE 1

Veeam have confirmed that this issue is related to Veeam Backup & Replication.  They’re working on a fix for the general availability (GA) release of the Windows Server 2012 R2 Update.  Follow this link on the Veeam forum for updates.

UPDATE 2

Veeam have released the hotfix and it’s available through support.

UPDATE 3

Veeam have identified a second issue.  Restoring Windows 2008 and earlier backups made on an updated Hyper-V 2012 R2 host fails.  The Veeam KB article is available here 

Having installed the Windows Server 2012 R2 Update, my Veeam Hyper-V backups failed with errors as shown below.

03/04/2014 20:58:28 :: Processing ‘SERVERNAME’ Error: Client error: The system cannot find the file specified.

Failed to open file [\\?\Volume{36db85be-bb6a-11e3-80ce-000af72f0333}\SERVERNAMEWINDOWS SERVER 2012 R2 STANDARD_DISK_1_117A5A0B-5976-4766-B256-3707A2D8CFA7-AutoRecovery.avhdx] in readonly mode.

Veeam Error

The engineer at Veeam (Anatoly) was excellent and was able to prove that the error wasn’t related to Veeam by running Windows backup commands using PowerShell.  Following the uninstall of KB 2919355, the only one of the six hot fixes comprising the Windows Server 2012 R2 Update listed in Add Remove Programs, I was able to run backups as before.  If you’re thinking of installing this update ensure you test it well.

For reference the configuration is:

  • 4x Hyper-V 2012 R2 hosts
  • Dell EqualLogic SAN 6.0.7
  • HIT Kit 4.7 EPA
  • Veeam Backup & Replication 7.0.0.839

Item Level Recovery Of SharePoint 2010 Using System Center Data Protection Manager 2010

In order to carry out item level recovery of SharePoint 2010 using System Center Data Protection Manager 2010 (DPM) you may need to update the DPM catalog.

I was testing SharePoint 2010 recovery using DPM and got a shock when a site and page I setup for the testing weren’t listed as being available for recovery despite the fact I’d just created a recovery point.  It turns out they had been backed up, but the DPM catalog was out of date.  The steps below detail how to update the catalog.

Open the DPM Management Shell and enter the commands below, substituting X for you DPM server name

$PG = Get-ProtectionGroup -DPMServerName X

Press Enter, type $PG and press Enter again and you should see a table with two columns.

Type $DS = Get-Datasource $PG[X] and press Enter.  Substitute X with the number of the datasource in the list above.  In my case I want to update the catalog for the “SharePoint 2010” datasource, which is third in the list, so I enter [2] as the list is zero based.   Type $DS and press Enter.

The output above shows the details for the selected datasource.  I want to update the catalog for the Sharepoint Farm, so type the command below and press Enter.  $DS[0] is the value of the datasource item within the list.  This list is also zero based.

Start-CreateCatalog -Datasource $DS[0]

Allow a few minutes for the catalog to update and the items will be available for recovery.

 

 

Remove SharePoint 2007 Backups

If you take a backup from Central Administration -> Operations -> Perform Backup, there’s no built-in way to remove old backups.  To clear up unwanted backups:

  • Navigate to the backup share and take a backup of spbrtoc.xml.  This file contains information about the backups.
  • Within spbrtoc.xml each SPHistoryObject node  corresponds to a backup.  Take a note of the SPDirectoryName and SPStartTime values within each SPHistoryObject  node.  SPDirectoryName is the directory the backup was stored and SPStartTime is the start time of the backup.
  • Delete the directories of the backups you wish to remove and delete the corresponding SPHistoryObject nodes from spbrtoc.xml.
  • Navigate to  Central Administration -> Operations -> Backup and restore history and the delete backups shouldn’t be visible.

ID 30101 Details: The request could not be performed because of an I/O device error (0x8007045D)

Leave a reply

On System Center Data Protection Manager (DPM) 2010 with March 2011 hotfix I got the following error when backing up to tape:

ID 30101 Details: The request could not be performed because of an I/O device error (0x8007045D)

This is an error that was originally fixed in DPM 2007 with a hotfix http://support.microsoft.com/kb/970868 , but reappeared on my 2010 install.  It can occur with large tape backups that span more than one tape.  I fixed the error by adding the following registry key, a reboot wasn’t necessary:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent]
"BufferQueueSize"=dword:00000001

[UPDATE]

I’ve had the same problem with DPM 2012 SP1 and the registry still fix worked.

System Center Data Protection Manager 2010 Doesn’t Detect Tape Drives In Library

[UPDATE – June 2013]

I’ve just encountered the same issue using Windows Server 2012, System Center Data Protection Manager 2012 SP1 and an Overland REO 4600 VTL.  As with DPM 2007, the fix is to change the tape drive driver to the generic Microsoft LTO tape driver.

I’ve used System Center Data Protection Manager (DPM) since the 2007 beta, primarily for SharePoint and SQL backups.  I have a primary DPM server which backs up to disk and a secondary server which backs up to disk and additionally to a Overland Storage REO 4500 iSCSI Virtual Tape Library (VTL) for long term protection.  Both servers have run Windows Server 2008 R2 with DPM 2010 since its release.

I’ve never had any major issues with DPM until I recently rebuilt the secondary server from scratch with a clean OS install of Windows Server 2008 R2 SP1 and DPM 2010.  What I found was that although DPM installed correctly it couldn’t see the tape drives in the VTL.  All the tape drives and the medium changer were visible in Device Manager and none were showing errors.  The medium changer and driver were automatically installed by Windows and the tape drive driver came from Windows Update.  The REO 4500 emulates an HP LTO2 tape drive and this is what was displayed in Device Manager.

In the DPM console under Management -> Libraries, the library was listed, but Total Drives was 0, rather than the expected 16.  The DPM console showed the following error:

Tape Library Overland Neo Series Medium Changers is not functioning efficiently for the following reason:

The drive information for Tape Library Overland Neo Series Medium Changers is incorrect and needs to be refreshed. (ID: 3302)

I followed the instructions to remap the drives here: http://technet.microsoft.com/en-us/library/bb795782.aspx.  The DPMLA.xml file produced by the tool was correct, showing the correct SCSI IDs, serial numbers etc. but DPM still couldn’t see the drives.

I found the solution to the problem on the DPM forum here: http://social.technet.microsoft.com/Forums/en-US/dpmtapebackuprecovery/thread/1d599443-7bf6-437a-bf12-52847fa7c8e5/ What I did is update the tape drive driver to the Inbox LTO driver as described on the forum and below:

1) Open device manager
2) Locate the tape drive
3) Right-click and look at the properties.
3) Under the DRIVER tab, select UPDATE DRIVER
a) Select the Install from list or specific location (Advanvced) – next.
b) Select Don’t search. I will choose the driver to install. – next.
c) Uncheck the Show comtabile hardware checkbox.
d) Highlight LTO under the manufacturer.
e) Highlight the LTO tape drive under model – then next.
f) This should install the Microsoft ltotape.sys driver.
4) Rescan the tape library in the DPM console – try to take another backup.

After following the above steps, all 16 tape drives were visible in the  DPM console.

Backup Exec and Autonomy IDOL

Autonomy IDOL enables you to scheduled a backup of its index to local disk.  I wanted to use Backup Exec to backup the Autonomy backup.  The Backup Exec agent was installed, but the service was refusing to start.  By default the Backup Exec agent service communicates on port 10,000, which was being used by IDOL.  Changing the Backup Exec agent port enabled the service to start.  See here for details: http://www.symantec.com/business/support/index?page=content&id=TECH24256