Category Archives: Backup

Hyper-V VM Stuck In Backing Up… State

In this post I’ll show how to resolve the issue of a VM that’s stuck in the “backing up…” state as shown by Hyper-V Manager, without having to reboot the virtual host.

BackingUp0

 

If a VM is stuck in the backing up… state it’s probably due to an error with the Microsoft Hyper-V VSS Writer.  Open an elevated command prompt and run “vssadmin list writers”.  The output should look like it does below, with no errors listed.

BackingUp1

 

The Microsoft Hyper-V VSS Writer runs within the Hyper-V Virtual Machine Management service, so in order to restart the VSS writer and clear the error, you have to restart the Hyper-V Virtual Machine Management service.  I’ve restarted this service without any issues, but please test this on a test server first.

BackingUp2

You can restart the service from the Services mmc, but if the Hyper-V VSS Writer is in an error state the service may hang on shutdown, in this case you’ll have to kill the vmms.exe process from Task Manager.

BackingUp3

When you do this VMs will disappear from Hyper-V Manager, but will reappear when you restart the Hyper-V Virtual Machine Management service.  Following the service restart the VM should no longer by in a backing up… state.

 

Resolving SharePoint 2013 VSS Errors With Veeam And SCDPM

Backing up a virtual SharePoint 2013 environment using Veeam and MS System Center Data Protection Manager 2012 SP1 (SCDPM) failed with VSS errors.  Veeam is being used to create VM image based backups and SCDPM is being used to provide SharePoint backups with item level recovery.

The Veeam backup failed with:

Unable to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing). Details: Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to create VSS snapshot. –tr:Failed to perform pre-backup tasks.
Make sure VM does not have ‘iSCSI Software Target Storage Provider’ feature installed.

The SCDPM backup ran, but logged the warning:

Backup metadata enumeration failed

DPM could not obtain backup metadata information for SharePoint Farm Sharepoint Farm\%SQL_SERVER%\SharePoint_Config on %SHAREPOINT_SERVER%. If the data source is a SharePoint farm then a valid recovery point has been created. However, content databases from this recovery point can be recovered using the alternate location option only. (ID 3134)

The SharePoint web front end server had two VSS errors in the Windows Application Event Log:

Event ID 8194

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
PrepareForSnapshot Event

Context:
Execution Context: Writer
Writer Class Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Name: SharePoint Services Writer
Writer Instance ID: {e3f3a187-6600-4750-b96d-b5579dc09e9d}

Event ID 8193

Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,…). hr = 0x80070005, Access is denied.
.

Operation:
Initializing Writer

Context:
Writer Class Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Name: SharePoint Services Writer
Writer Instance ID: {4d6e7cca-8810-41f2-a56f-ac87156c426f}

One of two SharePoint servers running SharePoint Search had one VSS error in the Windows Application Event Log:

Event ID 8213

Volume Shadow Copy Service error: The process that hosts the writer with name OSearch15 VSS Writer and ID {0ff1ce15-0201-0000-0000-000000000000} does not run under a user with sufficient access rights. Consider running this process under a local account which is either Local System, Administrator, Network Service, or Local Service.

Operation:
Initializing Writer

Context:
Writer Class Id: {0ff1ce15-0201-0000-0000-000000000000}
Writer Name: OSearch15 VSS Writer
Writer Instance Name: OSearch15 Replication Service

When configuring SCDPM SharePoint protection you have to run the ConfigureSharePoint.exe command, see here for more details.  You provide this command with the SharePoint Farm Administrator account and this account is not normally an Windows Administrator on the server.  Event ID 8193 indicated that permissions were an issue.

In order to resolve the VSS errors, open registry editor on the web front end and search servers.  Navigate to HKLM\SYSTEM\CurrentControlSet\Services\VSS\Diag right-click on the key and choose permissions.

On the web front end server I granted the SharePoint Farm Administrator account full control to the key.

On both SharePoint Search servers I granted the SharePoint Search and the SharePoint Farm Administrator accounts full control to the key.  On one of the search servers under HKLM\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl, both the SharePoint Farm Administrator account and the the SharePoint Search were listed, but on the other only the SharePoint Search account was listed.  I added a REG_DWORD key under HKLM\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl and set the value to 1 for the SharePoint Farm Administrator account.

Following a reboot of the servers the backups worked correctly in both Veeam and SCDPM and the Event Log errors disappeared.

Veeam Tape Backup Fails With Unable to start tape backup session: StartSessionTape failed

Following an upgrade to Veeam Backup & Replication 7.0 Patch 4, tape backups failed with the error “Unable to start tape backup session: StartSessionTape failed”.

In order to get the backups working again I upgraded the firmware on the tape library, which simultaneously upgraded the firmware on the tape drives.  I then upgraded the drivers for the tape library and the tape drives.  Following these steps the tape jobs ran successfully again.

Windows Server 2012 R2 Update – Hyper-V Backup Issue

UPDATE 1

Veeam have confirmed that this issue is related to Veeam Backup & Replication.  They’re working on a fix for the general availability (GA) release of the Windows Server 2012 R2 Update.  Follow this link on the Veeam forum for updates.

UPDATE 2

Veeam have released the hotfix and it’s available through support.

UPDATE 3

Veeam have identified a second issue.  Restoring Windows 2008 and earlier backups made on an updated Hyper-V 2012 R2 host fails.  The Veeam KB article is available here 

Having installed the Windows Server 2012 R2 Update, my Veeam Hyper-V backups failed with errors as shown below.

03/04/2014 20:58:28 :: Processing ‘SERVERNAME’ Error: Client error: The system cannot find the file specified.

Failed to open file [\\?\Volume{36db85be-bb6a-11e3-80ce-000af72f0333}\SERVERNAMEWINDOWS SERVER 2012 R2 STANDARD_DISK_1_117A5A0B-5976-4766-B256-3707A2D8CFA7-AutoRecovery.avhdx] in readonly mode.

Veeam Error

The engineer at Veeam (Anatoly) was excellent and was able to prove that the error wasn’t related to Veeam by running Windows backup commands using PowerShell.  Following the uninstall of KB 2919355, the only one of the six hot fixes comprising the Windows Server 2012 R2 Update listed in Add Remove Programs, I was able to run backups as before.  If you’re thinking of installing this update ensure you test it well.

For reference the configuration is:

  • 4x Hyper-V 2012 R2 hosts
  • Dell EqualLogic SAN 6.0.7
  • HIT Kit 4.7 EPA
  • Veeam Backup & Replication 7.0.0.839

Item Level Recovery Of SharePoint 2010 Using System Center Data Protection Manager 2010

In order to carry out item level recovery of SharePoint 2010 using System Center Data Protection Manager 2010 (DPM) you may need to update the DPM catalog.

I was testing SharePoint 2010 recovery using DPM and got a shock when a site and page I setup for the testing weren’t listed as being available for recovery despite the fact I’d just created a recovery point.  It turns out they had been backed up, but the DPM catalog was out of date.  The steps below detail how to update the catalog.

Open the DPM Management Shell and enter the commands below, substituting X for you DPM server name

$PG = Get-ProtectionGroup -DPMServerName X

Press Enter, type $PG and press Enter again and you should see a table with two columns.

Type $DS = Get-Datasource $PG[X] and press Enter.  Substitute X with the number of the datasource in the list above.  In my case I want to update the catalog for the “SharePoint 2010” datasource, which is third in the list, so I enter [2] as the list is zero based.   Type $DS and press Enter.

The output above shows the details for the selected datasource.  I want to update the catalog for the Sharepoint Farm, so type the command below and press Enter.  $DS[0] is the value of the datasource item within the list.  This list is also zero based.

Start-CreateCatalog -Datasource $DS[0]

Allow a few minutes for the catalog to update and the items will be available for recovery.

 

 

Remove SharePoint 2007 Backups

If you take a backup from Central Administration -> Operations -> Perform Backup, there’s no built-in way to remove old backups.  To clear up unwanted backups:

  • Navigate to the backup share and take a backup of spbrtoc.xml.  This file contains information about the backups.
  • Within spbrtoc.xml each SPHistoryObject node  corresponds to a backup.  Take a note of the SPDirectoryName and SPStartTime values within each SPHistoryObject  node.  SPDirectoryName is the directory the backup was stored and SPStartTime is the start time of the backup.
  • Delete the directories of the backups you wish to remove and delete the corresponding SPHistoryObject nodes from spbrtoc.xml.
  • Navigate to  Central Administration -> Operations -> Backup and restore history and the delete backups shouldn’t be visible.

ID 30101 Details: The request could not be performed because of an I/O device error (0x8007045D)

On System Center Data Protection Manager (DPM) 2010 with March 2011 hotfix I got the following error when backing up to tape:

ID 30101 Details: The request could not be performed because of an I/O device error (0x8007045D)

This is an error that was originally fixed in DPM 2007 with a hotfix http://support.microsoft.com/kb/970868 , but reappeared on my 2010 install.  It can occur with large tape backups that span more than one tape.  I fixed the error by adding the following registry key, a reboot wasn’t necessary:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent]
"BufferQueueSize"=dword:00000001

[UPDATE]

I’ve had the same problem with DPM 2012 SP1 and the registry still fix worked.