Category Archives: Virtualisation

Hyper-V VM Stuck In Backing Up… State

In this post I’ll show how to resolve the issue of a VM that’s stuck in the “backing up…” state as shown by Hyper-V Manager, without having to reboot the virtual host.

BackingUp0

 

If a VM is stuck in the backing up… state it’s probably due to an error with the Microsoft Hyper-V VSS Writer.  Open an elevated command prompt and run “vssadmin list writers”.  The output should look like it does below, with no errors listed.

BackingUp1

 

The Microsoft Hyper-V VSS Writer runs within the Hyper-V Virtual Machine Management service, so in order to restart the VSS writer and clear the error, you have to restart the Hyper-V Virtual Machine Management service.  I’ve restarted this service without any issues, but please test this on a test server first.

BackingUp2

You can restart the service from the Services mmc, but if the Hyper-V VSS Writer is in an error state the service may hang on shutdown, in this case you’ll have to kill the vmms.exe process from Task Manager.

BackingUp3

When you do this VMs will disappear from Hyper-V Manager, but will reappear when you restart the Hyper-V Virtual Machine Management service.  Following the service restart the VM should no longer by in a backing up… state.

 

SCVMM 2012 R2 – Dynamic Optimization Cannot Be Performed At This Time

When trying to run the Optimize Hosts wizard within SCVMM 2012 R2 I received the error “Dynamic Optimization Cannot Be Performed At This Time” and “Object reference not set to an instance of an object”.

The Application Event Log on the SCVMM server contained a Windows Error Reporting event from the same time.  Opening the event showed a link to the error log.

SCVMMDO0

 

Opening the error log showed that the error was related to a logical network issue on the cluster.  This cluster has a converged network switch to which all virtual machines (VMs) connect.  However, two additional logical networks are mapped to the switch to enable the migration of VMs which were connected to logical networks of a different name on a legacy cluster.

SCVMMDO1

What I found is that some VMs were connected to the “Hyper-V External Access ” logical networks, rather than the ConvergedNetworkSwitch.  Changing the network mapping of the affected VMs to ConvergedNetworkSwitch enabled me to run the Dynamic Optimization wizard.

SCVMMDO2

Unable To Live Migrate A Virtual Machine – “There currently are no network adapters with network optimization available on host”

Having moved a virtual machine (VM) from a Hyper-V cluster where network optimization was available, to a cluster where it isn’t I was unable to live migrate the VM because “There currently are no network adapters with network optimization available on host”.

MigrateVM1

As network optimizations aren’t available on the host cluster, the tick box to disable virtual switch optimizations isn’t available.

MigrateVM2

On the original Hyper-V cluster where network optimization is available you can see a check box to “Enable virtual switch optimizations”.

MigrateVM2a

Luckily, PowerShell can help. Running the command below from the SCVMM PowerShell console lists the properties of the VM’s network adapter.

$VM = Get-SCVirtualMachine -Name “VMNAME”
Get-SCVirtualNetworkAdapter -VM $VM

VMNetworkOptimizationEnabled is set to true.

MigrateVM3

You can use PowerShell to disable VM Network Optimization.

$VM = Get-SCVirtualMachine -Name “VMNAME”
$Adapter = Get-SCVirtualNetworkAdapter -VM $VM
Set-SCVirtualNetworkAdapter -VirtualNetworkAdapter $Adapter -EnableVMNetworkOptimization $false

You can now live migrate the VM.

Improve Virtual Machine Migration Performance

During the migration from a Windows Server 2008 R2 Hyper-V cluster to one based on Windows Server 2012 R2 I have had to migrate a lot of virtual machines.  The migration speed was lower than I expected given the available source and destination disk performance.  On a 1 Gbps network connection, the migration was only using about 25% of the available bandwidth.

In order to improve the performance I disabled TCP Connection Offload (TOE) on the source hosts.  This is because TOE can slow down the VM migration process, which takes place over BITS using HTTPS.  The screenshot below shows the setting in the Broadcom software.  Having disabled TOE the migration used anything up to 80% of the available bandwidth, significantly improving the migration times.

VMMigrationSpeed

 

 

Event ID 17140 & 15030 On Windows Server 2012 R2 Hyper-V

Looking through the Event Logs on a Windows Server 2012 R2 Hyper-V host I could see multiple entries for event IDs 17140 & 15030 as below

17140 – The specified account name is not valid: ‘%USER SID%’.

15030 – ‘%SERVER NAME%’ failed to modify settings. (Virtual machine ID %VM ID%)

These errors can occur if you’re using System Center Virtual Machine Manager (SCVMM).  In the SCVMM console, open Settings -> Security -> User Roles.  Right-click on each user role and choose Properties.  Open the Members tab and look for the user SID you see in event ID 17140.

SCVMM SID

Highlight the SID and click Remove.

Windows Server 2012 R2 Update – Hyper-V Backup Issue

UPDATE 1

Veeam have confirmed that this issue is related to Veeam Backup & Replication.  They’re working on a fix for the general availability (GA) release of the Windows Server 2012 R2 Update.  Follow this link on the Veeam forum for updates.

UPDATE 2

Veeam have released the hotfix and it’s available through support.

UPDATE 3

Veeam have identified a second issue.  Restoring Windows 2008 and earlier backups made on an updated Hyper-V 2012 R2 host fails.  The Veeam KB article is available here 

Having installed the Windows Server 2012 R2 Update, my Veeam Hyper-V backups failed with errors as shown below.

03/04/2014 20:58:28 :: Processing ‘SERVERNAME’ Error: Client error: The system cannot find the file specified.

Failed to open file [\\?\Volume{36db85be-bb6a-11e3-80ce-000af72f0333}\SERVERNAMEWINDOWS SERVER 2012 R2 STANDARD_DISK_1_117A5A0B-5976-4766-B256-3707A2D8CFA7-AutoRecovery.avhdx] in readonly mode.

Veeam Error

The engineer at Veeam (Anatoly) was excellent and was able to prove that the error wasn’t related to Veeam by running Windows backup commands using PowerShell.  Following the uninstall of KB 2919355, the only one of the six hot fixes comprising the Windows Server 2012 R2 Update listed in Add Remove Programs, I was able to run backups as before.  If you’re thinking of installing this update ensure you test it well.

For reference the configuration is:

  • 4x Hyper-V 2012 R2 hosts
  • Dell EqualLogic SAN 6.0.7
  • HIT Kit 4.7 EPA
  • Veeam Backup & Replication 7.0.0.839

Hyper-V Virtual Machine Connectivity Issue

Following an issue with the storage used by our Hyper-V cluster, one node in our five node cluster became partially unresponsive.  The virtual machines (VMs) running on the unresponsive node were automatically moved to other cluster nodes and service was resumed withing a couple of minutes.  At first everything appeared to be fine, but within a few minutes our monitoring system started to report connectivity issues to the VMs that had failed over.

I RDP’d onto one of the VMs that was having connectivity issues, but found the connection kept dropping out, so I connected to the console through System Center Virtual Machine Manager (SCVMM).  I found I was unable to ping any server on the physical network.  I took a look at the event log on one of the virtual hosts and saw the error below:

Port ‘BF392932-9AE4-453A-8E13-26671BB556D9′ was prevented from using MAC address ’00-14-22-18-7F-DC’ because it is pinned to port ‘SCVMM-C26227E3-D6AB-4818-B8BF-4CCF923C’.

The error message implied another VM was using the MAC of the VM that was having connectivity issues.  As the VM had a dynamic MAC that was managed by SCVMM I knew that couldn’t be the case.  I decided to reboot the unresponsive cluster node.  After waiting 30 minutes for the node to shutdown I killed the power via a DRAC.  As soon as I killed the power to the node the MAC address errors in the event log disappeared and all the VMs resumed normal connectivity.  I believe the cluster node that became unresponsive was keeping some kind of lock on the MAC addresses of the VMs that were running on the node when it became unresponsive.  Killing the power to the node freed the locks enabling connectivity to resume.