Author Archives: richardstk

Azure VM Agent Status – Not Ready

In a secure sandbox environment withing Azure I encountered and issue whereby the Azure VM Agent status was reporting as not ready and the agent version as unknown.

Agent0

Opening the VM Agent log in C:\WindowsAzure\Logs\WaAppAgent.log I saw the error:

[ERROR] GetVersions() failed with exception: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://168.63.129.16/?comp=versions that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 168.63.129.16:80
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
— End of inner exception stack trace —
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
— End of inner exception stack trace —

Research showed that the IP address 168.63.129.16 is a virtual IP address used for Azure VM Agent communication amongst other things.  See here for more detail.

In this case traffic was flowing through an Azure security appliance and a rule needed to be created to allow traffic to and from 168.63.129.16.  Once this was done the VM Agent functioned correctly.

Advertisements

The volume shadow copy provider is not registered in the system

On a Windows Server 2012 R2 VM protected by Veeam Backup & Replication, Veeam’s application aware processing was failing with the error below.

Retrying snapshot creation attempt (Could not create backup checkpoint for virtual machine ‘%VMNAME%’: The volume shadow copy provider is not registered in the system. (0x80042304). (Virtual machine ID 464FEFAA-0EF5-4E9F-98BF-D76EBE72717B))
Failed to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing with failover) Details: Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.
Retrying snapshot creation attempt (Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.)
Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

The VM is also protected by Azure Site Recovery/Hyper-V Replica and the virtual host running the VM logged a “The volume shadow copy provider is not registered in the system” error each time an application consistent recovery point was attempted.

I checked the registry key HKLM\SYSTEM\CurrentControlSet\Services\VSS\Providers and noticed one of the providers ({74600e39-7dc5-4567-a03b-f091d6c7b092}) was missing a CLSID key.

Reg1

I found another VM running Windows Server 2012 R2 and navigated to the same registry key.  In this case the CLSID key was populated.  I copied the CLSID key from the working system and applied to the VM with the issue.

Reg2

Next I restarted the Hyper-V Volume Shadow Copy Requestor on the VM and found that the VSS error was resolved.

Server 2012 R2 – VSSAdmin Waiting for response & VSSAdmin List Writers is empty

On a Windows Server 2012 R2 Hyper-V cluster I had an issue whereby backups were failing with VSS errors.

When running VSSAdmin List Writers I received the response: “Waiting for responses. These may be delayed if a shadow copy is being prepared.”

VSS0

In order to resolve this I restarted the COM+ Event System service.  Following this VSSAdmin List Writers failed to return anything.

VSS1

When restarting the COM+ Event System service there are a number of dependant services.  I noticed that the COM+ System Application service hadn’t automatically restarted.  I started the service, then restarted the services below.  Following this VSSAdmin List Writers returned the expected list of writers.

  • EqualLogic VSS Requestor – Specific to my use of EqualLogic storage
  • EqualLogic VSS Service – Specific to my use of EqualLogic storage
  • Hyper-V Virtual Machine Management
  • Microsoft Software Shadow Copy Provider
  • Volume Shadow Copy

Disabling WPAD on Windows Server 2012 R2 for SCDPM, Azure Backup Server and Azure Site Recovery

When using SCDPM 2016 with an Azure Recovery Services Vault I needed to bypass the web proxy for data transfer from SCDPM to the Azure Recovery Services Vault.  I also needed to bypass the proxy for Azure Site Recovery replication traffic on some Hyper-V hosts.

In the case of SCDPM I used the Configure option under Online Protection in the DPM Administration Console to disable the proxy.

I also checked the CBSettings.xml file for proxy information in the location scratch location, which can be found in the reg key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Azure Backup\Config.  As per this KB.

In the case of Azure Site Recovery I ran the Azure Site Recovery Configurator and selected to bypass the proxy server.

In the case of both SCDPM and Azure Site Recovery the replication traffic continued to flow via the web proxy. I removed the DNS suffix used to find the web proxy via WPAD and used the FindProxyForURL toolset to check that WPAD wouldn’t find the proxy.  However, the proxy continued to be used.  In the end I performed the following steps to prevent WPAD from working.

  • Remove the DNS suffix used to find WPAD from the network adapter
  • Ping WPAD to verify the host cannot be found
  • Delete the cached WPAD files from C:\Windows\ServiceProfiles\LocalService\winhttp
  • Delete DefaultConnectionSettings and SavedLegacySettings from HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
  • Delete the sub key under HKEY_USERS\S-1-5-19\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Wpad
  • Restart the WinHTTP Web Proxy Auto-Discovery Service

These are some other useful links:

 

Remotely disable Network Level Authentication (NLA)

If you try to RDP to a machine, but can’t because you receive the error below, you can use PSExec to remotely disable the requirement for NLA.

“The remote computer that you are trying to connect to requires Network Level Authentication (NLA), but your Windows domain controller cannot be contacted to perform NLA.  If you are an administrator on the remote computer, you can disable NLA by using the options on the Remote tab of the System Properties dialog box.”

Download PSExec from TechNet.  Run the code below updating the following values.

\\VMNAME – The name of the machine on which you want to disable NLA

VMNAME\ADMIN_ACCOUNT – The username of a local administrator on the machine on which you want to disable NLA, e.g. pc1\admin

psexec \\VMNAME -u VMNAME\ADMIN_ACCOUNT -p PASSWORD reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp” /f /v SecurityLayer /t REG_DWORD /d 0

 

 

 

DPM services fail to start following an upgrade to Windows Server 2016

I had SCDPM 2012 R2 running on Windows Server 2012 R2 with SQL Server 2012.  I followed the instructions here to upgrade to SCDPM 2016 UR 2 running on Windows Server 2016 with SQL Server 2016.

Everything was working fine until the final step to upgrade from Windows Server 2012 R2 to Windows Server 2016.  Following this the DPM services failed to start and the Event Log was full of errors like the below:

System.Exception: -2147023107

ATL Failure in Initializing Security of msdpm

Microsoft.Internal.EnterpriseStorage.Dls.Utils.DlsException: exception —> System.Runtime.InteropServices.COMException: Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE))

The resolution was to change the Netlogon service from Manual to Automatic.  Following this change and a reboot everything started working again.

 

SCCM stuck in downloading state

The 1610 update for SCCM had been stuck in the downloading state, with a last update time of three days ago.  There’s a note within the SCCM console that if an update is stuck in the downloading state for a long period of time you should restart the SMS_EXECUTIVE (smsexec) service on the standalone primary or central administration site server (CAS).

I restarted this service, but nothing changed.  The next step was to check dmpdownloader.log, which can be found in C:\Program Files\Microsoft Configuration Manager\Logs.  This contained the error below, where I’ve replaced the proxy and SCCM server name with placeholders.

ERROR: Failed to download redist for 91406b1d-7c14-42d8-a68b-484be5c5e9b8 with command  /RedistUrl http://go.microsoft.com/fwlink/?LinkID=831290 /LnManifestUrl http://go.microsoft.com/fwlink/?LinkID=831291 /RedistVersion 112015 /ProxyUri http://%PROXYURL%/ /NoUI  “\\%SCCMSERVERNAME%\EasySetupPayload\91406b1d-7c14-42d8-a68b-484be5c5e9b8\redist” .

Looking in the C:\Program Files\Microsoft Configuration Manager\EasySetupPayload\91406b1d-7c14-42d8-a68b-484be5c5e9b8\redist folder, I could see the last modified date was three days ago, equal to the last update time in the console.  I was able to manually download the two CAB files in the above error, so there wasn’t an access issue.  I restarted the SMS_EXECUTIVE (smsexec) service again and this time I could see some more files had been download to the redist folder.  Unfortunately, the update was still listed as downloading.  I restarted the SMS_EXECUTIVE (smsexec) service again, which again resulted in some more files appearing in the redist folder and the update became available.  So the take away from this is that if the update is stuck in a downloading state you may have to restart the SMS_EXECUTIVE (smsexec) service more than once.

Update 14/02/18

Beginning with version 1706 there’s an update reset tool which can be used to fix issues when in-console updates have problems downloading or replicating.  Fore more information see here.