Author Archives: richardstk

The volume shadow copy provider is not registered in the system

On a Windows Server 2012 R2 VM protected by Veeam Backup & Replication, Veeam’s application aware processing was failing with the error below.

Retrying snapshot creation attempt (Could not create backup checkpoint for virtual machine ‘%VMNAME%’: The volume shadow copy provider is not registered in the system. (0x80042304). (Virtual machine ID 464FEFAA-0EF5-4E9F-98BF-D76EBE72717B))
Failed to create snapshot (Dell EqualLogic VSS HW Provider) (mode: Veeam application-aware processing with failover) Details: Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.
Retrying snapshot creation attempt (Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.)
Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

The VM is also protected by Azure Site Recovery/Hyper-V Replica and the virtual host running the VM logged a “The volume shadow copy provider is not registered in the system” error each time an application consistent recovery point was attempted.

I checked the registry key HKLM\SYSTEM\CurrentControlSet\Services\VSS\Providers and noticed one of the providers ({74600e39-7dc5-4567-a03b-f091d6c7b092}) was missing a CLSID key.

Reg1

I found another VM running Windows Server 2012 R2 and navigated to the same registry key.  In this case the CLSID key was populated.  I copied the CLSID key from the working system and applied to the VM with the issue.

Reg2

Next I restarted the Hyper-V Volume Shadow Copy Requestor on the VM and found that the VSS error was resolved.

Advertisement

Server 2012 R2 – VSSAdmin Waiting for response & VSSAdmin List Writers is empty

On a Windows Server 2012 R2 Hyper-V cluster I had an issue whereby backups were failing with VSS errors.

When running VSSAdmin List Writers I received the response: “Waiting for responses. These may be delayed if a shadow copy is being prepared.”

VSS0

In order to resolve this I restarted the COM+ Event System service.  Following this VSSAdmin List Writers failed to return anything.

VSS1

When restarting the COM+ Event System service there are a number of dependant services.  I noticed that the COM+ System Application service hadn’t automatically restarted.  I started the service, then restarted the services below.  Following this VSSAdmin List Writers returned the expected list of writers.

  • EqualLogic VSS Requestor – Specific to my use of EqualLogic storage
  • EqualLogic VSS Service – Specific to my use of EqualLogic storage
  • Hyper-V Virtual Machine Management
  • Microsoft Software Shadow Copy Provider
  • Volume Shadow Copy

Disabling WPAD on Windows Server 2012 R2 for SCDPM, Azure Backup Server and Azure Site Recovery

When using SCDPM 2016 with an Azure Recovery Services Vault I needed to bypass the web proxy for data transfer from SCDPM to the Azure Recovery Services Vault.  I also needed to bypass the proxy for Azure Site Recovery replication traffic on some Hyper-V hosts.

In the case of SCDPM I used the Configure option under Online Protection in the DPM Administration Console to disable the proxy.

I also checked the CBSettings.xml file for proxy information in the location scratch location, which can be found in the reg key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Azure Backup\Config.  As per this KB.

In the case of Azure Site Recovery I ran the Azure Site Recovery Configurator and selected to bypass the proxy server.

In the case of both SCDPM and Azure Site Recovery the replication traffic continued to flow via the web proxy. I removed the DNS suffix used to find the web proxy via WPAD and used the FindProxyForURL toolset to check that WPAD wouldn’t find the proxy.  However, the proxy continued to be used.  In the end I performed the following steps to prevent WPAD from working.

  • Remove the DNS suffix used to find WPAD from the network adapter
  • Ping WPAD to verify the host cannot be found
  • Delete the cached WPAD files from C:\Windows\ServiceProfiles\LocalService\winhttp
  • Delete DefaultConnectionSettings and SavedLegacySettings from HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
  • Delete the sub key under HKEY_USERS\S-1-5-19\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Wpad
  • Restart the WinHTTP Web Proxy Auto-Discovery Service

These are some other useful links:

 

Remotely disable Network Level Authentication (NLA)

If you try to RDP to a machine, but can’t because you receive the error below, you can use PSExec to remotely disable the requirement for NLA.

“The remote computer that you are trying to connect to requires Network Level Authentication (NLA), but your Windows domain controller cannot be contacted to perform NLA.  If you are an administrator on the remote computer, you can disable NLA by using the options on the Remote tab of the System Properties dialog box.”

Download PSExec from TechNet.  Run the code below updating the following values.

\\VMNAME – The name of the machine on which you want to disable NLA

VMNAME\ADMIN_ACCOUNT – The username of a local administrator on the machine on which you want to disable NLA, e.g. pc1\admin

psexec \\VMNAME -u VMNAME\ADMIN_ACCOUNT -p PASSWORD reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp” /f /v SecurityLayer /t REG_DWORD /d 0

 

 

 

DPM services fail to start following an upgrade to Windows Server 2016

I had SCDPM 2012 R2 running on Windows Server 2012 R2 with SQL Server 2012.  I followed the instructions here to upgrade to SCDPM 2016 UR 2 running on Windows Server 2016 with SQL Server 2016.

Everything was working fine until the final step to upgrade from Windows Server 2012 R2 to Windows Server 2016.  Following this the DPM services failed to start and the Event Log was full of errors like the below:

System.Exception: -2147023107

ATL Failure in Initializing Security of msdpm

Microsoft.Internal.EnterpriseStorage.Dls.Utils.DlsException: exception —> System.Runtime.InteropServices.COMException: Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE))

The resolution was to change the Netlogon service from Manual to Automatic.  Following this change and a reboot everything started working again.

 

SCCM stuck in downloading state

The 1610 update for SCCM had been stuck in the downloading state, with a last update time of three days ago.  There’s a note within the SCCM console that if an update is stuck in the downloading state for a long period of time you should restart the SMS_EXECUTIVE (smsexec) service on the standalone primary or central administration site server (CAS).

I restarted this service, but nothing changed.  The next step was to check dmpdownloader.log, which can be found in C:\Program Files\Microsoft Configuration Manager\Logs.  This contained the error below, where I’ve replaced the proxy and SCCM server name with placeholders.

ERROR: Failed to download redist for 91406b1d-7c14-42d8-a68b-484be5c5e9b8 with command  /RedistUrl http://go.microsoft.com/fwlink/?LinkID=831290 /LnManifestUrl http://go.microsoft.com/fwlink/?LinkID=831291 /RedistVersion 112015 /ProxyUri http://%PROXYURL%/ /NoUI  “\\%SCCMSERVERNAME%\EasySetupPayload\91406b1d-7c14-42d8-a68b-484be5c5e9b8\redist” .

Looking in the C:\Program Files\Microsoft Configuration Manager\EasySetupPayload\91406b1d-7c14-42d8-a68b-484be5c5e9b8\redist folder, I could see the last modified date was three days ago, equal to the last update time in the console.  I was able to manually download the two CAB files in the above error, so there wasn’t an access issue.  I restarted the SMS_EXECUTIVE (smsexec) service again and this time I could see some more files had been download to the redist folder.  Unfortunately, the update was still listed as downloading.  I restarted the SMS_EXECUTIVE (smsexec) service again, which again resulted in some more files appearing in the redist folder and the update became available.  So the take away from this is that if the update is stuck in a downloading state you may have to restart the SMS_EXECUTIVE (smsexec) service more than once.

Update 14/02/18

Beginning with version 1706 there’s an update reset tool which can be used to fix issues when in-console updates have problems downloading or replicating.  Fore more information see here.

SharePoint 2013 Save Document Library As Template Missing

If you view a document library settings page and “Save document library is a template” is missing try this work around.

Take the document library settings page URL, e.g. http://SharePoint/_layouts/15/listedit.aspx?List=58d3d056… and change listedit.aspx to savetmpl.aspx.  Press enter to load the page and you should be prompted to save the document library as a template.

 

WSUS Integrated With SCCM 2012 R2 Fails To Start

On SCCM 2012 R2 with WSUS integrated for software update deployments I was unable to open the WSUS console.  The WSUS service had stopped and the event log showed event ID 507 Windows Server Update Services, Update Services failed its initialization and stopped.  WSUS is configured to use a SQL database and the SQL Server Logs contained multiple errors for Login failed for user ‘DOMAIN\SCCMSERVERNAME$’. Reason: Could not find a login matching the name provided. [CLIENT: <local machine>]

Opening SQL Server Management Studio, I navigated to the SUSDB.  Expanding users showed a user for NT AUTHORITY\NETWORK SERVICE.

SCCMWSUS1

I next opened Security -> Logins and could not see a login for NT AUTHORITY\NETWORK SERVICE.  I added a login for NT AUTHORITY\NETWORK SERVICE and started the WSUS service.  this resolved the problem.

SCCMWSUS2windowwindows

 

Missing Search Results In SharePoint 2013 When Using A Content Enrichment Web Service

Users reported that they couldn’t find a few specific documents when searching within SharePoint 2013.  Checking the crawl log showed documents with the error “The item failed due to an error occurring when sending or receiving data to the external content processing enrichment web service.”  I searched the ULS for one of the effected documents and found the error “System.Net.WebException: The remote server returned an error: (413) Request Entity Too Large.”  Checking the web.config for the content enrichment web service showed the maxReceivedMessageSize value was configured to 8 MB.

TWCEWS0

 

To resolve the error I increased the value of the maxReceivedMessageSize property and re-indexed the document library.

The interesting thing about this case is that the content enrichment web service is designed to add additional metadata to documents.  When the content enrichment web service failed to receive documents over 8 MB the documents didn’t appear within search at all, it wasn’t just the case that they appeared but without the extra metadata.

Dell Firmware Update Fails With “Creation of Dynamic Partition Failed.”

When attempting to run a Dell firmware update you receive the error “Creation of Dynamic Partition Failed.”

DellFWUpdate0

In order to resolve the error reset the Dell iDRAC from the command prompt.  Open a command prompt as administrator and run racadm racreset.

DellFWUpdate1

Once the iDRAC has reset retry the update.