System state backups failing

Im currently working on a script to automate system state backups and in my testing I encountered a issue, namely System state backups fail on my 2008 domain controller with the following error message

ERROR – Volume Shadow Copy Service operation error ( 0x800423f4). The Write experienced a non-transient error. If the backup is retried the error is likely to occur

Where to start with this one……..

Well the Hex error code indicates that the problem is with VSS failing to complete the read of data so the next port of call is check VSS. This can be done via powershell and the following command run from a administrative shell

vssadmin List Writers

which produces the following output

VSS

Which as you can see this confirmed that the NTDS VSS writer failed, which would be expected as we were backing up the system state. The first step in troubleshooting VSS failures is basic enough, restart the services and test. If that doesnt help then restart the server. This had no effect on the problem so it was time to dig a little deeper.

As always the best place to start is the event logs, Microsoft have really increased the level of logging on the servers and it is far more useful than in 2003.A quick perusal of the event logs showed that the backup ran until it tried to use the Extensible Storage Engine API ( ESENT) to read the shadowcopy headers of the Active directory database it then logged the following error

Log Name: Application
Source: ESENT
Date: <date & time>
Event ID: 412
Task Category: Logging/Recovery
Level: Error
Keywords: Classic
User: N/A
Computer: <computer name>
Description:
Lsass(640) Unable to read the header of logfile \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\Windows\NTDS\edb.log  Error -546.

This error points to a known issue with Windows server 2008 ( which my Domain controller is) and applications that use ESENT. Microsoft have released a hotfix for this issue  : http://support.microsoft.com/kb/2470478

Once this hotfix was applied there were no further ESENT Errors logged and the VSS portion of the backup completed successfully.