Failing Azure Recovery Service VM restore jobs and how to resurrect your backups

Recently we’ve had a bit of a crisis situation when one of our Azure VM’s decided to lose a bunch of data. Fortunately backup jobs were set up through Azure’s Recovery Services and I’ve already used that a few times to restore or make copies of various VM’s without any problems. A few clicks and you’re ready to go. This was supposed to be an easy 20 minute task but this time was different.

For whatever reason instead of getting a restored VM, I started getting the following restore job fail message:

Restore failed with an internal error.
Please retry the operation in a few minutes. If the problem persists, contact Microsoft Support.

Not very descriptive, and not really helpful. :/

The data transfer part of the job would succeed each time, but the “Create the Restored VM” kept on failing. I tried using different restore points from a day, week, or even a month back, but it made no difference. It all came to a point where we had to submit a ticket to Microsoft to resolve this issue.

The two possible solutions that were presented to us were:

• either restore the VM under a new Azure Cloud Service – this worked fine, but wasn’t really what we wanted to do (you don’t really want to pile up additional Cloud Services just to do a simple restore, it makes no sense and leaves a messy infrastructure behind)
• restore the VM through Azure Powershell – this was a bit trickier, but it worked great in the end

So after a bit of research I realized that the Azure Web Portal doesn’t actually use the exact same back-end infrastructure as Powershell which is a bit weird and should probably be emphasized a bit more throughout Azure documentation.

Microsoft support told us to follow this documentation page to restore the VM using powershell, but the tutorial wasn’t without its kinks either.

Perhaps this got resolved by now but for the whole thing to work, I first needed Azure Powershell v1. That ended up being a bit of a pain because it required the regular Powershell v3 where Windows 8.1 comes with Powershell v4 and the downgrade was another mission impossible… In the end I somehow managed to resolve this issue by installing the latest Azure Powershell using Microsoft Web Platform Installer. That gave me the much needed Azure tooling for Powershell. Yay!

Now to code – these few Powershell commands will extract the VHD from the backup:

> Select-AzureRmSubscription -SubscriptionName YourSubscription
> $backupvault = Get-AzureRmBackupVault -Name "YourBackupVault"
> $backupitem = Get-AzureRMBackupContainer -Vault $backupvault -Type AzureVM -name "YourVmName" | Get-AzureRMBackupItem
> $rp = Get-AzureRMBackupRecoveryPoint -Item $backupitem
# change the $rp number to select the recovery point you want here
> $restorejob = Restore-AzureRMBackupItem -StorageAccountName "yourStorageAccountName" -RecoveryPoint $rp[6]
> $restorejob = Get-AzureRMBackupJob -Job $restorejob
> $details = Get-AzureRMBackupJobDetails -Job $restorejob

From here I finished the process using the Azure Portal as the rest of the process / powershell commands from the documentation seemed to be out of date and didn’t work.

To complete the process, you should go to the Azure Portal, to VM section, and then select the “Disks” tab. From there you’ll be able to create an unprovisioned disk which you will then use to create a new VM from. Afterwards click the + icon in the bottom left corner, and choose “create a VM – from gallery”, you will see an option to use your newly created disk. Finish the setup and you’re good to go. 🙂

Hope this helps you if you find yourself in a similar situation. Cheers!