I'm currently using SCVMM in tandem with TFS Lab Management to virtualize our TFS testing lab for a SharePoint deployment. In doing so, I've created a 3 machine environment in Network Isolation consisting of a SP 2010 Farm Stand Alone Server, a Domain Controller and a testing client. I've built this environment up to a "test-ready" state and snapshotted it.
I'm now using this environment in a nightly build-deploy-test cycle that reverts to a snapshot, deploys SharePoint solutions and executes tests. This has been working great for weeks.
While I was out on vacation last week, we hit the following reproducible error during our nightly executions:
"Environment message: Type=Error; Message=TF259175: Team Foundation Server failed to restore the following environment to snapshot: <removed for privacy> . The restore operation failed on one or more virtual machines, and as a result the root
of the environment snapshot tree has been reset. To recover from this condition, attempt to restore the environment again either to the same or a different snapshot.;
Machine messages:
Machine name: TFSLM-CLIENT
Machine message: Type=Error; Message=2606: Unable to perform the job because one or more of the selected objects are locked by another job.
Additional information: Failed task ID at SCVMM server is a9c9c330-dc9d-4b44-a0d9-bea5ed13ea97.
Corresponding result name is Lab_97.;
--------------------------------------------------
Environment message: Type=Error; Message=TF259113: Team Foundation Server placed the environment in mixed state. A mixed state indicates that not all virtual machines are in the same state. Start (or stop) the environment to recover from this state.
If the problem persists, contact your system administrator to bring the affected virtual machines into a usable state.;
--------------------------------------------------"
Further investigation of this error in SCVMM via both UI and PowerShell get-job functions indicated there NO jobs currently in a "Running" state that could be locking this machine.
I then queried the SCVMM Database tables directly and found that the tbl_VMM_Lock table does indeed show a lock on the VM in question:
Query
SELECT [ObjectID]
,[ObjectType]
,[TaskID]
,[LockType]
FROM [SCVMM_DB].[dbo].[tbl_VMM_Lock]
WHERE ObjectID = '3DF8ACE0-AB46-42B7-82FE-9DEFB361594C'
-- this is the failing VM's GUID.
Output:
ObjectID ObjectType TaskID LockType
3DF8ACE0-AB46-42B7-82FE-9DEFB361594C VM 9BBE38FA-80D1-487F-A802-46B3F773E959 2
The TaskID denoted in the Lock doesn't exist in any of the tbl_TR_* tables as a task so I feel like it's a ghost row. I deleted it and a replacement row just reappears.
Help?