Based on how many threads there are on this topic it seems that the error handling for bad data in sql is very poor. I'm not really sure why there wouldn't be a database cleaning tool or logic to delete a table entry that raises an exception, or at the very least, to log the table that references the bad data. (end rant)
Running vmm 2012sp1ru1. Database is sql2012 on the same server.
Getting an exception when the following function runs.
Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherVirtualNetworkInformation(Host host, List`1 networkAdapters, List`1 vNics, ITaskContext taskContext)
I did add some additional networks as well as two hosts. There also seems to be a bug that wipes out the settings on the hosts virtual nic when adding another logical switch when there are physical nics on the host that are disconnected. Since the virtual nic that got wiped was my management network I suspect this caused the inconsistency due to vmm losing connection with the host while it was updating the network settings.
I have however backed out both new hyper-v hosts and all the new network configuration that was added when the crashing started. I also scoured the vmm db to try and find any stranded network config and was unable to. Here are the events and the stack trace. I appreciate any suggestions. Since I don't have too much loaded in vmm yet I may just backup the database and start over. If anyone knows what table might be causing the issue I will still go back into the db backup and see if i can find the entry. My major concern is that this could happen again once I have all of our virtual infrastructure moved over. I will, of course, have backups running regularly by then :D TIA
Events
Log Name: Application
Source: .NET Runtime
Date: 1/29/2013 1:25:38 AM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: VMMGR02.amd.appriver.corp
Description:
Application: vmmservice.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
Stack:
at Microsoft.VirtualManager.Engine.RefreshDriver`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].RefreshThreadFunction(System.Object)
at Microsoft.VirtualManager.Utils.Multithreading.ThrottledThreadPool+<>c__DisplayClass3.<ScheduleNextThread>b__2(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name=".NET Runtime" />
<EventID Qualifiers="0">1026</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-01-29T07:25:38.000000000Z" />
<EventRecordID>5770</EventRecordID>
<Channel>Application</Channel>
<Computer>VMMGR02.amd.appriver.corp</Computer>
<Security />
</System>
<EventData>
<Data>Application: vmmservice.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
Stack:
at Microsoft.VirtualManager.Engine.RefreshDriver`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].RefreshThreadFunction(System.Object)
at Microsoft.VirtualManager.Utils.Multithreading.ThrottledThreadPool+<>c__DisplayClass3.<ScheduleNextThread>b__2(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
</Data>
</EventData>
</Event>
Log Name: Application
Source: Application Error
Date: 1/29/2013 1:25:38 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: VMMGR02.amd.appriver.corp
Description:
Faulting application name: vmmservice.exe, version: 3.1.6011.0, time stamp: 0x50aaba3f
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000007fca25ec487
Faulting process id: 0x1198
Faulting application start time: 0x01cdfdef7f4084a7
Faulting application path: C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\Bin\vmmservice.exe
Faulting module path: unknown
Report Id: 13f22b67-69e5-11e2-9402-782bcb049272
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-01-29T07:25:38.000000000Z" />
<EventRecordID>5771</EventRecordID>
<Channel>Application</Channel>
<Computer>VMMGR02.amd.appriver.corp</Computer>
<Security />
</System>
<EventData>
<Data>vmmservice.exe</Data>
<Data>3.1.6011.0</Data>
<Data>50aaba3f</Data>
<Data>unknown</Data>
<Data>0.0.0.0</Data>
<Data>00000000</Data>
<Data>c0000005</Data>
<Data>000007fca25ec487</Data>
<Data>1198</Data>
<Data>01cdfdef7f4084a7</Data>
<Data>C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\Bin\vmmservice.exe</Data>
<Data>unknown</Data>
<Data>13f22b67-69e5-11e2-9402-782bcb049272</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>
Stack trace
----------------------------------------------------------------------- Error Report -------------------
----------------------------------------------------
Error report created 1/28/2013 5:03:54 PM
CLR is terminating
----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=3.1.6018.0
P3(assemblyName)=Engine.Adhc.Operations.dll
P4(assemblyVer)=3.1.6018.0
P5(methodName)=Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherVirtualNetworkInformation
P6(exceptionType)=System.NullReferenceException
P7(callstackHash)=e69d
SCVMM Version=3.1.6018.0
SCVMM flavor=C-buddy-RTL-AMD64
Default Assembly Version=3.1.6018.0
Executable Name=vmmservice.exe
Executable Version=3.1.6011.0
Base Exception Target Site=8781617125368
Base Exception Assembly name=Engine.Adhc.Operations.dll
Base Exception Method Name=Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherVirtualNetworkInformation
Exception Message=Object reference not set to an instance of an object.
EIP=0x000007fca23d3a07
Build bit-size=64
----------------------------------------------------
------------ exceptionObject.ToString() ------------
----------------------------------------------------
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherVirtualNetworkInformation(Host host, List`1 networkAdapters, List`1 vNics, ITaskContext taskContext)
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherAllInformation(Host host, Object agentRefreshSyncObj, Boolean checkIfClustered, Boolean refreshEventCapabilities, String& clusterName, Guid taskID, ITaskContext taskContext)
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.RefreshLockedHost(Host host, Guid taskID, ITaskContext taskContext, Boolean checkClusterStatus)
at Microsoft.VirtualManager.Engine.Adhc.RefreshVmHostSubtask.RunSubtask()
at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Run()
at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)
----------------------------------------------------
--------------- exception.StackTrace ---------------
----------------------------------------------------
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherVirtualNetworkInformation(Host host, List`1 networkAdapters, List`1 vNics, ITaskContext taskContext)
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.GatherAllInformation(Host host, Object agentRefreshSyncObj, Boolean checkIfClustered, Boolean refreshEventCapabilities, String& clusterName, Guid taskID, ITaskContext taskContext)
at Microsoft.VirtualManager.Engine.Adhc.HostRefresher.RefreshLockedHost(Host host, Guid taskID, ITaskContext taskContext, Boolean checkClusterStatus)
at Microsoft.VirtualManager.Engine.Adhc.RefreshVmHostSubtask.RunSubtask()
at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Run()
at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)
----------------------------------------------------
------------- StackTrace from handler --------------
----------------------------------------------------
This is the call stack from where the exception was caught, not where it was thrown.
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.WriteReportTextFile(TextWriter reportFile)
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonExceptionReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonCenter.ReportException(Exception e, WERReportOptions options)
at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()