I’ve been involved to nail down one CPU 100% issue. per my understanding, the ETW trace API failed and caused the System to run the refresh in an infinite loop.
what’s the symbol of the event?
there are 9 HOSTs in one Biztalk server. only 1 Host kept hitting 100% cpu. even there was no business activity. ( no message came in, no orchestration get invoked. )
what did I do to locate the problem.
Run a memory Dump, then open the dump in Windbg, see which thread is CPU hungry.
91:1235 0 days 3:35:28.796
92:9740 0 days 3:34:24.140
threads 91 and 92 are the top threads which consume a lot CPU time.
then Run a clrstack to see what are those two threads doing.
|!91s //switch to thread 91 |
!clrstack –a //view the stacktrace, and show the instance of the variable
Both 91 and 92 get the same result.
00000000143fc810 00000642783437fa System.Diagnostics.StackTrace.ToString(TraceFormat)
Let’s see the stacktrace from bottom up, the closed Biz code is the Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread
the update by default should be invoked every 60 seconds. why it keep invoking. ???
since memory dump is a static analysis and snapshot of the memory. then a run a cordbg, what surprised me is that the process keep throwing exceptions.
first chance exception generated: (0xe00a18b8) <System.Threading.ThreadAbortExcetion>
then I turned on the “unhanled exceptino on” by run “ca e” get the exception details
first chance exception generated: (0x120128fd0) <System.Threading.ThreadAbortExc
is it the same stacktrace, yes it is. run W
then I use the ILDASM , and Open the assembly, go the the 0x 23 Lines.
IL_001c: callvirt instance void [Microsoft.BizTalk.Tracing]Microsoft.BizTalk.Tracing.Trace/HackTraceProvider::TraceMessage(uint32,
it is a ETW trace API.
Trace.Tracer.TraceMessage(4, "MgmtDbAccessEntity: Entering RunCacheUpdates fn for host " + this.hostName, new object)
what a bad design for the code that Tracing API (non-functional) broke the functional application, and even didn't catch the unhandled exception
More troubleshooting entries.
- SQlServer StreamInside , Unable to Create Application Microsoft.ComplexEventProcessing.ConnectionException: Access is denied
- windbg:PDB symbol for mscorwks.dll not loaded when you debug dump of .net app or silverlight apps and How to configure WinDbg to run other versions of the .NET
- Biztalk 64bit Host CPU 100% , Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread