Layman's Quick Guide on Crashdump Debugging20 Dec 2020
In distributed computing, we rely on traces and metrics to understand the runtime behavior of programs. However, in some cases we still need assistance from debuggers for live-site issues. For instance, if the service crashes all of sudden and no trace offers any clue, we need to load crashdump into debugger. Or some exception is raised but traces are insufficient to understand the nature of the problem, we may need to capture a full state of the process.
In old days, at least starting in Windows 3.1, there was a Dr. Watson to collect the error information following a process crash, mainly the crash dump file. Every time I saw it, something bad happened. Nowadays it has been under the new name of Windows Error Reporting, or WER. Inside the platform, there is still a “watson” service to collect all the crashdumps created by the platform code, process it, assign to the right owner, and send alerts as configured. Some times during live-site investigation, we can also request a dump file collection using “Node Diagnostiics”, then the file will be taken over by Watson (assuming your hand isn’t fast enough to move the file somewhere else).
Like it or not, to look at the dump file you have to use windbg. You can choose cdb or windbgx but they are not really different. If you are too busy to learn how windbg works, particularly managed code debugging using SOS, then you may use this quick guide to save some time.
Download sosex from Steve’s TechSpot and save the DLL in the extension directory.
Download mex from Microsoft download and save the DLL in the extension directory.
To find the extension directory, find the directory at where windbg.exe is located using Task Manager, then go to
Exit windbg: enter
qd or simply Alt-F4.
Display process environment block
Wou will see where the execution image is, all the environment variables which contains the machine name, processor ID, count, etc.
To check which threads have consumed how much CPU time:
To check CPU utilization, thread pool worker thread and completion port thread usage:
List of threads: check if how many threads there are, any threads are terminated or hitting some exception, etc.
If you click the blue underlined link you can switch to that thread, then use the following to see the native stack trace:
or see the managed stack trace
To check the object on the stack run the following:
To check the local variables of a specific frame (use the frame number in “k” output):
Object count: to get the statistics of objects in the managed heap.
If you want to get the live objects (the objects that cannot be garbage collected), add
-live parameter. If you want
to get the dead object, add
Find object by type name: firstly find the list of types with statistics by the type name (either full name of partial):
!dumpheap -stat -type MyClassName
Then click the method table link, which is essentially:
!dumpheap /d -mt [MethodTableAddress]
You can click the address link to dump the object, or
A better way to browse the object properties is to use sosex:
To know why it’s live, or the GC root:
or use sosex
Check the current symbol path, you use use menu or
Add a directory where PDB files (symbols) are located, use menu or
Find all the class names and properties with a particular string (use your own wildcard string):
List of all modules loaded:
To get the details about a module, click the link in above output or:
Here you can see the file version, product version string, timestamp, etc. For the files from many repos, you can see the branch name and commit hash. If you are interested in the module info:
To show disassembled IL code, firstly switch to a managed frame, then run mu:
!sosex.mframe 70 !sosex.mu
Find unique stack traces: this will go through the stack trace of all threads, group them by identical ones, and show you which stack has shown up how many times:
Often times you can see lock contentions or slow transaction isuse, etc.
Find all exceptions:
Dump all async task objects:
If you have to debug memory related issue, refer to my previous post.
Many debugging topics are not covered, for instance finalization, deadlock, locking, etc. If quick guidance is insufficient, please spend some time starting from Getting Started With Windows Debugging or the book Advanced .NET Debugging.