Software Diagnostics Library

This part follows Dumps for Dummies (Part 2) and here I’ll try to explain crashes, dumps and postmortem debuggers. 

Sometimes a computer (CPU, Central Processing Unit) cannot perform its job because the instruction it gets to do some calculations, read or write data is wrong. Imagine a situation when you get an address to deliver a message to and you find that it doesn’t exist…  The following idealized picture shows this situation (if memory locations/addresses are indexed from 0 then -1 is obviously wrong address):

When referencing invalid address CPU executes special sequence of actions (called trap) that ultimately leads to saving memory so you could later examine its contents and find out which instruction was invalid. If crash happens inside Windows operating system then you see blue screen and then a kernel memory or full computer physical memory is saved in a file (called either kernel or complete memory dump respectively). If you have a crash in a running application or service then its memory contents are saved in a file (called user dump). The latter file is also called a postmortem dump and we call a program which saves it a postmortem debugger. There can be several such programs and the one which is specified in the registry to execute whenever a crash happens in a running application or service is called a default postmortem debugger. The following picture illustrates this (here spooler service, spoolsv.exe, crashed by faulty printer driver):

By default it is Dr. Watson (drwtsn32.exe) but sometimes it doesn’t work in terminal services environment and has limitations so we always recommend setting NTSD (ntsd.exe) as a default postmortem debugger:

How to Set NTSD as a Default Windows Postmortem Debugger

I prefer to call both user and kernel/complete memory dumps postmortem (not only user dumps) because they are saved after application, service or system is dead already (crash or fatal error already happened). This distinguishes them from live memory dumps saved manually whenever we want them. This brings us to dump classification that I show you in forthcoming parts. 

- Dmitry Vostokov @ DumpAnalysis.org -

Can the smallest program that crashes be smaller than the smallest program that doesn’t? Depends on a platform and a compiler/linker set. I’ve chosen x64 and MASM64 for my experiments. The smallest working program I came up first was this:

; ml64 /Zi TheSmallestProgram64.asm /link
;   /entry:main /SUBSYSTEM:CONSOLE

_text SEGMENT
main PROC
ret
main ENDP
_text ENDS

END

It compiles and links to an executable with only one byte instruction in its main function:

0:000> u main
TheSmallestProgram64!main:
00000000`00401010 c3              ret
00000000`00401011 cc              int     3
00000000`00401012 cc              int     3
00000000`00401013 cc              int     3
00000000`00401014 cc              int     3
00000000`00401015 cc              int     3
00000000`00401016 cc              int     3
00000000`00401017 cc              int     3

Then I thought about removing ret instruction and supposed that if we compile and link and try to execute the program with 0 bytes we get straight to int 3 instruction and in my case (I have NTSD set as a default postmortem debugger) a dump will be saved. So I did that but I found that unfortunately compiler inserts ret instruction if the procedure body is empty. So I cheated them by putting nop instruction (which is also one byte) and got my dump!

; ml64 /Zi TheSmallestProgramWithBug64.asm /link
;    /entry:main /SUBSYSTEM:CONSOLE

_text SEGMENT
main PROC
nop
main ENDP
_text ENDS

END

Loading Dump File [new_2006-10-25_12-40-06-500_076C.dmp]
…

0:000> kL
TheSmallestProgramWithBug64!main+0×1
kernel32!BaseProcessStart+0×29

0:000> u main
TheSmallestProgramWithBug64!main:
00000000`00401010 90              nop
00000000`00401011 cc              int     3
00000000`00401012 cc              int     3
00000000`00401013 cc              int     3
00000000`00401014 cc              int     3
00000000`00401015 cc              int     3
00000000`00401016 cc              int     3
00000000`00401017 cc              int     3

So one answer to my question: The smallest working program and the smallest crashing program have the same size unless we use some binary editors

Then I tried MS Visual C++ (this time 32-bit project) and came up with the following C or C++ program without any prolog and epilog code:

__declspec(naked) void Main ()
{
}

I changed entry point from standard main function to my own capitalized Main function and compiler/link options:

Compiler:

/Od /GL /D "WIN32" /D "NDEBUG" /D "_CONSOLE"
/D "_UNICODE" /D "UNICODE" /D "_AFXDLL"
/FD /MD /GS- /Fo"Release\"
/Fd"Releasevc80.pdb" /W3 /nologo /c
/Wp64 /Zi /TP /errorReport:prompt

Linker:

/OUT:"SmallestProgram.exe" /INCREMENTAL:NO
/NOLOGO /MANIFEST:NO /NODEFAULTLIB /DEBUG
/PDB:"SmallestProgram.pdb" /SUBSYSTEM:CONSOLE
/OPT:REF /OPT:ICF /LTCG /ENTRY:"Main"
/ERRORREPORT:PROMPT

The program crashes immediately because the body is empty:

Loading Dump File [new_2006-10-25_15-18-03-109_13B0.dmp]

0:000> u Main
SmallestProgram!Main:
00401000 cc              int     3
00401001 0000            add     byte ptr [eax],al
00401003 0000            add     byte ptr [eax],al
00401005 0000            add     byte ptr [eax],al
00401007 0000            add     byte ptr [eax],al
00401009 0000            add     byte ptr [eax],al
0040100b 0000            add     byte ptr [eax],al
0040100d 0000            add     byte ptr [eax],al

0:000> kL
ChildEBP RetAddr
002cfff0 00000000 SmallestProgram!Main

0:000> dds esp
002cffc4  7d4e992a kernel32!BaseProcessStart+0x28
002cffc8  00000000
002cffcc  00000000
002cffd0  7efdf000
002cffd4  80000003
002cffd8  002cffc8
002cffdc  002cfbbc
002cffe0  ffffffff
002cffe4  7d4d8998 kernel32!_except_handler3
002cffe8  7d4e9938 kernel32!`string'+0x28
002cffec  00000000
002cfff0  00000000
002cfff4  00000000
002cfff8  00401000 SmallestProgram!Main
002cfffc  00000000

So another answer to my question: The smallest crashing program can be less than the smallest working program and is actually 0 bytes

- Dmitry Vostokov -

Crash dumps (and live debugging) can be very useful for reverse engineering component dependencies. Let’s look at MS Video Driver Architecture UML component diagram (synthesized after reading various articles from OSR and DDK):

Coupled with this understanding and armed with Citrix symbol files (which are freely downloadable from Citrix support and you don’t really need them to see component dependencies) I was able to transform this thread stack below and other similar stacks into the following UML component diagram (some functions are shown as module!xxx and offsets are removed for clarity):

nt!KiSwapContext
nt!KiSwapThread
nt!KeWaitForSingleObject
tcpip!xxx
tcpip!TCPDispatch
nt!IofCallDriver
nt!xxx
nt!xxx
TDTCP!xxx
TDTCP!xxx
TDTCP!TdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
pdrframe!xxx
pdrframe!PdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
pdcrypt1!xxx
pdcrypt1!PdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!WdIoctl
termdd!IcaCallStack
termdd!IcaCallDriver
termdd!IcaDeviceControlChannel
termdd!IcaDeviceControl
termdd!IcaDispatch
win32k!GreDeviceIoControl
win32k!EngDeviceIoControl
vdtw30!xxx
vdtw30!xxx
win32k!vMovePointer
win32k!GreMovePointer
win32k!xxxMoveEventAbsolute
win32k!ProcessMouseInput
win32k!InputApc
nt!KiDeliverApc
nt!KiSwapThread
nt!KeWaitForMultipleObjects
win32k!xxxMsgWaitForMultipleObjects
win32k!xxxDesktopThread
win32k!xxxCreateSystemThreads
win32k!NtUserCallOneParam
nt!KiSystemServiceCopyEnd
nt!KiSwapThread
nt!KeWaitForSingleObject
win32k!EngWaitForSingleObject
vdtw30!xxx
vdtw30!xxx
vdtw30!xxx
vdtw30!DrvTw2SaveScreenBits
win32k!GreSaveScreenBits
win32k!CreateSpb
win32k!zzzChangeStates
win32k!zzzBltValidBits
win32k!xxxEndDeferWindowPosEx
win32k!xxxSetWindowPos
win32k!xxxShowWindow
win32k!NtUserShowWindow
nt!KiSystemService
USER32!NtUserShowWindow
USER32!InternalDialogBox
USER32!DialogBoxIndirectParamAorW
USER32!DialogBoxParamW

We replace MS components with Citrix ones:

• Video Display with vdtw30.dll
• Video Miniport with icacdd.sys
• Hardware and HAL with Terminal Services stack components (MS termdd.sys, Citrix wdica.sys, etc)

- Dmitry Vostokov -

If you need to attach NTSD to a process on x64 Windows and to save a dump remember that there are two versions of NTSD: x86 (32-bit) and x64. The former is located in \Windows\SysWOW64 and should be used for attaching to 32-bit applications and services. IMA service, for example, is 32-bit in Citrix Presentation Server for x64, so if it hangs you need 32-bit debugger. For explanation why you need different versions of NTSD please refer to the first picture in my previous post: Dumps, Debuggers and Virtualization 

If you use WinDbg for that purpose install both 32-bit and 64-bit versions.

If you want to install NTSD or WinDbg as a default postmortem debugger use Wow6432Node registry hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug

Debugger = ntsd -p %ld -e %ld -g -c ".dump /ma /u c:\TEMP\new.dmp; q"

Refer to the following Citrix support articles explaining and describing in more detail how to set NTSD and WinDbg as default postmortem debuggers:

How to Set NTSD as a Default Windows Postmortem Debugger

How to Set WinDbg as a Default Windows Postmortem Debugger

They will be updated soon to include WOW64 information.

- Dmitry Vostokov -

Following the announced Troubleshooting Tool Ideas database Ramzy Mansour from Citrix Technical Support came up with a brilliant idea about Dump Monitor Suite and its two useful components for Citrix administrators:

DumpStats:

- Monitors and displays a graphical chart showing which services and processes crashed or hanged on an individual Citrix server, their crash time and date, dump location, dump type, crash signature, modules where crashes happened, etc.

- Aggregates and displays statistics for the whole Citrix farm

DumpAlerts:

- Sends an e-mail alert and/or an SMS message to a cell phone when any crash or hang happens

- Configures alerts based on severity and specific processes

Additionally Dump Monitor Suite will include the following components (some of them already exist and will be enhanced):

DumpChecks: 

- Enhanced and improved version of Citrix DumpCheck Explorer extension and its command line version

DumpProperties:

- New Explorer extension (Properties dialog) which shows various data extracted from a dump, like process name, module list, whether heap checking was enabled, module name where crash happened, etc.

DumpDepends:

- Integrated and enhanced version of SystemDump which allows to dump dependent processes

We are currently finalizing functional specs and architecture. More information about this Suite will be posted soon.

- Dmitry Vostokov -

There is an idea to interpret a process or a system dump as a picture (similar to interpreting it as a giant wave file: Dump2Wave). I would like to extend this idea and present it as a Dump Tomography - a combination of images taken from a dump when looking at it from different perspectives - memory, resources, subsystem hierarchy, etc. I’m going to include some simple pictorial interpretations and representations in forthcoming DumpPlayer.

Dump Analysis becomes both Medicine and Art. You can finally hear how corruption sounds and how it looks

- Dmitry Vostokov -

Dump2Wave command line tool is available for free download:

Download Dump2Wave

Simply run it from the command prompt and specify full paths to dump file and output WAV file. Dump file will be converted by default into 44.1KHz 16bit stereo WAV file (CD quality). You can specify you own conversion parameters like samples per second (22050, 11025, etc), bits per sample (8 or 16) and the number of channels (1 - mono, 2 - stereo):

For example, I converted sndrec32.dmp to sndrec32.wav:

The dump was taken after sndrec32.exe played “Windows XP Logon Sound.wav” file from \Windows\Media folder and that wave file was originally sampled as 22050Hz 16bits stereo. By listening to sndrec32.dmp I was able to hear a fragment from that logon sound because it was stored in a buffer inside sndrec32.exe process.

Note: Dump2Wave will not convert a dump file which is greater than 4Gb. Forthcoming DumpPlayer will be able to play large complete memory dumps in real-time without conversion and you could graphically choose a region to play.

Just a reminder on how you can save a dump manually (unless you have a dump from application crash or BSOD):

Dumping Processes Without Breaking Them

Microsoft User Mode Process Dumper

How to Attach NTSD to a Process and Save a Dump

How to Attach WinDbg to a Process

- Dmitry Vostokov -

Have you ever tried to debug a debugger when it debugs a debuggee? Is it possible? Good question. I never asked it to myself until today. And tried. And it works! First I tried to attach WinDbg.exe to an instance of WinDbg.exe executing ‘!analyze -v’ command and got these stacks:

0:002> ~*kL 100
0 Id: 1ff0.104c Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr
0006df38 7739d02f ntdll!KiFastSystemCallRet
0006ff7c 01055e36 USER32!NtUserWaitMessage+0xc
0006ffc0 77e523e5 windbg!_initterm_e+0x170
0006fff0 00000000 kernel32!BaseProcessStart+0x23

# 2 Id: 1ff0.116c Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
00fdffc8 7c845ea0 ntdll!DbgBreakPoint
00fdfff4 00000000 ntdll!DbgUiRemoteBreakin+0x36

Next I thought, wait a moment, we are debugging dump analysis session. Can we debug a debugger debugging a running process? So I attached WinDbg.exe to an instance of WinDbg.exe attached to an instance of notepad.exe and got these stacks:

0:002> ~*kL
0 Id: 11f0.164c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0006df38 7739d02f ntdll!KiFastSystemCallRet
0006ff7c 01055e36 USER32!NtUserWaitMessage+0xc
0006ffc0 77e523e5 windbg!_initterm_e+0x170
0006fff0 00000000 kernel32!BaseProcessStart+0x23

Given that many functions from dbghelp.dll and dbgeng.dll are described in WinDbg help you can quickly reverse engineer WinDbg.exe and its extensions code.

- Dmitry Vostokov -

After listening to “An Anthology of Noise and Electronic Music”

and remembering that long time ago I was asked to convert stock charts into sound waves an idea came to me to convert memory dump files into WAV files by appending an appropriate header in front of them. So depending on imposed sampling frequency (Hz), quantization level (bits) and mono/stereo settings you can enjoy listening to dumps. Long time ago I wrote a component for hard-disk recording while working on my voice recognition projects and I’m going to reuse it now.

Forthcoming DumpPlayer will be released this weekend for free download so stay tuned. 

If it goes well I promise to write a WinDbg extension to listen to a given memory range.

- Dmitry Vostokov -

I’ve been thinking about this since my previous DDV post and came up with the following example of general DDV architecture:

So we need:

• A “Debugger” to debug “Virtualization layer”
• A “Debugger plugin” to help the “Debugger” to understand the “Subject of virtualization”
• Not to mention various virtualized “debuggers” debugging their virtualized subjects

- Dmitry Vostokov -

Everyone now speaks about virtualization and its benefit. New horizons. I would like to add my 2 cents from dump analysis and debugging perspective. There will be more complex debugging environment as my recent experience with WOW64 tells me:

So if we generalize this to virtualization environment we would come up with the following picture:

- Dmitry Vostokov -

Part 2 follows the discussion of various dump types depicted here: Dumps for Dummies (Part 1) 

So the question arises: how to make sure the customer got the right dump? And if the dump type is not what you asked for provide a recommendation for further actions. Troubled with such questions during my first years in Citrix technical support I decided to develop a lightweight Explorer extension and a command line version of dump checking tool called Citrix DumpCheck:

Here it does basic checks for dump validity and shows the dump type: Complete memory dump

If it found small mini dump type (64Kb) the tool would have suggested to change settings in Control Panel.

The extension can be downloaded from Citrix support web site:

Citrix DumpCheck Explorer Extension version 1.4 

FAQ:

Q. Is it possible to show more information like process name in a user dump or whether full page heap was enabled?

A. Certainly it is possible to include. However it requires access to OS symbol files during runtime and most customers don’t have them installed or downloaded from MS symbol server. So the design decision was not to include these checks in version 1.x. I consider to include this in next versions 2.x.

Q. The customer doesn’t want to modify environment by installing extension. Is there any command line version of this tool?

A. Yes, there is. The following article contains a download link to a command line version of Citrix DumpCheck:

Citrix DumpCheck Utility (Command Line) version 1.4   

Q. Does this extension work in 64-bit Windows?

A. No, but you can use command line equivalent shown in the answer to the previous question. Also I’m planning to port this extension to 64-bit soon and will announce as soon as I release it.

- Dmitry Vostokov  @ DumpAnalysis.org -

Recently I discovered wonderful WinDbg commands dpu (UNICODE strings) and dpa (ASCII strings). Look at WinDbg help for other d** equivalents like dpp.

I needed to examine raw stack data and check if any pointers on stack were pointing to strings. For example:

0:143> !teb
TEB at 7ff2b000
...
    StackBase:            05e90000
    StackLimit:           05e89000
...
...
...
0:143> dpu 05e89000 05e90000
05e8f58c  00120010 ""
...
...
...
05e8f590  77e7723c "Debugger"
05e8f594  00000000
05e8f598  08dc0154
05e8f59c  01000040
05e8f5a0  05e8f5dc "G:\WINDOWS\system32\faultrep.dll"
05e8f5a4  0633adf0 ""
05e8f5a8  00000000
05e8f5ac  00000001
05e8f5b0  00000012
05e8f5b4  7c8723e0
05e8f5b8  ffffffff
05e8f5bc  00000004
05e8f5c0  69500000
05e8f5c4  00000000
05e8f5c8  00000aac
05e8f5cc  00000002
05e8f5d0  05e8f740
05e8f5d4  0633adfc "drwtsn32 -p %ld -e %ld -g"
05e8f5d8  00000000
...
...
...


Of course, you can apply these commands to any memory range, not only stack.

- Dmitry Vostokov -

There is much confusion among MS and therefore Citrix customers about different dump types - Windows has 3 major dump types (not including various mini-dumps): complete, kernel and user. Long time ago I created a hand-crafted picture showing how various parts of computer memory are saved in a dump and I want to share it with a wider part of Citrix community and perhaps with the rest of the world:

- Dmitry Vostokov  @ DumpAnalysis.org -

I got the impression after reading numerous books and articles about device drivers that UML is almost never used in describing kernel and device driver design and architecture. Everything is described either by words or using proprietary notations. If you don’t know about UML (Unified Modeling Language) it is time to learn because it is an industry standard general purpose modeling language with graphical notation. You can find many good tutorials on the Web and I can recommend the book to start:

UML Distilled: A Brief Guide to the Standard Object Modeling Language, Third Edition

Recently I created some diagrams based on my past experience in using UML to describe and communicate architecture and design:

0. Component diagram depicting major driver interfaces 

1. Class and object diagram depicting relationship between drivers and devices

2. Component diagram showing dependencies and interfaces when calling Win32 API function ReadFile

3. Component diagram showing IRP flow in a driver stack (driver-to-driver communication)

Actually I found that some driver books incorrectly depict the ordering of I/O stack locations in IRP stack corresponding to driver or device stack. The correct layout is depicted above. IRP I/O stack locations grow down (to lower addresses) in memory like any other Wintel stack. You can see it from kernel dumps or the following macro from DDK header file wdm.h which shows that next IRP I/O stack location is down in memory (1 is subtracted from current stack location pointer):

#define IoGetNextIrpStackLocation( Irp ) (\
    (Irp)->Tail.Overlay.CurrentStackLocation - 1 )

Dumps (and live debugging) are good in studying component relationships, reconstructing sequence diagrams, etc. For example, this edited fragment below is from crash dump and it shows who calls whom and component dependencies be reconstructed from call stack of Win32 API function GetDriveType: SHELL32 (calls it) -> kernel32 -> ntdll -> nt (ntoskrnl.exe). You can also see various Citrix hooks and filters here (CtxSbxXXX):

kd> kL
CtxSbx!xxx
nt!IovCallDriver
nt!IofCallDriver
CtxAltStr!xxx
nt!IovCallDriver
nt!IofCallDriver
nt!IopParseDevice
nt!ObpLookupObjectName
nt!ObOpenObjectByName
nt!IopCreateFile
nt!IoCreateFile
nt!NtOpenFile
nt!KiSystemService
SharedUserData!SystemCallStub
ntdll!ZwOpenFile
CtxSbxHook!xxx
kernel32!GetDriveTypeW
SHELL32!CMountPoint::_EnumMountPoints

- Dmitry Vostokov -

Noticed as one engineer was frequently switching between WinDbg and Calc. Forget about using calc.exe during debugging or dump analysis sessions. Save your valuable time. Don’t multiprocess. Use ? and .formats commands:

0:000> ? 2 + 2
Evaluate expression: 4 = 00000004
0:000> .formats 4
Evaluate expression:
  Hex:     00000004
  Decimal: 4
  Octal:   00000000004
  Binary:  00000000 00000000 00000000 00000100
  Chars:   ....
  Time:    Thu Jan 01 00:00:04 1970
  Float:   low 5.60519e-045 high 0
  Double:  1.97626e-323


Now you can do your finance calculations in WinDbg too.

The WinDbg Way!

- Dmitry Vostokov -

If this scenario happens after some event or user action most likely some notification hooks were involved. WinDbg !thread command on the current processor will most likely catch running thread than IdleLoop thread and !process command will show the current process context. Then from thread stack you can make an educated guess which components were likely responsible for that.

To change the current processor when looking at the dump from multiprocessor platform use ~”p”s command where “p” is zero-based processor number, for example, ~1s changes the current processor to the second processor. Remember that every processor has its own thread and process context. If processor has nothing to do it is looping in KiIdleLoop thread which belongs to Idle process.

- Dmitry Vostokov -

The post was updated and can be found here:

http://www.dumpanalysis.org/blog/index.php/2007/08/26/moving-to-kernel-space-updated-references/

- Dmitry Vostokov -

HTML version of selected lecture materials from my forthcoming book is available on my personal web site:

http://www.dumpanalysis.org/blog/index.php/foundations-of-debugging-x86/

Topics available online include:

• Memory, registers and simple arithmetic 
• Number representations and pointers
• Bytes, words, double words and pointers to memory
• Instruction pointer and disassembling a program with pointers
• Memory and stacks
• Frame pointer and local variables
• Function parameters
• Function pointer parameters
• Virtual memory, processes and threads
• Arrays and structures in memory

In addition to x86 (32-bit) the book covers assembler, C/C++ and crash dump analysis on the following platforms: 

• x64 Windows
• Windows Mobile (ARM processors)

It will include an educational version of x86/x64/ARM4 assembler and interpreter written using modern C++ and STL (including UML diagrams of all classes and components) and much more! Although all this bonus stuff is still under development it is actually working, compiling, loading and executing/interpreting small assembly language programs.

I’m also planning to publish an Internet version of this assembly language interpreter to run small educational assembly language programs anytime, anywhere. The latter Citrix-like goal is actually the part of my Masters degree project. 

- Dmitry Vostokov -

Not really new as it was previously called CtxBSOD v2.1 but was renamed to better show its purpose. In addition to renaming I added a command line option to dump a system remotely or from a command line locally without using its GUI interface. The main motivation for me to write this tool was the absence of similar tools for 64-bit Windows. SystemDump can dump a 64-bit server too! 

You can download it form Citrix support web site (requires free registration):

http://support.citrix.com/article/CTX111072

Main features:

• The tool has both GUI and command line interfaces.
• You can type a message/text (or copy it from clipboard) before forcing a memory dump. This message is saved in a dump and a support engineer can read it after loading the dump in WinDbg.exe. This is implemented to encourage writing the symptoms and conditions explaining why the dump has to be forced.
• The tool can stay on top of any window (if you need this to quickly dump the server after a reproduction or during the process of an activity).
• It is supplied with Program Database (PDB) symbols for the driver (32-bit and 64-bit) which is useful when you want to have all symbols present on the bugcheck thread.
• The bugcheck clearly shows that the dump is manually generated.
• The tool can force a memory dump on both 32-bit and 64-bit platforms.
• Before forcing a fatal error on a server, the tool warns about potential damaging consequences: Users are disconnected and all the data which is not saved will be lost. It asks for a confirmation.
• You can specify a period of time (in minutes) when to force a memory dump.

The latter feature is implemented entirely in kernel. Additional command that not covered in the article is

>SystemDump.exe abort

allows you to abort the action if you ran the tool using command line options.

I attached the UML component diagram showing the architecture of this tool. I recently developed a presentation about device drivers architecture and Citrix kernel drivers where I used this tool as one of examples.

- Dmitry Vostokov -