Archive for the ‘Debugging’ Category

Tracing Win32 API while debugging a process

Wednesday, January 3rd, 2007

Load an executable or attach WinDbg to an existing process and use logexts debugging extension (in output below all API parameters and return values are omitted for visual clarity):

0:001> !logexts.loge
0:001> !logc e *
All categories enabled.
0:001> !logo e d
  Debugger            Enabled
  Text file           Disabled
  Verbose log         Enabled
0:001> g
Thrd 7c0 77555B59 BeginPaint( 0x001103AA) ...
Thrd 7c0 77555B65 GetClientRect( 0x001103AA) ...
Thrd 7c0 77555B96 DrawEdge( 0x01010072 ...) ...
Thrd 7c0 77555C8A DrawFrameControl( 0x01010072 ...) ...
Thrd 7c0 77555CE1 EndPaint( 0x001103AA ... ) ...
Thrd 7c0 004165F2 TlsGetValue( 0x00000006) ...
Thrd 7c0 4B8D54B5 CallNextHookEx( ... ) ...
Thrd 7c0 0040D7CC GetMessageW( ... ) ...

You can break in and put a breakpoint at a return address:

0:001> bp 0040D7CC
0:001> g
Thrd 7c0 0040D7CC GetMessageW( ... ) ...
Breakpoint 0 hit
ProcessHistory+0xd7cc:
0040d7cc 85c0            test    eax,eax
0:000> u 0040D7C0 0040D7CC
ProcessHistory+0xd7c0:
0040d7c0 50              push    eax
0040d7c1 50              push    eax
0040d7c2 8d7730          lea     esi,[edi+30h]
0040d7c5 56              push    esi
0040d7c6 ff15f8434300    call    dword ptr
[ProcessHistory+0x343f8 (004343f8)]
0:000> dd 004343f8
004343f8  3c001950 3c0018c4 3c00193c 3c0014dc
0:000> u 3c001950
3c001950 b889020000      mov     eax,289h
3c001955 e98e410014      jmp     logexts!LogHook
(50005ae8)
3c00195a b88a020000      mov     eax,28Ah
3c00195f e984410014      jmp     logexts!LogHook
(50005ae8)
3c001964 b88b020000      mov     eax,28Bh
3c001969 e97a410014      jmp     logexts!LogHook
(50005ae8)
3c00196e b88c020000      mov     eax,28Ch
3c001973 e970410014      jmp     logexts!LogHook
(50005ae8)

Here we can see that logexts patches import table.

And you can trace different API categories:

0:001> !logexts.logc
Categories:
  1 AdvApi32                        Enabled
  2 AtomFunctions                   Enabled
  3 AVIFileExports                  Enabled
  4 Clipboard                       Enabled
  5 ComponentObjectModel            Enabled
  6 DebuggingAndErrorHandling       Enabled
  7 DeviceFunctions                 Enabled
  8 Direct3D                        Enabled
  9 DirectDraw                      Enabled
 10 DirectPlay                      Enabled
 11 DirectSound                     Enabled
 12 GDI                             Enabled
 13 HandleAndObjectFunctions        Enabled
 14 HookingFunctions                Enabled
 15 IOFunctions                     Enabled
 16 MemoryManagementFunctions       Enabled
 17 Multimedia                      Enabled
 18 Printing                        Enabled
 19 ProcessesAndThreads             Enabled
 20 RegistryFunctions               Enabled
 21 Shell                           Enabled
 22 StringManipulation              Enabled
 23 ThreadLocalStorage              Enabled
 24 User32                          Enabled
 25 User32StringExports             Enabled
 26 Version                         Enabled
 27 WinSock2                        Enabled

- Dmitry Vostokov -

WindowHistory 3.0

Monday, January 1st, 2007

WindowHistory tool has been significantly rewritten and improved to make it better for troubleshooting and debugging GUI. What’s new in this version:

  • Real-time support: windows are tracked as they are created and destroyed, their position and size are changed, etc.
  • Dramatically improved speed, no matter how many windows you have in your session WindowHistory is fast and has minimum impact on the system (O(log(n)))
  • Better formatted output
  • Fixed bugs found in previous version
  • Easter egg (hold <Shift> key and click on About button)

 

It is a native Windows application written in C++/STL/MFC/Win32.

There are two packages: WindowHistory32 and WindowHistory64. Both can be downloaded from Citrix support web site:

To use download, unpack and run WindowHistory(64).exe.

To uninstall just remove files.

Note: although 32-bit version will run on x64 Windows too, real-time support for 64-bit application windows will not be available. For x64 Windows please use WindowHistory64 which correctly handles both 64-bit and 32-bit application windows.

The following UML collaboration diagram depicts schematically how WindowHistory64 gets notifications from 32-bit windows:

If you want to track window messages and processes simultaneously run it with MessageHistory and ProcessHistory tools.

 - Dmitry Vostokov -

Notes about NMI_HARDWARE_FAILURE

Saturday, December 23rd, 2006

WinDbg help states that NMI_HARDWARE_FAILURE (0×80) bugcheck 80 indicates a hardware fault. This description can easily lead to a conclusion that a kernel or complete crash dump you just got from your customer doesn’t worth examining. But hardware malfunction is not always the case especially if your customer mentions that their system was hanging and they forced a manual dump. Here I would advise to check whether they have a special hardware for debugging purposes, for example, a card or an integrated iLO chip (Integrated Lights-Out) for remote server administration. Both can generate NMI (Non Maskable Interrupt) on demand and therefore bugcheck the system. If this is the case then it is worth examining their dump to see why the system was hanging.

- Dmitry Vostokov -

Crash Dump Analysis Blog

Saturday, December 23rd, 2006

Welcome to the new blog location at dumpanalysis.org/blog/ 

Its feed address is

http://feeds.feedburner.com/CrashDumpAnalysis

The blog has been moved from its original location at

citrite.org/blogs/dmitryv/ 

in order to bring all crash dump analysis and debugging information to one place including www.dumpanalysis.org/forum and the forthcoming online encyclopedia about assembly languages:

www.asmpedia.org

Thank you and sorry for any inconvenience this might have caused.

Merry Christmas and Happy Debugging in New Year!

- Dmitry Vostokov -

Crash Dump Analysis Patterns (Part 6)

Monday, December 18th, 2006

Now it’s time to ”introduce” Invalid Pointer pattern. It’s just a number saved in a register or in a memory location and when we try to interpret it as a memory address itself and follow it (dereference) to fetch memory contents (value) it points to, OS with the help of hardware tells us that the address doesn’t exist or inaccessible due to security restrictions. The following two slides from my old presentation depict the concept of a pointer:

Pointer definition
Pointers depicted

In Windows you have your process memory partitioned into two big regions: kernel space and process space. Space partition is a different concept than execution mode (kernel or user, ring 0 or ring 3) which is a processor state. Code executing in kernel mode (a driver or OS, for example) can access memory that belongs to user space.

Based on this we can make distinction between invalid pointers containing kernel space addresses (start from 0×80000000 on x86, no /3Gb switch) and invalid pointers containing user space addresses (below 0×7FFFFFFF).

On Windows x64 user space addresses are below 0×0000070000000000 and kernel space addresses start from 0xFFFF080000000000.

When you dereference invalid kernel space address you get bugcheck immediately:

UNEXPECTED_KERNEL_MODE_TRAP (7f)

PAGE_FAULT_IN_NONPAGED_AREA (50)

There is no way you can catch it in your code (by using SEH).

However when you dereference user space address the course of action depends on whether your processor is in kernel mode (ring 0) or in user mode (ring 3). In any mode you can catch the exception (by using appropriate SEH handler) or leave this to the operating system or debugger. If there was no component willing to process the exception when it happened in user mode you get your process crash and in kernel mode you get bugchecks:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e) 

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)

I summarized all of this on the following diagram: 

NULL pointer is a special class of user space pointers. Usually its value is in the range of 0×00000000 - 0×0000FFFF. You can see them used in instructions like

mov   esi, dword ptr [ecx+0×10] 

and ecx value is 0×00000000 so you try to access the value located at 0×00000010 memory address.

When you get a crash dump and you see an invalid pointer pattern the next step is to interpret the pointer value which should help in understanding possible steps that led to the crash. Pointer value interpretation is the subject of the next part.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 5)

Friday, December 15th, 2006

The next pattern I would like to talk about is Optimized Code. If you have such cases you should not trust your crash dump analysis tools like WinDbg. Always suspect that compiler generated code might have been optimized if you see any suspicious or strange behaviour of your tool. Let’s consider this fragment of stack:

Args to Child
77e44c24 000001ac 00000000 ntdll!KiFastSystemCallRet
000001ac 00000000 00000000 ntdll!NtFsControlFile+0xc
00000034 00000bb8 0013e3f4 kernel32!WaitNamedPipeW+0x2c3
0016fc60 00000000 67c14804 MyModule!PipeCreate+0x48

3rd-party function PipeCreate from MyModule opens a named pipe and its first parameter (0016fc60) points to a pipe name L”\\.\pipe\MyPipe”. Inside the source code it calls Win32 API function WaitNamedPipeW (to wait for the pipe to be available for connection) and passes the same pipe name. But we see that  the first parameter to WaitNamedPipeW is 00000034 which cannot be the pointer to a valid Unicode string. And the program should have been crashed if 00000034 were a pointer value.

Everything becomes clear if we look at WaitNamedPipeW disassembly (comments are mine):

0:000> uf kernel32!WaitNamedPipeW
mov     edi,edi
push    ebp
mov     ebp,esp
sub     esp,50h
push    dword ptr [ebp+8]  ; Use pipe name
lea     eax,[ebp-18h]
push    eax
call    dword ptr [kernel32!_imp__RtlCreateUnicodeString (77e411c8)]




call    dword ptr [kernel32!_imp__NtOpenFile (77e41014)]
cmp     dword ptr [ebp-4],edi
mov     esi,eax
jne     kernel32!WaitNamedPipeW+0×1d5 (77e93316)
cmp     esi,edi
jl      kernel32!WaitNamedPipeW+0×1ef (77e93331)
movzx   eax,word ptr [ebp-10h]
mov     ecx,dword ptr fs:[18h]
add     eax,0Eh
push    eax
push    dword ptr [kernel32!BaseDllTag (77ecd14c)]
mov     dword ptr [ebp+8],eax  ; reuse parameter slot

As we know [ebp+8] is the first function parameter in non-FPO calls:

Parameters and Local Variables

And we see it is reused because after we convert LPWSTR to UNICODE_STRING and call NtOpenFile to get a handle we no longer need our parameter slot and the compiler can reuse it to store other information.

There is another compiler optimization we should be aware of and it is called OMAP. It moves the code inside the code section and puts the most frequently accessed code fragments together. In that case if you type in WinDbg, for example, 

0:000> uf nt!someFunction

you get different code than if you type (assuming f4794100 is the address of the function you obtained from stack or disassembly)

0:000> uf f4794100

In conclusion the advice is to be alert and conscious during crash dump analysis and inspect any inconsistencies closer.

Happy debugging!

- Dmitry Vostokov @ DumpAnalysis.org -

New TestDefaultDebugger Tool

Wednesday, December 6th, 2006

It often happens that Citrix support advises customers to change their default postmortem debugger to NTSD. But there is no way to test new settings unless some application crashes again. And some customers come back saying dumps are not saved despite new settings and we don’t know whether dumps were not saved because a crash hadn’t yet happened or default debugger hadn’t been configured properly or something else happened.

In addition the arrival of 64-bit Windows brings another problem: there are 2 default postmortem debuggers on 64-bit Windows (for 32-bit and 64-bit applications respectively):

NTSD on x64 Windows

The new tool TestDefaultDebugger forces a crash on itself to test the presence and configuration of default postmortem debugger (Dr. Watson, NTSD or other). Then if the default postmortem debugger is configured properly OS will launch it to save a dump of TestDefaultDebugger.exe process.

 

If you enabled NTSD as a default postmortem debugger (CTX105888) the following console window will briefly appear:

Postmortem debuggers are explained here:

Dumps for Dummies (Part 3)

On 64-bit Windows you can run both 32-bit TestDefaultDebugger.exe and 64-bit TestDefaultDebugger64.exe applications and then open crash dumps to see whether both postmortem debuggers have been configured properly. The tool has also command line interface so you can use it remotely:

c:\>TestDefaultDebugger.exe now

You can download the tool from Citrix support web site:

TestDefaultDebugger v1.0 for 32-bit and 64-bit platforms

- Dmitry Vostokov @ DumpAnalysis.org -

Dumps and Systems Theory

Friday, November 24th, 2006

The environment where Citrix software operates is so complex that some education in Systems Theory and basic understanding of “cause and effect” and impossibility of “action at a distance” is needed. In forthcoming mini-series I would try to highlight some notions of that.

- Dmitry Vostokov -

Inside Citrix - November 2006

Wednesday, November 22nd, 2006

Welcome to Inside Citrix. This monthly column gives a glimpse of different aspects of Citrix through our people. Our guests have different areas of responsibility and expertise to give you an idea of what is happening behind the scenes. We discuss items of interest with people from Product Readiness, Escalation, Technical Support, and Engineering just to name a few.

In this installment of Inside Citrix, we discuss the meaning of life with Dmitry Vostokov, EMEA Development Analysis Team Lead.

Q: Hello Dmitry, how are you? I am very happy to conduct this interview as you are a creative and prolific worker. I wonder…has fame caught up to you yet, due to your creativity?

A: I’m fine, thank you! I believe there is a synergistic effect going on here. I make the company famous and the company makes me famous.

Q: So, before I get too far ahead of myself, please tell everyone a bit of your history. Where are you from? What did you do before Citrix? How long have you been with us? What kinds of things have you been doing at Citrix during your tenure?

A: I’m from Russia. I was born near Moscow and I spent 14 years there after enrolling at Moscow State University to study chemistry. In that university, I saw a computer and immediately started programming. My first program was written in FORTRAN and had almost 200 lines. My second program had commercial success: I ported 800 FORTRAN lines to about 2000 PDP-11 assembler lines and achieved a 25 percent increase in speed (the program calculated rocket fuel properties for weeks). Since then I’d been working from home for some U.S. and Russian ISV companies (mostly in speech and image processing domains) until 1999, when I went to work in an office to see a large software factory from the inside out.

In 2001 I went to Ireland to learn English. My first job in Ireland was with Ericsson in a small town as a Senior Software Designer. The title sounded great to me, but I heard rumors that the only engineers in Ericsson were hardware engineers. So that job didn’t last long because I was headhunted by a company called Programming Research and I relocated to Dublin. I spent 1.5 years there and after working briefly for a security company (that company is extinct now) I was hired by Citrix. I’ve already spent 3.16 years here. For Citrix I analyze crash dumps and provide recommendations. It’s like being a computer psychologist assessing brain damage. I also do a bit of escalation work when I have time. I like to provide full escalation and software maintenance cycles whenever I have sufficient resources to analyze the problem, contact the customer, and provide the resolution. I also have an opportunity here to apply my software design and programming skills by writing various troubleshooting tools.

Q: Most people probably didn’t know all of that. I guarantee you that Escalation knows you well. How is the blogging going? How can readers get to your blog?

A: I love blogging. I didn’t even think about blogging until I suddenly realized its potential in information sharing. When I joined the company there was no sufficient information available about crash dump analysis, so I had to learn on my own. Now I’m happy to share what I have learnt to everyone.

One topic I like to write about in my blog at the moment is crash dump analysis patterns and anti-patterns, where I summarize general solutions you can apply or should not apply in specific contexts to common recurrent dump analysis problems.

More will come…

Q: And the tools that you create, very useful! Can you take a moment to talk about each of the ones you have created? Which ones have you gotten the best feedback about? Which ones have been the most useful?

A: Thanks! I use them too. The tool I got the most complaints about is RepairCBDChain; the tool with the fewest complaints is SystemDump. I got the best feedback about PDBFinder.

All of them are useful in certain troubleshooting scenarios. I’m preparing a presentation about all these tools and I will present it to the EMEA TRM team in December. I’ll definitely publish it as soon as I get feedback about that training.

Here are brief descriptions of these tools (most of them have different versions for various platforms, and some were even ported to Windows Mobile):

• RepairCBDChain: Repairs clipboard functionality and magically you are able to copy/paste again (not always actually – I promise to write a blog post explaining why).

• ADSCleaner: Cleans Windows NT File System (NTFS) file streams created by Citrix memory optimization code if you no longer need this feature (it also frees disk space, by the way).

• ProcessHistory: Tracks processes, threads, and modules on 32-bit and 64-bit platforms. I’m going to release a Windows Mobile version soon.

• MessageHistory: Tracks window messages. It’s similar to Spy++ but much easier to use for troubleshooting and it works on 64-bit platforms too.

• WindowHistory: Tracks windows as they change their appearance, are created, and are destroyed and saves a log file. This is what Spy++ lacks and it was the primary motivation to write this tool.

• SystemDump: Forces a dump immediately or after a specified period of time. This can be done remotely too. It works on both 32-bit and 64-bit Windows! My primary motivation was that the OSR “bang” tool doesn’t work on 64-bit Windows.

• PDBFinder: Helps to find symbol files if you have zillions of them.

• DumpCheck: Verifies that you have a valid dump and even provides recommendations to avoid common mistakes before sending dumps to support.

• CtxHidEx32: Can hide any annoying windows or message boxes and reduce unnecessary support calls. It also has a peculiar feature: you can specify an action to do before hiding the window. When the Media Player window appears it can send a message to your boss.

• Dump2Wave: My most controversial tool that allows you to hear the sound of memory corruption. Some people say it’s useless but I would say it is entertaining.

Some other upcoming tools I’m working days and nights on (when I have free time) are:

• DumpDepends: Helps to automate repetitive dumping.

• DumpAlerts: Provides notification whenever new dump is saved.

• SessionHistory: Tracks session information.

• HistoryToolbar: Organizes “History” tools into one coherent super tool.

• DumpPlayer: Plays musical dumps in real-time and provides visual images based on crash dump memory contents. I coined a term—Dump Tomography—for this.

Q: They must take some upkeep, as we see a lot of improvements, updates, and so on. I also see you provide a lot of training information on escalation techniques, debugging, analysis, and more. What do you believe is the most important characteristic of a successful escalation engineer?

A: As Winston Churchill said: “Never, never, never give up!”?

Q: Any advice for Citrix administrators who might be reading this on how to avoid trouble or have their environment best situated to speed resolution, should an issue occur?

A: If you are asked to generate and/or collect crash dumps, please tell support personnel how you got that dump. And ensure that you are sending the right dump for the right issue.

I started writing Dumps for Dummies blog posts to explain dumps and I promise to continue and expand them.

Q: What do you find most challenging about your job?

A: To work with enormous amounts of information and make quick decisions at the same time.

Q: Is there anything you can share with us about new Citrix products or technologies (not giving away confidential information) that you are excited about?

A: I would tell you that with whatever new technology comes along, crash dumps will be the same! And this gives me some optimism. Whether there will be more or less crash dumps in the future is pretty confidential though…

Q: Any plans to visit Citrix headquarters in Fort Lauderdale, Florida?

A: I’m actually visiting Citrix headquarters at the end of this month! See you there.

Q: Not so much a question, make us laugh!

A: One day we got a fax from a customer where all of the blue screen information was written down by hand—hundreds of digits… How long it took to copy all that from the screen and whether or not he made any mistakes, we will never know. The copy from that fax is still hanging on my desk wall.

Q: What do you do in your free time besides analyzing dumps, debugging and programming?

A: Read books. I read lots of them and about quite diverse subjects. However, my favorite subject for the last four years has been math—the more abstract the better.

It really helps in improving the critical thinking skills required for my job.

Thanks, Dmitry. People will know to look you up online…

Horrors of debugging legacy code

Wednesday, November 8th, 2006

We all know that macro definitions in C and C++ are evil. They cause maintenance nightmares by introducing subtle bugs. I never took that seriously until last weekend I was debugging my old code written 10 years ago which uses macros written 15 years ago :-) 

My Windows Mobile 5.0 application was crashing when I was using POOM COM interfaces (Pocket Outlook Object Model). The crash never pointed to my code. It always happened after pimstore.dll and other MS modules were loaded and COM interfaces started to return errors. I first suspected that I was using POOM incorrectly and rewrote all code several times and in different ways. No luck. Then I tried PoomMaster sample from Windows Mobile 5.0 SDK and it worked well. So I rewrote my code in exactly the same way as in that sample. No luck. My last hope was that moving code from my DLL to EXE (as in sample SDK project) would eliminate crashes but it didn’t help too. Then I slowly started to realize that the problem might have been in my old code and I also noticed that one old piece of code had never been used before. So I started debugging by elimination (commenting out less and less code) until I found a macro. I had to stare at it for couple of minutes until I realized that one pair of brackets was missing and that caused allocating less memory and worse: the returned pointer to allocated memory was multiplied by 2! So the net result was the pointer pointing to other modules and subsequent string copy was effectively overwriting their memory and eventually causing crashes inside MS dlls.  

Here is that legacy macro:

#define ALLOC(t, p, s)
((p)=(t)GlobalLock(GlobalAlloc(GHND, (s))))

It allocates memory and returns a pointer. It should have been called like this (size parameter is highlighted in blue):

if (ALLOC(LPWSTR,lpm->lpszEvents,
(lstrlen(lpszMacro)+1)*sizeof(WCHAR)))
{
lstrcpy(lpm->lpszEvents, lpszMacro);
    lpm->nEvents=lstrlen(lpm->lpszEvents)+1;
}

What I found is the missing bracket before lstrlen and last enclosing bracket (size parameter is highlighted in red):

if (ALLOC(LPWSTR,lpm->lpszEvents,
lstrlen(lpszMacro)+1)*sizeof(WCHAR))
{
lstrcpy(lpm->lpszEvents, lpszMacro);
    lpm->nEvents=lstrlen(lpm->lpszEvents)+1;
}

The resulted code after macro expansion looks like this

if (lpm->lpszEvents=(LPWSTR)GlobalLock(GlobalAlloc(GHND,
lstrlen(lpszMacro)+1))*sizeof(WCHAR))

You see that the pointer to allocated memory is multiplied by two and string copy is performed to a random place in the address space of other loaded dlls corrupting their data and causing the process to crash later.

- Dmitry Vostokov -

Debugging the Debugger

Friday, October 20th, 2006

Have you ever tried to debug a debugger when it debugs a debuggee? Is it possible? Good question. I never asked it to myself until today. And tried. And it works! First I tried to attach WinDbg.exe to an instance of WinDbg.exe executing ‘!analyze -v’ command and got these stacks:

0:002> ~*kL 100
0 Id: 1ff0.104c Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr
0006df38 7739d02f ntdll!KiFastSystemCallRet
0006ff7c 01055e36 USER32!NtUserWaitMessage+0xc
0006ffc0 77e523e5 windbg!_initterm_e+0x170
0006fff0 00000000 kernel32!BaseProcessStart+0x23

1 Id: 1ff0.1af8 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
00ac3448 030a5677 dbghelp!CAllPubNameTrav::next+0x1b
00ac345c 0301e16e
dbghelp!CDiaEnumTraversalCSymRow::Next+0x48
00ac44fc 0301e452 dbghelp!diaGetGlobals+0x8fe
00ac4524 0304967a dbghelp!diaGetSymbols+0x42
00ac453c 03045ca3 dbghelp!diaEnumSymbols+0x1a
00ac4554 03031e5a dbghelp!modEnumSymbols+0x43
00ac459c 030338a5 dbghelp!ModLoop+0x10a
00ac6570 030391d8 dbghelp!EnumSymbols+0x155
00ac65a0 0220947b dbghelp!SymEnumSymbolsW+0x48
00ac7600 0220a53d dbgeng!FindTypeInfoInMod+0x18b
00aca5cc 0220caa2 dbgeng!TypeInfoFound+0xced
00acb62c 0220c95f dbgeng!SymbolTypeDumpNew+0xa2
00acb654 0220d36b dbgeng!FastSymbolTypeDump+0xef
00acb700 0213c753 dbgeng!SymbolTypeDump+0xbb
00acc25c 0147d632 dbgeng!ExtIoctl+0x1073
00acc2f4 0150e10e ext!GetFieldData+0xe2
00accc14 014f9f00 ext!UaThread::_Extract_UIThread+0x34e
00accc24 014fa1f9 ext!UaThread::CallExtractors+0x20
00accc34 01511126 ext!UaThread::ExtractAttributes+0x99
00accd78 015212e2
ext!UserAnalyze::ExtractAttributes+0x376
00acd02c 01521467 ext!UeFillAnalysis+0x462
00acd10c 01521650 ext!UeAnalyze+0x147
00acd208 0147c90c ext!AnalyzeUserException+0x1a0
00acd23c 02141299 ext!analyze+0x28c
00acd2c8 021414d9 dbgeng!ExtensionInfo::CallA+0x2e9
00acd458 021415a2 dbgeng!ExtensionInfo::Call+0x129
00acd474 0213feb1 dbgeng!ExtensionInfo::CallAny+0x72
00acd8ec 02181698 dbgeng!ParseBangCmd+0x661
00acd9dc 02182b29 dbgeng!ProcessCommands+0x508
00acda20 020c9049 dbgeng!ProcessCommandsAndCatch+0x49
00acdeb8 020c92aa dbgeng!Execute+0x2b9
00acdee8 010283bf dbgeng!DebugClient::ExecuteWide+0x6a
00acdf88 0102883b windbg!ProcessCommand+0xff
00acffa4 0102aabc windbg!ProcessEngineCommands+0x8b
00acffb8 77e6608b windbg!EngineLoop+0x3dc
00acffec 00000000 kernel32!BaseThreadStart+0x34

# 2 Id: 1ff0.116c Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
00fdffc8 7c845ea0 ntdll!DbgBreakPoint
00fdfff4 00000000 ntdll!DbgUiRemoteBreakin+0x36

Next I thought, wait a moment, we are debugging dump analysis session. Can we debug a debugger debugging a running process? So I attached WinDbg.exe to an instance of WinDbg.exe attached to an instance of notepad.exe and got these stacks:

0:002> ~*kL
0 Id: 11f0.164c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0006df38 7739d02f ntdll!KiFastSystemCallRet
0006ff7c 01055e36 USER32!NtUserWaitMessage+0xc
0006ffc0 77e523e5 windbg!_initterm_e+0x170
0006fff0 00000000 kernel32!BaseProcessStart+0x23

1 Id: 11f0.1bb0 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
00adff0c 7c822124 ntdll!KiFastSystemCallRet
00adff10 77e6bad8 ntdll!NtWaitForSingleObject+0xc
00adff80 020bf8aa kernel32!WaitForSingleObjectEx+0xac
00adffa0 0102aa42
dbgeng!DebugClient::DispatchCallbacks+0x4a
00adffb8 77e6608b windbg!EngineLoop+0x362
00adffec 00000000 kernel32!BaseThreadStart+0x34

# 2 Id: 11f0.100c Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr
00beffc8 7c845ea0 ntdll!DbgBreakPoint
00befff4 00000000 ntdll!DbgUiRemoteBreakin+0x36

Given that many functions from dbghelp.dll and dbgeng.dll are described in WinDbg help you can quickly reverse engineer WinDbg.exe and its extensions code.

- Dmitry Vostokov -