Software Diagnostics Library

Suppose we got a kernel, complete or minidump from Windows running under Xen hypervisor. How would we distinguish it from a memory dump of Windows running on non-virtualized hardware? We can check machine id:

kd> !sysinfo machineid
Machine ID Information [From Smbios 2.4, DMIVersion 36, Size=348]
BiosMajorRelease = 3
BiosMinorRelease = 1
BiosVendor = Xen
BiosVersion = 3.1.0
SystemManufacturer = Xen
SystemProductName = HVM domU
SystemVersion = 3.1.0

and drivers:

kd> lm m *xen*
start end module name
f6012000 f605f000 dump_xenvbd (deferred)
f794b000 f795c000 xennet (deferred)
f82c0000 f830d000 xenvbd (deferred)
f845f000 f846b000 XENUTIL (deferred)
f84cf000 f84db000 dump_XENUTIL (deferred)

Note: similar information can be checked for VMWare and Virtual PC.

- Dmitry Vostokov @ DumpAnalysis.org -

Here is an additional kernel space example to my old Dynamic Memory Corruption pattern. If kernel pools are corrupt then calls that allocate or free memory result in bugchecks C2 or 19 and in other less frequent bugchecks (from Google stats):

Bug Checks 0xC2 and 0×19 have parameters in bugcheck arguments that tell the type of detected pool corruption. Refer to WinDbg help for details or use the variant of !analyze command where you can supply optional bugcheck arguments:

1: kd> !analyze -show c2
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request.  Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000000, The caller is requesting a zero byte pool allocation.
Arg2: 00000000, zero.
Arg3: 00000000, the pool type being allocated.
Arg4: 00000000, the pool tag being used.

1: kd> !analyze -show 19 2 1 1 1
BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 00000002, the verifier pool pattern check failed.  The owner has likely corrupted the pool block
Arg2: 00000001, the pool entry being checked.
Arg3: 00000001, size of the block.
Arg4: 00000001, 0.

If we enable special pool on suspected drivers we might get these bugchecks too with the following Google frequency:

Here is one example of nonpaged pool corruption detected during free operation with the following !analyze -v output:

BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 00000020, a pool block header size is corrupt.
Arg2: a34583b8, The pool entry we were looking for within the page.
Arg3: a34584f0, The next pool entry.
Arg4: 0a270001, (reserved)

POOL_ADDRESS:  a34583b8 Nonpaged pool

PROCESS_NAME:  process.exe

CURRENT_IRQL:  2

STACK_TEXT: 
b80a60cc 808927bb nt!KeBugCheckEx+0x1b
b80a6134 80892b6f nt!ExFreePoolWithTag+0x477
b80a6144 b9591400 nt!ExFreePool+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
b80a615c b957b954 driver+0x38400
b80a617c b957d482 driver+0x22954
b80a61c0 b957abf4 driver+0x24482
b80a6260 b957ccef driver+0x21bf4
b80a62a8 8081df65 driver+0x23cef
b80a62bc f721ac45 nt!IofCallDriver+0x45
b80a62e4 8081df65 fltMgr!FltpDispatch+0x6f
b80a62f8 b99de70b nt!IofCallDriver+0x45
b80a6308 b99da6ee filter!Dispatch+0xfb
b80a6318 8081df65 filter!dispatch+0x6e
b80a632c b9bdebfe nt!IofCallDriver+0x45
b80a6334 8081df65 2ndfilter!Redirect+0x7ea
b80a6348 b9bd1756 nt!IofCallDriver+0x45
b80a6374 b9bd1860 3rdfilter!PassThrough+0x136
b80a6384 8081df65 3rdfilter!Dispatch+0x80
b80a6398 808f5437 nt!IofCallDriver+0x45
b80a63ac 808ef963 nt!IopSynchronousServiceTail+0x10b
b80a63d0 8088978c nt!NtQueryDirectoryFile+0x5d
b80a63d0 7c8285ec nt!KiFastCallEntry+0xfc
00139524 7c8274eb ntdll!KiFastSystemCallRet
00139528 77e6ba40 ntdll!NtQueryDirectoryFile+0xc
00139830 77e6bb5f kernel32!FindFirstFileExW+0x3d5
00139850 6002665e kernel32!FindFirstFileW+0x16
00139e74 60026363 process+0x2665e
0013a328 60027852 process+0x26363
0013a33c 60035b58 process+0x27852
0013b104 600385ff process+0x35b58
0013b224 612cb643 process+0x385ff
0013b988 612cc109 dll!FileDialog+0xc53
0013bba0 612cb47b dll!FileDialog+0x1719
0013c2c0 7739b6e3 dll!FileDialog+0xa8b
0013c2ec 77395f82 USER32!InternalCallWinProc+0x28
0013c368 77395e22 USER32!UserCallDlgProcCheckWow+0x147
0013c3b0 7739c9c6 USER32!DefDlgProcWorker+0xa8
0013c3d8 7c828536 USER32!__fnDWORD+0x24
0013c3d8 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
b80a66b8 8091d6d1 nt!KiCallUserMode+0x4
b80a6710 bf8a2622 nt!KeUserModeCallback+0x8f
b80a6794 bf8a2517 win32k!SfnDWORD+0xb4
b80a67dc bf8a13d9 win32k!xxxSendMessageToClient+0x133
b80a6828 bf85ae67 win32k!xxxSendMessageTimeout+0x1a6
b80a684c bf8847a1 win32k!xxxWrapSendMessage+0x1b
b80a6868 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
b80a68a0 8088978c win32k!NtUserMessageCall+0xc0
b80a68a0 7c8285ec nt!KiFastCallEntry+0xfc
0013c3d8 7c828536 ntdll!KiFastSystemCallRet
0013c3d8 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
b80a6b7c 8091d6d1 nt!KiCallUserMode+0x4
b80a6bd4 bf8a2622 nt!KeUserModeCallback+0x8f
b80a6c58 bf8a23a0 win32k!SfnDWORD+0xb4
b80a6ca0 bf8a13d9 win32k!xxxSendMessageToClient+0x118
b80a6cec bf85ae67 win32k!xxxSendMessageTimeout+0x1a6
b80a6d10 bf8c148c win32k!xxxWrapSendMessage+0x1b
b80a6d40 8088978c win32k!NtUserMessageCall+0x9d
b80a6d40 7c8285ec nt!KiFastCallEntry+0xfc
0013f474 7c828536 ntdll!KiFastSystemCallRet
0013f4a0 7739d1ec ntdll!KiUserCallbackDispatcher+0x2e
0013f4dc 7738cf29 USER32!NtUserMessageCall+0xc
0013f4fc 612d3276 USER32!SendMessageA+0x7f
0013f63c 611add41 dll!SubWindow+0x3dc6
0013f658 7739b6e3 dll!SetWindowText+0x37a1
0013f684 7739b874 USER32!InternalCallWinProc+0x28
0013f6fc 7739ba92 USER32!UserCallWinProcCheckWow+0x151
0013f764 7739bad0 USER32!DispatchMessageWorker+0x327
0013f774 61221ca8 USER32!DispatchMessageW+0xf
0013f7e0 0040156d dll!MainLoop+0x2c8
0013ff24 00401dfa process+0x156d
0013ffc0 77e6f23b process+0x1dfa
0013fff0 00000000 kernel32!BaseProcessStart+0x23

MODULE_NAME: driver

IMAGE_NAME:  driver.sys

We see that WinDbg pointed to driver.sys by using a procedure described in one of my old minidump analysis posts: BugCheck C2 Minidump Analysis. 

However any OS component could corrupt the pool prior to detection as the bugcheck description says: “The pool is already corrupt at the time of the current request.”. What other evidence can reinforce our belief in driver.sys? Let’s look at our pool entry tag first:

1: kd> !pool a34583b8
Pool page a34583b8 region is Nonpaged pool
 a3458000 size:  270 previous size:    0  (Allocated)  Thre (Protected)
 a3458270 size:   10 previous size:  270  (Free)       RxIr
 a3458280 size:   40 previous size:   10  (Allocated)  Vadl
 a34582c0 size:   98 previous size:   40  (Allocated)  File (Protected)
 a3458358 size:    8 previous size:   98  (Free)       Vadl
 a3458360 size:   50 previous size:    8  (Allocated)  Gsem
 a34583b0 size:    8 previous size:   50  (Free)       CcSc
*a34583b8 size:  138 previous size:    8  (Allocated) *DRIV
  Owning component : Unknown (update pooltag.txt)
a34584f0 is not a valid large pool allocation, checking large session pool…
a34584f0 is freed (or corrupt) pool
Bad allocation size @a34584f0, zero is invalid

***
*** An error (or corruption) in the pool was detected;
*** Attempting to diagnose the problem.
***
*** Use !poolval a3458000 for more details.
***

Pool page [ a3458000 ] is __inVALID.

Analyzing linked list...
[ a34583b8 --> a34583d8 (size = 0x20 bytes)]: Corrupt region
[ a34583f8 --> a34585e8 (size = 0x1f0 bytes)]: Corrupt region

Scanning for single bit errors...

None found

We see that the tag is DRIV and we know either from association or from similar problems in the past that it belongs to driver.sys. Let’s dump our pool entry contents to see if there are any symbolic hints in it:

1: kd> dps a34583b8
a34583b8 0a270001
a34583bc 5346574e
a34583c0 00000000
a34583c4 00000000
a34583c8 b958f532 driver+0×36532
a34583cc a3471010
a34583d0 0000012e
a34583d4 00000001
a34583d8 00041457
a34583dc 05af0026
a34583e0 00068002
a34583e4 7b9ec6f5
a34583e8 ffffff00
a34583ec 73650cff
a34583f0 7461445c
a34583f4 97a10061
a34583f8 ff340004
a34583fc c437862a
a3458400 6a000394
a3458404 00000038
a3458408 00000000
a345840c bf000000
a3458410 bf0741b5
a3458414 f70741b5
a3458418 00000000
a345841c 00000000
a3458420 00000000
a3458424 00000000
a3458428 05000000
a345842c 34303220
a3458430 31323332
a3458434 ff322d36

Indeed we see the possible code pointer driver+0×36532 and the code around this address looks normal:

3: kd> .asm no_code_bytes
Assembly options: no_code_bytes

3: kd> u b958f532
driver+0x36532:
b958f532 push    2Ch
b958f534 push    offset driver+0x68d08 (b95c1d08)
b958f539 call    driver+0x65c50 (b95bec50)
b958f53e mov     byte ptr [ebp-19h],0
b958f542 and     dword ptr [ebp-24h],0
b958f546 call    dword ptr [driver+0x65f5c (b95bef5c)]
b958f54c mov     ecx,dword ptr [ebp+0Ch]
b958f54f cmp     eax,ecx

3: kd> ub b958f532
driver+0x36528:
b958f528 leave
b958f529 ret     18h
b958f52c int     3
b958f52d int     3
b958f52e int     3
b958f52f int     3
b958f530 int     3
b958f531 int     3

- Dmitry Vostokov @ DumpAnalysis.org -

I read science fiction from time to time now (I was a big fan of it back to school and university days) but I cannot recall memory dumps mentioned explicitly in such books. I’ve just finished reading Dan Simmons’s The Fall of Hyperion book (the sequel to Hyperion book that I read previously) and I recall in chapter 33 on page 303 a poetic description of a process crash (italics are mine):

“Johnny twists a second in the AI’s massive grip (fault injection?), and then his analog - Keats’s small but beautiful body (GUI?) - is torn, compacted, smashed into an unrecognizable mass (corrupt dump?) which Ummon sets against his megalith flesh (private bytes?), absorbing the analogs’s remains (overwriting discarded pages?) back into the orange-and-red depths of itself (working set?).”  

PS: Hyperion and The Fall of Hyperion is the best science fiction I have ever read and highly recommend:

Hyperion

The Fall of Hyperion

I continue looking for crash dumps in Dan Simmons’s Endymion and Rise of Endymion books which are on my lunch time reading list as soon as I finish Global Conspiracy book I’m reading now.  

- Dmitry Vostokov @ DumpAnalysis.org -

259 bugchecks are documented in WinDbg help. I did Google search for every one and here is their search results distribution graph cut off for data with less than 10 matches:

Of course there is some noise and matches do not always correspond to WinDbg bugcheck analysis output but we can get rough idea about bugcheck frequency. For example, unhandled exceptions in kernel mode, IRQL contract violation, pool corruption and hardware failures are the most frequent. Here is the full table:

- Dmitry Vostokov @ DumpAnalysis.org -

Now it’s time to write about Wild Pointer pattern I briefly mentioned in Local Buffer Overflow post which shows examples of pointers having UNICODE structure in their values. I have also observed pointers with ASCII structure as .formats WinDbg command indicates:

FAILED_INSTRUCTION_ADDRESS: 
65747379`735c5357 ??              ???

IP_ON_HEAP:  65747379735c5357

0:012> .formats rip
Evaluate expression:
  Hex:     65747379`735c5357
  Decimal: 7310595060592825175
  Octal:   0625643467456327051527
  Binary:  01100101 01110100 01110011 01111001 01110011 01011100 01010011 01010111
  Chars:   etsys\SW
  Time:    Wed May 10 22:00:59.282 24767 (GMT+1)
  Float:   low 1.7456e+031 high 7.21492e+022
  Double:  5.30388e+180

Here is another example of a pointer having UNICODE structure:

FAILED_INSTRUCTION_ADDRESS: 
0045004c`00490046 ??              ???

0:014> .formats rip
Evaluate expression:
  Hex:     0045004c`00490046
  Decimal: 19422099815333958
  Octal:   0001050004600022200106
  Binary:  00000000 01000101 00000000 01001100 00000000 01001001 00000000 01000110
  Chars:   .E.L.I.F
  Time:    Wed Jul 19 07:46:21.533 1662 (GMT+1)
  Float:   low 6.70409e-039 high 6.33676e-039
  Double:  2.33646e-307

When we have EIP or RIP pointers I have another pattern to name when the value is coincidentally lies inside some valid region of memory: Wild Code. Here is one example of the latter pattern I’m going to write about soon:

IP_ON_STACK:
+e11ffa8

STACK_TEXT:
0e11ff7c 098eeef2 0xe11ffa8
0e11ff84 77b6b530 dll!StartWorking+0xcab
0e11ffb8 7c826063 msvcrt!_endthreadex+0xa3
0e11ffec 00000000 kernel32!BaseThreadStart+0×34

0:020> u
0e11ffa8 dcff fdiv st(7),st
…
…
…

We see that EIP is very close to EBP/ESP and this explains why !analyze -v reports IP_ON_STACK. Clearly floating-point code is not what we should expect. This example shows that wild pointers sometimes are valid but either through code chain or pointer chain execution reaches Invalid Pointer and a process or a system crashes.

- Dmitry Vostokov @ DumpAnalysis.org -

My colleagues and friends have been divided (50-50) over the opinion which draft cover is better (some polishing is required for both of them):

1. Original Draft Cover
2. New Draft Cover

Your comments are much appreciated (you can just respond with the numbers 1 or 2 corresponding to cover links above). You can really influence the cover of the book!

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced draft cover for Memory Dump Analysis Anthology has got new look and feel:

Books symbolize modular memory structure found in Windows memory dumps (see Dump2Picture paintings) and blue strip separating cover text and books symbolizes familiar blue screen).

- Dmitry Vostokov @ DumpAnalysis.org -

I was wondering whether there are any tools that allow to step through unmanaged and native code backwards when we do crash dump analysis or live debugging on Windows platforms and found these interesting references from Microsoft Research:

• iDNA: Time Travel Debugging (PPT)

• Framework for Instruction-level Tracing and Analysis of Program Executions (PDF)

It would be really good if Microsoft integrates this into Debugging Tools for Windows. 

- Dmitry Vostokov @ DumpAnalysis.org -

They know buzzwords like heap corruption, buffer overflow and multi-threading, talk about designing maintainable software but unable to cope with real-life debugging scenarios. I remember old days in one company where the executable Rational Rose Real-Time model with IDebug interface crashed and the whole team of software designers couldn’t find the defect in code and when I pointed them to the problem source code line after loading and running the executable under the Visual Studio debugger I was nominated as an expert in implementation.

- Dmitry Vostokov @ DumpAnalysis.org -

Q. Every time you open the specific Microsoft Word 2007 document on Vista WER error message box appears on the screen (click on the picture to enlarge):

What might be the cause of it?

You can find the correct answer in the comments to this post. 

- Dmitry Vostokov @ DumpAnalysis.org -

I decided to elaborate a bit on how to see whether the specific synchronization object has signaled or not. If that object has signaled (is in signaled state) then any wait on it will be satisfied immediately. All such objects like kernel events, processes and threads are structures in memory and have _OBJECT_HEADER structure followed by _DISPATCHER_HEADER structure and then by the actual object body (if it is necessary). These objects can be found in the output of !process or !thread commands where they are listed as being waited for, for example:

THREAD 88865db0  Cid 05d0.5ea8  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89a65a68  SynchronizationEvent
Not impersonating
DeviceMap                 e1536358
Owning Process            8abbdd88       Image:         svchost.exe
Wait Start TickCount      5303768        Ticks: 1132797 (0:04:54:59.953)
Context Switch Count      193            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×7c82b5f3)
Stack Init b8ca2000 Current b8ca1c60 Base b8ca2000 Limit b8c9f000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b8ca1c78 80832f7a nt!KiSwapContext+0×26
b8ca1ca4 8082925c nt!KiSwapThread+0×284
b8ca1cec 80937e6a nt!KeWaitForSingleObject+0×346
b8ca1d50 80888c7c nt!NtWaitForSingleObject+0×9a
b8ca1d50 7c94ed54 nt!KiFastCallEntry+0xfc
00c6fea0 7c942124 ntdll!KiFastSystemCallRet
00c6fea4 7c95970f ntdll!NtWaitForSingleObject+0xc
00c6fee0 7c959620 ntdll!RtlpWaitOnCriticalSection+0×19c
00c6ff00 7c95c1e7 ntdll!RtlEnterCriticalSection+0xa8
00c6ffa8 7c8261d6 ntdll!LdrShutdownThread+0×33
00c6ffb8 7c826090 kernel32!ExitThread+0×2f
00c6ffec 00000000 kernel32!BaseThreadStart+0×39

We can see them as parameters of KeWaitForSingle(Multiple)Objects functions:

0: kd> !thread 88865db0
THREAD 88865db0  Cid 05d0.5ea8  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89a65a68  SynchronizationEvent
Not impersonating
DeviceMap                 e1536358
Owning Process            8abbdd88       Image:         svchost.exe
Wait Start TickCount      5303768        Ticks: 1132797 (0:04:54:59.953)
Context Switch Count      193            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×7c82b5f3)
Stack Init b8ca2000 Current b8ca1c60 Base b8ca2000 Limit b8c9f000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr  Args to Child             
b8ca1c78 80832f7a 88865e28 88865db0 88865e58 nt!KiSwapContext+0×26
b8ca1ca4 8082925c 00000000 00000000 00000000 nt!KiSwapThread+0×284
b8ca1cec 80937e6a 89a65a68 00000006 00000001 nt!KeWaitForSingleObject+0×346
b8ca1d50 80888c7c 00000470 00000000 00000000 nt!NtWaitForSingleObject+0×9a
b8ca1d50 7c94ed54 00000470 00000000 00000000 nt!KiFastCallEntry+0xfc
00c6fea0 7c942124 7c95970f 00000470 00000000 ntdll!KiFastSystemCallRet
00c6fea4 7c95970f 00000470 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
00c6fee0 7c959620 00000000 00000004 009171a8 ntdll!RtlpWaitOnCriticalSection+0×19c
00c6ff00 7c95c1e7 7c9a9d94 00000000 7ffaa000 ntdll!RtlEnterCriticalSection+0xa8
00c6ffa8 7c8261d6 00000000 00000000 00c6ffec ntdll!LdrShutdownThread+0×33
00c6ffb8 7c826090 00000000 00000000 00000000 kernel32!ExitThread+0×2f
00c6ffec 00000000 77c2de6d 009171a8 00000000 kernel32!BaseThreadStart+0×39

We can also see them as part of other structures, for example:

0: kd> !irp 88f4fd28 1
Irp is active with 1 stacks 1 is current (= 0x88f4fd98)
 No Mdl: No System Buffer: Thread 8a9449a0:  Irp stack trace. 
Flags = 00000800
ThreadListEntry.Flink = 8a944ba8
ThreadListEntry.Blink = 8a944ba8
IoStatus.Status = 00000000
IoStatus.Information = 00000000
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00dddf04
UserEvent = 8a033c88
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00dddf04
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = f75c815c   Npfs!NpCancelListeningQueueIrp
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = 88f4fd68
Tail.Overlay.Thread = 8a9449a0
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = e311f7d4
Tail.Overlay.ListEntry.Blink = e311f7d4
Tail.Overlay.CurrentStackLocation = 88f4fd98
Tail.Overlay.OriginalFileObject = 884bc8a0
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
>[  d, 0]   5  1 8a937398 884bc8a0 00000000-00000000    pending
        \FileSystem\Npfs
   Args: 00000000 00000000 00110008 00000000

0: kd> !object 8a033c88
Object: 8a033c88  Type: (8ad7a990) Event
    ObjectHeader: 8a033c70 (old version)
    HandleCount: 1  PointerCount: 3

They can also be found in process or kernel (System process) handle tables: 

0: kd> !process 0 0
...
...
...
PROCESS 8a8fa8c0  SessionId: 0  Cid: 037c    Peb: 7ffd8000  ParentCid: 04ac
    DirBase: f3b10360  ObjectTable: e1c276b8  HandleCount: 500.
    Image: MyService.exe
...
...
...

0: kd> !kdexts.handle 0 3 037c
...
...
...
0510: Object: 89237d88  GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88  Type: (8ad84900) Process
    ObjectHeader: 89237d70 (old version)
        HandleCount: 1  PointerCount: 2
…
…
…

Let’s look at the last process object 89237d88. This is the start of object-specific structure whose first member is _DISPATCHER_HEADER. Every object-specific structure is preceded in memory by _OBJECT_HEADER structure and in our case its address is 89237d70. To illustrate this memory layout we can write this inheritance relationship in C++ pseudo-code where I put relevant structure members:    

struct _KPROCESS : _DISPATCHER_HEADER
{
   // ... members
};

struct _DISPATCHER_HEADER
{
   UChar Type;
   // ... members
   Int4B SignalState; 
   // ... members
};

struct _OBJECT_HEADER
{
   Int4B PointerCount;
   Int4B HandleCount; 
   // ... members
   _OBJECT_TYPE *Type;
   // ... members
   _QUAD Body; // Int8B - first 8 bytes of _DISPATCHER_HEADER  
};

HandleCount is the number of open handles and PointerCount is the number of dereferences.  

I also put all this on the following simplified UML diagram:

Here is the detailed data for our process object: 

0: kd> dt _OBJECT_HEADER 89237d70
ntdll!_OBJECT_HEADER
   +0x000 PointerCount     : 2
   +0x004 HandleCount      : 1
   +0x004 NextToFree       : 0x00000001
   +0x008 Type             : 0x8ad84900 _OBJECT_TYPE
   +0x00c NameInfoOffset   : 0 ''
   +0x00d HandleInfoOffset : 0 ''
   +0x00e QuotaInfoOffset  : 0 ''
   +0x00f Flags            : 0x20 ' '
   +0x010 ObjectCreateInfo : 0x808ad180 _OBJECT_CREATE_INFORMATION
   +0x010 QuotaBlockCharged : 0x808ad180
   +0x014 SecurityDescriptor : 0xe1002bd6
   +0x018 Body             : _QUAD

0: kd> dt _DISPATCHER_HEADER 89237d88
ntdll!_DISPATCHER_HEADER
   +0x000 Type             : 0x3 ''
   +0x001 Absolute         : 0 ''
   +0x001 NpxIrql          : 0 ''
   +0x002 Size             : 0x1e ''
   +0x002 Hand             : 0x1e ''
   +0x003 Inserted         : 0 ''
   +0x003 DebugActive      : 0 ''
   +0x000 Lock             : 1966083
   +0×004 SignalState      : 1
   +0×008 WaitListHead     : _LIST_ENTRY [ 0×89237d90 - 0×89237d90 ]

0: kd> dt _OBJECT_TYPE 0x8ad84900
ntdll!_OBJECT_TYPE
   +0x000 Mutex            : _ERESOURCE
   +0x038 TypeList         : _LIST_ENTRY [ 0x8ad84938 - 0x8ad84938 ]
   +0×040 Name             : _UNICODE_STRING “Process”
   +0×048 DefaultObject    : (null)
   +0×04c Index            : 5
   +0×050 TotalNumberOfObjects : 0×18d
   +0×054 TotalNumberOfHandles : 0×35d
   +0×058 HighWaterNumberOfObjects : 0×2d4
   +0×05c HighWaterNumberOfHandles : 0xd66
   +0×060 TypeInfo         : _OBJECT_TYPE_INITIALIZER
   +0×0ac Key              : 0×636f7250
   +0×0b0 ObjectLocks      : [4] _ERESOURCE

We see that SignalState is 1 and this means that the process is in signaled state (exited) and the wait on it will be satisfied immediately (WaitForSingleObject will return). If we close its handle the object will be destroyed.

If we look at other objects threads are waiting for we would see SignalState equal to 0. For example, in the following thread below event handle 4b0 corresponds to a34d9940 _KEVENT address and the latter object is inherited from _DISPATCHER_HEADER:

3: kd> !thread a34369d0
THREAD a34369d0  Cid 1fc8.1e88  Teb: 7ffae000 Win32Thread: bc6d5818 WAIT: (Unknown) UserMode Non-Alertable
    a34d9940  SynchronizationEvent
    a3436a48  NotificationTimer
Not impersonating
DeviceMap                 e12256a0
Owning Process            a3340a10       Image:         IEXPLORE.EXE
Wait Start TickCount      1450409        Ticks: 38 (0:00:00:00.593)
Context Switch Count      7091                 LargeStack
UserTime                  00:00:01.015
KernelTime                00:00:02.250
Win32 Start Address mshtml!CExecFT::StaticThreadProc (0×7fab1061)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init f252b000 Current f252ac60 Base f252b000 Limit f2528000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
f252ac78 80833465 a34369d0 a3436a78 00000003 nt!KiSwapContext+0×26
f252aca4 80829a62 00000000 f252ad14 00000000 nt!KiSwapThread+0×2e5
f252acec 80938d0c a34d9940 00000006 f252ac01 nt!KeWaitForSingleObject+0×346
f252ad50 8088978c 000004b0 00000000 f252ad14 nt!NtWaitForSingleObject+0×9a
f252ad50 7c8285ec 000004b0 00000000 f252ad14 nt!KiFastCallEntry+0xfc (TrapFrame @ f252ad64)
030dff08 7c827d0b 77e61d1e 000004b0 00000000 ntdll!KiFastSystemCallRet
030dff0c 77e61d1e 000004b0 00000000 030dff50 ntdll!NtWaitForSingleObject+0xc
030dff7c 77e61c8d 000004b0 000927c0 00000000 kernel32!WaitForSingleObjectEx+0xac
030dff90 7fab08a3 000004b0 000927c0 00000000 kernel32!WaitForSingleObject+0×12
030dffa8 7fab109c 00000000 7fab106e 030dffec mshtml!CDwnTaskExec::ThreadExec+0xae
030dffb0 7fab106e 030dffec 77e64829 02714f30 mshtml!CExecFT::ThreadProc+0×28
030dffb8 77e64829 02714f30 00000000 00000000 mshtml!CExecFT::StaticThreadProc+0xd
030dffec 00000000 7fab1061 02714f30 00000000 kernel32!BaseThreadStart+0×34

3: kd> dt _DISPATCHER_HEADER a34d9940
cutildll!_DISPATCHER_HEADER
   +0x000 Type             : 0x1 ''
   +0x001 Absolute         : 0xda ''
   +0x002 Size             : 0x4 ''
   +0x003 Inserted         : 0xa3 ''
   +0x003 DebugActive      : 0xa3 ''
   +0x000 Lock             : -1559963135
   +0×004 SignalState      : 0
   +0×008 WaitListHead     : _LIST_ENTRY [ 0xa3436a78 - 0xa3436a78 ]

For real life example of signaled process objects please see Zombie Processes pattern case study.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes when listing processes we see the so called Zombie Processes. They are better visible in the output of !vm command as processes with zero private memory:

0: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:      999294 (   3997176 Kb)
 Page File: \??\C:\pagefile.sys
   Current:   5995520 Kb  Free Space:   5324040 Kb
   Minimum:   5995520 Kb  Maximum:      5995520 Kb
 Available Pages:      626415 (   2505660 Kb)
 ResAvail Pages:       902639 (   3610556 Kb)
 Locked IO Pages:         121 (       484 Kb)
 Free System PTEs:     201508 (    806032 Kb)
 Free NP PTEs:          32766 (    131064 Kb)
 Free Special NP:           0 (         0 Kb)
 Modified Pages:          256 (      1024 Kb)
 Modified PF Pages:       256 (      1024 Kb)
 NonPagedPool Usage:    12304 (     49216 Kb)
 NonPagedPool Max:      65359 (    261436 Kb)
 PagedPool 0 Usage:     18737 (     74948 Kb)
 PagedPool 1 Usage:      2131 (      8524 Kb)
 PagedPool 2 Usage:      2104 (      8416 Kb)
 PagedPool 3 Usage:      2140 (      8560 Kb)
 PagedPool 4 Usage:      2134 (      8536 Kb)
 PagedPool Usage:       27246 (    108984 Kb)
 PagedPool Maximum:     67072 (    268288 Kb)
 Shared Commit:         60867 (    243468 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:        14359 (     57436 Kb)
 PagedPool Commit:      27300 (    109200 Kb)
 Driver Commit:          1662 (      6648 Kb)
 Committed pages:      501592 (   2006368 Kb)
 Commit limit:        2456879 (   9827516 Kb)

 Total Private:        368810 (   1475240 Kb)
...
...
...
         3654 explorer.exe      2083 (      8332 Kb)
         037c MyService.exe     2082 (      8328 Kb)
         315c explorer.exe      2045 (      8180 Kb)
…
…
…
         0588 svchost.exe        360 (      1440 Kb)
         3f94 csrss.exe          288 (      1152 Kb)
         0acc svchost.exe        245 (       980 Kb)
         0380 smss.exe            38 (       152 Kb)
         0004 System               7 (        28 Kb)
         6ee8 cmd.exe              0 (         0 Kb)
         6d7c cmd.exe              0 (         0 Kb)
         6ca8 cmd.exe              0 (         0 Kb)
         6b48 IEXPLORE.EXE         0 (         0 Kb)
         6ac4 cmd.exe              0 (         0 Kb)
         69e8 cmd.exe              0 (         0 Kb)
         69dc cmd.exe              0 (         0 Kb)
         68dc AcroRd32.exe         0 (         0 Kb)
         6860 cmd.exe              0 (         0 Kb)
         6858 cmd.exe              0 (         0 Kb)
         67d8 cmd.exe              0 (         0 Kb)
         6684 AcroRd32.exe         0 (         0 Kb)
         6484 cmd.exe              0 (         0 Kb)
         6464 cmd.exe              0 (         0 Kb)
         6288 cmd.exe              0 (         0 Kb)
         626c cmd.exe              0 (         0 Kb)
         6260 cmd.exe              0 (         0 Kb)
         6258 cmd.exe              0 (         0 Kb)
         620c IEXPLORE.EXE         0 (         0 Kb)
         60f0 cmd.exe              0 (         0 Kb)
         5fa4 cmd.exe              0 (         0 Kb)
         5f60 cmd.exe              0 (         0 Kb)
         5eec cmd.exe              0 (         0 Kb)
         5d24 IEXPLORE.EXE         0 (         0 Kb)
         5bd4 cmd.exe              0 (         0 Kb)
         5b9c cmd.exe              0 (         0 Kb)
         5b10 cmd.exe              0 (         0 Kb)
         5b08 cmd.exe              0 (         0 Kb)
         5a4c cmd.exe              0 (         0 Kb)
         5a08 cmd.exe              0 (         0 Kb)
         5934 cmd.exe              0 (         0 Kb)
         58b8 cmd.exe              0 (         0 Kb)
         56dc cmd.exe              0 (         0 Kb)
         558c cmd.exe              0 (         0 Kb)
         5588 cmd.exe              0 (         0 Kb)
         5574 cmd.exe              0 (         0 Kb)
         5430 cmd.exe              0 (         0 Kb)
         5424 cmd.exe              0 (         0 Kb)
         53b0 cmd.exe              0 (         0 Kb)
         5174 explorer.exe         0 (         0 Kb)
         5068 cmd.exe              0 (         0 Kb)
         5028 IEXPLORE.EXE         0 (         0 Kb)
         5004 cmd.exe              0 (         0 Kb)
         4f3c javaw.exe            0 (         0 Kb)
         4de4 cmd.exe              0 (         0 Kb)
         4dd8 cmd.exe              0 (         0 Kb)
         4c50 cmd.exe              0 (         0 Kb)
         4c48 cmd.exe              0 (         0 Kb)
         4c08 cmd.exe              0 (         0 Kb)
         4a8c cmd.exe              0 (         0 Kb)
         49ac cmd.exe              0 (         0 Kb)
         4938 cmd.exe              0 (         0 Kb)
         4928 cmd.exe              0 (         0 Kb)
         491c cmd.exe              0 (         0 Kb)
         4868 POWERPNT.EXE         0 (         0 Kb)
         4724 cmd.exe              0 (         0 Kb)
         46cc cmd.exe              0 (         0 Kb)
         44a8 cmd.exe              0 (         0 Kb)
         43cc cmd.exe              0 (         0 Kb)
         4350 cmd.exe              0 (         0 Kb)
         4208 cmd.exe              0 (         0 Kb)
         41f4 cmd.exe              0 (         0 Kb)
         41ec cmd.exe              0 (         0 Kb)
         4170 cmd.exe              0 (         0 Kb)
         40bc cmd.exe              0 (         0 Kb)
         3ddc cmd.exe              0 (         0 Kb)
         3dcc cmd.exe              0 (         0 Kb)
         3db8 cmd.exe              0 (         0 Kb)
         3d88 cmd.exe              0 (         0 Kb)
         3d10 cmd.exe              0 (         0 Kb)
         3cac cmd.exe              0 (         0 Kb)
         3ca4 cmd.exe              0 (         0 Kb)
         3c88 cmd.exe              0 (         0 Kb)
         337c cmd.exe              0 (         0 Kb)
         3310 cmd.exe              0 (         0 Kb)
         3308 cmd.exe              0 (         0 Kb)
         32f0 cmd.exe              0 (         0 Kb)
         32b8 cmd.exe              0 (         0 Kb)
         2ed0 cmd.exe              0 (         0 Kb)
         2eb8 cmd.exe              0 (         0 Kb)
         2e28 cmd.exe              0 (         0 Kb)
         2d44 AcroRd32.exe         0 (         0 Kb)
         2d24 cmd.exe              0 (         0 Kb)
         2c94 cmd.exe              0 (         0 Kb)
         2c54 IEXPLORE.EXE         0 (         0 Kb)
         2a28 cmd.exe              0 (         0 Kb)
         29e4 cmd.exe              0 (         0 Kb)
         2990 cmd.exe              0 (         0 Kb)
         28c0 cmd.exe              0 (         0 Kb)
         25a0 cmd.exe              0 (         0 Kb)
         2558 cmd.exe              0 (         0 Kb)
         2478 cmd.exe              0 (         0 Kb)
         244c cmd.exe              0 (         0 Kb)
         23dc cmd.exe              0 (         0 Kb)
         2320 cmd.exe              0 (         0 Kb)
         2280 cmd.exe              0 (         0 Kb)
         2130 cmd.exe              0 (         0 Kb)
         205c cmd.exe              0 (         0 Kb)
         2014 cmd.exe              0 (         0 Kb)
         1fd8 cmd.exe              0 (         0 Kb)
         1fa0 cmd.exe              0 (         0 Kb)
         1eb8 cmd.exe              0 (         0 Kb)
         1d68 IEXPLORE.EXE         0 (         0 Kb)
         1cb8 cmd.exe              0 (         0 Kb)
         1c9c cmd.exe              0 (         0 Kb)
         1c50 cmd.exe              0 (         0 Kb)
         1a74 cmd.exe              0 (         0 Kb)
         1954 cmd.exe              0 (         0 Kb)
         1948 cmd.exe              0 (         0 Kb)
         06e4 cmd.exe              0 (         0 Kb)
         0650 cmd.exe              0 (         0 Kb)

We see lots of cmd.exe processes. Let’s examine a few of them:

0: kd> !process 0650
Searching for Process with Cid == 650
PROCESS 89237d88  SessionId: 0  Cid: 0650    Peb: 7ffde000  ParentCid: 037c
    DirBase: f3b31940  ObjectTable: 00000000  HandleCount:   0.
    Image: cmd.exe
    VadRoot 00000000 Vads 0 Clone 0 Private 0. Modified 2. Locked 0.
    DeviceMap e10038a8
    Token                             e4eb5b98
    ElapsedTime                       1 Day 00:16:11.706
    UserTime                          00:00:00.015
    KernelTime                        00:00:00.015
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (7, 50, 345) (28KB, 200KB, 1380KB)
    PeakWorkingSetSize                588
    VirtualSize                       11 Mb
    PeakVirtualSize                   14 Mb
    PageFaultCount                    663
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      0

No active threads

0: kd> !process 2130
Searching for Process with Cid == 2130
PROCESS 89648020  SessionId: 0  Cid: 2130    Peb: 7ffdc000  ParentCid: 037c
    DirBase: f3b31060  ObjectTable: 00000000  HandleCount:   0.
    Image: cmd.exe
    VadRoot 00000000 Vads 0 Clone 0 Private 0. Modified 2. Locked 0.
    DeviceMap e10038a8
    Token                             e5167bb8
    ElapsedTime                       15:40:17.643
    UserTime                          00:00:00.015
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (7, 50, 345) (28KB, 200KB, 1380KB)
    PeakWorkingSetSize                545
    VirtualSize                       11 Mb
    PeakVirtualSize                   14 Mb
    PageFaultCount                    621
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      0

No active threads

We see that most of them have Parent PID as 037c which is MyService.exe. Let’s peek inside its handle table:

0: kd> !kdexts.handle 0 3 037c
processor number 0, process 0000037c
Searching for Process with Cid == 37c
PROCESS 8a8fa8c0  SessionId: 0  Cid: 037c    Peb: 7ffd8000  ParentCid: 04ac
    DirBase: f3b10360  ObjectTable: e1c276b8  HandleCount: 500.
    Image: MyService.exe

Handle table at e272d000 with 500 Entries in use
0004: Object: e1000638  GrantedAccess: 00000003 Entry: e1caf008
Object: e1000638  Type: (8ad79ad0) KeyedEvent
    ObjectHeader: e1000620 (old version)
        HandleCount: 151  PointerCount: 152
        Directory Object: e1001898  Name: CritSecOutOfMemoryEvent

0008: Object: 8a8cfdf8  GrantedAccess: 001f0003 Entry: e1caf010
Object: 8a8cfdf8  Type: (8ad7a990) Event
    ObjectHeader: 8a8cfde0 (old version)
        HandleCount: 1  PointerCount: 1

000c: Object: e186d690  GrantedAccess: 00000003 Entry: e1caf018
Object: e186d690  Type: (8ad84e70) Directory
    ObjectHeader: e186d678 (old version)
        HandleCount: 150  PointerCount: 181
        Directory Object: e1003b28  Name: KnownDlls

0010: Object: 8a8d1328  GrantedAccess: 00100020 (Inherit) Entry: e1caf020
Object: 8a8d1328  Type: (8ad74900) File
    ObjectHeader: 8a8d1310 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \WINDOWS\system32 {HarddiskVolume1}
...
...
...
0484: Object: 89648020  GrantedAccess: 001f0fff Entry: e1caf908
Object: 89648020  Type: (8ad84900) Process
    ObjectHeader: 89648008 (old version)
        HandleCount: 1  PointerCount: 2
…
…
…
0510: Object: 89237d88  GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88  Type: (8ad84900) Process
    ObjectHeader: 89237d70 (old version)
        HandleCount: 1  PointerCount: 2
…
…
…
 
Therefore we may guess that MyService.exe probably forgot to close process handles either after launching cmd.exe or after waiting for their exit when process objects become signaled:

0510: Object: 89237d88  GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88  Type: (8ad84900) Process
    ObjectHeader: 89237d70 (old version)
        HandleCount: 1  PointerCount: 2 

0: kd> dt _DISPATCHER_HEADER 89237d88
ntdll!_DISPATCHER_HEADER
   +0x000 Type             : 0x3 '' ; PROCESS OBJECT
   +0x001 Absolute         : 0 ''
   +0x001 NpxIrql          : 0 ''
   +0x002 Size             : 0x1e ''
   +0x002 Hand             : 0x1e ''
   +0x003 Inserted         : 0 ''
   +0x003 DebugActive      : 0 ''
   +0x000 Lock             : 1966083
   +0×004 SignalState      : 1
   +0×008 WaitListHead     : _LIST_ENTRY [ 0×89237d90 - 0×89237d90 ]

This pattern can also be seen a specialization of a more general Handle Leak pattern.

- Dmitry Vostokov @ DumpAnalysis.org -

Next monthly summary of my Literate Scientist blog:

• Classical and Nonclassical Logics
• Not Even Wrong and the Trouble With Physics
• The Naked Capitalist and Tragedy & Hope
• Discrete Thoughts
• Fashionable Nonsense
• Deep Down Things and The Great Design
• The Road to Reality

- Dmitry Vostokov @ DumpAnalysis.org -

Next monthly summary of my Management Bits and Tips blog:

• Two Faces of Mess and Management
• New Management Bits Theme
• Process Parasites
• Management Bit and Tip 0×80
• Project Failure Analysis Patterns (Part 3)
• Management Bits
• Management Bit and Tip 0×100
• Bullshit Bibliophilia

- Dmitry Vostokov @ DumpAnalysis.org -

We often say that a particular thread is blocked and/or it blocks other threads. At the same time we know that almost all threads are “blocked” to some degree except those currently running on processors. They are either preempted and in the ready lists, voluntary yielded their execution or they are waiting for some synchronization object. Therefore the notion of Blocked Thread is highly context and problem dependent and usually we notice them when comparing current thread stack traces with their expected normal stack traces. Here reference guides are indispensible especially those created for troubleshooting concrete products. Forthcoming Debugger Log Analyzer was partially incepted to address this problem.  

To show the diversity of “blocked” threads I propose the following thread classification:

Running threads

Their EIP (RIP) points to some function different from KiSwapContext:

3: kd> !running

System Processors f (affinity mask)
  Idle Processors 0

     Prcb      Current   Next   
  0  ffdff120  a30a9350            ................
  1  f7727120  a3186448            ................
  2  f772f120  a59a1b40            ................
  3  f7737120  a3085888            ................

3: kd> !thread a59a1b40
THREAD a59a1b40  Cid 0004.00b8  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 2
Not impersonating
DeviceMap                 e10028b0
Owning Process            a59aa648       Image:         System
Wait Start TickCount      1450446        Ticks: 1 (0:00:00:00.015)
Context Switch Count      308765            
UserTime                  00:00:00.000
KernelTime                00:00:01.250
Start Address nt!ExpWorkerThread (0×80880356)
Stack Init f7055000 Current f7054cec Base f7055000 Limit f7052000 Call 0
Priority 12 BasePriority 12 PriorityDecrement 0
ChildEBP RetAddr
f7054bc4 8093c55c nt!ObfReferenceObject+0×1c
f7054ca0 8093d2ae nt!ObpQueryNameString+0×2ba
f7054cbc 808f7d0f nt!ObQueryNameString+0×18
f7054d80 80880441 nt!IopErrorLogThread+0×197
f7054dac 80949b7c nt!ExpWorkerThread+0xeb
f7054ddc 8088e062 nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

3: kd> .thread a59a1b40
Implicit thread is now a59a1b40

3: kd> r
Last set context:
eax=00000028 ebx=e1000228 ecx=e1002b30 edx=e1000234 esi=e1002b18 edi=0000001a
eip=8086c73e esp=f7054bc4 ebp=f7054ca0 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000202
nt!ObfReferenceObject+0×1c:
8086c73e 40              inc     eax
 
These threads can also be identified by RUNNING attribute in the output of !process 0 ff command applied for complete and kernel memory dumps.

Ready threads  

These threads can be seen in the output of !ready command or identified by READY attribute in the output of !process 0 ff command:

3: kd> !ready
Processor 0: No threads in READY state
Processor 1: Ready Threads at priority 11
    THREAD a3790790  Cid 0234.1108  Teb: 7ffab000 Win32Thread: 00000000 READY
    THREAD a32799a8  Cid 0234.061c  Teb: 7ff83000 Win32Thread: 00000000 READY
    THREAD a3961798  Cid 0c04.0c68  Teb: 7ffab000 Win32Thread: bc204ea8 READY
Processor 1: Ready Threads at priority 10
    THREAD a32bedb0  Cid 1fc8.1a30  Teb: 7ffad000 Win32Thread: bc804468 READY
Processor 1: Ready Threads at priority 9
    THREAD a52dcd48  Cid 0004.04d4  Teb: 00000000 Win32Thread: 00000000 READY
Processor 2: Ready Threads at priority 11
    THREAD a37fedb0  Cid 0c04.11f8  Teb: 7ff8e000 Win32Thread: 00000000 READY
Processor 3: Ready Threads at priority 11
    THREAD a5683db0  Cid 0234.0274  Teb: 7ffd6000 Win32Thread: 00000000 READY
    THREAD a3151b48  Cid 0234.2088  Teb: 7ff88000 Win32Thread: 00000000 READY
    THREAD a5099d80  Cid 0ecc.0d60  Teb: 7ffd4000 Win32Thread: 00000000 READY
    THREAD a3039498  Cid 0c04.275c  Teb: 7ff7d000 Win32Thread: 00000000 READY

If we look at these threads we would see that they were either scheduled to run because of a signaled object they were waiting for:

3: kd> !thread a3039498
THREAD a3039498  Cid 0c04.275c  Teb: 7ff7d000 Win32Thread: 00000000 READY
IRP List:
    a2feb008: (0006,0094) Flags: 00000870  Mdl: 00000000
Not impersonating
DeviceMap                 e10028b0
Owning Process            a399a770       Image:         svchost.exe
Wait Start TickCount      1450447        Ticks: 0
Context Switch Count      1069            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x001e4f22
LPC Server thread working on message Id 1e4f22
Start Address KERNEL32!BaseThreadStartThunk (0x77e617ec)
Stack Init f171b000 Current f171ac60 Base f171b000 Limit f1718000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
f171ac78 80833465 a3039498 a3039540 e7561930 nt!KiSwapContext+0x26
f171aca4 80829a62 00000000 00000000 00000000 nt!KiSwapThread+0x2e5
f171acec 80938d0c a301cad8 00000006 f171ad01 nt!KeWaitForSingleObject+0×346
f171ad50 8088978c 00000c99 00000000 00000000 nt!NtWaitForSingleObject+0×9a
f171ad50 7c8285ec 00000c99 00000000 00000000 nt!KiFastCallEntry+0xfc
03d9efa8 00000000 00000000 00000000 00000000 ntdll!KiFastSystemCallRet

3: kd> !object a301cad8
Object: a301cad8  Type: (a59a0720) Event
    ObjectHeader: a301cac0 (old version)
    HandleCount: 1  PointerCount: 3

Or they were boosted in priority:

3: kd> !thread a3790790
THREAD a3790790  Cid 0234.1108  Teb: 7ffab000 Win32Thread: 00000000 READY
IRP List:
    a2f8b7f8: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e10028b0
Owning Process            a554bcc8       Image:         lsass.exe
Wait Start TickCount      1450447        Ticks: 0
Context Switch Count      384            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c7b0f5)
Start Address KERNEL32!BaseThreadStartThunk (0x77e617ec)
Stack Init f3ac1000 Current f3ac0ce8 Base f3ac1000 Limit f3abe000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
f3ac0d00 80831266 a3790790 a50f1870 a3186448 nt!KiSwapContext+0x26
f3ac0d20 8082833a 00000000 a50f1870 8098b56c nt!KiExitDispatcher+0xf8
f3ac0d3c 8098b5b9 a50f1870 00000000 00f5f8d0 nt!KeSetEventBoostPriority+0×156
f3ac0d58 8088978c a50f1870 00f5f8d4 7c8285ec nt!NtSetEventBoostPriority+0×4d
f3ac0d58 7c8285ec a50f1870 00f5f8d4 7c8285ec nt!KiFastCallEntry+0xfc
00f5f8d4 00000000 00000000 00000000 00000000 ntdll!KiFastSystemCallRet

3: kd> !object a50f1870
Object: a50f1870  Type: (a59a0720) Event
    ObjectHeader: a50f1858 (old version)
    HandleCount: 1  PointerCount: 15

Or were interrupted and queued to be run again:

3: kd> !thread a5683db0
THREAD a5683db0 Cid 0234.0274 Teb: 7ffd6000 Win32Thread: 00000000 READY
IRP List:
  a324d498: (0006,0094) Flags: 00000900 Mdl: 00000000
  a2f97a20: (0006,0094) Flags: 00000900 Mdl: 00000000
  a50c3e70: (0006,0190) Flags: 00000000 Mdl: a50a22d0
  a5167750: (0006,0094) Flags: 00000800 Mdl: 00000000
Not impersonating
DeviceMap e10028b0
Owning Process a554bcc8 Image: lsass.exe
Wait Start TickCount 1450447 Ticks: 0
Context Switch Count 9619
UserTime 00:00:00.156
KernelTime 00:00:00.234
Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c7b0f5)
Start Address KERNEL32!BaseThreadStartThunk (0x77e617ec)
Stack Init f59f3000 Current f59f2d00 Base f59f3000 Limit f59f0000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
f59f2d18 80a5c1ae nt!KiDispatchInterrupt+0xb1
f59f2d48 80a5c577 hal!HalpDispatchSoftwareInterrupt+0x5e
f59f2d54 80a59902 hal!HalEndSystemInterrupt+0x67
f59f2d54 77c6928d hal!HalpIpiHandler+0xd2 (TrapFrame @ f59f2d64)
00c5f908 00000000 RPCRT4!OSF_SCALL::GetBuffer+0×37

3: kd> .thread a5683db0
Implicit thread is now a5683db0

3: kd> r
Last set context:
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=8088dba1 esp=f59f2d0c ebp=f59f2d2c iopl=0 nv up di pl nz na po nc
cs=0008 ss=0010 ds=0000 es=0000 fs=0000 gs=0000 efl=00000000
nt!KiDispatchInterrupt+0xb1:
8088dba1 b902000000 mov ecx,2

3: kd> ub
nt!KiDispatchInterrupt+0x8f:
8088db7f mov dword ptr [ebx+124h],esi
8088db85 mov byte ptr [esi+4Ch],2
8088db89 mov byte ptr [edi+5Ah],1Fh
8088db8d mov ecx,edi
8088db8f lea edx,[ebx+120h]
8088db95 call nt!KiQueueReadyThread (80833490)
8088db9a mov cl,1
8088db9c call nt!SwapContext (8088dbd0)

3: kd> u
nt!KiDispatchInterrupt+0xb1:
8088dba1 mov ecx,2
8088dba6 call dword ptr [nt!_imp_KfLowerIrql (80801108)]
8088dbac mov ebp,dword ptr [esp]
8088dbaf mov edi,dword ptr [esp+4]
8088dbb3 mov esi,dword ptr [esp+8]
8088dbb7 add esp,0Ch
8088dbba pop ebx
8088dbbb ret

We can get user space thread stack by using .trap command but we need to switch to its process context first:

3: kd> .process /r /p a554bcc8
Implicit process is now a554bcc8
Loading User Symbols

3: kd> kL
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
00c5f908 77c7ed60 RPCRT4!OSF_SCALL::GetBuffer+0x37
00c5f924 77c7ed14 RPCRT4!I_RpcGetBufferWithObject+0x7f
00c5f934 77c7f464 RPCRT4!I_RpcGetBuffer+0xf
00c5f944 77ce3470 RPCRT4!NdrGetBuffer+0x2e
00c5fd44 77ce35c4 RPCRT4!NdrStubCall2+0x35c
00c5fd60 77c7ff7a RPCRT4!NdrServerCall2+0x19
00c5fd94 77c8042d RPCRT4!DispatchToStubInCNoAvrf+0x38
00c5fde8 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x11f
00c5fe0c 77c68e0d RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
00c5fe40 77c68cb3 RPCRT4!OSF_SCALL::DispatchHelper+0x149
00c5fe54 77c68c2b RPCRT4!OSF_SCALL::DispatchRPCCall+0x10d
00c5fe84 77c68b5e RPCRT4!OSF_SCALL::ProcessReceivedPDU+0x57f
00c5fea4 77c6e8db RPCRT4!OSF_SCALL::BeginRpcCall+0x194
00c5ff04 77c6e7b4 RPCRT4!OSF_SCONNECTION::ProcessReceiveComplete+0x435
00c5ff18 77c7b799 RPCRT4!ProcessConnectionServerReceivedEvent+0x21
00c5ff84 77c7b9b5 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
00c5ff8c 77c8872d RPCRT4!ProcessIOEventsWrapper+0xd
00c5ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
00c5ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
00c5ffec 00000000 kernel32!BaseThreadStart+0x34

Waiting threads (wait originated from user space)

THREAD a34369d0  Cid 1fc8.1e88  Teb: 7ffae000 Win32Thread: bc6d5818 WAIT: (Unknown) UserMode Non-Alertable
    a34d9940  SynchronizationEvent
    a3436a48  NotificationTimer
Not impersonating
DeviceMap                 e12256a0
Owning Process            a3340a10       Image:         IEXPLORE.EXE
Wait Start TickCount      1450409        Ticks: 38 (0:00:00:00.593)
Context Switch Count      7091                 LargeStack
UserTime                  00:00:01.015
KernelTime                00:00:02.250
Win32 Start Address mshtml!CExecFT::StaticThreadProc (0x7fab1061)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f252b000 Current f252ac60 Base f252b000 Limit f2528000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
f252ac78 80833465 nt!KiSwapContext+0x26
f252aca4 80829a62 nt!KiSwapThread+0x2e5
f252acec 80938d0c nt!KeWaitForSingleObject+0x346
f252ad50 8088978c nt!NtWaitForSingleObject+0x9a
f252ad50 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ f252ad64)
030dff08 7c827d0b ntdll!KiFastSystemCallRet
030dff0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
030dff7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
030dff90 7fab08a3 kernel32!WaitForSingleObject+0×12
030dffa8 7fab109c mshtml!CDwnTaskExec::ThreadExec+0xae
030dffb0 7fab106e mshtml!CExecFT::ThreadProc+0×28
030dffb8 77e64829 mshtml!CExecFT::StaticThreadProc+0xd
030dffec 00000000 kernel32!BaseThreadStart+0×34

If we had taken user dump of iexplore.exe we would have seen the following stack trace there:

030dff08 7c827d0b ntdll!KiFastSystemCallRet
030dff0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
030dff7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
030dff90 7fab08a3 kernel32!WaitForSingleObject+0×12
030dffa8 7fab109c mshtml!CDwnTaskExec::ThreadExec+0xae
030dffb0 7fab106e mshtml!CExecFT::ThreadProc+0×28
030dffb8 77e64829 mshtml!CExecFT::StaticThreadProc+0xd
030dffec 00000000 kernel32!BaseThreadStart+0×34

Another example:

THREAD a31f2438  Cid 1fc8.181c  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    a30f8c20  NotificationEvent
    a5146720  NotificationEvent
    a376fbb0  NotificationEvent
Not impersonating
DeviceMap                 e12256a0
Owning Process            a3340a10       Image:         IEXPLORE.EXE
Wait Start TickCount      1419690        Ticks: 30757 (0:00:08:00.578)
Context Switch Count      2            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address USERENV!NotificationThread (0x76929dd9)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f5538000 Current f5537900 Base f5538000 Limit f5535000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
f5537918 80833465 nt!KiSwapContext+0x26
f5537944 80829499 nt!KiSwapThread+0x2e5
f5537978 80938f68 nt!KeWaitForMultipleObjects+0x3d7
f5537bf4 809390ca nt!ObpWaitForMultipleObjects+0x202
f5537d48 8088978c nt!NtWaitForMultipleObjects+0xc8
f5537d48 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ f5537d64)
0851fec0 7c827cfb ntdll!KiFastSystemCallRet
0851fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0851ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
0851ff88 76929e35 kernel32!WaitForMultipleObjects+0×18
0851ffb8 77e64829 USERENV!NotificationThread+0×5f
0851ffec 00000000 kernel32!BaseThreadStart+0×34

Waiting threads (wait originated from kernel space)

Examples include explicit wait as a result from calling potentially blocking API:

THREAD a33a9740  Cid 1980.1960  Teb: 7ffde000 Win32Thread: bc283ea8 WAIT: (Unknown) UserMode Non-Alertable
    a35e3168  SynchronizationEvent
Not impersonating
DeviceMap                 e689f298
Owning Process            a342d3a0       Image:         explorer.exe
Wait Start TickCount      1369801        Ticks: 80646 (0:00:21:00.093)
Context Switch Count      1667                 LargeStack
UserTime                  00:00:00.015
KernelTime                00:00:00.093
Win32 Start Address Explorer!ModuleEntry (0x010148a4)
Start Address kernel32!BaseProcessStartThunk (0x77e617f8)
Stack Init f258b000 Current f258ac50 Base f258b000 Limit f2585000 Call 0
Priority 13 BasePriority 10 PriorityDecrement 1
Kernel stack not resident.
ChildEBP RetAddr 
f258ac68 80833465 nt!KiSwapContext+0x26
f258ac94 80829a62 nt!KiSwapThread+0x2e5
f258acdc bf89abd3 nt!KeWaitForSingleObject+0×346
f258ad38 bf89da43 win32k!xxxSleepThread+0×1be
f258ad4c bf89e401 win32k!xxxRealWaitMessageEx+0×12
f258ad5c 8088978c win32k!NtUserWaitMessage+0×14
f258ad5c 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ f258ad64)
0007feec 7739bf53 ntdll!KiFastSystemCallRet
0007ff08 7c8fadbd USER32!NtUserWaitMessage+0xc
0007ff14 0100fff1 SHELL32!SHDesktopMessageLoop+0×24
0007ff5c 0101490c Explorer!ExplorerWinMain+0×2c4
0007ffc0 77e6f23b Explorer!ModuleEntry+0×6d
0007fff0 00000000 kernel32!BaseProcessStart+0×23

and implicit wait when a thread yields execution to another thread voluntarily via explicit context swap:

THREAD a3072b68  Cid 1fc8.1d94  Teb: 7ffaf000 Win32Thread: bc1e3c20 WAIT: (Unknown) UserMode Non-Alertable
    a37004d8  QueueObject
    a3072be0  NotificationTimer
IRP List:
    a322be00: (0006,01fc) Flags: 00000000  Mdl: a30b8e30
    a30bcc38: (0006,01fc) Flags: 00000000  Mdl: a35bf530
Not impersonating
DeviceMap                 e12256a0
Owning Process            a3340a10       Image:         IEXPLORE.EXE
Wait Start TickCount      1447963        Ticks: 2484 (0:00:00:38.812)
Context Switch Count      3972                 LargeStack
UserTime                  00:00:00.140
KernelTime                00:00:00.250
Win32 Start Address ntdll!RtlpWorkerThread (0x7c839efb)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f1cc3000 Current f1cc2c38 Base f1cc3000 Limit f1cbf000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
f1cc2c50 80833465 nt!KiSwapContext+0x26
f1cc2c7c 8082b60f nt!KiSwapThread+0x2e5
f1cc2cc4 808ed620 nt!KeRemoveQueue+0×417
f1cc2d48 8088978c nt!NtRemoveIoCompletion+0xdc
f1cc2d48 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ f1cc2d64)
06ceff70 7c8277db ntdll!KiFastSystemCallRet
06ceff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
06ceffb8 77e64829 ntdll!RtlpWorkerThread+0×3d
06ceffec 00000000 kernel32!BaseThreadStart+0×34

THREAD a3612020  Cid 1980.1a48  Teb: 7ffd9000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Alertable
    a3612098  NotificationTimer
Not impersonating
DeviceMap                 e689f298
Owning Process            a342d3a0       Image:         explorer.exe
Wait Start TickCount      1346718        Ticks: 103729 (0:00:27:00.765)
Context Switch Count      4            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!RtlpTimerThread (0x7c83d3dd)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f2453000 Current f2452c80 Base f2453000 Limit f2450000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
f2452c98 80833465 nt!KiSwapContext+0x26
f2452cc4 80828f0b nt!KiSwapThread+0x2e5
f2452d0c 80994812 nt!KeDelayExecutionThread+0×2ab
f2452d54 8088978c nt!NtDelayExecution+0×84
f2452d54 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ f2452d64)
0149ff9c 7c826f4b ntdll!KiFastSystemCallRet
0149ffa0 7c83d424 ntdll!NtDelayExecution+0xc
0149ffb8 77e64829 ntdll!RtlpTimerThread+0×47
0149ffec 00000000 kernel32!BaseThreadStart+0×34

Explicit waits in kernel can be originated from GUI threads and their message loops, for example, Main Thread. Blocked GUI Thread pattern can be seen as an example of a genuine Blocked Thread. Some “blocked” threads are just really Passive Threads.

- Dmitry Vostokov @ DumpAnalysis.org -

Finally release dates are set for the following two books:

DebugWare: The Art and Craft of Writing Troubleshooting and Debugging Tools

• Author: Kapildev Ramlal, Dmitry Vostokov
• Paperback: 256 pages (*)
• ISBN-13: 978-0-9558328-3-3
• Publisher: Opentask (15 Nov 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

Windows® Crash Dump Analysis

• Author: Dmitry Vostokov
• Paperback: 512 pages (*)
• ISBN-13: 978-0-9558328-2-6
• Publisher: Opentask (01 Dec 2008)
• Language: English
• Product Dimensions: 22.86 x 15.24

The latter book will be shorter than planned initially and will contain references to Memory Dump Analysis Anthology, Volume I and Volume II.

(*) subject to change

- Dmitry Vostokov @ DumpAnalysis.org -

Self-publishing process is so fascinating and I feel designing and implementing something between software and hardware despite all necessary hard work. Modern print on demand technology allows to see incremental versions of the book in print. Here is the draft cover and table of contents for previously announced Memory Dump Analysis Anthology, Volume 1:

Table of Contents

I decided not to use banal digits and geometrical figures for the book cover and make something original although it might not be exactly the same as you see it now

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes processes dump themselves using Microsoft DbgHelp API when they encounter internal errors or for other debugging purposes. I separate this from unhandled exceptions which usually cause an external postmortem debugger process to dump memory contents as explained in my previous posts:

• Who calls the postmortem debugger?
• Inside Vista Error Reporting (Part 1)

It is important to understand that a process can dump itself for any reasons that came to mind of an application designer and not only as a reaction to hardware and software exceptions. In any case it is useful to look at raw stack dump of all threads and search for first chance exceptions like c0000005 and custom exception handlers. 

I call this pattern Self-Dump and consider it as a specialized version of Special Stack Trace pattern. The typical thread stack might look like (and might be Incorrect Stack Trace that requires manual reconstruction):

0:012> kL
ChildEBP RetAddr 
0151f0c4 77e61f0c ntdll!KiFastSystemCallRet
0151f0d4 08000000 kernel32!CreateFileMappingW+0xc8
WARNING: Frame IP not in any known module. Following frames may be wrong.
0151f0f0 7c82728b 0x8000000
0151f0f4 77e63e41 ntdll!NtMapViewOfSection+0xc
0151f12c 77e6440c kernel32!MapViewOfFileEx+0x71
0151f14c 7c826d2b kernel32!MapViewOfFile+0x1b
0151f1d4 028ca67c ntdll!ZwClose+0xc
0151f2a4 028cc2f1 dbghelp!GenGetClrMemory+0xec
0151f2b4 028c8e55 dbghelp!Win32LiveSystemProvider::CloseMapping+0×11
0151f414 00000000 dbghelp!GenAllocateModuleObject+0×3c5

Raw stack data should reveal DbgHelp API calls:

0151f944  00000000
0151f948  00000000
0151f94c  0151f9c0
0151f950  028c7662 dbghelp!MiniDumpWriteDump+0×1b2
0151f954  ffffffff
0151f958  00000cb0
0151f95c  00c21ea8
0151f960  00c21f88
0151f964  00c21e90
0151f968  00c21fa0
0151f96c  00000002
0151f970  00000000
0151f974  00000000
0151f978  00000000
0151f97c  7c829f60 ntdll!CheckHeapFillPattern+0×64
0151f980  ffffffff
0151f984  7c829f59 ntdll!RtlFreeHeap+0×70f
0151f988  7c34218a msvcr71!free+0xc3
0151f98c  00000000
0151f990  00000000
0151f994  00c21e90
0151f998  00c21fa0
0151f99c  00c21ea8
0151f9a0  00c21f88
0151f9a4  00000002
0151f9a8  021a00da
0151f9ac  001875d8
0151f9b0  7c3416db msvcr71!_nh_malloc+0×10
0151f9b4  0151f998
0151f9b8  001875d8
0151f9bc  00000000
0151f9c0  0151fbe4
0151f9c4  57b77d01 application!write_problem_report+0×18d1
0151f9c8  ffffffff
0151f9cc  00000cb0
0151f9d0  00000718
0151f9d4  00000002
0151f9d8  00000000
0151f9dc  00000000
0151f9e0  00000000
0151f9e4  00029722
0151f9e8  0151fc20

Partially reconstructed stack trace might look like:

0:012> k L=0151f94c
ChildEBP RetAddr 
0151f0c4 77e61f0c ntdll!KiFastSystemCallRet
0151f94c 028c7662 kernel32!CreateFileMappingW+0xc8
0151f9c0 57b77d01 dbghelp!MiniDumpWriteDump+0×1b2
0151fbe4 57b77056 application!write_problem_report+0×18d1
0151fc30 579c83af application!TerminateThread+0×18

- Dmitry Vostokov @ DumpAnalysis.org -

Finally the new update is available for download from Citrix support website.

What’s new in 3.6:

1. Fixed a bug related to secure string handling updates in MFC. PDBFinder now handles UNC pathnames exceeding 256 characters when building database.

2. Added additional filter to eliminate noise as shown on the following screenshot:

The motivation behind PDBFinder is covered in my old post:

Cons of Symbol Server

- Dmitry Vostokov @ DumpAnalysis.org -

Another suspicious threads in crash dumps are GUI threads executing message box code. Usually message boxes are displayed to show some error and we can see it by dumping the second and the third MessageBox parameters:

int MessageBox(     
    HWND hWnd,     
    LPCTSTR lpText,
    LPCTSTR lpCaption,
    UINT uType);

Sometimes message boxes block processes as shown in the example illustrating Coupled Process pattern. Other threads might point to possibly “hang” sessions and processes in memory dumps coming from terminal services environments. I call this pattern Message Box and it is another example of the forthcoming collective Blocked Thread pattern. 

Let’s look at one example where message box pointed to the right direction of troubleshooting. Some user process was reported hanging from time to time however it was not specified which one. Searching for MessageBox in the log of all threads in the system produced by !process 0 ff WinDbg command revealed the following thread:

THREAD 88b14da8  Cid 0a04.0c14  Teb: 7ffdd000 Win32Thread: e5e50ab0 WAIT: (WrUserRequest) UserMode Non-Alertable
    87b74358  SynchronizationEvent
IRP List:
    87a0ba00: (0006,0244) Flags: 00000000  Mdl: 00000000
Not impersonating
DeviceMap                 e14bec28
Owning Process            888ffb60       Image:         OUTLOOK.EXE
Wait Start TickCount      1275435        Ticks: 210 (0:00:00:03.281)
Context Switch Count      1050203                 LargeStack
UserTime                  00:00:16.812
KernelTime                00:00:18.000
Win32 Start Address OUTLOOK (0x30001084)
Start Address kernel32!BaseProcessStartThunk (0x7c810665)
Stack Init a0c98000 Current a0c97cb0 Base a0c98000 Limit a0c90000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 2 DecrementCount 16
ChildEBP RetAddr 
a0c97cc8 804e1bd2 nt!KiSwapContext+0x2f
a0c97cd4 804e1c1e nt!KiSwapThread+0x8a
a0c97cfc bf802f70 nt!KeWaitForSingleObject+0x1c2
a0c97d38 bf803776 win32k!xxxSleepThread+0x192
a0c97d4c bf803793 win32k!xxxRealWaitMessageEx+0x12
a0c97d5c 804dd99f win32k!NtUserWaitMessage+0x14
a0c97d5c 7c90eb94 nt!KiFastCallEntry+0xfc
0013f3a8 7e419418 ntdll!KiFastSystemCallRet
0013f3e0 7e42593f USER32!NtUserWaitMessage+0xc
0013f408 7e43a91e USER32!InternalDialogBox+0xd0
0013f6c8 7e43a284 USER32!SoftModalMessageBox+0x938
0013f818 7e4661d3 USER32!MessageBoxWorker+0x2ba
0013f870 7e466278 USER32!MessageBoxTimeoutW+0x7a
0013f8a4 7e450617 USER32!MessageBoxTimeoutA+0x9c
0013f8c4 7e4505cf USER32!MessageBoxExA+0x1b
0013f8e0 088098a9 USER32!MessageBoxA+0x45
...
...
...

0: kd> .process /r /p 888ffb60
Implicit process is now 888ffb60
Loading User Symbols

0: kd> !thread 88b14da8
...
...
...
ChildEBP RetAddr  Args to Child             
...
...
...
0013f8e0 088098a9 00000000 0013f944 088708f0 USER32!MessageBoxA+0×45
…
…
…

0: kd> dA 0013f944
0013f944  "Cannot contact database, Retry"

This immediately raised suspicion and looking at other threads in the same application revealed that many of them were trying to open a network connection, for example:

THREAD 87a70da8  Cid 0a04.0cc0  Teb: 7ff83000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
    87d690b0  NotificationEvent
    87a70e98  NotificationTimer
IRP List:
    87af7bc8: (0006,0244) Flags: 00000000  Mdl: 00000000
Not impersonating
DeviceMap                 e14bec28
Owning Process            888ffb60       Image:         OUTLOOK.EXE
Wait Start TickCount      1267130        Ticks: 8515 (0:00:02:13.046)
Context Switch Count      18            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address msmapi32!FOpenThreadImpersonationToken (0×35f76963)
Start Address kernel32!BaseThreadStartThunk (0×7c810659)
Stack Init 9fbc5000 Current 9fbc4ca0 Base 9fbc5000 Limit 9fbc2000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 16
Kernel stack not resident.
ChildEBP RetAddr 
9fbc4cb8 804e1bd2 nt!KiSwapContext+0×2f
9fbc4cc4 804e1c1e nt!KiSwapThread+0×8a
9fbc4cec 8056d2f9 nt!KeWaitForSingleObject+0×1c2
9fbc4d50 804dd99f nt!NtWaitForSingleObject+0×9a
9fbc4d50 7c90eb94 nt!KiFastCallEntry+0xfc
1bd8f52c 7c90e9c0 ntdll!KiFastSystemCallRet
1bd8f530 7c8025cb ntdll!ZwWaitForSingleObject+0xc
1bd8f594 7c802532 kernel32!WaitForSingleObjectEx+0xa8
1bd8f5a8 77eec4c6 kernel32!WaitForSingleObject+0×12
1bd8f6a4 77eec8b7 RPCRT4!WS_Open+0×31d
1bd8f7c8 77eec96d RPCRT4!TCPOrHTTP_Open+0×19e
1bd8f800 77e83e8d RPCRT4!TCP_Open+0×55
1bd8f84c 77e843f7 RPCRT4!OSF_CCONNECTION::TransOpen+0×5e
1bd8f8b4 77e84581 RPCRT4!OSF_CCONNECTION::OpenConnectionAndBind+0xbc
1bd8f8f8 77e844d0 RPCRT4!OSF_CCALL::BindToServer+0×104
1bd8f95c 77e7f99c RPCRT4!OSF_BINDING_HANDLE::AllocateCCall+0×2b0
1bd8f98c 77e791c1 RPCRT4!OSF_BINDING_HANDLE::NegotiateTransferSyntax+0×28
1bd8f9a4 77e791f8 RPCRT4!I_RpcGetBufferWithObject+0×5b
1bd8f9b4 77e79825 RPCRT4!I_RpcGetBuffer+0xf
1bd8f9c4 77ef460b RPCRT4!NdrGetBuffer+0×28
1bd8fda4 35bae645 RPCRT4!NdrClientCall2+0×195
…
…
…

Looking at IRP showed the possible problem with network at TDI level:

0: kd> !irp 87af7bc8
Irp is active with 4 stacks 2 is current (= 0x87af7c5c)  No Mdl: No System Buffer: Thread 87a70da8:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   
   Args: 00000000 00000000 00000000 00000000
>[  f, 3]   0 e1 897b0310 87bb24c8 a1ad7080-87b6f1f0 Success Error Cancel pending
        \Driver\Tcpip    MYTDI!WaitForNetwork
   Args: 00000000 87a33988 87a33af8 a1a57600
 [  f, 3]   0 e1 88c13020 87bb24c8 a1a6bea5-87a33988 Success Error Cancel pending
        \Driver\SYMTDI afd!AfdRestartSuperConnect
   Args: 00000000 87a33988 87a33af8 a1a57600
 [  e,31]   5  0 88bedf18 87d3af90 00000000-00000000   
        \Driver\AFD
   Args: 00000000 00000016 000120c7 1bd8f52c

- Dmitry Vostokov @ DumpAnalysis.org -