Software Diagnostics Library

“Everything is in a state of” memory.

Attributed to Heraclitus of Ephesus

- Dmitry Vostokov @ DumpAnalysis.org -

“For it is not enough to have a good” debugger: “one must use it well.”

René Descartes, Discourse on the Method

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes there is not enough physical memory and the system experiences the so called disk or page file thrashing trying to resolve page faults. This can be seen in some memory dumps coming from frozen environments showing signs of double traps in running threads, the first one is a normal memory access fault (blue) and the other is forced NMI bugcheck to save a memory dump (red):

1: kd> .bugcheck
Bugcheck code 00000080
Arguments 004f4454 00000000 00000000 00000000

1: kd> !thread
THREAD 88939b20  Cid 360.378  Teb: 7ffdb000  Win32Thread: a20a7ac8 RUNNING
IRP List:
    86be9e68: (0006,0100) Flags: 00000070  Mdl: 00000000
    88939e68: (0006,0100) Flags: 00000070  Mdl: 00000000
    88939128: (0006,0100) Flags: 00000070  Mdl: 00000000
Not impersonating
Owning Process 889456e0
Wait Start TickCount    2357431       Elapsed Ticks: 9
Context Switch Count    18267                   LargeStack
UserTime                  0:00:08.0218
KernelTime                0:12:28.0109
Start Address KERNEL32!BaseThreadStartThunk (0x7c57b740)
Win32 Start Address msafd!SockAsyncThread (0x74fd3113)
Stack Init bef9e000 Current bef9db60 Base bef9e000 Limit bef9b000 Call 0
Priority 11 BasePriority 11 PriorityDecrement 0 DecrementCount 0

ChildEBP RetAddr
8904aff0 80469211 hal!HalHandleNMI+0×193
8904aff0 80438621 nt!KiTrap02+0×41
bef9dc10 8043799a nt!MiTrimWorkingSet+0xa7
bef9dc38 804378ec nt!MiDoReplacement+0×2e
bef9dc50 804453cf nt!MiLocateAndReserveWsle+0×1e
bef9dc68 804444e0 nt!MiAddValidPageToWorkingSet+0×89
bef9dc8c 804443a2 nt!MiCompleteProtoPteFault+0xf6
bef9dcb8 804436e8 nt!MiResolveProtoPteFault+0×160
bef9dcfc 8044ccd0 nt!MiDispatchFault+0xfc
bef9dd4c 8046b063 nt!MmAccessFault+0xd1c
bef9dd4c 74fd31e0 nt!KiTrap0E+0xc7
016effb4 7c57b3bc msafd!SockAsyncThread+0xcd
016effec 00000000 KERNEL32!BaseThreadStart+0×52

If we check virtual memory stats we see the low number of available pages:

1: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:   524165   ( 2096660 Kb)
 Page File: \??\C:\pagefile.sys
    Current:   4190208Kb Free Space:   3298704Kb
    Minimum:   4190208Kb Maximum:      4190208Kb
 Page File: \??\E:\pagefile.sys
    Current:   4190208Kb Free Space:   3339860Kb
    Minimum:   4190208Kb Maximum:      4190208Kb
 Available Pages:     1098   (    4392 Kb)
 ResAvail Pages:    410646   ( 1642584 Kb)
 Modified Pages:    282384   ( 1129536 Kb)
 NonPagedPool Usage: 10046   (   40184 Kb)
 NonPagedPool Max:   68609   (  274436 Kb)
 PagedPool 0 Usage:  15391   (   61564 Kb)
 PagedPool 1 Usage:   1906   (    7624 Kb)
 PagedPool 2 Usage:   1925   (    7700 Kb)
 PagedPool 3 Usage:   1937   (    7748 Kb)
 PagedPool 4 Usage:   1892   (    7568 Kb)
 PagedPool Usage:    23051   (   92204 Kb)
 PagedPool Maximum:  87040   (  348160 Kb)
 Shared Commit:      16867   (   67468 Kb)
 Special Pool:           0   (       0 Kb)
 Free System PTEs:   65288   (  261152 Kb)
 Shared Process:     38655   (  154620 Kb)
 PagedPool Commit:   23051   (   92204 Kb)
 Driver Commit:       1060   (    4240 Kb)
 Committed pages:  1049592   ( 4198368 Kb)
 Commit limit:     2580155   (10320620 Kb)
[…]

In W2K dumps we can also see locking on a working set resource (I guess the name from Ws shortcut here):

 1: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****

Resource @ nt!MmSystemWsLock (0×804869c0)    Exclusively owned
    Contention Count = 33083
    NumberOfExclusiveWaiters = 237
[…]

and huge number of threads in Ready state for every thread priority.

Looking at the current process owning the running thread shows the large number of page faults and increased kernel CPU time compared to time spent in user mode:

1: kd> !process 889456e0
PROCESS 889456e0  SessionId: 0  Cid: 0360    Peb: 7ffdf000  ParentCid: 01a8
    DirBase: 102af000  ObjectTable: 88945c08  TableSize: 622.
    Image: Application.EXE
    VadRoot 88944468 Clone 0 Private 838. Modified 30691412. Locked 188.
    DeviceMap 89049288
    Token                             e28db550
    ElapsedTime                       10:13:30.0684
    UserTime                          0:00:12.0578
    KernelTime                        0:12:38.0625
    QuotaPoolUsage[PagedPool]         31568
    QuotaPoolUsage[NonPagedPool]      68266
    Working Set Sizes (now,min,max)  (49, 50, 345) (196KB, 200KB, 1380KB)
    PeakWorkingSetSize                1956
    VirtualSize                       131 Mb
    PeakVirtualSize                   131 Mb
    PageFaultCount                    46180598
    MemoryPriority                    BACKGROUND
    BasePriority                      10
    CommitCharge                      1247

- Dmitry Vostokov @ DumpAnalysis.org -

Publishing my old CV and salary expectations damaged my memory and yesterday night I experienced a dream where a courier arrived to my office. I opened a packet and saw an A4 plastic card with a cover displaying a VIP job offer from one company the name of which I cannot disclose here. The plastic job offer card also had 2 buttons: accept an offer and decline an offer. To the left of the buttons there was a picture of a tiger: I used to play with 2 x 2 tiger puzzle with my 2 year old son before that night… When I woke up the dream was so real that I searched around for that card

- Dmitry Vostokov @ DumpAnalysis.org -

I made a beautiful 100 x 18400 slice of pagefile.bmp generated by Dump2Picture using ImageMagick (1.5Mb JPEG image):

Wider 450 x 18400 slice (7Mb JPEG image) is available for viewing here: 

Page File Image Slice (7Mb JPEG)

- Dmitry Vostokov @ DumpAnalysis.org -

I was curious about how page file looks like when represented as a bitmap picture image like I previously did with memory dumps and I expected it to look like a picture of a complete memory dump due to its purpose as a backup to physical memory. It looks similar indeed. Here is a picture of a 1.3Gb pagefile.sys from my home computer after running Vista for last 2 weeks, generated by Dump2Picture tool and resized from 18400 x 18400 32-bit bitmap by ImageMagick:

- Dmitry Vostokov @ DumpAnalysis.org -

“Look for” a bug “in everything and you will find it.”

Pierre-Jules Renard, Journal

- Dmitry Vostokov @ DumpAnalysis.org -

… taking into account one second because of slowing Earth rotation. I wish everyone a Happy New Year and successful debugging and memory analysis sessions in virtuality and reality!

Dmitry Vostokov @ 45474150 504d5544  PAGEDUMP....(...
0bc6c000 81000000 8054d030 8054f098  ........0.T...T.
0000014c 00000001 000000e2 00000000  L...............
00000000 00000000 00000000 45474100  .............AGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474100  PAGEPAGEPAGE.AGE
8053eee0 00000003 0000f67d 00000001  ..S.....}.......
0000009e 00000100 00000eff 00001000  ................
0000e6e0 45474150 45474150 45474150  ....PAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
45474150 45474150 45474150 45474150  PAGEPAGEPAGEPAGE
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
00000000 00000000 00000000 00000000  ................
[...]


To test various postmortem debuggers and WER to their fullest potential and especially Crash2Hang I wrote another small program that models multiple exceptions in several threads. It is free and can be downloaded with full PDB and source code from here:

Download MTCrash

The source code is simple as possible:

// MTCrash (Multithreaded crash)
// Copyright (c) 2009 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt

#include <windows.h>
#include <process.h>
#include <iostream>

bool twice = false;

void thread_one(void *)
{
 Sleep(1000);
 std::cout << "Thread 1 is about to experience an AV exception..." << std::endl;
 *(int *)NULL = 0;
}

void thread_two(void *)
{
 Sleep(2000);
 if (twice)
 {
  std::cout << "Thread 2 is about to experience an AV exception..." << std::endl;
  *(int *)NULL = 0;
 }

 while (true)
 {
  std::cout << "Thread 2 is still running..." << std::endl;
  Sleep(1000);
 }
}

int main(int argc, WCHAR* argv[])
{
 if (argc > 1) twice = true;

 _beginthread(thread_two, 0, NULL);
 _beginthread(thread_one, 0, NULL);

 while(true)
 {
  std::cout << "Main Thread is still running..." << std::endl;
  Sleep(1000);
 }

 return 0;
}

It creates 2 additional threads and the first of them tries to access a NULL pointer:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(d3c.e94): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=005a4660 ecx=0041948c edx=00419ef0 esi=00419488 edi=00000000
eip=004013bd esp=007eff7c ebp=007effb4 iopl=0 nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00010246
MTCrash!thread_one+0×6d:
004013bd c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:002> ~*kL

   0  Id: d3c.eb4 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr 
002dfee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15
002dff4c 7d4d14ef kernel32!SleepEx+0x68
002dff5c 0040157a kernel32!Sleep+0xf
002dff70 004046ac MTCrash!main+0xaa
002dffc0 7d4e7d2a MTCrash!__tmainCRTStartup+0x15f
002dfff0 00000000 kernel32!BaseProcessStart+0x28

   1  Id: d3c.ebc Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr  
006afee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15
006aff4c 7d4d14ef kernel32!SleepEx+0x68
006aff5c 004014c5 kernel32!Sleep+0xf
006aff7c 00404352 MTCrash!thread_two+0xf5
006affb4 004043eb MTCrash!_callthreadstart+0x1b
006affb8 7d4dfe21 MTCrash!_threadstart+0x73
006affec 00000000 kernel32!BaseThreadStart+0x34

#  2  Id: d3c.e94 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr 
007effb8 7d4dfe21 MTCrash!thread_one+0×6d
007effe4 00000000 kernel32!BaseThreadStart+0×34

   3  Id: d3c.f0c Suspend: 1 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr 
0083ffc8 7d665081 ntdll!DbgBreakPoint+0x1
0083fff4 00000000 ntdll!DbgUiRemoteBreakin+0x2d

The second thread and main thread continue to run:

C:\Crash2Hang>MTCrash.exe
Main Thread is still running...
Thread 1 is about to experience an AV exception...
Main Thread is still running...
Thread 2 is still running...
Main Thread is still running...
Thread 2 is still running...
Main Thread is still running...
Thread 2 is still running...
[...]

If launched with any parameter the second thread also experiences an unhandled exception (in red) while the first one is suspended by an unhandled exception filter (in blue):

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(ca4.cb0): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=005a4668 ecx=0041948c edx=00419ef0 esi=00419488 edi=00000000
eip=004013bd esp=007eff7c ebp=007effb4 iopl=0 nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00010246
MTCrash!thread_one+0×6d:
004013bd c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:002> ~*kL

   0  Id: ca4.ca0 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr 
002dfee4 7d4d0ec5 ntdll!ZwDelayExecution+0x15
002dff4c 7d4d14ef kernel32!SleepEx+0x68
002dff5c 0040157a kernel32!Sleep+0xf
002dff70 004046ac MTCrash!main+0xaa
002dffc0 7d4e7d2a MTCrash!__tmainCRTStartup+0x15f
002dfff0 00000000 kernel32!BaseProcessStart+0x28

   1  Id: ca4.ca8 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr  
006af8cc 7d5357f3 ntdll!ZwRaiseHardError+0×12
006afb38 7d508f4e kernel32!UnhandledExceptionFilter+0×519
006afb40 7d4d8a25 kernel32!BaseThreadStart+0×4a (FPO: [SEH])
006afb68 7d61ec2a kernel32!_except_handler3+0×61
006afb8c 7d61ebfb ntdll!ExecuteHandler2+0×26
006afc34 7d61ea36 ntdll!ExecuteHandler+0×24
006afc34 0040144f ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 006afc9c)
006aff7c 00404352 MTCrash!thread_two+0×7f
006affb4 004043eb MTCrash!_callthreadstart+0×1b
006affb8 7d4dfe21 MTCrash!_threadstart+0×73
006affec 00000000 kernel32!BaseThreadStart+0×34

#  2  Id: ca4.cb0 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr 
007effb8 7d4dfe21 MTCrash!thread_one+0×6d
007effe4 00000000 kernel32!BaseThreadStart+0×34

0:002> .cxr 006afc9c
eax=00000000 ebx=005a4448 ecx=0041948c edx=00419ef0 esi=00419488 edi=00000000
eip=0040144f esp=006aff68 ebp=7d4d14e0 iopl=0  nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00010246
MTCrash!thread_two+0×7f:
0040144f c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

However as soon as we dismiss the first error message box or if Auto is set to 1 in AeDebug registry key MTCrash terminates. If Crash2Hang is set as a default postmortem debugger then we get two instances of it running and MTCrash hangs even if we dismiss the first message. The main thread continues to run:

C:\Crash2Hang>MTCrash.exe 1
Main Thread is still running...
Thread 1 is about to experience an AV exception...
Main Thread is still running...
Main Thread is still running...
Thread 2 is about to experience an AV exception...
Main Thread is still running...
Main Thread is still running...
Main Thread is still running...
Main Thread is still running...
[...]

- Dmitry Vostokov @ DumpAnalysis.org -

I didn’t know that such challenge and contest exists until I came across this blog:

http://blog.flexilis.com/2008/12/the-2008-malware-challenge/

I’m always interested in malware and reverse engineering because sites about these topics usually contain hard-to-find Windows internals information. May be I try next year if such opportunity arises again.

- Dmitry Vostokov @ DumpAnalysis.org -

Here is the front cover image for the first issue of previously announced Debugged! MZ/PE magazine:

- Dmitry Vostokov @ DumpAnalysis.org -

“Where are you” debugging “tonight?”

Arthur Evelyn St. John Waugh, The Diaries of Evelyn Waugh, edited by Michael Davie

- Dmitry Vostokov @ DumpAnalysis.org -

Last quarter was very busy for me and to keep up with schedule I now employ pipeline book writing techniques borrowed from CPU of my laptop to work simultaneously on 10 books. I also feel more relaxed with take it easy attitude towards writing and publishing: TIEP - TIE Publishing. In summary: the following book is planned to be released in Q1, 2009 where I’m an author:

• - Windows® Debugging: Practical Foundations (ISBN: 978-1906717100)

A magazine issue is planned for Q1 where I’m an editor:

• - March issue of Debugged! MZ/PE: MagaZine for/from Practicing Engineers (ISBN: 978-1906717384)

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes there is a need to preserve a crashing application or a service from termination and keep it in memory without showing any GUI dialogs or message boxes. Here Crash2Hang tool comes handy. It is free and can be downloaded from here:

Download Crash2Hang

The source code is simple as possible:

// Crash2Hang
// Copyright (c) 2009 Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt

#include <windows.h>

int main(int argc, WCHAR* argv[])
{
 if (argc > 1)
  MessageBox(NULL, L"One of processes has called a postmortem debugger!", L"Crash2Hang", MB_OK | MB_ICONSTOP | MB_SETFOREGROUND);
 else
  Sleep(INFINITE);
 return 0;
}

The tool can be used as a postmortem debugger specified in AeDebug registry key, for example, instead of CDB. Any argument specified to Crash2Hang.exe causes it to display a message box when launched

and exit process upon its dismissal. If several threads in a problem process experience an unhandled exception then Crash2Hang process is launched several times which may result in several such message boxes. Without arguments Crash2Hang process hangs infinitely causing the problem thread with an unhandled exception to hang indefinitely too (see my old post Who calls the postmortem debugger? for explanation).  

- Dmitry Vostokov @ DumpAnalysis.org -

Bugs “are useful to attract attention to” programs.

Mandell Creighton, Life and Letters of Mandell Creighton by Louise Creighton

- Dmitry Vostokov @ DumpAnalysis.org -

It had always been my dream since I left Moscow State University to be associated with a research institute. Until yesterday it became a reality with the announcement of

Memory Analysis & Debugging Institute (MA_&D^I).

From: http://www.dumpanalysis.org/madinstitute-announcement

- Dmitry Vostokov @ DumpAnalysis.org -

OpenTask publisher plans to expand its offering of computer memory visualization titles:

http://www.opentask.com/memory-visualization-titles

More details will be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org -

In one complete memory dump taken from the system refusing user connections but not hung completely we can see the signs of past pool allocation failures (see Insufficient Memory):

0: kd> !vm
[...]
       PagedPool Usage:       47391 (    189564 Kb)
       PagedPool Maximum:     67584 (    270336 Kb)

       ********** 981 pool allocations have failed **********
[…]

We check paged pool usage but the output is inconsistent (shown in magenta color):

0: kd> !poolused 4
   Sorting by  Paged Pool Consumed

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 LSmi        0        0        -1 4294967240 BlockTypeMisc
 PpEE        0        0        -1 4294967040 PNP_DEVICE_EVENT_ENTRY_TAG , Binary: nt!pnp
 CM         58     2320        -1 4294967000 Configuration Manager (registry) , Binary: nt!cm
 SeSc        0        0       -65 4294966112 Captured Security Descriptor , Binary: nt!se
 RxMs        1     1096       -99 4294947312 misc.
 CM38        0        0        -2 4294942720 Internal Configuration manager allocations , Binary: nt!cm
 RxFc        0        0        -8 4294879664 FCB
 Lfs         0        0      -907 4294872976 Lfs allocations
 xSMB        0        0      -179 4293500928 IFSKIT sample SMB mini-redirector , Binary: smbmrx.sys
 AAAA        4      224       581 51639048 UNKNOWN pooltag ‘AAAA’, please update pooltag.txt
 BBBB        2    65664      2582 16362984 UNKNOWN pooltag ‘BBBB’, please update pooltag.txt
 MmSt        0        0     10718 14944776 Mm section object prototype ptes , Binary: nt!mm
[…]

However we see that drivers using AAAA and BBBB consumed almost 65Mb and we can search for them as described here.

Dumping processes we notice signs of possible handle leak:

0: kd> !process 0 0
[...]
PROCESS 89ac09c0  SessionId: 0  Cid: 04b0    Peb: 7ffd5000  ParentCid: 0480
    DirBase: cc210000  ObjectTable: e13991a0  HandleCount: 3329.
    Image: csrss.exe

PROCESS 89ae4508  SessionId: 0  Cid: 07a8    Peb: 7ffdf000  ParentCid: 04f4
    DirBase: cb330000  ObjectTable: e115c220  HandleCount: 4476.
    Image: svchost.exe

PROCESS 868d1d88  SessionId: 0  Cid: 4120    Peb: 7ffd8000  ParentCid: 04f4
    DirBase: 95558000  ObjectTable: e1135428  HandleCount: 2255.
    Image: AppC.exe
[…]

We see lots of threads in the process 89ae4508 waiting for LPC reply:

0: kd> !thread 86b64388 1f
THREAD 86b64388  Cid 07a8.0fbc  Teb: 7ff73000 Win32Thread: bc173ac0 WAIT: (Unknown) UserMode Non-Alertable
    86b64574  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 06345018:
Current LPC port e169dd90
Impersonation token:  e492a6c0 (Level Impersonation)
DeviceMap                 e1603ce8
Owning Process            89ae4508       Image:         svchost.exe
Wait Start TickCount      148053822      Ticks: 23982406 (4:08:05:25.093)
Context Switch Count      11                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address DllA!ThreadEntry (0×752e27fe)
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 901e5000 Current 901e4c08 Base 901e5000 Limit 901e2000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

Now we follow LPC wait chain:

0: kd> !lpc message 06345018
Searching message 6345018 in threads ...
Client thread 86b64388 waiting a reply from 6345018                         
    Server thread 87f0d790 is working on message 6345018
[…]

0: kd> !thread 87f0d790 1f
THREAD 87f0d790  Cid 0de4.5b2c  Teb: 7ff8f000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    87f0d97c  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 0634501d:
Current LPC port eb2e6450
Impersonation token:  e93ff870 (Level Impersonation)
DeviceMap                 e1603ce8
Owning Process            89c36690       Image:         AppA.exe
Wait Start TickCount      113650910      Ticks: 58385318 (10:13:24:30.593)
Context Switch Count      373            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Win32 Start Address 0×06345018
LPC Server thread working on message Id 6345018
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8b0e6000 Current 8b0e5c08 Base 8b0e6000 Limit 8b0e3000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

0: kd> !lpc message 0634501d
Searching message 634501d in threads ...
Client thread 87f0d790 waiting a reply from 634501d                         
    Server thread 89137780 is working on message 634501d
[…]

0: kd> !thread 89137780 1f
THREAD 89137780  Cid 1884.41f8  Teb: 7ff90000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    8913796c  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064aa11b:
Current LPC port ea3fc860
Impersonation token:  e93ff870 (Level Impersonation)
DeviceMap                 e1603ce8
Owning Process            8a608020       Image:         AppB.exe
Wait Start TickCount      148002015      Ticks: 24034213 (4:08:18:54.578)
Context Switch Count      700            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Win32 Start Address 0×0634501d
LPC Server thread working on message Id 634501d
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8b749000 Current 8b748c08 Base 8b749000 Limit 8b746000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

0: kd> !lpc message 064aa11b
Searching message 64aa11b in threads ...
Client thread 89137780 waiting a reply from 64aa11b                         
    Server thread 87acb728 is working on message 64aa11b
[…]

0: kd> !thread 87acb728 1f
THREAD 87acb728  Cid 4120.4078  Teb: 7ff3f000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    87acb914  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064aa127:
Current LPC port e7ec63f0
Not impersonating
DeviceMap                 e1003910
Owning Process            868d1d88       Image:         AppC.exe
Wait Start TickCount      147996856      Ticks: 24039372 (4:08:20:15.187)
Context Switch Count      440            
UserTime                  00:00:00.812
KernelTime                00:00:00.015
Win32 Start Address 0×064aa11b
LPC Server thread working on message Id 64aa11b
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8ba35000 Current 8ba34c08 Base 8ba35000 Limit 8ba32000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

0: kd> !lpc message 064aa127
Searching message 64aa127 in threads ...
    Server thread 899e1750 is working on message 64aa127
Client thread 87acb728 waiting a reply from 64aa127                         
[…]

0: kd> !thread 899e1750 1f
THREAD 899e1750  Cid 0a0c.6cfc  Teb: 7ff16000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89293390  Mutant - owning thread 89a9dc38
Not impersonating
DeviceMap                 e1003910
Owning Process            892b8a38       Image:         svchost.exe
Wait Start TickCount      148115996      Ticks: 23920232 (4:07:49:13.625)
Context Switch Count      166            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×064aa127
LPC Server thread working on message Id 64aa127
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 8c7b1000 Current 8c7b0c60 Base 8c7b1000 Limit 8c7ae000 Call 0
Priority 13 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

We finally come to a thread waiting for a mutant and we inspect its owner:

0: kd> !thread 89a9dc38 1f
THREAD 89a9dc38  Cid 0a0c.185c  Teb: 7ff8d000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89a9de24  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 064a90f1:
Current LPC port e15992a8
Not impersonating
DeviceMap                 e1003910
Owning Process            892b8a38       Image:         svchost.exe
Wait Start TickCount      148115996      Ticks: 23920232 (4:07:49:13.625)
Context Switch Count      29043            
UserTime                  00:00:01.046
KernelTime                00:00:00.968
Win32 Start Address 0×064a8fb6
LPC Server thread working on message Id 64a8fb6
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 92201000 Current 92200c08 Base 92201000 Limit 921fe000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

We see it is waiting for an LPC reply from the thread waiting for the mutant we already saw:

0: kd> !lpc message 064a90f1
Searching message 64a90f1 in threads ...
Client thread 89a9dc38 waiting a reply from 64a90f1                      
    Server thread 88806a28 is working on message 64a90f1
[…]

0: kd> !thread 88806a28 1f
THREAD 88806a28  Cid 0a0c.10b8  Teb: 7ff82000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89293390  Mutant - owning thread 89a9dc38
Not impersonating
DeviceMap                 e1003910
Owning Process            892b8a38       Image:         svchost.exe
Wait Start TickCount      148115996      Ticks: 23920232 (4:07:49:13.625)
Context Switch Count      532            
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Win32 Start Address 0×064a90f1
LPC Server thread working on message Id 64a90f1
Start Address kernel32!BaseThreadStartThunk (0×77e6b5f3)
Stack Init 94ef3000 Current 94ef2c60 Base 94ef3000 Limit 94ef0000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.

We have

… -> Thread(899e1750) -> ^{Mutant(89293390)} Thread(89a9dc38) -> ^{LPC(064a90f1)} Thread(88806a28) -> ^{Mutant(89293390)} Thread(89a9dc38)

This looks like a deadlock although we cannot examine stack traces which are not resident (stack data resides in a page file).

We also notice the system uptime which might suggest that all these abnormalities had been gradually accumulated (see Overaged System):

- Dmitry Vostokov @ DumpAnalysis.org -

Seems every respectable publisher now comes with its own debugging book title. Now it is Pragmatic Bookshelf:

Debug It!: Find, Repair, and Prevent Bugs in Your Code

Curiously enough the title sounds similar to Debugged! magazine from Dump Analysis Portal… 

Debugged! MZ/PE: Magazine for/from Practicing Engineers

Despite this similarity both titles also have a pragmatic difference: Debug it! is an imperative but Debugged! is a statement of success

- Dmitry Vostokov @ DumpAnalysis.org -

The analogy between learning a complex tool with its own language and a foreign natural language has been developed further after the release of WinDbg Learning Cards and finally culminated in “WinDbg In Use” book series with the first book to be published during the 1st quarter of 2009:

• Title: WinDbg In Use: Debugging Exercises (Elementary and Intermediate Level)
• Author: Dmitry Vostokov
• Publisher: Opentask (15 March 2009)
• Language: English
• Product Dimensions: 23.5 x 19.1
• ISBN-13: 978-1-906717-50-6
• Paperback: 200 pages
• Book Annotation: Includes 60 programmed exercises from real life debugging and crash dump analysis scenarios and multiple-choice questions with full answers, comments and suggestions for further reading.

Some example exercises will be published on this blog from time to time. I also plan a corresponding column in the forthcoming Debugged! magazine. 

- Dmitry Vostokov @ DumpAnalysis.org -