Software Diagnostics Library

This is a forthcoming book about .NET debugging seen in a wider context than .NET runtime environment (CLR). There is the whole new generation of .NET software developers, designers and architects thinking in terms of managed code and associated concepts. However CLR runs in unmanaged environment which finally interfaces with native code. Therefore understanding unmanaged and native code is vital for successful debugging of real customer problems. Preliminary information is:

• Title: Unmanaged Code: Escaping the Matrix of .NET
• Author: Dmitry Vostokov
• Paperback: 512 pages (*)
• ISBN-13: 978-0-9558328-6-4
• Publisher: Opentask (1 Feb 2009)
• Language: English
• Product Dimensions: 22.86 x 15.24

(*) subject to change

- Dmitry Vostokov @ DumpAnalysis.org -

Thanks to everyone responded online and privately to proposed draft covers for forthcoming MDAA volumes. As some commented the original one looks good on the shelf so the final decision will be made when I print both variants for comparison. Supporters of the original cover proposed the following modification:

Also there will be surprise on the back cover of the book

PS. I decided to read couple of books about book design and manufacturing to get clearer picture about the whole process.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m coming back to my old Insufficient Memory pattern range (see Committed Memory, Handle Leak, Kernel Pool, PTE). This post discusses the user space case when we don’t have enough virtual memory available for reservation due to memory fragmentation. For example, a java virtual machine is pre-allocating memory for its garbage-collected heap. However after installing some 3rd-party software the amount of pre-allocated memory is less than expected. In such cases it is possible to do comparative memory dump analysis to see the difference in virtual address spaces. Original memory dump has this module distribution in memory:

0:000> lm
start    end        module name
00400000 0040b000   javaw      (deferred)            
009e0000 009e7000   hpi        (deferred)            
00a30000 00a3e000   verify     (deferred)            
00a40000 00a59000   java       (deferred)            
00a60000 00a6d000   zip        (deferred)            
03ff0000 03fff000   net        (deferred)            
040a0000 040a8000   nio        (deferred)            
040b0000 0410a000   hnetcfg    (deferred)            
041d0000 042e2000   awt        (deferred)            
04540000 04591000   fontmanager   (deferred)            
04620000 04670000   msctf      (deferred)            
047c0000 047de000   jpeg       (deferred)            
05820000 05842000   dcpr       (deferred)            
05920000 05932000   pkcs11wrapper   (deferred)            
08000000 08139000   jvm        (deferred)            
10000000 100e0000   moduleA    (deferred)            
68000000 68035000   rsaenh     (deferred)
6e220000 6e226000   RMProcessLink   (deferred)            
71ae0000 71ae8000   wshtcpip   (deferred)            
71b20000 71b61000   mswsock    (deferred)            
71bf0000 71bf8000   ws2help    (deferred)            
71c00000 71c17000   ws2_32     (deferred)            
71c20000 71c32000   tsappcmp   (deferred)            
71c40000 71c97000   netapi32   (deferred)            
73070000 73097000   winspool   (deferred)            
76290000 762ad000   imm32      (deferred)            
76920000 769e2000   userenv    (deferred)            
76aa0000 76acd000   winmm      (deferred)            
76b70000 76b7b000   psapi      (deferred)            
76ed0000 76efa000   dnsapi     (deferred)            
76f10000 76f3e000   wldap32    (deferred)            
76f50000 76f63000   secur32    (deferred)            
76f70000 76f77000   winrnr     (deferred)            
76f80000 76f85000   rasadhlp   (deferred)            
77380000 77411000   user32     (deferred)            
77670000 777a9000   ole32      (deferred)            
77ba0000 77bfa000   msvcrt     (deferred)            
77c00000 77c48000   gdi32      (deferred)            
77c50000 77cef000   rpcrt4     (deferred)            
77e40000 77f42000   kernel32   (deferred)            
77f50000 77feb000   advapi32   (deferred)            
78130000 781cb000   msvcr80    (deferred)            
7c800000 7c8c0000   ntdll      (pdb symbols)

We see the big gap between 100e0000 and 68000000 addresses. This means that it is theoretically possible to reserve and/or commit up to 57F20000 bytes (about 1.4Gb). !address WinDbg command shows that at least 1.1Gb region (shown in bold below) was reserved indeed:

0:000> !address
00000000 : 00000000 - 00010000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00010000 : 00010000 - 00001000
              Type     00020000 MEM_PRIVATE
              Protect  00000004 PAGE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageEnvironmentBlock
00011000 : 00011000 - 0000f000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00020000 : 00020000 - 00001000
              Type     00020000 MEM_PRIVATE
              Protect  00000004 PAGE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageProcessParametrs
00021000 : 00021000 - 0000f000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
00030000 : 00030000 - 00003000
              Type     00020000 MEM_PRIVATE
              Protect  00000140 <unk>
              State    00001000 MEM_COMMIT
              Usage    RegionUsageStack
              Pid.Tid  97c.1b3c
...
...
...
100e0000 : 100e0000 - 000a0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         10180000 - 05cc0000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
         15e40000 - 004f3000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         16333000 - 45bad000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
         5bee0000 - 00ac0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         5c9a0000 - 03540000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
5fee0000 : 5fee0000 - 00120000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
…
…
…

Looking at the problem memory dump we see that the gap is smaller (less than 1.1Gb):

0:000> lm
start    end        module name
00400000 0040b000   javaw      (deferred)            
08000000 08139000   jvm        (deferred)            
10000000 10007000   hpi        (deferred)            
51120000 511bb000   msvcr80  # (private pdb symbols)
520f0000 520fe000   verify     (deferred)            
52100000 52119000   java       (deferred)            
52120000 5212d000   zip        (deferred)            
52130000 5213f000   net        (deferred)            
52140000 52148000   nio        (deferred)            
52150000 52262000   awt        (deferred)            
52270000 522c1000   fontmanager   (deferred)            
522d0000 52320000   MSCTF      (deferred)            
52330000 5234e000   jpeg       (deferred)            
52350000 52372000   dcpr       (deferred)            
52510000 52522000   pkcs11wrapper   (deferred)            
5f270000 5f2ca000   hnetcfg    (deferred)            
60000000 60029000   3rdPartyHook   (deferred)            
61e80000 61e86000   detoured   (export symbols)
68000000 68035000   rsaenh     (deferred)            
71ae0000 71ae8000   wshtcpip   (deferred)            
71b20000 71b61000   mswsock    (deferred)            
71bf0000 71bf8000   ws2help    (deferred)            
71c00000 71c17000   ws2_32     (deferred)            
71c20000 71c32000   tsappcmp   (deferred)            
71c40000 71c97000   netapi32   (deferred)            
73070000 73097000   winspool   (deferred)            
76290000 762ad000   imm32      (deferred)            
76920000 769e2000   userenv    (deferred)            
76aa0000 76acd000   winmm      (deferred)            
76b70000 76b7b000   psapi      (deferred)            
76ed0000 76efa000   dnsapi     (deferred)            
76f10000 76f3e000   wldap32    (deferred)            
76f50000 76f63000   secur32    (deferred)            
76f70000 76f77000   winrnr     (deferred)            
76f80000 76f85000   rasadhlp   (deferred)            
77380000 77411000   user32     (pdb symbols)         
77670000 777a9000   ole32      (deferred)            
77ba0000 77bfa000   msvcrt     (deferred)            
77c00000 77c48000   gdi32      (deferred)            
77c50000 77cef000   rpcrt4     (deferred)            
77e40000 77f42000   kernel32   (pdb symbols)         
77f50000 77feb000   advapi32   (pdb symbols)         
7c340000 7c396000   msvcr71    (deferred)            
7c800000 7c8c0000   ntdll      (pdb symbols)

!address command shows that less memory was reserved in the latter case (about 896Mb):

0:000> !address
...
...
...
10010000 : 10010000 - 000a0000
             Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         100b0000 - 04a70000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
         14b20000 - 004a6000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
         14fc6000 - 3804a000
             Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
          4d010000 - 00ac0000
              Type     00020000 MEM_PRIVATE
              Protect  00000040 PAGE_EXECUTE_READWRITE
              State    00001000 MEM_COMMIT
              Usage    RegionUsageIsVAD
          4dad0000 - 03540000
              Type     00020000 MEM_PRIVATE
              State    00002000 MEM_RESERVE
              Usage    RegionUsageIsVAD
51010000 : 51010000 - 00110000
              Type     00000000
              Protect  00000001 PAGE_NOACCESS
              State    00010000 MEM_FREE
              Usage    RegionUsageFree
…
…
…

Looking at module list again we notice that most java runtime modules were shifted to 50000000 address range. We also notice that new 3rdPartyHook and detoured modules appear in our problem memory dump. 

Module information is missing for detoured module:

0:000> lmv m detoured
start    end        module name
61e80000 61e86000   detoured   (deferred)            
    Image path: C:\WINDOWS\system32\detoured.dll
    Image name: detoured.dll
    Timestamp:        Thu Feb 07 04:14:16 2008 (47AA8598)
    CheckSum:         0000EF91
    ImageSize:        00006000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        0.0 Unknown
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

However applying Unknown Component pattern we see that is Microsoft Research Detours Package:

0:000> db 61e80000 61e86000
61e80000  MZ..............
61e80010  ........@.......
61e80020  ................
61e80030  ................
61e80040  ........!..L.!Th
61e80050  is program canno
61e80060  t be run in DOS
61e80070  mode....$.......
61e80080  5...q...q...q...
61e80090  ....r...q...p...
61e800a0  V%..p...V%..p...
61e800b0  V%..p...V%..p...
61e800c0  Richq...........
61e800d0  ................
61e800e0  PE..L......G....
61e800f0  .......!........
61e80100  ................
...
...
...
...
61e84390  ..P.r.o.d.u.c.t.
61e843a0  N.a.m.e…..M.i.
61e843b0  c.r.o.s.o.f.t. .
61e843c0  R.e.s.e.a.r.c.h.
61e843d0  .D.e.t.o.u.r.s.
61e843e0  .P.a.c.k.a.g.e.
61e843f0  ….j.#…P.r.o.
61e84400  d.u.c.t.V.e.r.s.
61e84410  i.o.n…P.r.o.f.
61e84420  e.s.s.i.o.n.a.l.
61e84430  .V.e.r.s.i.o.n.
61e84440  .2…1. .B.u.i.
61e84450  l.d._.2.1.0…..
61e84460  D…..V.a.r.F.i.
…
…
…

0:000> du 61e843a0+C
61e843ac  "Microsoft Research Detours Packa"
61e843ec  "ge"

We can also see that 3rdPartyHook module imports this library and lots of kernel32 API related to memory allocation, file mapping and loading DLLs (see No Component Symbols pattern):

0:000> !dh 60000000
...
...
...
OPTIONAL HEADER VALUES
     10B magic #
    8.00 linker version
   18000 size of code
    F000 size of initialized data
       0 size of uninitialized data
   13336 address of entry point
    1000 base of code
         ----- new -----
60000000 image base
    1000 section alignment
    1000 file alignment
       2 subsystem (Windows GUI)
    4.00 operating system version
    0.00 image version
    4.00 subsystem version
   29000 size of image
    1000 size of headers
   3376F checksum
00100000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
       0 [       0] address [size] of Export Directory
   218CC [      8C] address [size] of Import Directory
   25000 [     5F4] address [size] of Resource Directory
       0 [       0] address [size] of Exception Directory
   28000 [    19E0] address [size] of Security Directory
   26000 [    2670] address [size] of Base Relocation Directory
   19320 [      1C] address [size] of Debug Directory
       0 [       0] address [size] of Description Directory
       0 [       0] address [size] of Special Directory
       0 [       0] address [size] of Thread Storage Directory
   1F3C0 [      40] address [size] of Load Configuration Directory
       0 [       0] address [size] of Bound Import Directory
   19000 [     2B0] address [size] of Import Address Table Directory
       0 [       0] address [size] of Delay Import Directory
       0 [       0] address [size] of COR20 Header Directory
       0 [       0] address [size] of Reserved Directory
…
…
…

0:000> dds 60000000+19000 60000000+19000+2B0
60019064  7c82b0dc ntdll!RtlReAllocateHeap
60019068  77e4ec39 kernel32!HeapDestroy
6001906c  77e41fba kernel32!GetSystemTimeAsFileTime
60019070  77e619d1 kernel32!GetTickCount
60019074  77e69577 kernel32!QueryPerformanceCounter
60019078  7c82a9be ntdll!RtlSizeHeap
6001907c  77e82060 kernel32!SetUnhandledExceptionFilter
60019080  77e7690d kernel32!UnhandledExceptionFilter
60019084  77e42004 kernel32!TerminateProcess
60019088  7c82a136 ntdll!RtlRestoreLastWin32Error
6001908c  77e77a5f kernel32!SuspendThread
60019090  77e76a26 kernel32!SetThreadContext
60019094  77e77ae3 kernel32!GetThreadContext
60019098  77e73347 kernel32!FlushInstructionCache
6001909c  77e5f38b kernel32!ResumeThread
600190a0  77e616a8 kernel32!InterlockedCompareExchange
600190a4  77e645a9 kernel32!VirtualAlloc
600190a8  77e41fe3 kernel32!VirtualProtect
600190ac  77e66ed1 kernel32!VirtualQuery
600190b0  77e44960 kernel32!GetLogicalDriveStringsA
600190b4  77eab401 kernel32!GetVolumeNameForVolumeMountPointA
600190b8  77e6794d kernel32!GetACP
600190bc  77e6f3cf kernel32!GetLocaleInfoA
600190c0  77e622b7 kernel32!GetThreadLocale
600190c4  77e69d74 kernel32!GetVersionExA
600190c8  77e4beab kernel32!RaiseException
600190cc  77e60037 kernel32!GetSystemDirectoryA
600190d0  77e52bf4 kernel32!GetWindowsDirectoryA
600190d4  77e5c7a8 kernel32!lstrcmpA
600190d8  77e46c99 kernel32!OutputDebugStringA
600190dc  77e5bd7d kernel32!CreateEventA
600190e0  77e62311 kernel32!SetEvent
600190e4  77e51281 kernel32!ExpandEnvironmentStringsA
600190e8  77e9f365 kernel32!MoveFileA
600190ec  77e5da00 kernel32!IsDebuggerPresent
600190f0  77e9e4b1 kernel32!QueryDosDeviceA
600190f4  7c829e08 ntdll!RtlGetLastWin32Error
600190f8  77e63d7a kernel32!GetProcAddress
600190fc  77e41dc6 kernel32!LoadLibraryA
60019100  7c829e17 ntdll!RtlFreeHeap
60019104  77e62419 kernel32!LocalFree
60019108  7c829fd6 ntdll!RtlAllocateHeap
6001910c  77e63ec7 kernel32!GetProcessHeap
60019110  77ea2186 kernel32!VerifyVersionInfoA
60019114  7c81379f ntdll!VerSetConditionMask
60019118  77e63143 kernel32!WideCharToMultiByte
6001911c  77e70550 kernel32!SizeofResource
60019120  77e6b11b kernel32!SetHandleCount
60019124  77e69bf9 kernel32!LoadResource
60019128  77e511e1 kernel32!FindResourceA
6001912c  77e7388c kernel32!FindResourceExA
60019130  77e5be30 kernel32!lstrlenA
60019134  77e424de kernel32!Sleep
60019138  77ea2cb1 kernel32!WaitNamedPipeA
6001913c  77e63e6f kernel32!CloseHandle
60019140  77e5e123 kernel32!OpenEventA
60019144  77e622c9 kernel32!lstrlenW
60019148  77e62fc7 kernel32!GetCurrentThreadId
6001914c  77e5fdd4 kernel32!OpenProcess
60019150  77e63c78 kernel32!GetCurrentProcessId
60019154  7c81a3ab ntdll!RtlLeaveCriticalSection
60019158  7c81a360 ntdll!RtlEnterCriticalSection
6001915c  77e4cabf kernel32!GetComputerNameA
60019160  77e6f032 kernel32!ProcessIdToSessionId
60019164  77e645ff kernel32!GetModuleFileNameA
60019168  77e6474a kernel32!GetModuleHandleA
6001916c  77e62f9d kernel32!GetCurrentProcess
60019170  77e49968 kernel32!GetCurrentDirectoryA
60019174  77e61c7b kernel32!WaitForSingleObject
60019178  77e63868 kernel32!GetCurrentThread
6001917c  7c82c988 ntdll!RtlDeleteCriticalSection
60019180  77e67861 kernel32!InitializeCriticalSection
60019184  77e6b1a1 kernel32!FreeLibrary
60019188  77e63f41 kernel32!UnmapViewOfFile
6001918c  77e643f1 kernel32!MapViewOfFile
60019190  77e6b65f kernel32!OpenFileMappingA
60019194  77e61694 kernel32!InterlockedExchange
60019198  77e4d2fb kernel32!DeleteFileA
...
...
...
60019294  00000000
60019298  61e81000 detoured!Detoured
6001929c  00000000
…
…
…

This warrants the suspicion that 3rdPartyHook somehow optimized the virtual address space for its own purposes and this resulted in more fragmented virtual address space.

- Dmitry Vostokov @ DumpAnalysis.org -

Suppose we got a kernel, complete or minidump from Windows running under Xen hypervisor. How would we distinguish it from a memory dump of Windows running on non-virtualized hardware? We can check machine id:

kd> !sysinfo machineid
Machine ID Information [From Smbios 2.4, DMIVersion 36, Size=348]
BiosMajorRelease = 3
BiosMinorRelease = 1
BiosVendor = Xen
BiosVersion = 3.1.0
SystemManufacturer = Xen
SystemProductName = HVM domU
SystemVersion = 3.1.0

and drivers:

kd> lm m *xen*
start end module name
f6012000 f605f000 dump_xenvbd (deferred)
f794b000 f795c000 xennet (deferred)
f82c0000 f830d000 xenvbd (deferred)
f845f000 f846b000 XENUTIL (deferred)
f84cf000 f84db000 dump_XENUTIL (deferred)

Note: similar information can be checked for VMWare and Virtual PC.

- Dmitry Vostokov @ DumpAnalysis.org -

Here is an additional kernel space example to my old Dynamic Memory Corruption pattern. If kernel pools are corrupt then calls that allocate or free memory result in bugchecks C2 or 19 and in other less frequent bugchecks (from Google stats):

Bug Checks 0xC2 and 0×19 have parameters in bugcheck arguments that tell the type of detected pool corruption. Refer to WinDbg help for details or use the variant of !analyze command where you can supply optional bugcheck arguments:

1: kd> !analyze -show c2
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request.  Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000000, The caller is requesting a zero byte pool allocation.
Arg2: 00000000, zero.
Arg3: 00000000, the pool type being allocated.
Arg4: 00000000, the pool tag being used.

1: kd> !analyze -show 19 2 1 1 1
BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 00000002, the verifier pool pattern check failed.  The owner has likely corrupted the pool block
Arg2: 00000001, the pool entry being checked.
Arg3: 00000001, size of the block.
Arg4: 00000001, 0.

If we enable special pool on suspected drivers we might get these bugchecks too with the following Google frequency:

Here is one example of nonpaged pool corruption detected during free operation with the following !analyze -v output:

BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 00000020, a pool block header size is corrupt.
Arg2: a34583b8, The pool entry we were looking for within the page.
Arg3: a34584f0, The next pool entry.
Arg4: 0a270001, (reserved)

POOL_ADDRESS:  a34583b8 Nonpaged pool

PROCESS_NAME:  process.exe

CURRENT_IRQL:  2

STACK_TEXT: 
b80a60cc 808927bb nt!KeBugCheckEx+0x1b
b80a6134 80892b6f nt!ExFreePoolWithTag+0x477
b80a6144 b9591400 nt!ExFreePool+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
b80a615c b957b954 driver+0x38400
b80a617c b957d482 driver+0x22954
b80a61c0 b957abf4 driver+0x24482
b80a6260 b957ccef driver+0x21bf4
b80a62a8 8081df65 driver+0x23cef
b80a62bc f721ac45 nt!IofCallDriver+0x45
b80a62e4 8081df65 fltMgr!FltpDispatch+0x6f
b80a62f8 b99de70b nt!IofCallDriver+0x45
b80a6308 b99da6ee filter!Dispatch+0xfb
b80a6318 8081df65 filter!dispatch+0x6e
b80a632c b9bdebfe nt!IofCallDriver+0x45
b80a6334 8081df65 2ndfilter!Redirect+0x7ea
b80a6348 b9bd1756 nt!IofCallDriver+0x45
b80a6374 b9bd1860 3rdfilter!PassThrough+0x136
b80a6384 8081df65 3rdfilter!Dispatch+0x80
b80a6398 808f5437 nt!IofCallDriver+0x45
b80a63ac 808ef963 nt!IopSynchronousServiceTail+0x10b
b80a63d0 8088978c nt!NtQueryDirectoryFile+0x5d
b80a63d0 7c8285ec nt!KiFastCallEntry+0xfc
00139524 7c8274eb ntdll!KiFastSystemCallRet
00139528 77e6ba40 ntdll!NtQueryDirectoryFile+0xc
00139830 77e6bb5f kernel32!FindFirstFileExW+0x3d5
00139850 6002665e kernel32!FindFirstFileW+0x16
00139e74 60026363 process+0x2665e
0013a328 60027852 process+0x26363
0013a33c 60035b58 process+0x27852
0013b104 600385ff process+0x35b58
0013b224 612cb643 process+0x385ff
0013b988 612cc109 dll!FileDialog+0xc53
0013bba0 612cb47b dll!FileDialog+0x1719
0013c2c0 7739b6e3 dll!FileDialog+0xa8b
0013c2ec 77395f82 USER32!InternalCallWinProc+0x28
0013c368 77395e22 USER32!UserCallDlgProcCheckWow+0x147
0013c3b0 7739c9c6 USER32!DefDlgProcWorker+0xa8
0013c3d8 7c828536 USER32!__fnDWORD+0x24
0013c3d8 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
b80a66b8 8091d6d1 nt!KiCallUserMode+0x4
b80a6710 bf8a2622 nt!KeUserModeCallback+0x8f
b80a6794 bf8a2517 win32k!SfnDWORD+0xb4
b80a67dc bf8a13d9 win32k!xxxSendMessageToClient+0x133
b80a6828 bf85ae67 win32k!xxxSendMessageTimeout+0x1a6
b80a684c bf8847a1 win32k!xxxWrapSendMessage+0x1b
b80a6868 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
b80a68a0 8088978c win32k!NtUserMessageCall+0xc0
b80a68a0 7c8285ec nt!KiFastCallEntry+0xfc
0013c3d8 7c828536 ntdll!KiFastSystemCallRet
0013c3d8 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
b80a6b7c 8091d6d1 nt!KiCallUserMode+0x4
b80a6bd4 bf8a2622 nt!KeUserModeCallback+0x8f
b80a6c58 bf8a23a0 win32k!SfnDWORD+0xb4
b80a6ca0 bf8a13d9 win32k!xxxSendMessageToClient+0x118
b80a6cec bf85ae67 win32k!xxxSendMessageTimeout+0x1a6
b80a6d10 bf8c148c win32k!xxxWrapSendMessage+0x1b
b80a6d40 8088978c win32k!NtUserMessageCall+0x9d
b80a6d40 7c8285ec nt!KiFastCallEntry+0xfc
0013f474 7c828536 ntdll!KiFastSystemCallRet
0013f4a0 7739d1ec ntdll!KiUserCallbackDispatcher+0x2e
0013f4dc 7738cf29 USER32!NtUserMessageCall+0xc
0013f4fc 612d3276 USER32!SendMessageA+0x7f
0013f63c 611add41 dll!SubWindow+0x3dc6
0013f658 7739b6e3 dll!SetWindowText+0x37a1
0013f684 7739b874 USER32!InternalCallWinProc+0x28
0013f6fc 7739ba92 USER32!UserCallWinProcCheckWow+0x151
0013f764 7739bad0 USER32!DispatchMessageWorker+0x327
0013f774 61221ca8 USER32!DispatchMessageW+0xf
0013f7e0 0040156d dll!MainLoop+0x2c8
0013ff24 00401dfa process+0x156d
0013ffc0 77e6f23b process+0x1dfa
0013fff0 00000000 kernel32!BaseProcessStart+0x23

MODULE_NAME: driver

IMAGE_NAME:  driver.sys

We see that WinDbg pointed to driver.sys by using a procedure described in one of my old minidump analysis posts: BugCheck C2 Minidump Analysis. 

However any OS component could corrupt the pool prior to detection as the bugcheck description says: “The pool is already corrupt at the time of the current request.”. What other evidence can reinforce our belief in driver.sys? Let’s look at our pool entry tag first:

1: kd> !pool a34583b8
Pool page a34583b8 region is Nonpaged pool
 a3458000 size:  270 previous size:    0  (Allocated)  Thre (Protected)
 a3458270 size:   10 previous size:  270  (Free)       RxIr
 a3458280 size:   40 previous size:   10  (Allocated)  Vadl
 a34582c0 size:   98 previous size:   40  (Allocated)  File (Protected)
 a3458358 size:    8 previous size:   98  (Free)       Vadl
 a3458360 size:   50 previous size:    8  (Allocated)  Gsem
 a34583b0 size:    8 previous size:   50  (Free)       CcSc
*a34583b8 size:  138 previous size:    8  (Allocated) *DRIV
  Owning component : Unknown (update pooltag.txt)
a34584f0 is not a valid large pool allocation, checking large session pool…
a34584f0 is freed (or corrupt) pool
Bad allocation size @a34584f0, zero is invalid

***
*** An error (or corruption) in the pool was detected;
*** Attempting to diagnose the problem.
***
*** Use !poolval a3458000 for more details.
***

Pool page [ a3458000 ] is __inVALID.

Analyzing linked list...
[ a34583b8 --> a34583d8 (size = 0x20 bytes)]: Corrupt region
[ a34583f8 --> a34585e8 (size = 0x1f0 bytes)]: Corrupt region

Scanning for single bit errors...

None found

We see that the tag is DRIV and we know either from association or from similar problems in the past that it belongs to driver.sys. Let’s dump our pool entry contents to see if there are any symbolic hints in it:

1: kd> dps a34583b8
a34583b8 0a270001
a34583bc 5346574e
a34583c0 00000000
a34583c4 00000000
a34583c8 b958f532 driver+0×36532
a34583cc a3471010
a34583d0 0000012e
a34583d4 00000001
a34583d8 00041457
a34583dc 05af0026
a34583e0 00068002
a34583e4 7b9ec6f5
a34583e8 ffffff00
a34583ec 73650cff
a34583f0 7461445c
a34583f4 97a10061
a34583f8 ff340004
a34583fc c437862a
a3458400 6a000394
a3458404 00000038
a3458408 00000000
a345840c bf000000
a3458410 bf0741b5
a3458414 f70741b5
a3458418 00000000
a345841c 00000000
a3458420 00000000
a3458424 00000000
a3458428 05000000
a345842c 34303220
a3458430 31323332
a3458434 ff322d36

Indeed we see the possible code pointer driver+0×36532 and the code around this address looks normal:

3: kd> .asm no_code_bytes
Assembly options: no_code_bytes

3: kd> u b958f532
driver+0x36532:
b958f532 push    2Ch
b958f534 push    offset driver+0x68d08 (b95c1d08)
b958f539 call    driver+0x65c50 (b95bec50)
b958f53e mov     byte ptr [ebp-19h],0
b958f542 and     dword ptr [ebp-24h],0
b958f546 call    dword ptr [driver+0x65f5c (b95bef5c)]
b958f54c mov     ecx,dword ptr [ebp+0Ch]
b958f54f cmp     eax,ecx

3: kd> ub b958f532
driver+0x36528:
b958f528 leave
b958f529 ret     18h
b958f52c int     3
b958f52d int     3
b958f52e int     3
b958f52f int     3
b958f530 int     3
b958f531 int     3

- Dmitry Vostokov @ DumpAnalysis.org -

I read science fiction from time to time now (I was a big fan of it back to school and university days) but I cannot recall memory dumps mentioned explicitly in such books. I’ve just finished reading Dan Simmons’s The Fall of Hyperion book (the sequel to Hyperion book that I read previously) and I recall in chapter 33 on page 303 a poetic description of a process crash (italics are mine):

“Johnny twists a second in the AI’s massive grip (fault injection?), and then his analog - Keats’s small but beautiful body (GUI?) - is torn, compacted, smashed into an unrecognizable mass (corrupt dump?) which Ummon sets against his megalith flesh (private bytes?), absorbing the analogs’s remains (overwriting discarded pages?) back into the orange-and-red depths of itself (working set?).”  

PS: Hyperion and The Fall of Hyperion is the best science fiction I have ever read and highly recommend:

Hyperion

The Fall of Hyperion

I continue looking for crash dumps in Dan Simmons’s Endymion and Rise of Endymion books which are on my lunch time reading list as soon as I finish Global Conspiracy book I’m reading now.  

- Dmitry Vostokov @ DumpAnalysis.org -

259 bugchecks are documented in WinDbg help. I did Google search for every one and here is their search results distribution graph cut off for data with less than 10 matches:

Of course there is some noise and matches do not always correspond to WinDbg bugcheck analysis output but we can get rough idea about bugcheck frequency. For example, unhandled exceptions in kernel mode, IRQL contract violation, pool corruption and hardware failures are the most frequent. Here is the full table:

- Dmitry Vostokov @ DumpAnalysis.org -

Now it’s time to write about Wild Pointer pattern I briefly mentioned in Local Buffer Overflow post which shows examples of pointers having UNICODE structure in their values. I have also observed pointers with ASCII structure as .formats WinDbg command indicates:

FAILED_INSTRUCTION_ADDRESS: 
65747379`735c5357 ??              ???

IP_ON_HEAP:  65747379735c5357

0:012> .formats rip
Evaluate expression:
  Hex:     65747379`735c5357
  Decimal: 7310595060592825175
  Octal:   0625643467456327051527
  Binary:  01100101 01110100 01110011 01111001 01110011 01011100 01010011 01010111
  Chars:   etsys\SW
  Time:    Wed May 10 22:00:59.282 24767 (GMT+1)
  Float:   low 1.7456e+031 high 7.21492e+022
  Double:  5.30388e+180

Here is another example of a pointer having UNICODE structure:

FAILED_INSTRUCTION_ADDRESS: 
0045004c`00490046 ??              ???

0:014> .formats rip
Evaluate expression:
  Hex:     0045004c`00490046
  Decimal: 19422099815333958
  Octal:   0001050004600022200106
  Binary:  00000000 01000101 00000000 01001100 00000000 01001001 00000000 01000110
  Chars:   .E.L.I.F
  Time:    Wed Jul 19 07:46:21.533 1662 (GMT+1)
  Float:   low 6.70409e-039 high 6.33676e-039
  Double:  2.33646e-307

When we have EIP or RIP pointers I have another pattern to name when the value is coincidentally lies inside some valid region of memory: Wild Code. Here is one example of the latter pattern I’m going to write about soon:

IP_ON_STACK:
+e11ffa8

STACK_TEXT:
0e11ff7c 098eeef2 0xe11ffa8
0e11ff84 77b6b530 dll!StartWorking+0xcab
0e11ffb8 7c826063 msvcrt!_endthreadex+0xa3
0e11ffec 00000000 kernel32!BaseThreadStart+0×34

0:020> u
0e11ffa8 dcff fdiv st(7),st
…
…
…

We see that EIP is very close to EBP/ESP and this explains why !analyze -v reports IP_ON_STACK. Clearly floating-point code is not what we should expect. This example shows that wild pointers sometimes are valid but either through code chain or pointer chain execution reaches Invalid Pointer and a process or a system crashes.

- Dmitry Vostokov @ DumpAnalysis.org -

My colleagues and friends have been divided (50-50) over the opinion which draft cover is better (some polishing is required for both of them):

1. Original Draft Cover
2. New Draft Cover

Your comments are much appreciated (you can just respond with the numbers 1 or 2 corresponding to cover links above). You can really influence the cover of the book!

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced draft cover for Memory Dump Analysis Anthology has got new look and feel:

Books symbolize modular memory structure found in Windows memory dumps (see Dump2Picture paintings) and blue strip separating cover text and books symbolizes familiar blue screen).

- Dmitry Vostokov @ DumpAnalysis.org -

I was wondering whether there are any tools that allow to step through unmanaged and native code backwards when we do crash dump analysis or live debugging on Windows platforms and found these interesting references from Microsoft Research:

• iDNA: Time Travel Debugging (PPT)

• Framework for Instruction-level Tracing and Analysis of Program Executions (PDF)

It would be really good if Microsoft integrates this into Debugging Tools for Windows. 

- Dmitry Vostokov @ DumpAnalysis.org -

They know buzzwords like heap corruption, buffer overflow and multi-threading, talk about designing maintainable software but unable to cope with real-life debugging scenarios. I remember old days in one company where the executable Rational Rose Real-Time model with IDebug interface crashed and the whole team of software designers couldn’t find the defect in code and when I pointed them to the problem source code line after loading and running the executable under the Visual Studio debugger I was nominated as an expert in implementation.

- Dmitry Vostokov @ DumpAnalysis.org -

Q. Every time you open the specific Microsoft Word 2007 document on Vista WER error message box appears on the screen (click on the picture to enlarge):

What might be the cause of it?

You can find the correct answer in the comments to this post. 

- Dmitry Vostokov @ DumpAnalysis.org -

I decided to elaborate a bit on how to see whether the specific synchronization object has signaled or not. If that object has signaled (is in signaled state) then any wait on it will be satisfied immediately. All such objects like kernel events, processes and threads are structures in memory and have _OBJECT_HEADER structure followed by _DISPATCHER_HEADER structure and then by the actual object body (if it is necessary). These objects can be found in the output of !process or !thread commands where they are listed as being waited for, for example:

THREAD 88865db0  Cid 05d0.5ea8  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89a65a68  SynchronizationEvent
Not impersonating
DeviceMap                 e1536358
Owning Process            8abbdd88       Image:         svchost.exe
Wait Start TickCount      5303768        Ticks: 1132797 (0:04:54:59.953)
Context Switch Count      193            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×7c82b5f3)
Stack Init b8ca2000 Current b8ca1c60 Base b8ca2000 Limit b8c9f000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b8ca1c78 80832f7a nt!KiSwapContext+0×26
b8ca1ca4 8082925c nt!KiSwapThread+0×284
b8ca1cec 80937e6a nt!KeWaitForSingleObject+0×346
b8ca1d50 80888c7c nt!NtWaitForSingleObject+0×9a
b8ca1d50 7c94ed54 nt!KiFastCallEntry+0xfc
00c6fea0 7c942124 ntdll!KiFastSystemCallRet
00c6fea4 7c95970f ntdll!NtWaitForSingleObject+0xc
00c6fee0 7c959620 ntdll!RtlpWaitOnCriticalSection+0×19c
00c6ff00 7c95c1e7 ntdll!RtlEnterCriticalSection+0xa8
00c6ffa8 7c8261d6 ntdll!LdrShutdownThread+0×33
00c6ffb8 7c826090 kernel32!ExitThread+0×2f
00c6ffec 00000000 kernel32!BaseThreadStart+0×39

We can see them as parameters of KeWaitForSingle(Multiple)Objects functions:

0: kd> !thread 88865db0
THREAD 88865db0  Cid 05d0.5ea8  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89a65a68  SynchronizationEvent
Not impersonating
DeviceMap                 e1536358
Owning Process            8abbdd88       Image:         svchost.exe
Wait Start TickCount      5303768        Ticks: 1132797 (0:04:54:59.953)
Context Switch Count      193            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×7c82b5f3)
Stack Init b8ca2000 Current b8ca1c60 Base b8ca2000 Limit b8c9f000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr  Args to Child             
b8ca1c78 80832f7a 88865e28 88865db0 88865e58 nt!KiSwapContext+0×26
b8ca1ca4 8082925c 00000000 00000000 00000000 nt!KiSwapThread+0×284
b8ca1cec 80937e6a 89a65a68 00000006 00000001 nt!KeWaitForSingleObject+0×346
b8ca1d50 80888c7c 00000470 00000000 00000000 nt!NtWaitForSingleObject+0×9a
b8ca1d50 7c94ed54 00000470 00000000 00000000 nt!KiFastCallEntry+0xfc
00c6fea0 7c942124 7c95970f 00000470 00000000 ntdll!KiFastSystemCallRet
00c6fea4 7c95970f 00000470 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
00c6fee0 7c959620 00000000 00000004 009171a8 ntdll!RtlpWaitOnCriticalSection+0×19c
00c6ff00 7c95c1e7 7c9a9d94 00000000 7ffaa000 ntdll!RtlEnterCriticalSection+0xa8
00c6ffa8 7c8261d6 00000000 00000000 00c6ffec ntdll!LdrShutdownThread+0×33
00c6ffb8 7c826090 00000000 00000000 00000000 kernel32!ExitThread+0×2f
00c6ffec 00000000 77c2de6d 009171a8 00000000 kernel32!BaseThreadStart+0×39

We can also see them as part of other structures, for example:

0: kd> !irp 88f4fd28 1
Irp is active with 1 stacks 1 is current (= 0x88f4fd98)
 No Mdl: No System Buffer: Thread 8a9449a0:  Irp stack trace. 
Flags = 00000800
ThreadListEntry.Flink = 8a944ba8
ThreadListEntry.Blink = 8a944ba8
IoStatus.Status = 00000000
IoStatus.Information = 00000000
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00dddf04
UserEvent = 8a033c88
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00dddf04
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = f75c815c   Npfs!NpCancelListeningQueueIrp
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = 88f4fd68
Tail.Overlay.Thread = 8a9449a0
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = e311f7d4
Tail.Overlay.ListEntry.Blink = e311f7d4
Tail.Overlay.CurrentStackLocation = 88f4fd98
Tail.Overlay.OriginalFileObject = 884bc8a0
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
>[  d, 0]   5  1 8a937398 884bc8a0 00000000-00000000    pending
        \FileSystem\Npfs
   Args: 00000000 00000000 00110008 00000000

0: kd> !object 8a033c88
Object: 8a033c88  Type: (8ad7a990) Event
    ObjectHeader: 8a033c70 (old version)
    HandleCount: 1  PointerCount: 3

They can also be found in process or kernel (System process) handle tables: 

0: kd> !process 0 0
...
...
...
PROCESS 8a8fa8c0  SessionId: 0  Cid: 037c    Peb: 7ffd8000  ParentCid: 04ac
    DirBase: f3b10360  ObjectTable: e1c276b8  HandleCount: 500.
    Image: MyService.exe
...
...
...

0: kd> !kdexts.handle 0 3 037c
...
...
...
0510: Object: 89237d88  GrantedAccess: 001f0fff Entry: e1cafa20
Object: 89237d88  Type: (8ad84900) Process
    ObjectHeader: 89237d70 (old version)
        HandleCount: 1  PointerCount: 2
…
…
…

Let’s look at the last process object 89237d88. This is the start of object-specific structure whose first member is _DISPATCHER_HEADER. Every object-specific structure is preceded in memory by _OBJECT_HEADER structure and in our case its address is 89237d70. To illustrate this memory layout we can write this inheritance relationship in C++ pseudo-code where I put relevant structure members:    

struct _KPROCESS : _DISPATCHER_HEADER
{
   // ... members
};

struct _DISPATCHER_HEADER
{
   UChar Type;
   // ... members
   Int4B SignalState; 
   // ... members
};

struct _OBJECT_HEADER
{
   Int4B PointerCount;
   Int4B HandleCount; 
   // ... members
   _OBJECT_TYPE *Type;
   // ... members
   _QUAD Body; // Int8B - first 8 bytes of _DISPATCHER_HEADER  
};

HandleCount is the number of open handles and PointerCount is the number of dereferences.  

I also put all this on the following simplified UML diagram:

Here is the detailed data for our process object: 

0: kd> dt _OBJECT_HEADER 89237d70
ntdll!_OBJECT_HEADER
   +0x000 PointerCount     : 2
   +0x004 HandleCount      : 1
   +0x004 NextToFree       : 0x00000001
   +0x008 Type             : 0x8ad84900 _OBJECT_TYPE
   +0x00c NameInfoOffset   : 0 ''
   +0x00d HandleInfoOffset : 0 ''
   +0x00e QuotaInfoOffset  : 0 ''
   +0x00f Flags            : 0x20 ' '
   +0x010 ObjectCreateInfo : 0x808ad180 _OBJECT_CREATE_INFORMATION
   +0x010 QuotaBlockCharged : 0x808ad180
   +0x014 SecurityDescriptor : 0xe1002bd6
   +0x018 Body             : _QUAD

0: kd> dt _DISPATCHER_HEADER 89237d88
ntdll!_DISPATCHER_HEADER
   +0x000 Type             : 0x3 ''
   +0x001 Absolute         : 0 ''
   +0x001 NpxIrql          : 0 ''
   +0x002 Size             : 0x1e ''
   +0x002 Hand             : 0x1e ''
   +0x003 Inserted         : 0 ''
   +0x003 DebugActive      : 0 ''
   +0x000 Lock             : 1966083
   +0×004 SignalState      : 1
   +0×008 WaitListHead     : _LIST_ENTRY [ 0×89237d90 - 0×89237d90 ]

0: kd> dt _OBJECT_TYPE 0x8ad84900
ntdll!_OBJECT_TYPE
   +0x000 Mutex            : _ERESOURCE
   +0x038 TypeList         : _LIST_ENTRY [ 0x8ad84938 - 0x8ad84938 ]
   +0×040 Name             : _UNICODE_STRING “Process”
   +0×048 DefaultObject    : (null)
   +0×04c Index            : 5
   +0×050 TotalNumberOfObjects : 0×18d
   +0×054 TotalNumberOfHandles : 0×35d
   +0×058 HighWaterNumberOfObjects : 0×2d4
   +0×05c HighWaterNumberOfHandles : 0xd66
   +0×060 TypeInfo         : _OBJECT_TYPE_INITIALIZER
   +0×0ac Key              : 0×636f7250
   +0×0b0 ObjectLocks      : [4] _ERESOURCE

We see that SignalState is 1 and this means that the process is in signaled state (exited) and the wait on it will be satisfied immediately (WaitForSingleObject will return). If we close its handle the object will be destroyed.

If we look at other objects threads are waiting for we would see SignalState equal to 0. For example, in the following thread below event handle 4b0 corresponds to a34d9940 _KEVENT address and the latter object is inherited from _DISPATCHER_HEADER:

3: kd> !thread a34369d0
THREAD a34369d0  Cid 1fc8.1e88  Teb: 7ffae000 Win32Thread: bc6d5818 WAIT: (Unknown) UserMode Non-Alertable
    a34d9940  SynchronizationEvent
    a3436a48  NotificationTimer
Not impersonating
DeviceMap                 e12256a0
Owning Process            a3340a10       Image:         IEXPLORE.EXE
Wait Start TickCount      1450409        Ticks: 38 (0:00:00:00.593)
Context Switch Count      7091                 LargeStack
UserTime                  00:00:01.015
KernelTime                00:00:02.250
Win32 Start Address mshtml!CExecFT::StaticThreadProc (0×7fab1061)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init f252b000 Current f252ac60 Base f252b000 Limit f2528000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
f252ac78 80833465 a34369d0 a3436a78 00000003 nt!KiSwapContext+0×26
f252aca4 80829a62 00000000 f252ad14 00000000 nt!KiSwapThread+0×2e5
f252acec 80938d0c a34d9940 00000006 f252ac01 nt!KeWaitForSingleObject+0×346
f252ad50 8088978c 000004b0 00000000 f252ad14 nt!NtWaitForSingleObject+0×9a
f252ad50 7c8285ec 000004b0 00000000 f252ad14 nt!KiFastCallEntry+0xfc (TrapFrame @ f252ad64)
030dff08 7c827d0b 77e61d1e 000004b0 00000000 ntdll!KiFastSystemCallRet
030dff0c 77e61d1e 000004b0 00000000 030dff50 ntdll!NtWaitForSingleObject+0xc
030dff7c 77e61c8d 000004b0 000927c0 00000000 kernel32!WaitForSingleObjectEx+0xac
030dff90 7fab08a3 000004b0 000927c0 00000000 kernel32!WaitForSingleObject+0×12
030dffa8 7fab109c 00000000 7fab106e 030dffec mshtml!CDwnTaskExec::ThreadExec+0xae
030dffb0 7fab106e 030dffec 77e64829 02714f30 mshtml!CExecFT::ThreadProc+0×28
030dffb8 77e64829 02714f30 00000000 00000000 mshtml!CExecFT::StaticThreadProc+0xd
030dffec 00000000 7fab1061 02714f30 00000000 kernel32!BaseThreadStart+0×34

3: kd> dt _DISPATCHER_HEADER a34d9940
cutildll!_DISPATCHER_HEADER
   +0x000 Type             : 0x1 ''
   +0x001 Absolute         : 0xda ''
   +0x002 Size             : 0x4 ''
   +0x003 Inserted         : 0xa3 ''
   +0x003 DebugActive      : 0xa3 ''
   +0x000 Lock             : -1559963135
   +0×004 SignalState      : 0
   +0×008 WaitListHead     : _LIST_ENTRY [ 0xa3436a78 - 0xa3436a78 ]

For real life example of signaled process objects please see Zombie Processes pattern case study.

- Dmitry Vostokov @ DumpAnalysis.org -