Software Diagnostics Library

Not My Version is another basic pattern of DLL Hell variety. It is when we look at the component timestamps and paths and realize that that one of the modules from the production environment is older than we had during development and testing. The lmft WinDbg command will produce the necessary output. If there are many modules we might want to create a CAD graph (Component Age Diagram) to spot anomalies. Component version check is one of the basic troubleshooting and system administration activities that will be fully illustrated in the forthcoming book “Crash Dump Analysis for System Administrators and Support Engineers” (ISBN-13: 978-1-906717-02-5). Here is one example (module start and end load addresses are removed for visual clarity):

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012fed8 00000001`40001093 MyDLL!fnMyDLL
00000000`0012fee0 00000001`40001344 2DLLs+0×1093
00000000`0012ff10 00000000`773acdcd 2DLLs+0×1344
00000000`0012ff60 00000000`774fc6e1 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d


0:000> lmft
module name
MyDLL    C:\OLD\MyDLL.dll Wed Jun 18 14:49:13 2004
user32   C:\Windows\System32\user32.dll Thu Feb 15 05:22:33 2007
kernel32 C:\Windows\System32\kernel32.dll Thu Nov 02 11:14:48 2006
ntdll    C:\Windows\System32\ntdll.dll Thu Nov 02 11:16:02 2006
2DLLs    C:\2DLLs\2DLLs.exe Thu Jun 19 10:46:44 2008 (485A2B04)
uxtheme  C:\Windows\System32\uxtheme.dll Thu Nov 02 11:15:07 2006
rpcrt4   C:\Windows\System32\rpcrt4.dll Tue Jul 17 05:21:15 2007
lpk      C:\Windows\System32\lpk.dll Thu Nov 02 11:12:33 2006
oleaut32 C:\Windows\System32\oleaut32.dll Thu Dec 06 05:09:35 2007
usp10    C:\Windows\System32\usp10.dll Thu Nov 02 11:15:03 2006
ole32    C:\Windows\System32\ole32.dll Thu Nov 02 11:14:31 2006
advapi32 C:\Windows\System32\advapi32.dll Thu Nov 02 11:11:35 2006
gdi32    C:\Windows\System32\gdi32.dll Thu Feb 21 04:40:51 2008
msvcrt   C:\Windows\System32\msvcrt.dll Thu Nov 02 11:13:37 2006
imm32    C:\Windows\System32\imm32.dll Thu Nov 02 11:13:15 2006
msctf    C:\Windows\System32\msctf.dll Thu Nov 02 11:13:42 2006

This pattern should be checked when we have instances of Module Variety and, especially, Duplicated Module. Note that this pattern can also easily become an anti-pattern when applied to an unknown component: Alien Component.

- Dmitry Vostokov @ DumpAnalysis.org -

In addition to Module Variety there is another DLL Hell pattern that I call Duplicated Module. Here the same module is loaded at least twice and we can detect this when we see the module load address appended to its name in the output of lm commands (this is done to make the name of the module unique):

0:000> lm
start    end        module name
00b20000 0147f000   MSO_b20000
30000000 309a7000   EXCEL
30c90000 31848000   mso
71c20000 71c32000   tsappcmp
745e0000 7489e000   msi  
76290000 762ad000   imm32 
76b70000 76b7b000   psapi            
76f50000 76f63000   secur32            
77380000 77411000   user32
77670000 777a9000   ole32
77ba0000 77bfa000   msvcrt
77c00000 77c48000   gdi32          
77c50000 77cef000   rpcrt4           
77da0000 77df2000   shlwapi            
77e40000 77f42000   kernel32
77f50000 77feb000   advapi32           
7c800000 7c8c0000   ntdll

Usually his happens when the DLL is loaded from different locations. It can be exactly the same DLL version. The problems usually surface when there are different DLL versions and the new code loads the old version of the DLL and uses it. This may result in interface incompatibility issues and ultimately in application fault like an access violation.

In order to provide a dump to play with I created a small toy program called 2DLLs to model the worst case scenario similar to the one I encountered in a production environment a couple of days ago. The program periodically loads MyDLL module to call one of its functions. Unfortunately in one place it uses hardcoded relative path:

HMODULE hLib = LoadLibrary(L".\\DLL\\MyDLL.dll");

and in another place it relies on DLL search order:

hLib = LoadLibrary(L".\\MyDLL.dll");

PATH variable directories are used for search if this DLL was not found in other locations specified by DLL search order. We see that the problem can happen when another application is installed which uses the old version of that DLL and modifies PATH variable to point to its location. To model interface incompatibility I compiled the version of MyDLL that causes NULL pointer access violation when the same function is called from it. The DLL was placed into a separate folder and the PATH variable was modified to reference that folder:

C:\>set PATH=C:\OLD;%PATH%

The application crashes and the installed default postmortem debugger (CDB) saves its crash dump. If we open it we would see that it crashed in MyDLL_1e60000 module which should trigger suspicion:

0:000> r
rax=0000000001e61010 rbx=0000000000000000 rcx=0000775dcac00000
rdx=0000000000000000 rsi=0000000000000006 rdi=0000000000001770
rip=0000000001e61010 rsp=000000000012fed8 rbp=0000000000000000
 r8=0000000000000000  r9=000000000012fd58 r10=0000000000000001
r11=000000000012fcc0 r12=0000000000000000 r13=0000000000000002
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010200
MyDLL_1e60000!fnMyDLL:
00000000`01e61010 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012fed8 00000001`40001093 MyDLL_1e60000!fnMyDLL
00000000`0012fee0 00000001`40001344 2DLLs+0×1093
00000000`0012ff10 00000000`773acdcd 2DLLs+0×1344
00000000`0012ff60 00000000`774fc6e1 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Looking at the list of modules we see two versions of MyDLL loaded from two different folders:

0:000> lm
start             end                 module name
00000000`01e60000 00000000`01e71000   MyDLL_1e60000
00000000`772a0000 00000000`7736a000   user32
00000000`77370000 00000000`774a1000   kernel32
00000000`774b0000 00000000`7762a000   ntdll
00000001`40000000 00000001`40010000   2DLLs
00000001`80000000 00000001`80011000   MyDLL
000007fe`fc9e0000 000007fe`fca32000   uxtheme
000007fe`fe870000 000007fe`fe9a9000   rpcrt4
000007fe`fe9b0000 000007fe`fe9bc000   lpk
000007fe`fea10000 000007fe`feae8000   oleaut32
000007fe`fecd0000 000007fe`fed6a000   usp10
000007fe`fedd0000 000007fe`fefb0000   ole32
000007fe`fefb0000 000007fe`ff0af000   advapi32
000007fe`ff0d0000 000007fe`ff131000   gdi32
000007fe`ff2e0000 000007fe`ff381000   msvcrt
000007fe`ff390000 000007fe`ff3b8000   imm32
000007fe`ff4b0000 000007fe`ff5b4000   msctf

0:000> lmv m MyDLL_1e60000
start             end                 module name
00000000`01e60000 00000000`01e71000   MyDLL_1e60000
    Loaded symbol image file: MyDLL.dll
    Image path: C:\OLD\MyDLL.dll
    Image name: MyDLL.dll
    Timestamp:        Wed Jun 18 14:49:13 2008 (48591259)
[…]

0:000> lmv m MyDLL
start             end                 module name
00000001`80000000 00000001`80011000   MyDLL
    Image path: C:\2DLLs\DLL\MyDLL.dll
    Image name: MyDLL.dll
    Timestamp:        Wed Jun 18 14:50:56 2008 (485912C0)
[...]

We can also see that the old version of MyDLL was the last loaded DLL:

0:000> !dlls -l

0x002c2680: C:\2DLLs\2DLLs.exe
      Base   0x140000000  EntryPoint  0x1400013b0  Size        0x00010000
      Flags  0x00004000  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_ENTRY_PROCESSED

[...]

0x002ea9b0: C:\2DLLs\DLL\MyDLL.dll
      Base   0x180000000  EntryPoint  0x1800013d0  Size        0x00011000
      Flags  0x00084004  LoadCount   0x00000001  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_PROCESS_ATTACH_CALLED

[...]

0x002ec430: C:\OLD\MyDLL.dll
      Base   0×01e60000  EntryPoint  0×01e613e0  Size        0×00011000
      Flags  0×00284004  LoadCount   0×00000001  TlsIndex    0×00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_PROCESS_ATTACH_CALLED
             LDRP_IMAGE_NOT_AT_BASE

We can also see that the PATH variable points to its location and this might explain why it was loaded:

0:000> !peb
PEB at 000007fffffd6000
[...]
Path=C:\OLD;C:\Windows\system32;C:\Windows;[…]
[…]

We might think that the module having address in its name was loaded the last but this is not true. If we save another copy of the dump from the existing one using .dump command and load the new dump file we would see that order of the module names is reversed:

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012fed8 00000001`40001093 MyDLL!fnMyDLL
00000000`0012fee0 00000001`40001344 2DLLs+0×1093
00000000`0012ff10 00000000`773acdcd 2DLLs+0×1344
00000000`0012ff60 00000000`774fc6e1 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> lm
start             end                 module name
00000000`01e60000 00000000`01e71000   MyDLL
00000000`772a0000 00000000`7736a000   user32
00000000`77370000 00000000`774a1000   kernel32
00000000`774b0000 00000000`7762a000   ntdll
00000001`40000000 00000001`40010000   2DLLs
00000001`80000000 00000001`80011000   MyDLL_180000000
000007fe`fc9e0000 000007fe`fca32000   uxtheme
000007fe`fe870000 000007fe`fe9a9000   rpcrt4
000007fe`fe9b0000 000007fe`fe9bc000   lpk
000007fe`fea10000 000007fe`feae8000   oleaut32
000007fe`fecd0000 000007fe`fed6a000   usp10
000007fe`fedd0000 000007fe`fefb0000   ole32
000007fe`fefb0000 000007fe`ff0af000   advapi32
000007fe`ff0d0000 000007fe`ff131000   gdi32
000007fe`ff2e0000 000007fe`ff381000   msvcrt
000007fe`ff390000 000007fe`ff3b8000   imm32
000007fe`ff4b0000 000007fe`ff5b4000   msctf

0:000> !dlls -l

[...]

0x002ec430: C:\OLD\MyDLL.dll
      Base   0×01e60000  EntryPoint  0×01e613e0  Size        0×00011000
      Flags  0×00284004  LoadCount   0×00000001  TlsIndex    0×00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_PROCESS_ATTACH_CALLED
             LDRP_IMAGE_NOT_AT_BASE

The postprocessed dump file can be downloaded from FTP to play with:

ftp://dumpanalysis.org/pub/CDAPatternDuplicatedModule.zip 

- Dmitry Vostokov @ DumpAnalysis.org -

Analysis of computer memory snapshots (memory dumps) and their evolution is the domain of memoretics. Computer memory semiotics (memiotics or memosemiotics) is the branch of memoretics that studies the interpretation of computer memory, its meaning, signs and symbols.

- Dmitry Vostokov @ DumpAnalysis.org -     

Previously I introduced Missing Component pattern where the example and emphasis was on dynamically loaded modules. In this part I cover statically linked modules. Failure for a loader to find one of them results in a software exception. The most frequent of them are (numbers were taken from Google search):

C0000142 918
C0000143 919
C0000145 1,530
C0000135 24,900

0:001> !error c0000142
Error code: (NTSTATUS) 0xc0000142 (3221225794) - {DLL Initialization Failed}  Initialization of the dynamic link library %hs failed. The process is terminating abnormally.

0:001> !error c0000143
Error code: (NTSTATUS) 0xc0000143 (3221225795) - {Missing System File}  The required system file %hs is bad or missing.

0:001> !error c0000145
Error code: (NTSTATUS) 0xc0000145 (3221225797) - {Application Error}  The application failed to initialize properly (0x%lx). Click on OK to terminate the application.

0:000> !error c0000135
Error code: (NTSTATUS) 0xc0000135 (3221225781) - {Unable To Locate Component}  This application has failed to start because %hs was not found. Re-installing the application may fix this problem.

In this part I only consider user mode exceptions. If we have a default debugger configured it will usually save a crash dump. To model this problem I modified one of my applications by changing all occurrences of KERNEL32.DLL to  KERNEL32.DL using Visual Studio Binary Editor. CDB was configured as a default postmortem debugger (see Custom postmortem debuggers on Vista). When the application was launched CDB attached to it and saved a crash dump. If we open it in WinDbg we get characteristic Special Stack Trace involving loader functions:

Loading Dump File [C:\UserDumps\CDAPatternMissingComponent.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Vista Version 6000 MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Debug session time: Thu Jun 12 12:03:28.000 2008 (GMT+1)
System Uptime: 1 days 8:46:23.167
Process Uptime: 0 days 0:00:48.000

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(da4.f60): Wake debugger - code 80000007 (first/second chance not available)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!_LdrpInitialize+0x6d:
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

0:000> kL
ChildEBP RetAddr 
0017fd00 77b937ea ntdll!_LdrpInitialize+0×6d
0017fd10 00000000 ntdll!LdrInitializeThunk+0×10

Verbose analysis command doesn’t give us an indication of what had happened so we need to dig further:

0:000> !analyze -v
[...]

FAULTING_IP:
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000007 (Wake debugger)
  ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR:  80000007

PROCESS_NAME:  StackOverflow.exe

ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened}  the system debugger was awakened by an interrupt.

NTGLOBALFLAG:  400

APPLICATION_VERIFIER_FLAGS:  0

DERIVED_WAIT_CHAIN: 

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   0   da4.f60 Unknown               

WAIT_CHAIN_COMMAND:  ~0s;k;;

BLOCKING_THREAD:  00000f60

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BusyHang

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BusyHang

LAST_CONTROL_TRANSFER:  from 77b937ea to 77bcf1d1

FAULTING_THREAD:  00000000

STACK_TEXT: 
0017fd00 77b937ea 0017fd24 77b60000 00000000 ntdll!_LdrpInitialize+0x6d
0017fd10 00000000 0017fd24 77b60000 00000000 ntdll!LdrInitializeThunk+0x10

FOLLOWUP_IP:
ntdll!_LdrpInitialize+6d
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!_LdrpInitialize+6d

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4549bdf8

STACK_COMMAND:  ~0s ; kb

BUCKET_ID:  80000007_ntdll!_LdrpInitialize+6d

FAILURE_BUCKET_ID:  ntdll.dll!_LdrpInitialize_80000007_APPLICATION_HANG_BusyHang

Followup: MachineOwner

Last event and error code are not helpful too: 

0:000> .lastevent
Last event: da4.f60: Wake debugger - code 80000007 (first/second chance not available)
  debugger time: Thu Jun 12 15:04:38.917 2008 (GMT+1)

0:000> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0

Let’s search thread raw stack data for any signs of exceptions:

0:000> !teb
TEB at 7efdd000
    ExceptionList:        0017fcf0
    StackBase:            00180000
    StackLimit:           0017e000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efdd000
    EnvironmentPointer:   00000000
    ClientId:             00000da4 . 00000f60
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 0017e000 00180000
[...]
0017f8d8  7efdd000
0017f8dc  0017f964
0017f8e0  77c11c78 ntdll!_except_handler4
0017f8e4  00000000
0017f8e8  0017f988
0017f8ec  0017f900
0017f8f0  77ba1ddd ntdll!RtlCallVectoredContinueHandlers+0x15
0017f8f4  0017f988
0017f8f8  0017f9d8
0017f8fc  77c40370 ntdll!RtlpCallbackEntryList
0017f900  0017f970
0017f904  77ba1db5 ntdll!RtlDispatchException+0×11f
0017f908  0017f988
0017f90c  0017f9d8
0017f910  7efde028
0017f914  00000001
0017f918  77630000 kernel32!_imp___aullrem <PERF> (kernel32+0×0)
0017f91c  00000001
0017f920  776ced81 kernel32!_DllMainCRTStartupForGS2+0×10
0017f924  0017f938
0017f928  7765d4d9 kernel32!BaseDllInitialize+0×18
0017f92c  76042340 user32!$$VProc_ImageExportDirectory
0017f930  00000001
0017f934  00000000
0017f938  0017f9e0
0017f93c  77b8f890 ntdll!LdrpSnapThunk+0xc9
0017f940  0040977a StackOverflow+0×977a
0017f944  0000030b
0017f948  76030000 user32!_imp__RegSetValueExW <PERF> (user32+0×0)
0017f94c  76042f94 user32!$$VProc_ImageExportDirectory+0xc54
0017f950  77bb8881 ntdll!LdrpSnapThunk+0×40d
0017f954  0017bb30
0017f958  00409770 StackOverflow+0×9770
0017f95c  00881a50
0017f960  004098b2 StackOverflow+0×98b2
0017f964  77bac282 ntdll!ZwRaiseException+0×12
0017f968  00180000
0017f96c  0017fc48
0017f970  0017fd00
0017f974  77bac282 ntdll!ZwRaiseException+0×12
0017f978  77b7ee72 ntdll!KiUserExceptionDispatcher+0×2a
0017f97c  0017f988 ; exception record
0017f980  0017f9d8 ; exception context
0017f984  00000000
0017f988  c0000135
0017f98c  00000001
0017f990  00000000
0017f994  77bcf1d1 ntdll!_LdrpInitialize+0×6d
0017f998  00000000
0017f99c  77c11c78 ntdll!_except_handler4
0017f9a0  77b8dab8 ntdll!RtlpRunTable+0×218
0017f9a4  fffffffe
0017f9a8  77ba2515 ntdll!vDbgPrintExWithPrefixInternal+0×214
0017f9ac  77ba253b ntdll!DbgPrintEx+0×1e
0017f9b0  77b7f356 ntdll! ?? ::FNODOBFM::`string’
0017f9b4  00000055
0017f9b8  00000003
0017f9bc  77b809c2 ntdll! ?? ::FNODOBFM::`string’
0017f9c0  0017fc9c
0017f9c4  00000001
0017f9c8  0017fd00
0017f9cc  77bcf28e ntdll!_LdrpInitialize+0×12a
0017f9d0  00000055
0017f9d4  75ce3cf9
0017f9d8  0001003f
0017f9dc  00000000
0017f9e0  00000000
0017f9e4  00000000
0017f9e8  00000000
0017f9ec  00000000
0017f9f0  00000000
0017f9f4  0000027f
[…]

We see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get one:

0:000> .exr 0017f988
ExceptionAddress: 77bcf1d1 (ntdll!_LdrpInitialize+0x0000006d)
   ExceptionCode: c0000135
  ExceptionFlags: 00000001
NumberParameters: 0

Error c0000135 means that the loader was unable to locate a component. Now we try to examine the same raw stack data for any string patterns. For example, the following UNICODE pattern is clearly visible:

0017f2fc  00000000
0017f300  00880ec4
0017f304  77b910d7 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x344
0017f308  00000000
0017f30c  43000043
0017f310  0042002a
0017f314  0017f33c
0017f318  00000000
0017f31c  00000002
0017f320  00000008
0017f324  00000000
0017f328  0000008c
0017f32c  000a0008
0017f330  77b91670 ntdll!`string'
0017f334  00b92bd6
0017f338  0017f5d4
0017f33c  003a0043
0017f340  0050005c
0017f344  006f0072
0017f348  00720067
0017f34c  006d0061
0017f350  00460020
0017f354  006c0069
0017f358  00730065
0017f35c  00280020
0017f360  00380078
0017f364  00290036
0017f368  0043005c
0017f36c  006d006f
0017f370  006f006d
0017f374  0020006e
0017f378  00690046
0017f37c  0065006c
0017f380  005c0073
0017f384  006f0052
0017f388  00690078
0017f38c  0020006f
0017f390  00680053
0017f394  00720061
0017f398  00640065
0017f39c  0044005c
0017f3a0  004c004c
0017f3a4  00680053
0017f3a8  00720061
0017f3ac  00640065
0017f3b0  004b005c
0017f3b4  00520045
0017f3b8  0045004e
0017f3bc  0033004c
0017f3c0  002e0032
0017f3c4  006c0064
0017f3c8  00000000
0017f3cc  00000000

It is a path to DLL that was probably missing:

0:000> du 0017f33c
0017f33c  "C:\Program Files (x86)\Common Fi"
0017f37c  "les\Roxio Shared\DLLShared\KERNE"
0017f3bc  "L32.dl"

I think the loader was trying to find KERNEL32.dl following the DLL search order and this was the last path element:

0:000> !peb
PEB at 7efde000
    InheritedAddressSpace:    No
    ReadImageFileExecOptions: No
    BeingDebugged:            Yes
    ImageBaseAddress:         00400000
    Ldr                       77c40080
    Ldr.Initialized:          Yes
    Ldr.InInitializationOrderModuleList: 00881ad0 . 008831b8
    Ldr.InLoadOrderModuleList:           00881a50 . 00883dc8
    Ldr.InMemoryOrderModuleList:         00881a58 . 00883dd0
            Base TimeStamp                     Module
[...]
    Environment:  00881de8
[...] 
Path=C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
[…]

In  similar situations !dlls command might help that shows the load order (-l switch) and points to the last processed DLL:

0:001> !dlls -l

0x004740e8: C:\Program Files\Application\Application.exe
      Base   0x012a0000  EntryPoint  0x012b0903  Size        0x00057000
      Flags  0x00004010  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_ENTRY_PROCESSED

0x00474158: C:\Windows\SysWOW64\ntdll.dll
      Base   0x77d00000  EntryPoint  0x00000000  Size        0x00160000
      Flags  0x00004014  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED

0x00474440: C:\Windows\syswow64\kernel32.dll
      Base   0x77590000  EntryPoint  0x775a1f3e  Size        0x00110000
      Flags  0x00084014  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_PROCESS_ATTACH_CALLED

[…] 

0x00498ff8: C:\Windows\WinSxS\x86_microsoft.windows.common-controls_...\comctl32.dll
      Base   0x74d90000  EntryPoint  0x74dc43e5  Size        0x0019e000
      Flags  0x100c4014  LoadCount   0x00000003  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_DONT_CALL_FOR_THREADS
             LDRP_PROCESS_ATTACH_CALLED

0x004991b8: C:\Windows\WinSxS\x86_microsoft.vc80.mfcloc_...\MFC80ENU.DLL
      Base   0x71b10000  EntryPoint  0x00000000  Size        0x0000e000
      Flags  0x10004014  LoadCount   0x00000001  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED

If it is difficult to identify what had really happened in crash dumps we can enable loader snaps using gflags and run the application under a debugger. For example, for notepad.exe we have:

Microsoft (R) Windows Debugger Version 6.8.0004.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Windows\notepad.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00000000`ffac0000 00000000`ffaef000   notepad.exe
ModLoad: 00000000`779b0000 00000000`77b2a000   ntdll.dll
LDR: NEW PROCESS
     Image Path: C:\Windows\notepad.exe (notepad.exe)
     Current Directory: C:\Program Files\Debugging Tools for Windows 64-bit\
     Search Path: C:\Windows; C:\Windows\system32; C:\Windows\system; C:\Windows;.; C:\Program Files\Debugging Tools for Windows 64-bit\winext\arcade; C:\Windows\system32;C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
LDR: LdrLoadDll, loading kernel32.dll from
ModLoad: 00000000`777a0000 00000000`778d1000   C:\Windows\system32\kernel32.dll
LDR: kernel32.dll bound to ntdll.dll
LDR: kernel32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from kernel32.dll
LDR: LdrGetProcedureAddress by NAME - BaseThreadInitThunk
[3d8,1278] LDR: Real INIT LIST for process C:\Windows\notepad.exe pid 984 0x3d8
[3d8,1278]    C:\Windows\system32\kernel32.dll init routine 00000000777DC960
[3d8,1278] LDR: kernel32.dll loaded - Calling init routine at 00000000777DC960
LDR: notepad.exe bound to ADVAPI32.dll
ModLoad: 000007fe`fe520000 000007fe`fe61f000   C:\Windows\system32\ADVAPI32.dll
LDR: ADVAPI32.dll bound to ntdll.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from ADVAPI32.dll
LDR: ADVAPI32.dll bound to KERNEL32.dll
LDR: ADVAPI32.dll has stale binding to KERNEL32.dll
LDR: ADVAPI32.dll bound to ntdll.dll via forwarder(s) from kernel32.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind KERNEL32.dll from ADVAPI32.dll
LDR: LdrGetProcedureAddress by NAME - RtlAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlReAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlEncodePointer
LDR: LdrGetProcedureAddress by NAME - RtlDecodePointer
LDR: LdrGetProcedureAddress by NAME - RtlSizeHeap
LDR: LdrGetProcedureAddress by NAME - RtlDeleteCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlEnterCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlLeaveCriticalSection
LDR: ADVAPI32.dll bound to RPCRT4.dll
[...]

This technique only works for native platform loader snaps. For example, it doesn’t show loader snaps for 32-bit modules loaded under WOW64 (to the best of my knowledge):

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Apps\StackOverflow.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00418000   StackOverflow.exe
ModLoad: 77b60000 77cb0000   ntdll.dll
LDR: NEW PROCESS
     Image Path: C:\Apps\StackOverflow.exe (StackOverflow.exe)
[...]
LDR: Loading (STATIC, NON_REDIRECTED) C:\Windows\system32\wow64cpu.dll
LDR: wow64cpu.dll bound to ntdll.dll
LDR: wow64cpu.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from wow64cpu.dll
LDR: wow64cpu.dll bound to wow64.dll
LDR: wow64cpu.dll has stale binding to wow64.dll
LDR: Stale Bind wow64.dll from wow64cpu.dll
LDR: wow64.dll has stale binding to wow64cpu.dll
LDR: Stale Bind wow64cpu.dll from wow64.dll
LDR: Refcount wow64cpu.dll (1)
LDR: Refcount wow64.dll (2)
LDR: Refcount wow64win.dll (1)
LDR: Refcount wow64.dll (3)
LDR: LdrGetProcedureAddress by NAME - Wow64LdrpInitialize
[...]
ModLoad: 77630000 77740000   C:\Windows\syswow64\kernel32.dll
ModLoad: 76030000 76100000   C:\Windows\syswow64\USER32.dll
ModLoad: 775a0000 77630000   C:\Windows\syswow64\GDI32.dll
ModLoad: 76d00000 76dbf000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 76df0000 76ee0000   C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75d60000 75dc0000   C:\Windows\syswow64\Secur32.dll
(1ec.1290): Unknown exception - code c0000135 (first chance)
(1ec.1290): Unknown exception - code c0000135 (!!! second chance !!!)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!_LdrpInitialize+0×6d:
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

The dump file that I used was modified to remove sensitive information (see Data Hiding in Crash Dumps for this technique). It can be downloaded from FTP to play with:

ftp://dumpanalysis.org/pub/CDAPatternMissingComponent.zip

- Dmitry Vostokov @ DumpAnalysis.org -

I’ve just found that although I covered Stack Overflow in kernel mode I didn’t do this for user mode. In fact this is one of the simplest patterns to see in crash dumps. It has its own characteristic exception code and stack trace:

FAULTING_IP:
StackOverflow!SoFunction+27
00401317 6a00            push    0

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00401300 (StackOverflow!SoFunction+0x00000010)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00082ffc

0:000> kL
ChildEBP RetAddr 
00083000 00401317 StackOverflow!SoFunction+0x10
00083010 00401317 StackOverflow!SoFunction+0×27
00083020 00401317 StackOverflow!SoFunction+0×27
00083030 00401317 StackOverflow!SoFunction+0×27
00083040 00401317 StackOverflow!SoFunction+0×27
00083050 00401317 StackOverflow!SoFunction+0×27
00083060 00401317 StackOverflow!SoFunction+0×27
00083070 00401317 StackOverflow!SoFunction+0×27
00083080 00401317 StackOverflow!SoFunction+0×27
00083090 00401317 StackOverflow!SoFunction+0×27
000830a0 00401317 StackOverflow!SoFunction+0×27
000830b0 00401317 StackOverflow!SoFunction+0×27
000830c0 00401317 StackOverflow!SoFunction+0×27
000830d0 00401317 StackOverflow!SoFunction+0×27
000830e0 00401317 StackOverflow!SoFunction+0×27
000830f0 00401317 StackOverflow!SoFunction+0×27
00083100 00401317 StackOverflow!SoFunction+0×27
00083110 00401317 StackOverflow!SoFunction+0×27
00083120 00401317 StackOverflow!SoFunction+0×27
00083130 00401317 StackOverflow!SoFunction+0×27

There could be thousands of stack frames:

0:000> kL 2000
[...]
000a2fa0 00401317 StackOverflow!SoFunction+0x27
000a2fb0 00401317 StackOverflow!SoFunction+0x27
000a2fc0 00401317 StackOverflow!SoFunction+0x27
000a2fd0 00401317 StackOverflow!SoFunction+0x27
000a2fe0 00401317 StackOverflow!SoFunction+0x27
000a2ff0 00401317 StackOverflow!SoFunction+0x27

To reach the bottom and avoid over scrolling we can dump the raw stack data, search for the end of the repeating pattern of StackOverflow!SoFunction+0×27 and try to manually reconstruct the bottom of the stack trace:

0:000> !teb
TEB at 7efdd000
    ExceptionList:        0017fdf0
    StackBase:            00180000
    StackLimit:           00081000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efdd000
    EnvironmentPointer:   00000000
    ClientId:             00001dc4 . 00001b74
    RpcHandle:            00000000
    Tls Storage:          7efdd02c
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00081000 00180000
[...]
0017fc74  00401317 StackOverflow!SoFunction+0×27
0017fc78  00000000
0017fc7c  a3a8ea65
0017fc80  0017fc90
0017fc84  00401317 StackOverflow!SoFunction+0×27
0017fc88  10001843
0017fc8c  a3a8ea95
0017fc90  0017fca0
0017fc94  00401317 StackOverflow!SoFunction+0×27
0017fc98  0017fcb8
0017fc9c  a3a8ea85
0017fca0  0017fcb0
0017fca4  00401317 StackOverflow!SoFunction+0×27
0017fca8  00000003
0017fcac  a3a8eab5
0017fcb0  0017fcc0
0017fcb4  00401317 StackOverflow!SoFunction+0×27
0017fcb8  76c68738 user32!_EndUserApiHook+0×11
0017fcbc  a3a8eaa5
0017fcc0  0017fcd0
0017fcc4  00401317 StackOverflow!SoFunction+0×27
0017fcc8  76c6a6cc user32!DefWindowProcW+0×94
0017fccc  a3a8ead5
0017fcd0  0017fce0
0017fcd4  00401317 StackOverflow!SoFunction+0×27
0017fcd8  0037311e
0017fcdc  a3a8eac5
0017fce0  0017fcf0
0017fce4  00401317 StackOverflow!SoFunction+0×27
0017fce8  0017fcd0
0017fcec  a3a8eaf5
0017fcf0  0017fd00
0017fcf4  00401317 StackOverflow!SoFunction+0×27
0017fcf8  76c6ad0f user32!NtUserBeginPaint+0×15
0017fcfc  a3a8eae5
0017fd00  0017fd5c
0017fd04  00401272 StackOverflow!WndProc+0xe2
0017fd08  00401190 StackOverflow!WndProc
0017fd0c  00000003
0017fd10  cf017ada
[…]

We use the extended version of k WinDbg command and supply EBP, ESP and EIP to see in what function it started:

0:000> r
eax=a3b739e5 ebx=00000000 ecx=ac430000 edx=ffefd944 esi=0037311e edi=00000000
eip=00401300 esp=00082ff8 ebp=00083000 iopl=0         nv up ei ng nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
StackOverflow!SoFunction+0×10:
00401300 89442404        mov     dword ptr [esp+4],eax ss:002b:00082ffc=00000000

0:000> k L=0017fcf0 00082ff8 00401300
ChildEBP RetAddr 
0017fcb0 00401317 StackOverflow!SoFunction+0×10
0017fd00 00401272 StackOverflow!SoFunction+0×27
0017fd5c 76c687af StackOverflow!WndProc+0xe2
0017fd88 76c68936 user32!InternalCallWinProc+0×23
0017fe00 76c6a571 user32!UserCallWinProcCheckWow+0×109
0017fe5c 76c6a5dd user32!DispatchClientMessage+0xe0
0017fe98 77ccee2e user32!__fnDWORD+0×2b
0017fedc 0040107d ntdll!KiUserCallbackDispatcher+0×2e
0017ff08 0040151e StackOverflow!wWinMain+0×7d
00402ba0 20245c8b StackOverflow!__tmainCRTStartup+0×176

- Dmitry Vostokov @ DumpAnalysis.org -

Another pattern that we need to pay attention is called Thread Starvation. This happens when some threads or processes have higher priority and favored by OS thread dispatcher effectively starving other threads. If prioritized threads are CPU-bound we also see Spiking Thread pattern. However, if their thread priorities were normal they would have been preempted by other threads and latter threads would not be starved. Here is one example where 2 threads from 2 different applications but from one user session are spiking on 2 processors (threads from other processors have above normal and normal priority):

System Uptime: 0 days 6:40:41.143

0: kd> !running

System Processors f (affinity mask)
  Idle Processors None

     Prcb      Current   Next    
  0  f773a120  89864c86 ................
  1  f773d120  89f243d2 ................
  2  f7737120  89f61398 ................
  3  f773f120  897228a0 ................

0: kd> .thread /r /p 89f61398
Implicit thread is now 89f61398
Implicit process is now 88bcc2e0
Loading User Symbols

0: kd> !thread 89f61398 1f
THREAD 89f61398  Cid 16f8.2058  Teb: 7ffdf000 Win32Thread: bc41aea8 RUNNING on processor 2
Not impersonating
DeviceMap                 e48a6508
Owning Process            88bcc2e0       Image:         application.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      7201654                 LargeStack
UserTime                  01:24:06.687
KernelTime                00:14:53.828
Win32 Start Address application (0×0040a52c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba336000 Current ba335d00 Base ba336000 Limit ba330000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012e09c 762c3b7d USER32!IsWindowVisible
0012e0c4 762d61bb MSVBVM50!RbyCountVisibleDesks+0×3c
0012e0d0 004831f6 MSVBVM50!rtcDoEvents+0×7
0012e348 0046d1ae application+0×831f6
0012e3a0 762ce5a9 application+0×6d1ae
0012e3dc 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012e3f8 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012e434 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012e514 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012e55c 762b1aa3 MSVBVM50!EvtErrFire+0×18
0012e5ac 7739bffa MSVBVM50!CThreadPool::GetThreadData+0xf
0012e58c 762cd13b USER32!CallHookWithSEH+0×21
0012e5ac 7739bffa MSVBVM50!VBDefControlProc_2787+0xad
0012e618 762d3348 USER32!CallHookWithSEH+0×21
0012e640 762cda44 MSVBVM50!PushCtlProc+0×2e
0012e668 762cd564 MSVBVM50!CommonGizWndProc+0×4e
0012e6b8 7739b6e3 MSVBVM50!StdCtlWndProc+0×171
0012e6e4 7739b874 USER32!InternalCallWinProc+0×28
0012e75c 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012e7c4 773a16e5 USER32!DispatchMessageWorker+0×327
0012e7d4 762d616e USER32!DispatchMessageA+0xf
0012e828 762d6054 MSVBVM50!ThunderMsgLoop+0×97
0012e874 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012e8b4 004831f6 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012e8c0 00d3b3c8 application+0×831f6
00184110 00000000 0xd3b3c8

0: kd> .thread /r /p 897228a0
Implicit thread is now 897228a0
Implicit process is now 897348a8
Loading User Symbols

0: kd> !thread 897228a0 1f 100
THREAD 897228a0  Cid 2984.2988  Teb: 7ffdf000 Win32Thread: bc381488 RUNNING on processor 3
IRP List:
    89794bb8: (0006,0220) Flags: 00000000  Mdl: 8a145878
Not impersonating
DeviceMap                 e3ec0360
Owning Process            897348a8       Image:         application2.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      10239625                 LargeStack
UserTime                  02:38:18.890
KernelTime                00:29:36.187
Win32 Start Address application2 (0×00442e4c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init f1d90000 Current f1d8fd00 Base f1d90000 Limit f1d88000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012f66c 762d61bb USER32!_SEH_prolog+0×5
0012f678 00fdb0b9 MSVBVM50!rtcDoEvents+0×7
0012f92c 00fca760 application2+0xbdb0b9
0012fa20 762ce5a9 application2+0xbca760
0012fa40 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012fa5c 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012fa98 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012fb78 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012fb90 76330b2b MSVBVM50!EvtErrFire+0×18
0012fbf0 762cd13b MSVBVM50!ErrDefMouse_100+0×16d
0012fca4 762cda44 MSVBVM50!VBDefControlProc_2787+0xad
0012fccc 7631c826 MSVBVM50!CommonGizWndProc+0×4e
0012fd08 762cd523 MSVBVM50!StdCtlPreFilter_50+0×9e
0012fd5c 7739b6e3 MSVBVM50!StdCtlWndProc+0×130
0012fd88 7739b874 USER32!InternalCallWinProc+0×28
0012fe00 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012fe68 773a16e5 USER32!DispatchMessageWorker+0×327
0012fe78 762d616e USER32!DispatchMessageA+0xf
0012fea8 762bb78f MSVBVM50!ThunderMsgLoop+0×97
0012feb8 762d60cb MSVBVM50!MsoFInitPx+0×39
0012fecc 762d6054 MSVBVM50!CMsoCMHandler::FPushMessageLoop+0×1a
0012ff18 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012fef8 762d5f8e nt!KiDeliverApc+0×11f
0012ff18 762d5f55 MSVBVM50!SCM_MsoCompMgr::FPushMessageLoop+0×2f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ff18 762d5f55 nt!KiDeliverApc+0×11f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ffd4 0012ffc8 nt!KiDeliverApc+0×11f
00000000 00000000 0×12ffc8

What we see here is unusually high Priority and BasePriority values (24 and 24). This means that the base priority for these processes was most likely artificially increased to Realtime. Most processes have base priority 8 (Normal):

0: kd> !thread 88780db0 1f
THREAD 88780db0  Cid 44a8.1b8c  Teb: 7ffaf000 Win32Thread: bc315d20 WAIT: (Unknown) UserMode Non-Alertable
    887b8650  Semaphore Limit 0x7fffffff
    88780e28  NotificationTimer
Not impersonating
DeviceMap                 e1085298
Owning Process            889263a0       Image:         explorer.exe
Wait Start TickCount      1565543        Ticks: 4194 (0:00:01:05.531)
Context Switch Count      7                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6754000 Current b6753c0c Base b6754000 Limit b6750000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
b6753c24 80833e8d nt!KiSwapContext+0×26
b6753c50 80829b74 nt!KiSwapThread+0×2e5
b6753c98 809249cd nt!KeWaitForSingleObject+0×346
b6753d48 8088ac4c nt!NtReplyWaitReceivePortEx+0×521
b6753d48 7c8285ec nt!KiFastCallEntry+0xfc
01a2fe18 7c82783b ntdll!KiFastSystemCallRet
01a2fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
01a2ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×198
01a2ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
01a2ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
01a2ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
01a2ffec 00000000 kernel32!BaseThreadStart+0×34

Some important system processes like csrss.exe have base priority 13 (High) but their threads wait most of the time and this doesn’t create any problems:

0: kd> !thread 887eb3d0 1f
THREAD 887eb3d0  Cid 4cf4.2bd4  Teb: 7ffaf000 Win32Thread: bc141cc0 WAIT: (Unknown) UserMode Non-Alertable
    888769b0  SynchronizationEvent
Not impersonating
DeviceMap                 e1000930
Owning Process            8883f7c0       Image:         csrss.exe
Wait Start TickCount      1540456        Ticks: 29281 (0:00:07:37.515)
Context Switch Count      40                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address winsrv!ConsoleInputThread (0×75a81b18)
Stack Init b5c5a000 Current b5c59bac Base b5c5a000 Limit b5c55000 Call 0
Priority 15 BasePriority 13 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b5c59bc4 80833e8d nt!KiSwapContext+0×26
b5c59bf0 80829b74 nt!KiSwapThread+0×2e5
b5c59c38 bf89b1c3 nt!KeWaitForSingleObject+0×346
b5c59c94 bf89b986 win32k!xxxSleepThread+0×1be
b5c59cec bf89da22 win32k!xxxRealInternalGetMessage+0×46a
b5c59d4c 8088ac4c win32k!NtUserGetMessage+0×3f
b5c59d4c 7c8285ec nt!KiFastCallEntry+0xfc
00ffff64 7739c811 ntdll!KiFastSystemCallRet
00ffff84 75a81c47 USER32!NtUserGetMessage+0xc
00fffff4 00000000 winsrv!ConsoleInputThread+0×16c

However it is very unusual for a process to have Realtime base priority. I can speculate what had really happened before the system crash was forced. The system administrator noticed two applications consuming CPU over the long period of time and decided to intervene. Unfortunately his hand slipped when he was browsing Task Manager Set Priority menu and Realtime was selected instead of Low. Human error…

See also the similar starvation pattern resulted from threads with normal priority.

- Dmitry Vostokov @ DumpAnalysis.org -

Following the introductory division of memory analysis into two broad categories I decided to plan yet another book with the following title and preliminary product details:

• Title: Memory Analysis Forensics and Intelligence: An Integral Approach
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2009)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-05-6
• Paperback: 256 pages

As you might have noticed, I prefer to put 128, 256 or 512 pages in my book announcements. What that would say about the author’s background? I would like to set 1024 pages for my memory dump analysis anthology series but 740 or 800 pages is the limit for POD technology I use.

- Dmitry Vostokov @ DumpAnalysis.org -

Software ageing can be the cause of the problem. Sometimes a look at the following WinDbg output can give irresistible temptation to suggest periodic reboots:

Debug session time: Wed April 28 15:36:52.330 2008 (GMT+0)
System Uptime: 124 days 6:27:16.658

The suggested pattern name is Overaged System.

- Dmitry Vostokov @ DumpAnalysis.org -

Recently being interested in forensic science and intelligence I decided to elaborate on memory analysis classification and came up with the following division:

- Memory Analysis Forensics

Answering questions related to a committed computer crime. The suspect may be a human or a software / hardware component. Incident response, troubleshooting and debugging belong to this category. Postmortem memory analysis is usually analysis of dump files saved and detached from the original system or operating conditions.

- Memory Analysis Intelligence

Monitoring memory state for behavioural and structural patterns to prevent certain events from occurring. Usually done in situ. However digital dumpster divers and spies may also collect and analyze memory data that was detached from the original computer system.

Each category can be further subdivided into:

– Functional Memory Analysis

Tracing of events.

– Memoretics 

Analysis of memory states and their evolution.

The latter can be subdivided into: 

— Static Memory Analysis

Traditional memory dump analysis.

— Dynamic Memory Analysis

Live debugging.

- Dmitry Vostokov @ DumpAnalysis.org -

Invalid Handle exception (0xC0000008) can frequently be seen in crash dumps. It results from an invalid handle value passed to CloseHandle and other Win32 API or when a handle or return status is checked manually for validity and the same exception is raised via RaiseException or internally via RtlRaiseStatus. Therefore it merits its own separate crash dump analysis pattern with the same name.

For example, critical sections are implemented using events and invalid event handle can result in this exception:

STACK_TEXT:
025bff00 7c94243c c0000008 7c9010ed 00231af0 ntdll!RtlRaiseStatus+0×26
025bff80 7c90104b 0015b4ac 77e76a6f 0015b4ac ntdll!RtlpWaitForCriticalSection+0×204
025bff88 77e76a6f 0015b4ac 010d2040 00000000 ntdll!RtlEnterCriticalSection+0×46
025bffa8 77e76c0a 0015b420 025bffec 7c80b683 rpcrt4!BaseCachedThreadRoutine+0xad
025bffb4 7c80b683 001feae8 010d2040 00000000 rpcrt4!ThreadStartRoutine+0×1a
025bffec 00000000 77e76bf0 001feae8 00000000 kernel32!BaseThreadStart+0×37

By default, unless raised manually, this exception doesn’t result in a default postmortem debugger called to save a crash dump. In order to do this we need to run the application under a debugger and save a crash dump upon this exception or use exception monitoring tools that save first-chance exceptions like Debug Diagnostics, ADPlus or Exception Monitor (see Early Crash Dump pattern):

0:002> g
(7b0.d1c): Invalid handle - code c0000008 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400          ret     4

0:000> g
(7b0.d1c): Invalid handle - code c0000008 (!!! second chance !!!)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400          ret     4

In order to catch it using postmortem debuggers we can use Application Verifier and configure its basic checks to include invalid handles. Then we will have crash dumps if a postmortem debugger or WER is properly configured. The typical stack might look like this and pointing straight to the problem component:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 6b006369
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 00000000

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT

0:000> kL
ChildEBP RetAddr 
0301ff44 0489a480 ntdll!NtClose+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0301ff54 7d4d8e4f vfbasics+0xa480
0301ff60 04894df9 kernel32!CloseHandle+0×59
0301ff70 00401022 vfbasics+0×4df9
0301ffc0 7d4e7d2a BadHandle+0×1022
0301fff0 00000000 kernel32!BaseProcessStart+0×28

or like this:

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012ed58 00000000`01f9395a ntdll!DbgBreakPoint
00000000`0012ed60 00000000`023e29a7 vrfcore!VerifierStopMessageEx+0×846
00000000`0012f090 00000000`023d9384 vfbasics+0×129a7
00000000`0012f0f0 00000000`77f251ec vfbasics+0×9384
00000000`0012f180 00000000`77ee5f36 ntdll!RtlpCallVectoredHandlers+0×26f
00000000`0012f210 00000000`77ee6812 ntdll!RtlDispatchException+0×46
00000000`0012f8c0 00000000`77ef325a ntdll!RtlRaiseException+0xae
00000000`0012fe00 00000000`77d6e314 ntdll!KiRaiseUserExceptionDispatcher+0×3a
00000000`0012fed0 00000001`40001028 kernel32!CloseHandle+0×5f
00000000`0012ff00 00000001`40001294 BadHandle+0×1028
00000000`0012ff30 00000000`77d5964c BadHandle+0×1294
00000000`0012ff80 00000000`00000000 kernel32!BaseProcessStart+0×29

vfbasics and vrfcore are Application Verifier DLLs that might translate an invalid handle exception to a breakpoint exception and therefore trigger the launch of a postmortem debugger from an unhandled exception filter. Application Verifier version (x64 or x86) must match the application platform (64-bit or 32-bit).

If invalid handle exception is raised manually we get the status code and possibly problem component immediately from !analyze command:

FAULTING_IP:
kernel32!RaiseException+53
7d4e2366 5e              pop     esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7d4e2366 (kernel32!RaiseException+0x00000053)
   ExceptionCode: c0000008 (Invalid handle)
  ExceptionFlags: 00000000
NumberParameters: 0
Thread tried to close a handle that was invalid or illegal to close

DEFAULT_BUCKET_ID:  STATUS_INVALID_HANDLE

PROCESS_NAME:  BadHandle.exe

ERROR_CODE: (NTSTATUS) 0xc0000008 - An invalid HANDLE was specified.

STACK_TEXT: 
0012ff64 00401043 c0000008 00000000 00000000 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 BadHandle+0×1043
0012fff0 00000000 004012f9 00000000 00000000 kernel32!BaseProcessStart+0×28

FAULTING_THREAD:  00000b64

PRIMARY_PROBLEM_CLASS:  STATUS_INVALID_HANDLE

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_INVALID_HANDLE

Because we have WinDbg warning about stack unwind we can double check the disassembly of RaiseException return address:

0:000> ub 00401043
BadHandle+0×1029:
00401029 push    offset BadHandle+0×212c (0040212c)
0040102e push    0
00401030 call    esi
00401032 push    0
00401034 push    0
00401036 push    0
00401038 push    0C0000008h
0040103d call    dword ptr [BadHandle+0×2004 (00402004)]

0:000> dps 00402004 l1
00402004  7d4e2318 kernel32!RaiseException

Beware that in such cases the real problem might have been memory corruption overwriting stored valid handle values.

- Dmitry Vostokov @ DumpAnalysis.org -

After Bug Check Frequencies post I was curious enough to do the same Google counting procedure for exceptions. Here are my results for exceptions listed in Visual C++ Debug \ Exceptions dialog:

The corresponding graph:

- Dmitry Vostokov @ DumpAnalysis.org -

Component Age Diagram (CAD) helps to visualize and pinpoint anomalies in component timestamps. Excel helps here. We can import the output of lmt WinDbg command and get these graphs where peaks can be used to identify old modules. For example, here is a CAD from my Windows Vista SP1 running on MacMini:

Here is another CAD from Windows 2000 server where the oldest driver is easily identified:

The following CAD diagram is created from lmt output in Module Variety pattern example:

- Dmitry Vostokov @ DumpAnalysis.org -

Optimized VM Layout is a specialization of the general Changed Environment pattern where the whole modules are moved in virtual memory by changing their load order and load addresses. This can result in dormant bugs being exposed and one of workarounds usually is to disable such external optimization programs or services or adding applications that behave improperly to exclusion lists. Some optimized virtual memory cases can easily be detected by looking at module list where system DLLs are remapped to lower addresses instead of 0×7X000000 range:

0:000> lm
start    end        module name
00400000 00416000   Application
00470000 0050b000   advapi32
00520000 00572000   shlwapi
02340000 023cb000   oleaut32
04b80000 0523e000   System_Data_ni
1a400000 1a524000   urlmon
4dd60000 4df07000   GdiPlus
5f120000 5f12e000   ntlanman
5f860000 5f891000   netui1
5f8a0000 5f8b6000   netui0
637a0000 63d28000   System_Xml_ni
64890000 6498c000   System_Configuration_ni
64e70000 6515c000   System_Data
65ce0000 65ecc000   System_Web_Services_ni
71bd0000 71be1000   mpr           
71bf0000 71bf8000   ws2help
71c00000 71c17000   ws2_32
71c20000 71c32000   tsappcmp
71c40000 71c97000   netapi32
73070000 73097000   winspool
75e90000 75e97000   drprov
75ea0000 75eaa000   davclnt
76190000 761a2000   msasn1
761b0000 76243000   crypt32
76a80000 76a92000   atl
76b80000 76bae000   credui
76dc0000 76de8000   adsldpc
76df0000 76e24000   activeds
76f00000 76f08000   wtsapi32
76f10000 76f3e000   wldap32
771f0000 77201000   winsta
77670000 777a9000   ole32
77ba0000 77bfa000   msvcrt
78130000 781cb000   msvcr80
79000000 79046000   mscoree
79060000 790b6000   mscorjit
790c0000 79bf6000   mscorlib_ni
79e70000 7a3ff000   mscorwks
7a440000 7ac2a000   System_ni
7ade0000 7af7c000   System_Drawing_ni
7afd0000 7bc6c000   System_Windows_Forms_ni
7c340000 7c396000   msvcr71
7c8d0000 7d0ce000   shell32
7d4c0000 7d5f0000   kernel32
7d600000 7d6f0000   ntdll
7d800000 7d890000   gdi32
7d8d0000 7d920000   secur32
7d930000 7da00000   user32
7da20000 7db00000   rpcrt4
7dbd0000 7dcd3000   comctl32
7df50000 7dfc0000   uxtheme
7e020000 7e02f000   samlib

The similar address space reshuffling happens with ASLR-enabled applications with the difference that system modules are never remapped below 0×70000000.

 - Dmitry Vostokov @ DumpAnalysis.org -

In the pattern about NULL code pointer I created a simple program that crashes when we pass a NULL thread procedure pointer to CreateThread function. We might expect to see little in the raw stack data because there was no user-supplied thread code. In reality, if we dump it we would see lots of symbolic information for code and data including ASCII and UNICODE fragments that I call Execution Residue patterns and one of them is Exception Handling Residue we can use to check for hidden exceptions and differentiate between 1st and 2nd chance exceptions. Code residues are very powerful in reconstructing stack traces manually or looking for partial stack traces and historical information.

To show typical execution residues I created the small program with two additionally created threads based on Visual Studio Win32 project. After we dismiss About box we create the first thread and then we crash the process when creating the second thread because of the NULL thread procedure:

typedef DWORD (WINAPI *THREADPROC)(PVOID);

DWORD WINAPI ThreadProc(PVOID pvParam)
{
   for (unsigned int i = 0xFFFFFFFF; i; --i);
   return 0;
}

// Message handler for about box.
INT_PTR CALLBACK About(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
   UNREFERENCED_PARAMETER(lParam);
   switch (message)
   {
   case WM_INITDIALOG:
      return (INT_PTR)TRUE;

   case WM_COMMAND:
      if (LOWORD(wParam) == IDOK || LOWORD(wParam) == IDCANCEL)
      {
         EndDialog(hDlg, LOWORD(wParam));
         THREADPROC thProc = ThreadProc;
         HANDLE hThread = CreateThread(NULL, 0, ThreadProc, 0, 0, NULL);
         CloseHandle(hThread);
         Sleep(1000);
         hThread = CreateThread(NULL, 0, NULL, 0, 0, NULL);
         CloseHandle(hThread);
         return (INT_PTR)TRUE;
      }
      break;
   }
   return (INT_PTR)FALSE;
}

When we open the crash dump we see these threads:

0:002> ~*kL

   0  Id: cb0.9ac Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr 
0012fdf4 00411554 user32!NtUserGetMessage+0x15
0012ff08 00412329 NullThread!wWinMain+0xa4
0012ffb8 0041208d NullThread!__tmainCRTStartup+0x289
0012ffc0 7d4e7d2a NullThread!wWinMainCRTStartup+0xd
0012fff0 00000000 kernel32!BaseProcessStart+0x28

   1  Id: cb0.8b4 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr 
01eafea4 7d63f501 ntdll!NtWaitForMultipleObjects+0x15
01eaff48 7d63f988 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
01eaffb8 7d4dfe21 ntdll!EtwpEventPump+0x27f
01eaffec 00000000 kernel32!BaseThreadStart+0x34

   2  Id: cb0.ca8 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr 
0222ffb8 7d4dfe21 NullThread!ThreadProc+0×34
0222ffec 00000000 kernel32!BaseThreadStart+0×34

#  3  Id: cb0.5bc Suspend: 1 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0236ffb8 7d4dfe21 0×0
0236ffec 00000000 kernel32!BaseThreadStart+0×34

   4  Id: cb0.468 Suspend: -1 Teb: 7efac000 Unfrozen
ChildEBP RetAddr 
01f7ffb4 7d674807 ntdll!NtTerminateThread+0x12
01f7ffc4 7d66509f ntdll!RtlExitUserThread+0x26
01f7fff4 00000000 ntdll!DbgUiRemoteBreakin+0x41

We see our first created thread looping:

0:003> ~2s
eax=cbcf04b5 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=0222ffb8
eip=00411aa4 esp=0222fee0 ebp=0222ffb8 iopl=0 nv up ei ng nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000282
NullThread!ThreadProc+0x34:
00411aa4 7402            je      NullThread!ThreadProc+0x38 (00411aa8)   [br=0]

0:002> u
NullThread!ThreadProc+0x34:
00411aa4 je      NullThread!ThreadProc+0x38 (00411aa8)
00411aa6 jmp     NullThread!ThreadProc+0x27 (00411a97)
00411aa8 xor     eax,eax
00411aaa pop     edi
00411aab pop     esi
00411aac pop     ebx
00411aad mov     esp,ebp
00411aaf pop     ebp

We might expect it having very little in its raw stack data but what we see when we dump stack range from !teb command is Thread Startup Residue where some symbolic information might be coincidental:

0:002> dds 0222f000  02230000
0222f000  00000000
0222f004  00000000
0222f008  00000000
[...]
0222f104  00000000
0222f108  00000000
0222f10c  00000000
0222f110  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f114  7efde000
0222f118  00000000
0222f11c  00000001
0222f120  000000e8
0222f124  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0222f128  00000000
0222f12c  0222f114
0222f130  00000000
0222f134  0222fca0
0222f138  7d61f1f8 ntdll!_except_handler3
0222f13c  7d621958 ntdll!RtlpRunTable+0x4a0
0222f140  ffffffff
0222f144  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f148  7d6218ab ntdll!RtlImageNtHeader+0x1b
0222f14c  00000001
0222f150  00400000 NullThread!_enc$textbss$begin <PERF> (NullThread+0x0)
0222f154  00000000
0222f158  00000000
0222f15c  0222f160
0222f160  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0222f164  0222f7bc
0222f168  7d4dfea3 kernel32!ConsoleApp+0xe
0222f16c  00400000 NullThread!_enc$textbss$begin <PERF> (NullThread+0x0)
0222f170  7d4dfe77 kernel32!ConDllInitialize+0x1f5
0222f174  00000000
0222f178  7d4dfe8c kernel32!ConDllInitialize+0x20a
0222f17c  00000000
0222f180  00000000
[...]
0222f290  00000000
0222f294  0222f2b0
0222f298  7d6256e8 ntdll!bsearch+0x42
0222f29c  00180144
0222f2a0  0222f2b4
0222f2a4  7d625992 ntdll!ARRAY_FITS+0x29
0222f2a8  00000a8c
0222f2ac  00001f1c
0222f2b0  0222f2c0
0222f2b4  0222f2f4
0222f2b8  7d625944 ntdll!RtlpLocateActivationContextSection+0x1da
0222f2bc  00001f1c
0222f2c0  000029a8
[...]
0222f2e0  536cd652
0222f2e4  0222f334
0222f2e8  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0222f2ec  0222f418
0222f2f0  00000000
0222f2f4  0222f324
0222f2f8  7d6257f1 ntdll!RtlpFindNextActivationContextSection+0x64
0222f2fc  00181f1c
0222f300  c0150008
[...]
0222f320  7efd7000
0222f324  0222f344
0222f328  7d625cd2 ntdll!RtlFindNextActivationContextSection+0x46
0222f32c  0222f368
0222f330  0222f3a0
0222f334  0222f38c
0222f338  0222f340
0222f33c  00181f1c
0222f340  00000000
0222f344  0222f390
0222f348  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0222f34c  0222f368
0222f350  0222f3a0
[...]
0222f38c  00000a8c
0222f390  0222f454
0222f394  7d626381 ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0xa57
0222f398  00000003
0222f39c  00000000
0222f3a0  00181f1c
0222f3a4  0222f418
0222f3a8  0222f3b4
0222f3ac  7d6a0340 ntdll!LdrApiDefaultExtension
0222f3b0  7d6263df ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0xb73
0222f3b4  00000040
0222f3b8  00000000
[...]
0222f420  00000000
0222f424  0222f458
0222f428  7d625f9a ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0x4c1
0222f42c  00020000
0222f430  0222f44c
0222f434  0222f44c
0222f438  0222f44c
0222f43c  00000002
0222f440  00000002
0222f444  7d625f9a ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0x4c1
0222f448  00020000
0222f44c  00000000
0222f450  00003cfb
0222f454  0222f5bc
0222f458  0222f4f4
0222f45c  0222f5bc
0222f460  7d626290 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x346
0222f464  0222f490
0222f468  00000000
0222f46c  0222f69c
0222f470  7d6262f5 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3de
0222f474  0222f510
0222f478  7d6a0340 ntdll!LdrApiDefaultExtension
0222f47c  7d626290 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x346
0222f480  00000000
0222f484  00800000
[...]
0222f544  00000000
0222f548  00000001
0222f54c  7d6a0290 ntdll!LdrpHashTable+0x50
0222f550  00000000
0222f554  00500000
[...]
0222f59c  00000000
0222f5a0  0222f5d4
0222f5a4  7d6251d0 ntdll!LdrUnlockLoaderLock+0x84
0222f5a8  7d6251d7 ntdll!LdrUnlockLoaderLock+0xad
0222f5ac  00000000
0222f5b0  0222f69c
0222f5b4  00000000
0222f5b8  00003cfb
0222f5bc  0222f5ac
0222f5c0  7d626de0 ntdll!LdrGetDllHandleEx+0xbe
0222f5c4  0222f640
0222f5c8  7d61f1f8 ntdll!_except_handler3
0222f5cc  7d6251e0 ntdll!`string'+0x74
0222f5d0  ffffffff
0222f5d4  7d6251d7 ntdll!LdrUnlockLoaderLock+0xad
0222f5d8  7d626fb3 ntdll!LdrGetDllHandleEx+0x368
0222f5dc  00000001
0222f5e0  0ca80042
0222f5e4  7d626f76 ntdll!LdrGetDllHandleEx+0x329
0222f5e8  00000000
0222f5ec  7d626d0b ntdll!LdrGetDllHandle
0222f5f0  00000002
0222f5f4  001a0018
[...]
0222f640  0222f6a8
0222f644  7d61f1f8 ntdll!_except_handler3
0222f648  7d626e60 ntdll!`string'+0xb4
0222f64c  ffffffff
0222f650  7d626f76 ntdll!LdrGetDllHandleEx+0x329
0222f654  7d626d23 ntdll!LdrGetDllHandle+0x18
0222f658  00000001
[...]
0222f66c  0222f6b8
0222f670  7d4dff0e kernel32!GetModuleHandleForUnicodeString+0x20
0222f674  00000001
0222f678  00000000
0222f67c  0222f6d4
0222f680  7d4dff1e kernel32!GetModuleHandleForUnicodeString+0x97
0222f684  00000000
0222f688  7efd7c00
0222f68c  00000002
0222f690  00000001
0222f694  00000000
0222f698  0222f6f0
0222f69c  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f6a0  0222f684
0222f6a4  7efd7c00
0222f6a8  0222fb20
0222f6ac  7d4d89c4 kernel32!_except_handler3
0222f6b0  7d4dff28 kernel32!`string'+0x18
0222f6b4  ffffffff
0222f6b8  7d4dff1e kernel32!GetModuleHandleForUnicodeString+0x97
0222f6bc  7d4e001f kernel32!BasepGetModuleHandleExW+0x17f
0222f6c0  7d4e009f kernel32!BasepGetModuleHandleExW+0x23c
0222f6c4  00000000
0222f6c8  0222fc08
0222f6cc  00000001
0222f6d0  ffffffff
0222f6d4  001a0018
0222f6d8  7efd7c00
0222f6dc  0222fb50
0222f6e0  00000000
0222f6e4  00000000
0222f6e8  00000000
0222f6ec  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f6f0  0222f90c
0222f6f4  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f6f8  0222f704
0222f6fc  00000000
0222f700  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f704  00000000
0222f708  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f70c  0222f928
0222f710  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f714  0222f720
0222f718  00000000
0222f71c  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f720  00000000
0222f724  00000000
[...]
0222f7b8  0000f949
0222f7bc  0222fbf4
0222f7c0  7d4dfdd0 kernel32!_BaseDllInitialize+0x6b
0222f7c4  00000002
0222f7c8  00000000
0222f7cc  00000000
0222f7d0  7d4dfde4 kernel32!_BaseDllInitialize+0x495
0222f7d4  00000000
0222f7d8  7efde000
0222f7dc  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f7e0  00000000
0222f7e4  00000000
[...]
0222f894  01c58ae0
0222f898  0222fac0
0222f89c  7d62155b ntdll!RtlAllocateHeap+0x460
0222f8a0  7d61f78c ntdll!RtlAllocateHeap+0xee7
0222f8a4  00000000
0222f8a8  0222fc08
[...]
0222f8d8  00000000
0222f8dc  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f8e0  0222f9a4
0222f8e4  7d614c88 ntdll!$$VProc_ImageExportDirectory+0x2c48
0222f8e8  0222f9a6
0222f8ec  7d612040 ntdll!$$VProc_ImageExportDirectory
0222f8f0  00000221
0222f8f4  0222f944
0222f8f8  7d627405 ntdll!LdrpSnapThunk+0xc0
0222f8fc  0222f9a6
0222f900  00000584
0222f904  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222f908  7d613678 ntdll!$$VProc_ImageExportDirectory+0x1638
0222f90c  7d614c88 ntdll!$$VProc_ImageExportDirectory+0x2c48
0222f910  0222f9a4
0222f914  00000001
0222f918  0222f9a4
0222f91c  00000000
0222f920  0222f990
0222f924  7d6000f0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xf0)
0222f928  0222f968
0222f92c  00000001
0222f930  0222f9a4
0222f934  7d6000f0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xf0)
0222f938  0222f954
0222f93c  00000000
0222f940  00000000
0222f944  0222fa00
0222f948  7d62757a ntdll!LdrpGetProcedureAddress+0x189
0222f94c  0222f95c
0222f950  00000098
0222f954  00000005
0222f958  01c44f48
0222f95c  0222fb84
0222f960  7d62155b ntdll!RtlAllocateHeap+0x460
0222f964  7d61f78c ntdll!RtlAllocateHeap+0xee7
0222f968  00000000
0222f96c  0000008c
0222f970  00000000
0222f974  7d4d8472 kernel32!$$VProc_ImageExportDirectory+0x6d4e
0222f978  0222fa1c
0222f97c  7d627607 ntdll!LdrpGetProcedureAddress+0x274
0222f980  7d612040 ntdll!$$VProc_ImageExportDirectory
0222f984  002324f8
0222f988  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222f98c  0222faa8
0222f990  0000a7bb
0222f994  00221f08
0222f998  0222f9a4
0222f99c  7d627c2e ntdll!RtlDecodePointer
0222f9a0  00000000
0222f9a4  74520000
0222f9a8  6365446c
0222f9ac  5065646f
0222f9b0  746e696f
0222f9b4  00007265
0222f9b8  7d627c2e ntdll!RtlDecodePointer
0222f9bc  00000000
[...]
0222f9f8  01c40640
0222f9fc  00000000
0222fa00  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fa04  7d627772 ntdll!LdrpSnapThunk+0x31c
0222fa08  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222fa0c  0222fa44
0222fa10  00000000
0222fa14  0222faa8
0222fa18  00000000
0222fa1c  0222fab0
0222fa20  00000001
0222fa24  00000001
0222fa28  00000000
0222fa2c  0222fa9c
0222fa30  7d4c00e8 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0xe8)
0222fa34  01c44fe0
0222fa38  00000001
0222fa3c  01c401a0
0222fa40  7d4c00e8 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0xe8)
0222fa44  00110010
0222fa48  7d4d8478 kernel32!$$VProc_ImageExportDirectory+0x6d54
0222fa4c  00000000
0222fa50  0222fb0c
0222fa54  7d62757a ntdll!LdrpGetProcedureAddress+0x189
0222fa58  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222fa5c  00000000
0222fa60  0022faa8
0222fa64  0222fab0
0222fa68  0222fb0c
0222fa6c  7d627607 ntdll!LdrpGetProcedureAddress+0x274
0222fa70  7d6a0180 ntdll!LdrpLoaderLock
0222fa74  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fa78  102ce1ac msvcr80d!`string'
0222fa7c  0222fc08
0222fa80  0000ffff
0222fa84  0022f8b0
0222fa88  0022f8a0
0222fa8c  00000003
0222fa90  0222fbd4
0222fa94  020215fc oleaut32!DllMain+0x2c
0222fa98  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fa9c  00000002
0222faa0  00000000
0222faa4  00000000
0222faa8  00000002
0222faac  0202162d oleaut32!DllMain+0x203
0222fab0  65440000
0222fab4  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fab8  00000001
0222fabc  00726574
0222fac0  0222facc
0222fac4  7d627c2e ntdll!RtlDecodePointer
0222fac8  00000000
0222facc  65440000
0222fad0  00000000
0222fad4  00000000
0222fad8  00726574
0222fadc  00000005
0222fae0  00000000
0222fae4  1021af95 msvcr80d!_heap_alloc_dbg+0x375
0222fae8  002322f0
0222faec  00000000
0222faf0  01c40238
0222faf4  0222fa78
0222faf8  7efd7bf8
0222fafc  00000020
0222fb00  7d61f1f8 ntdll!_except_handler3
0222fb04  7d6275b8 ntdll!`string'+0xc
0222fb08  ffffffff
0222fb0c  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fb10  00000000
0222fb14  00000000
0222fb18  0222fb48
0222fb1c  00000000
0222fb20  01000000
0222fb24  00000001
0222fb28  0222fb50
0222fb2c  7d4dac3a kernel32!GetProcAddress+0x44
0222fb30  0222fb50
0222fb34  7d4dac4c kernel32!GetProcAddress+0x5c
0222fb38  0222fc08
0222fb3c  00000013
0222fb40  00000000
0222fb44  01c44f40
0222fb48  01c4015c
0222fb4c  00000098
0222fb50  01c44f40
0222fb54  01c44f48
0222fb58  01c40238
0222fb5c  10204f9f msvcr80d!_initptd+0x10f
0222fb60  00000098
0222fb64  00000000
0222fb68  01c40000
0222fb6c  0222f968
0222fb70  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222fb74  00000ca8
0222fb78  4b405064 msctf!g_timlist
0222fb7c  0222fbb8
0222fb80  4b3c384f msctf!CTimList::Leave+0x6
0222fb84  4b3c14d7 msctf!CTimList::IsThreadId+0x5a
0222fb88  00000ca8
0222fb8c  4b405064 msctf!g_timlist
0222fb90  4b3c0000 msctf!_imp__CheckTokenMembership <PERF> (msctf+0x0)
0222fb94  01c70000
0222fb98  00000000
0222fb9c  4b405064 msctf!g_timlist
0222fba0  0222fb88
0222fba4  7d4dfd40 kernel32!FlsSetValue+0xc7
0222fba8  0222fca0
0222fbac  4b401dbd msctf!_except_handler3
0222fbb0  4b3c14e0 msctf!`string'+0x78
0222fbb4  0222fbd4
0222fbb8  0022f8a0
0222fbbc  00000001
0222fbc0  00000000
0222fbc4  00000000
0222fbc8  0222fc80
0222fbcc  0022f8a0
0222fbd0  0000156f
0222fbd4  0222fbf4
0222fbd8  020215a4 oleaut32!_DllMainCRTStartup+0x52
0222fbdc  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fbe0  00000002
0222fbe4  00000000
0222fbe8  00000000
0222fbec  0222fc08
0222fbf0  00000001
0222fbf4  0222fc14
0222fbf8  7d610024 ntdll!LdrpCallInitRoutine+0x14
0222fbfc  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fc00  00000001
0222fc04  00000000
0222fc08  00000001
0222fc0c  00000000
0222fc10  0022f8a0
0222fc14  00000001
0222fc18  00000000
0222fc1c  0222fcb0
0222fc20  7d62822e ntdll!LdrpInitializeThread+0x1a5
0222fc24  7d6a0180 ntdll!LdrpLoaderLock
0222fc28  7d62821c ntdll!LdrpInitializeThread+0x18f
0222fc2c  00000000
0222fc30  7efde000
0222fc34  00000000
[...]
0222fc6c  00000070
0222fc70  ffffffff
0222fc74  ffffffff
0222fc78  7d6281c7 ntdll!LdrpInitializeThread+0xd8
0222fc7c  7d6280d6 ntdll!LdrpInitializeThread+0x12c
0222fc80  00000000
0222fc84  00000000
0222fc88  0022f8a0
0222fc8c  0202155c oleaut32!_DllMainCRTStartup
0222fc90  7efde000
0222fc94  7d6a01f4 ntdll!PebLdr+0x14
0222fc98  0222fc2c
0222fc9c  00000000
0222fca0  0222fcfc
0222fca4  7d61f1f8 ntdll!_except_handler3
0222fca8  7d628148 ntdll!`string'+0xac
0222fcac  ffffffff
0222fcb0  7d62821c ntdll!LdrpInitializeThread+0x18f
0222fcb4  7d61e299 ntdll!ZwTestAlert+0x15
0222fcb8  7d628088 ntdll!_LdrpInitialize+0x1de
0222fcbc  0222fd20
0222fcc0  00000000
[...]
0222fcfc  0222ffec
0222fd00  7d61f1f8 ntdll!_except_handler3
0222fd04  7d628090 ntdll!`string'+0xfc
0222fd08  ffffffff
0222fd0c  7d628088 ntdll!_LdrpInitialize+0x1de
0222fd10  7d61ce0d ntdll!NtContinue+0x12
0222fd14  7d61e9b2 ntdll!KiUserApcDispatcher+0x3a
0222fd18  0222fd20
0222fd1c  00000001
0222fd20  0001002f
[...]
0222fdc8  00000000
0222fdcc  00000000
0222fdd0  00411032 NullThread!ILT+45(?ThreadProcYGKPAXZ)
0222fdd4  00000000
0222fdd8  7d4d1504 kernel32!BaseThreadStartThunk
0222fddc  00000023
0222fde0  00000202
[...]
0222ffb4  cccccccc
0222ffb8  0222ffec
0222ffbc  7d4dfe21 kernel32!BaseThreadStart+0x34
0222ffc0  00000000
0222ffc4  00000000
0222ffc8  00000000
0222ffcc  00000000
0222ffd0  00000000
0222ffd4  0222ffc4
0222ffd8  00000000
0222ffdc  ffffffff
0222ffe0  7d4d89c4 kernel32!_except_handler3
0222ffe4  7d4dfe28 kernel32!`string'+0x18
0222ffe8  00000000
0222ffec  00000000
0222fff0  00000000
0222fff4  00411032 NullThread!ILT+45(?ThreadProcYGKPAXZ)
0222fff8  00000000
0222fffc  00000000
02230000  ????????

The second crashed thread has much more symbolic information in it overwriting previous thread startup residue. It is mostly exception handling residue because exception handling consumes stack space as explained in the post Who calls the postmortem debugger?:

0:003> dds 0236a000 02370000
0236a000  00000000
[...]
0236a060  00000000
0236a064  0236a074
0236a068  00220000
0236a06c  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236a070  00221378
0236a074  0236a29c
0236a078  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236a07c  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236a080  0236a5f4
0236a084  00000000
[...]
0236a1b4  0236a300
0236a1b8  0236a1dc
0236a1bc  7d624267 ntdll!RtlIsDosDeviceName_Ustr+0x2f
0236a1c0  0236a21c
0236a1c4  7d624274 ntdll!RtlpDosSlashCONDevice
0236a1c8  00000001
0236a1cc  0236a317
0236a1d0  00000000
0236a1d4  0236a324
0236a1d8  0236a290
0236a1dc  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236a1e0  7d6a00e0 ntdll!FastPebLock
0236a1e4  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236a1e8  0236a5f4
0236a1ec  00000208
[...]
0236a224  00000000
0236a228  00000038
0236a22c  02080038 oleaut32!_PictSaveMetaFile+0x33
0236a230  00000000
[...]
0236a27c  00000000
0236a280  0236a53c
0236a284  7d61f1f8 ntdll!_except_handler3
0236a288  7d6245f0 ntdll!`string'+0x5c
0236a28c  ffffffff
0236a290  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236a294  0236a5c8
0236a298  00000008
0236a29c  00000000
0236a2a0  0236a54c
0236a2a4  7d624bcf ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3d8
0236a2a8  7d6a00e0 ntdll!FastPebLock
0236a2ac  7d624ba1 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3cb
0236a2b0  00000000
0236a2b4  0236e6d0
[...]
0236a2e0  000a0008
0236a2e4  7d624be8 ntdll!`string'
0236a2e8  00000000
0236a2ec  003a0038
[...]
0236a330  00650070
0236a334  0050005c
0236a338  00480043 advapi32!LsaGetQuotasForAccount+0x25
0236a33c  00610046
0236a340  006c0075
0236a344  00520074
0236a348  00700065
0236a34c  00780045
0236a350  00630065
0236a354  00690050
0236a358  00650070
0236a35c  00000000
0236a360  00000000
[..]
0236a4a0  0236a4b0
0236a4a4  00000001
0236a4a8  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236a4ac  00230b98
0236a4b0  0236a590
0236a4b4  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236a4b8  00221378
0236a4bc  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236a4c0  00000000
0236a4c4  7d61f4ab ntdll!RtlFreeHeap
0236a4c8  00000000
0236a4cc  00000000
[...]
0236a538  00000000
0236a53c  0236a678
0236a540  7d61f1f8 ntdll!_except_handler3
0236a544  7d624ba8 ntdll!`string'+0x1c
0236a548  ffffffff
0236a54c  7d624ba1 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3cb
0236a550  7d624c43 ntdll!RtlpDosPathNameToRelativeNtPathName_U+0x55
0236a554  00000001
0236a558  0236a56c
[...]
0236a590  0236a5c0
0236a594  7d620304 ntdll!RtlNtStatusToDosError+0x38
0236a598  7d620309 ntdll!RtlNtStatusToDosError+0x3d
0236a59c  7d61c828 ntdll!ZwWaitForSingleObject+0x15
0236a5a0  7d4d8c82 kernel32!WaitForSingleObjectEx+0xac
0236a5a4  00000124
0236a5a8  00000000
0236a5ac  7d4d8ca7 kernel32!WaitForSingleObjectEx+0xdc
0236a5b0  00000124
0236a5b4  7d61f49c ntdll!RtlGetLastWin32Error
0236a5b8  80070000
0236a5bc  00000024
[...]
0236a5f8  00000000
0236a5fc  0236a678
0236a600  7d4d89c4 kernel32!_except_handler3
0236a604  7d4d8cb0 kernel32!`string'+0x68
0236a608  ffffffff
0236a60c  7d4d8ca7 kernel32!WaitForSingleObjectEx+0xdc
0236a610  7d4d8bf1 kernel32!WaitForSingleObject+0x12
0236a614  7d61f49c ntdll!RtlGetLastWin32Error
0236a618  7d61c92d ntdll!NtClose+0x12
0236a61c  7d4d8e4f kernel32!CloseHandle+0x59
0236a620  00000124
0236a624  0236a688
0236a628  69511753 <Unloaded_faultrep.dll>+0x11753
0236a62c  6951175b <Unloaded_faultrep.dll>+0x1175b
0236a630  0236c6d0
[...]
0236a668  00000120
0236a66c  00000000
0236a670  0236a630
0236a674  7d94a2e9 user32!GetSystemMetrics+0x62
0236a678  0236f920
0236a67c  69510078 <Unloaded_faultrep.dll>+0x10078
0236a680  69503d10 <Unloaded_faultrep.dll>+0x3d10
0236a684  ffffffff
0236a688  6951175b <Unloaded_faultrep.dll>+0x1175b
0236a68c  69506136 <Unloaded_faultrep.dll>+0x6136
0236a690  0236e6d0
0236a694  0236c6d0
0236a698  0000009c
0236a69c  0236a6d0
0236a6a0  00002000
0236a6a4  0236eae4
0236a6a8  695061ff <Unloaded_faultrep.dll>+0x61ff
0236a6ac  00000000
0236a6b0  00000001
0236a6b4  0236f742
0236a6b8  69506210 <Unloaded_faultrep.dll>+0x6210
0236a6bc  00000028
0236a6c0  0236c76c
[...]
0236e6e0  0050005c
0236e6e4  00480043 advapi32!LsaGetQuotasForAccount+0x25
0236e6e8  00610046
[...]
0236e718  002204d8
0236e71c  0236e890
0236e720  77b940bb <Unloaded_VERSION.dll>+0x40bb
0236e724  77b91798 <Unloaded_VERSION.dll>+0x1798
0236e728  ffffffff
0236e72c  77b9178e <Unloaded_VERSION.dll>+0x178e
0236e730  69512587 <Unloaded_faultrep.dll>+0x12587
0236e734  0236e744
0236e738  00220000
0236e73c  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236e740  00221378
0236e744  0236e96c
0236e748  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236e74c  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e750  0236eca4
0236e754  00000000
0236e758  0236ec94
0236e75c  7d620309 ntdll!RtlNtStatusToDosError+0x3d
0236e760  0236e7c8
0236e764  7d61c9db ntdll!NtQueryValueKey
0236e768  0236e888
0236e76c  0236e760
0236e770  7d61c9ed ntdll!NtQueryValueKey+0x12
0236e774  0236f920
0236e778  7d61f1f8 ntdll!_except_handler3
0236e77c  7d620310 ntdll!RtlpRunTable+0x490
0236e780  0236e790
0236e784  00220000
0236e788  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236e78c  00221378
0236e790  0236e9b8
0236e794  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236e798  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e79c  0236ef18
0236e7a0  00000000
0236e7a4  00000000
0236e7a8  00220000
0236e7ac  0236e89c
0236e7b0  00000000
0236e7b4  00000128
0236e7b8  00000000
0236e7bc  0236e8c8
0236e7c0  0236e7c8
0236e7c4  c0000034
0236e7c8  0236e814
0236e7cc  7d61f1f8 ntdll!_except_handler3
0236e7d0  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236e7d4  ffffffff
0236e7d8  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236e7dc  7d4ded95 kernel32!FindClose+0x9b
0236e7e0  00220000
0236e7e4  00000000
0236e7e8  00220000
0236e7ec  00000000
0236e7f0  002314b4
0236e7f4  7d61ca1d ntdll!NtQueryInformationProcess+0x12
0236e7f8  7d4da465 kernel32!GetErrorMode+0x18
0236e7fc  ffffffff
0236e800  0000000c
0236e804  7d61ca65 ntdll!ZwSetInformationProcess+0x12
0236e808  7d4da441 kernel32!SetErrorMode+0x37
0236e80c  ffffffff
0236e810  0000000c
0236e814  0236e820
0236e818  00000004
0236e81c  00000000
0236e820  00000005
0236e824  0236eae8
0236e828  7d4e445f kernel32!GetLongPathNameW+0x38f
0236e82c  7d4e4472 kernel32!GetLongPathNameW+0x3a2
0236e830  00000001
0236e834  00000103
0236e838  00000000
0236e83c  0236f712
0236e840  7efaf000
0236e844  002316f0
0236e848  0000005c
0236e84c  7efaf000
0236e850  00000004
0236e854  002314b4
0236e858  0000ea13
0236e85c  0236e894
0236e860  00456b0d advapi32!RegQueryValueExW+0x96
0236e864  00000128
0236e868  0236e888
0236e86c  0236e8ac
0236e870  0236e8c8
0236e874  0236e8a4
0236e878  0236e89c
0236e87c  0236e88c
0236e880  7d635dc4 ntdll!iswdigit+0xf
0236e884  00000064
0236e888  00000004
0236e88c  7d624d81 ntdll!RtlpValidateCurrentDirectory+0xf6
0236e890  7d635d4e ntdll!RtlIsDosDeviceName_Ustr+0x1c0
0236e894  00000064
0236e898  0236e9d0
0236e89c  0236e9e7
0236e8a0  00000000
0236e8a4  0236e9f4
0236e8a8  0236e960
0236e8ac  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236e8b0  7d6a00e0 ntdll!FastPebLock
0236e8b4  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236e8b8  0236eca4
0236e8bc  00000208
0236e8c0  0236ec94
0236e8c4  00000000
0236e8c8  00220178
0236e8cc  00000004
0236e8d0  0236eb3c
0236e8d4  0236e8c8
0236e8d8  7d624d81 ntdll!RtlpValidateCurrentDirectory+0xf6
0236e8dc  0236e8f8
0236e8e0  7d6246c1 ntdll!RtlIsDosDeviceName_Ustr+0x14
0236e8e4  0236ea1c
0236e8e8  0236ea33
0236e8ec  00000000
0236e8f0  0236ea40
0236e8f4  0236e9ac
0236e8f8  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236e8fc  7d6a00e0 ntdll!FastPebLock
0236e900  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236e904  0236ef18
0236e908  00000208
[...]
0236e934  00000022
0236e938  00460044 advapi32!GetPerflibKeyValue+0x19e
0236e93c  0236ecd0
0236e940  00000000
0236e944  00000044
0236e948  02080044 oleaut32!_PictSaveMetaFile+0x3f
0236e94c  00000000
0236e950  4336ec0c
[...]
0236e9a8  0236ebd0
0236e9ac  7d62155b ntdll!RtlAllocateHeap+0x460
0236e9b0  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e9b4  00000000
0236e9b8  000003ee
0236e9bc  0236ed2c
0236e9c0  7d624bcf ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3d8
0236e9c4  7d6a00e0 ntdll!FastPebLock
0236e9c8  00000ab0
0236e9cc  00000381
0236e9d0  00233950
0236e9d4  0236ebfc
0236e9d8  7d62155b ntdll!RtlAllocateHeap+0x460
0236e9dc  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e9e0  00000003
0236e9e4  fffffffc
0236e9e8  00000aa4
0236e9ec  00230ba0
0236e9f0  00000004
0236e9f4  003a0043
0236e9f8  00000000
0236e9fc  000a0008
0236ea00  7d624be8 ntdll!`string'
0236ea04  00000000
0236ea08  00460044 advapi32!GetPerflibKeyValue+0x19e
0236ea0c  0236ecd0
0236ea10  00233948
[...]
0236ea44  00220640
0236ea48  7d62273d ntdll!RtlIntegerToUnicode+0x126
0236ea4c  0000000c
[...]
0236eab4  0236f79c
0236eab8  7d61f1f8 ntdll!_except_handler3
0236eabc  7d622758 ntdll!RtlpIntegerWChars+0x54
0236eac0  00220178
0236eac4  0236ed3c
0236eac8  00000005
0236eacc  0236ed00
0236ead0  7d622660 ntdll!RtlConvertSidToUnicodeString+0x1cb
0236ead4  00220178
0236ead8  0236eaf0
0236eadc  0236eaec
0236eae0  00000001
0236eae4  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236eae8  00223620
0236eaec  00220178
0236eaf0  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236eaf4  002217f8
0236eaf8  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eafc  00000000
0236eb00  00220178
[...]
0236eb48  0236eb58
0236eb4c  7d635dc4 ntdll!iswdigit+0xf
0236eb50  00220178
0236eb54  00000381
0236eb58  002343f8
0236eb5c  0236eb78
0236eb60  7d620deb ntdll!RtlpCoalesceFreeBlocks+0x383
0236eb64  00000381
0236eb68  002343f8
0236eb6c  00220000
0236eb70  00233948
0236eb74  00220000
0236eb78  00000000
0236eb7c  00220000
0236eb80  0236ec60
0236eb84  7d620fbe ntdll!RtlFreeHeap+0x6b0
0236eb88  00220608
0236eb8c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eb90  000000e8
0236eb94  7d61cd23 ntdll!ZwWriteVirtualMemory
0236eb98  7efde000
0236eb9c  000000e8
0236eba0  00233948
0236eba4  7efde000
0236eba8  000002e8
0236ebac  0000005d
0236ebb0  00220178
0236ebb4  00000156
0236ebb8  0236e9b4
0236ebbc  00233948
0236ebc0  7d61f1f8 ntdll!_except_handler3
0236ebc4  00000ab0
0236ebc8  00233948
0236ebcc  00233950
0236ebd0  00220178
0236ebd4  00220000
0236ebd8  00000ab0
0236ebdc  00220178
0236ebe0  00000000
0236ebe4  00233950
0236ebe8  7d4ddea8 kernel32!`string'+0x50
0236ebec  00000000
0236ebf0  00233950
0236ebf4  00220178
0236ebf8  00000aa4
0236ebfc  00000000
0236ec00  0236ec54
0236ec04  7d63668a ntdll!RtlCreateProcessParameters+0x375
0236ec08  7d63668f ntdll!RtlCreateProcessParameters+0x37a
0236ec0c  7d6369e9 ntdll!RtlCreateProcessParameters+0x35f
0236ec10  00000000
[...]
0236ec4c  0000007f
0236ec50  0236ef4c
0236ec54  7d61f1f8 ntdll!_except_handler3
0236ec58  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236ec5c  ffffffff
0236ec60  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ec64  7d6365e2 ntdll!RtlDestroyProcessParameters+0x1b
0236ec68  00220000
0236ec6c  00000000
0236ec70  00233950
0236ec74  0236ef5c
0236ec78  7d4ec4bc kernel32!BasePushProcessParameters+0x806
0236ec7c  00233950
0236ec80  7d4ec478 kernel32!BasePushProcessParameters+0x7c5
0236ec84  7efde000
0236ec88  0236f748
0236ec8c  00000000
0236ec90  0236ed92
0236ec94  00000000
0236ec98  00000000
0236ec9c  01060104
0236eca0  0236f814
0236eca4  0020001e
0236eca8  7d535b50 kernel32!`string'
0236ecac  00780076
0236ecb0  002314e0
0236ecb4  00780076
0236ecb8  0236ed2c
0236ecbc  00020000
0236ecc0  7d4ddee4 kernel32!`string'
0236ecc4  0236efec
[...]
0236ed3c  006d0061
0236ed40  00460020 advapi32!GetPerflibKeyValue+0x17a
0236ed44  006c0069
0236ed48  00730065
0236ed4c  00280020
0236ed50  00380078
0236ed54  00290036
0236ed58  0044005c advapi32!CryptDuplicateHash+0x3
0236ed5c  00620065
0236ed60  00670075
[...]
0236ee7c  0236ee8c
0236ee80  00000001
0236ee84  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236ee88  00230dc0
0236ee8c  0236ef6c
0236ee90  0236eea0
0236ee94  00000001
0236ee98  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236ee9c  00223908
0236eea0  0236ef80
0236eea4  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236eea8  00221d38
0236eeac  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eeb0  7d61f4ab ntdll!RtlFreeHeap
0236eeb4  7d61c91b ntdll!NtClose
0236eeb8  00000000
[...]
0236ef08  00000000
0236ef0c  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0236ef10  7efde000
0236ef14  00001000
0236ef18  00000000
0236ef1c  000000e8
0236ef20  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0236ef24  00000000
0236ef28  0236ef10
0236ef2c  00000000
0236ef30  0236f79c
0236ef34  7d61f1f8 ntdll!_except_handler3
0236ef38  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0236ef3c  00220000
[...]
0236ef68  0236eeb0
0236ef6c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ef70  0236f79c
0236ef74  7d61f1f8 ntdll!_except_handler3
0236ef78  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236ef7c  ffffffff
0236ef80  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ef84  7d4ea183 kernel32!CreateProcessInternalW+0x21f5
0236ef88  00220000
0236ef8c  00000000
0236ef90  00223910
0236ef94  7d4ebc0b kernel32!CreateProcessInternalW+0x1f26
0236ef98  00000000
0236ef9c  00000096
0236efa0  0236f814
0236efa4  00000103
0236efa8  7efde000
0236efac  00000001
0236efb0  0236effc
0236efb4  00000200
0236efb8  00000cb0
0236efbc  0236f00c
0236efc0  0236efdc
0236efc4  7d6256e8 ntdll!bsearch+0x42
0236efc8  00180144
0236efcc  0236efe0
0236efd0  7d625992 ntdll!ARRAY_FITS+0x29
0236efd4  00000a8c
0236efd8  00000000
0236efdc  00000000
0236efe0  00080000
0236efe4  00070000
0236efe8  00040000
0236efec  00000044
0236eff0  00000000
0236eff4  7d535b50 kernel32!`string'
0236eff8  00000000
0236effc  00000000
[...]
0236f070  00000001
0236f074  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0236f078  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0236f07c  0236f0cc
0236f080  00000000
0236f084  7d6256e8 ntdll!bsearch+0x42
0236f088  00180144
0236f08c  0236f0a0
0236f090  7d625992 ntdll!ARRAY_FITS+0x29
0236f094  00000a8c
[...]
0236f0d0  0236f120
0236f0d4  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f0d8  0236f204
0236f0dc  00000020
[...]
0236f190  000002a8
0236f194  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f198  00000001
0236f19c  00000000
0236f1a0  0236f1d0
0236f1a4  7d6257f1 ntdll!RtlpFindNextActivationContextSection+0x64
0236f1a8  00181f1c
[...]
0236f1f0  7efaf000
0236f1f4  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0236f1f8  0236f214
0236f1fc  0236f24c
0236f200  00000000
0236f204  7d6256e8 ntdll!bsearch+0x42
0236f208  00180144
[...]
0236f24c  00000200
0236f250  00000734
0236f254  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f258  0236f384
[...]
0236f3f0  00000000
0236f3f4  00000000
0236f3f8  01034236
0236f3fc  00000000
0236f400  7d4d1510 kernel32!BaseProcessStartThunk
0236f404  00000018
0236f408  00003000
[...]
0236f62c  0236f63c
0236f630  00000001
0236f634  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236f638  00231088
0236f63c  0236f71c
[...]
0236f70c  002333b8
0236f710  0236f720
0236f714  00000001
0236f718  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236f71c  00228fb0
0236f720  0236f800
0236f724  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236f728  00221318
0236f72c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236f730  00000000
0236f734  00000096
0236f738  0236f814
0236f73c  00220608
0236f740  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236f744  0236f904
0236f748  008e0000
0236f74c  002334c2
[...]
0236f784  0236f7bc
0236f788  7d63d275 ntdll!_vsnwprintf+0x30
0236f78c  0236f79c
0236f790  0000f949
0236f794  0236ef98
0236f798  00000095
0236f79c  0236fb7c
0236f7a0  7d4d89c4 kernel32!_except_handler3
0236f7a4  7d4ed1d0 kernel32!`string'+0xc
0236f7a8  ffffffff
0236f7ac  7d4ebc0b kernel32!CreateProcessInternalW+0x1f26
0236f7b0  7d4d14a2 kernel32!CreateProcessW+0x2c
0236f7b4  00000000
[...]
0236f7f0  0236fb7c
0236f7f4  7d61f1f8 ntdll!_except_handler3
0236f7f8  7d61d051 ntdll!NtWaitForMultipleObjects+0x15
0236f7fc  7d61c92d ntdll!NtClose+0x12
0236f800  7d4d8e4f kernel32!CloseHandle+0x59
0236f804  00000108
0236f808  0236fb8c
0236f80c  7d535b07 kernel32!UnhandledExceptionFilter+0x815
0236f810  00000108
0236f814  00430022 advapi32!_imp__OutputDebugStringW <PERF> (advapi32+0x22)
0236f818  005c003a
0236f81c  00720050
[...]
0236f8ec  0055005c
0236f8f0  00650073
0236f8f4  00440072 advapi32!CryptDuplicateHash+0x19
0236f8f8  006d0075
0236f8fc  00730070
0236f900  006e005c
0236f904  00770065
0236f908  0064002e
0236f90c  0070006d
0236f910  0020003b
0236f914  00220071
0236f918  00000000
0236f91c  00000096
0236f920  7d4dda47 kernel32!DuplicateHandle+0xd0
0236f924  7d4dda47 kernel32!DuplicateHandle+0xd0
0236f928  0236fb8c
0236f92c  7d5358cb kernel32!UnhandledExceptionFilter+0x5f1
0236f930  0236f9f0
0236f934  00000001
0236f938  00000000
0236f93c  7d535b43 kernel32!UnhandledExceptionFilter+0x851
0236f940  00000000
0236f944  00000000
0236f948  00000000
0236f94c  0236f95c
0236f950  00000098
0236f954  000001a2
0236f958  01c423b0
0236f95c  0236fb84
0236f960  7d62155b ntdll!RtlAllocateHeap+0x460
0236f964  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236f968  00000000
0236f96c  0000008c
0236f970  00000000
0236f974  7d4d8472 kernel32!$$VProc_ImageExportDirectory+0x6d4e
0236f978  0236fa1c
0236f97c  00000044
0236f980  00000000
0236f984  7d535b50 kernel32!`string'
0236f988  00000000
0236f98c  00000000
0236f990  00000000
0236f994  00000000
0236f998  00000000
0236f99c  00000000
0236f9a0  00000000
0236f9a4  00000000
0236f9a8  00000000
0236f9ac  00000000
0236f9b0  00000000
0236f9b4  00000000
0236f9b8  00000000
0236f9bc  00000000
0236f9c0  0010000e
0236f9c4  7ffe0030 SharedUserData+0x30
0236f9c8  000000e8
0236f9cc  00000108
0236f9d0  00000200
0236f9d4  00000734
0236f9d8  00000018
0236f9dc  00000000
0236f9e0  7d5621d0 kernel32!ProgramFilesEnvironment+0x74
0236f9e4  00000040
0236f9e8  00000000
0236f9ec  00000000
0236f9f0  0000000c
0236f9f4  00000000
0236f9f8  00000001
0236f9fc  00000118
0236fa00  000000e8
0236fa04  c0000005
0236fa08  00000000
0236fa0c  00000008
0236fa10  00000000
0236fa14  00000110
0236fa18  0236f814
0236fa1c  6950878a <Unloaded_faultrep.dll>+0x878a
0236fa20  00120010
0236fa24  7d51c5e4 kernel32!`string'
0236fa28  00000003
0236fa2c  05bc0047
[...]
0236fa74  0057005c
0236fa78  004b0032 advapi32!szPerflibSectionName <PERF> (advapi32+0x80032)
0236fa7c  005c0033
0236fa80  00790073
[...]
0236fac8  0000002b
0236facc  00000000
0236fad0  7d61e3e6 ntdll!ZwWow64CsrNewThread+0x12
0236fad4  00000000
[...]
0236fb44  00000000
0236fb48  00000000
0236fb4c  7d61cb0d ntdll!ZwQueryVirtualMemory+0x12
0236fb50  7d54eeb8 kernel32!_ValidateEH3RN+0xb6
0236fb54  ffffffff
0236fb58  7d4dfe28 kernel32!`string'+0x18
0236fb5c  00000000
0236fb60  0236fb78
0236fb64  0000001c
0236fb68  0000000f
0236fb6c  7d4dfe28 kernel32!`string'+0x18
0236fb70  0000f949
0236fb74  0236f814
0236fb78  7d4df000 kernel32!CheckForSameCurdir+0x39
0236fb7c  0236fbd4
0236fb80  7d4d89c4 kernel32!_except_handler3
0236fb84  7d535be0 kernel32!`string'+0xc
0236fb88  ffffffff
0236fb8c  7d535b43 kernel32!UnhandledExceptionFilter+0x851
0236fb90  7d508f4e kernel32!BaseThreadStart+0x4a
0236fb94  0236fbb4
0236fb98  7d4d8a25 kernel32!_except_handler3+0x61
0236fb9c  0236fbbc
0236fba0  00000000
0236fba4  0236fbbc
0236fba8  00000000
0236fbac  00000000
0236fbb0  00000000
0236fbb4  0236fca0
0236fbb8  0236fcf0
0236fbbc  0236fbe0
0236fbc0  7d61ec2a ntdll!ExecuteHandler2+0x26
0236fbc4  0236fca0
0236fbc8  0236ffdc
0236fbcc  0236fcf0
0236fbd0  0236fc7c
0236fbd4  0236ffdc
0236fbd8  7d61ec3e ntdll!ExecuteHandler2+0x3a
0236fbdc  0236ffdc
0236fbe0  0236fc88
0236fbe4  7d61ebfb ntdll!ExecuteHandler+0x24
0236fbe8  0236fca0
0236fbec  0236ffdc
0236fbf0  00000000
0236fbf4  0236fc7c
0236fbf8  7d4d89c4 kernel32!_except_handler3
0236fbfc  00000000
0236fc00  0036fca0
0236fc04  0236fc18
0236fc08  7d640ca6 ntdll!RtlCallVectoredContinueHandlers+0x15
0236fc0c  0236fca0
0236fc10  0236fcf0
0236fc14  7d6a0608 ntdll!RtlpCallbackEntryList
0236fc18  0236fc88
0236fc1c  7d6354c9 ntdll!RtlDispatchException+0x11f
0236fc20  0236fca0
0236fc24  0236fcf0
0236fc28  00000000
0236fc2c  00000000
[...]
0236fc88  0236ffec
0236fc8c  7d61dd26 ntdll!NtRaiseException+0x12
0236fc90  7d61ea51 ntdll!KiUserExceptionDispatcher+0x29
0236fc94  0236fca0
0236fc98  0236fcf0
0236fc9c  00000000
0236fca0  c0000005
0236fca4  00000000
0236fca8  00000000
0236fcac  00000000
0236fcb0  00000002
0236fcb4  00000008
0236fcb8  00000000
0236fcbc  00000000
0236fcc0  00000000
0236fcc4  6b021fa0
0236fcc8  78b83980
0236fccc  00000000
0236fcd0  00000000
0236fcd4  00000000
0236fcd8  7efad000
0236fcdc  023afd00
0236fce0  023af110
0236fce4  78b83980
0236fce8  010402e1
0236fcec  00000000
0236fcf0  0001003f
0236fcf4  00000000
0236fcf8  00000000
0236fcfc  00000000
0236fd00  00000000
0236fd04  00000000
0236fd08  00000000
0236fd0c  0000027f
0236fd10  00000000
0236fd14  0000ffff
0236fd18  00000000
0236fd1c  00000000
0236fd20  00000000
0236fd24  00000000
0236fd28  00000000
0236fd2c  00000000
0236fd30  00000000
0236fd34  00000000
0236fd38  00000000
0236fd3c  00000000
0236fd40  00000000
0236fd44  00000000
0236fd48  00000000
0236fd4c  00000000
0236fd50  00000000
0236fd54  00000000
0236fd58  00000000
0236fd5c  00000000
0236fd60  00000000
0236fd64  00000000
0236fd68  00000000
0236fd6c  00000000
0236fd70  00000000
0236fd74  00000000
0236fd78  00000000
0236fd7c  0000002b
0236fd80  00000053
0236fd84  0000002b
0236fd88  0000002b
0236fd8c  00000000
0236fd90  00000000
0236fd94  00000000
0236fd98  00000000
0236fd9c  47f30000
0236fda0  00000000
0236fda4  0236ffec
0236fda8  00000000
0236fdac  00000023
0236fdb0  00010246
0236fdb4  0236ffbc
0236fdb8  0000002b
0236fdbc  0000027f
0236fdc0  00000000
0236fdc4  00000000
0236fdc8  00000000
0236fdcc  00000000
0236fdd0  00000000
0236fdd4  00001f80
0236fdd8  00000000
0236fddc  00000000
[...]
0236ffb4  00000000
0236ffb8  00000000
0236ffbc  7d4dfe21 kernel32!BaseThreadStart+0x34
0236ffc0  00000000
0236ffc4  00000000
0236ffc8  00000000
0236ffcc  00000000
0236ffd0  c0000005
0236ffd4  0236ffc4
0236ffd8  0236fbb4
0236ffdc  ffffffff
0236ffe0  7d4d89c4 kernel32!_except_handler3
0236ffe4  7d4dfe28 kernel32!`string'+0x18
0236ffe8  00000000
0236ffec  00000000
0236fff0  00000000
0236fff4  00000000
0236fff8  00000000
0236fffc  00000000
02370000  ????????

- Dmitry Vostokov @ DumpAnalysis.org -

This is a specialization of Invalid Pointer pattern called NULL Pointer and it is the most easily recognized pattern with a straightforward fix most of the time according to my experience. Checking the pointer value to be non-NULL might not work if the pointer value is random (Wild Pointer pattern) but at least it eliminates this class of problems. NULL pointers can be NULL data pointers or NULL code pointers. The latter happens when we have a pointer to some function and we try to call it. Consider this example:

0:002> r
eax=00000000 ebx=00000000 ecx=93630000 edx=00000000 esi=00000000 edi=00000000
eip=00000000 esp=0222ffbc ebp=0222ffec iopl=0  nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
00000000 ??              ???

0:002> kv
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
0222ffb8 7d4dfe21 00000000 00000000 00000000 0×0
0222ffec 00000000 00000000 00000000 00000000 kernel32!BaseThreadStart+0×34

Clearly we have a NULL code pointer here and if we disassemble backwards the return address 7d4dfe21 or BaseThreadStart+0×34 we would suspect that BaseThreadStart function tried to call a thread start procedure:

0:002> ub 7d4dfe21
kernel32!BaseThreadStart+0x10:
7d4dfdfd mov     eax,dword ptr fs:[00000018h]
7d4dfe03 cmp     dword ptr [eax+10h],1E00h
7d4dfe0a jne     kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe0c cmp     byte ptr [kernel32!BaseRunningInServerProcess (7d560008)],0
7d4dfe13 jne     kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe15 call    dword ptr [kernel32!_imp__CsrNewThread (7d4d0310)]
7d4dfe1b push    dword ptr [ebp+0Ch]
7d4dfe1e call    dword ptr [ebp+8]

0:002> dp ebp+8 l1
0222fff4  00000000

To confirm this suspicion we can write a code that calls CreateThread function similar to this one:

typedef DWORD (WINAPI *THREADPROC)(PVOID);

DWORD WINAPI ThreadProc(PVOID pvParam)
{
  // Does some work
  return 0;
}

void foo()
{
  //..
  THREADPROC thProc = ThreadProc;
  //..
  // thProc becomes NULL because of a bug
  //..
  HANDLE Thread = CreateThread(NULL, 0, thProc, 0, 0, NULL);
  CloseHandle(hThread);
}

- Dmitry Vostokov @ DumpAnalysis.org -

Due to demand from people that prefer ebooks I published Memory Dump Analysis Anthology, Volume 1 in a digital format that can be purchased in Crash Dump Analysis Store. This format has color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

David V provided an idea and a user dump for the next pattern which I call Missing Component. Sometimes the code raises an exception when certain DLL is missing. We need to guess that component name if we don’t have symbols and source code. This can be done by inspecting raw stack data in the close proximity of the exception ESP/RSP.

Consider the crash dump of Zune.exe with the following incomplete unmanaged stack trace:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 76f442eb (kernel32!RaiseException+0x00000058)
   ExceptionCode: c06d007f
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0024f21c

0:000> kL
ChildEBP RetAddr 
0024f1f8 6eb1081e kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
0024f260 6eac62fb ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f2ac 6ea9e269 ZuneNativeLib!ZuneLibraryExports::Phase2Initialization+0x24c9
0024f32c 79e74d79 ZuneNativeLib!ZuneLibraryExports::QueryDatabase+0x99da
0024f3d4 664bd6af mscorwks!MethodTable::IsValueType+0x35
0024f3e8 319cec9e ZuneShell_ni+0x2d6af
0024f3f4 31a15d19 UIX_ni+0x1ec9e
0024f3f8 00000000 UIX_ni+0x65d19

We can try to interpret the crash as Managed Code Exception but let’s first to check the exception code. Google search shows that the error code c06d007f means “DelayLoad Export Missing” and this definitely has to do with some missing DLL. It is not possible to tell which one was missing from the stack trace output. Additional digging is required.

Let’s look at the raw stack. First, we can try to see whether there are any calls to LoadLibrary on thread raw stack data:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0024f8c4
    StackBase:            00250000
    StackLimit:           00249000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             000012f4 . 00001080
    RpcHandle:            00000000
    Tls Storage:          004e8a18
    PEB Address:          7ffde000
    LastErrorValue:       126
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00249000 00250000
00249000 00000000
00249004 00000000
00249008 00000000
0024900c 00000000
00249010 00000000
00249014 00000000
00249018 00000000
[...]
0024f1a0 00000000
0024f1a4 00000000
0024f1a8 c06d007f
0024f1ac 00000000
0024f1b0 00000000
0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0x11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0x15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
[...]

There are no such calls in our crash dump. Then we can try to interpret raw stack data as a byte stream to see “.dll” strings:

0:000> db 00249000 00250000
00249000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249030 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
[...]

There are no such strings except “user32.dll”.

Now we can try to interpret every double word as a pointer to a Unicode string:

0:000> dpu 00249000 00250000
[...]

There are no strings with “.dll” inside. Finally, if we try to interpret every double word as a pointer to an ASCII string we get a few references to “ZuneService.dll”:

0:000> dpa 00249000 00250000
[...]
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 "MZ."
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8 ""
0024f1dc 76fb8e8f "..t-.E."
0024f1e0 6ecb9b40 “ZuneService.dll”
0024f1e4 0024f21c “$”
0024f1e8 0024f200 “.”
0024f1ec 6ec74e2a “..^.._]..”
0024f1f0 6ecb9b40 “ZuneService.dll”
0024f1f4 6ecb9ff0 “CreateServiceInstance”
0024f1f8 0024f260 “..$”
0024f1fc 6eb1081e “.]…….e.”
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268 “..$”
0024f210 00000000
0024f214 0024f2c8 “…n ..n<.$”
0024f218 6ecbe220 “”
0024f21c 00000024
0024f220 6ecb9960 “.”
0024f224 6ecbe05c “.c.n.2.n”
0024f228 6ecb9b40 “ZuneService.dll”
0024f22c 00000001
0024f230 6ecb9ff0 “CreateServiceInstance”
0024f234 ffffffff
0024f238 00000000

If we search for 0024f1e0 pointer in dps WinDbg command output we would see that it is in a close proximity to RaiseException call and it seems that all our pointers to “ZuneService.dll” string fall into ZuneNativeLib address range:

0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0×11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0×6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268
0024f210 00000000
0024f214 0024f2c8
0024f218 6ecbe220 ZuneNativeLib!ShutdownSingletonMgr+0×15f704
0024f21c 00000024
0024f220 6ecb9960 ZuneNativeLib!ShutdownSingletonMgr+0×15ae44
0024f224 6ecbe05c ZuneNativeLib!ShutdownSingletonMgr+0×15f540
0024f228 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f22c 00000001
0024f230 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f234 ffffffff
0024f238 00000000

When examining the system it was found that ZuneService.dll was missing there indeed.

- Dmitry Vostokov @ DumpAnalysis.org -

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

Among various memory leaks leading to Insufficient Memory pattern there is so called session pool leak briefly touched in the previous kernel pool leaks post. I decided to factor it out separately because it also involves GDI handles and structures allocated per user session that has the limit on how many of them can be created and this case should rather be called Handle Limit. Such leaks can result in poor visual application behavior after some time when drawing requests are not satisfied anymore. In severe cases, when the same bugs are present in a display driver, it can result in bugchecks like

BugCheck AB: SESSION_HAS_VALID_POOL_ON_EXIT

or, if a handle allocation request was not satisfied, it may result in a NULL pointer stored somewhere with the subsequent Invalid Pointer access:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)

CONTEXT:  b791e010 -- (.cxr 0xffffffffb791e010)
eax=00000000 ebx=bc43d004 ecx=a233add8 edx=00000000 esi=bc430fff edi=00000000
eip=bfe7d380 esp=b791e3dc ebp=b791e480 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
DisplayDriver+0×3e380:
bfe7d380 8a4702          mov     al,byte ptr [edi+2]        ds:0023:00000002=??

Rather than providing examples from specific kernel dump files I created 3 Win32 applications in Visual C++ that simulate GDI leaks. All of them create GDI objects in a loop and select them into their current graphics device context (DC) on Windows Server 2003 x64 SP2. Before running them I got the following session paged pool statistics:

lkd> !poolused c

   Sorting by Session Paged Pool Consumed

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       257  3629056 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 NV          0        0       203  1347648 nVidia video driver
 Ttfd        0        0       233  1053152 TrueType Font driver
 Gh05        0        0       391  1050400 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla1        0        0       348   785088 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gcac        0        0        25   640880 Gdi glyph cache
 Gla5        0        0       631   323072 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gdrs        0        0        33   172288 Gdi GDITAG_DRVSUP
 Gla:        0        0       212   139072 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla4        0        0       487   116880 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Usti        0        0       148    97088 THREADINFO , Binary: win32k!AllocateW32Thread
 Gla8        0        0       383    91920 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla@        0        0       339    70512 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gbaf        0        0        48    67584 UNKNOWN pooltag ‘Gbaf’, please update pooltag.txt
 knlf        0        0        20    66496 UNKNOWN pooltag ‘knlf’, please update pooltag.txt
 GDev        0        0         7    57344 Gdi pdev
 Usqu        0        0       152    53504 Q , Binary: win32k!InitQEntryLookaside
 Uscu        0        0       334    53440 CURSOR , Binary: win32k!_CreateEmptyCursorObject
 Bmfd        0        0        21    50224 Font related stuff
 Uspi        0        0       153    40000 PROCESSINFO , Binary: win32k!MapDesktop
 Gfnt        0        0        47    39856 UNKNOWN pooltag ‘Gfnt’, please update pooltag.txt
 Ggb         0        0        34    39088 Gdi glyph bits
 Gh08        0        0        33    38656 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ghab        0        0       228    32832 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ovfl        0        0         1    32768 The internal pool tag table has overflowed - usually this is a result of nontagged allocations being made
 Gpff        0        0        88    27712 Gdi physical font file
 Gpfe        0        0        88    27600 UNKNOWN pooltag ‘Gpfe’, please update pooltag.txt
 thdd        0        0         1    20480 DirectDraw/3D handle manager table
 Gebr        0        0        17    19776 Gdi ENGBRUSH
 Gh0@        0        0        86    19264 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gsp         0        0        79    18960 Gdi sprite
 HT40        0        0         2    16384 UNKNOWN pooltag ‘HT40′, please update pooltag.txt
 Gpat        0        0         4    16192 UNKNOWN pooltag ‘Gpat’, please update pooltag.txt
 Ggls        0        0       169    12944 Gdi glyphset
 Glnk        0        0       371    11872 Gdi PFELINK
 Gldv        0        0         9    11248 Gdi Ldev
 Gffv        0        0        84     9408 Gdi FONTFILEVIEW
 Gfsb        0        0         1     8192 Gdi font sustitution list
 Uskt        0        0         2     7824 KBDTABLE , Binary: win32k!ReadLayoutFile
 Gh04        0        0         7     5856 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gdcf        0        0        51     5712 UNKNOWN pooltag ‘Gdcf’, please update pooltag.txt
 Gh0<        0        0        88     5632 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gglb        0        0         1     4096 Gdi temp buffer
 Ustm        0        0        30     3360 TIMER , Binary: win32k!InternalSetTimer
 Gspm        0        0        39     3120 UNKNOWN pooltag ‘Gspm’, please update pooltag.txt
 Usac        0        0        16     3056 ACCEL , Binary: win32k!_CreateAcceleratorTable
 Usqm        0        0        25     2800 QMSG , Binary: win32k!InitQEntryLookaside
 Ghas        0        0         3     2592 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Uscl        0        0        20     2128 CLASS , Binary: win32k!ClassAlloc
 Uswl        0        0         1     2032 WINDOWLIST , Binary: win32k!BuildHwndList
 Gmul        0        0        19     1520 UNKNOWN pooltag ‘Gmul’, please update pooltag.txt
 Dddp        0        0         8     1472 UNKNOWN pooltag ‘Dddp’, please update pooltag.txt
 Ggdv        0        0         8     1472 Gdi GDITAG_GDEVICE
 UsDI        0        0         4     1408 DEVICEINFO , Binary: win32k!CreateDeviceInfo
 Vtfd        0        0         4     1312 Font file/context
 Ushk        0        0        20     1280 HOTKEY , Binary: win32k!_RegisterHotKey
 Gspr        0        0         3     1264 Gdi sprite grow range
 Gtmw        0        0        13     1248 Gdi TMW_INTERNAL
 Gxlt        0        0         8     1152 Gdi Xlate
 Gpft        0        0         2      944 Gdi font table
 Uspp        0        0         5      944 PNP , Binary: win32k!AllocateAndLinkHidTLCInf
 Ussm        0        0         7      896 SMS , Binary: win32k!InitSMSLookaside
 Gdbr        0        0        10      800 Gdi driver brush realization
 Usdc        0        0         8      768 DCE , Binary: win32k!CreateCacheDC
 Usct        0        0        12      768 CHECKPT , Binary: win32k!CkptRestore
 Usim        0        0         2      736 IME , Binary: win32k!CreateInputContext
 Usci        0        0         3      720 CLIENTTHREADINFO , Binary: win32k!InitSystemThread
 Gh09        0        0         1      640 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ussy        1       80         4      608 SYSTEM , Binary: win32k!xxxDesktopThread
 Urdr        0        0         9      576 REDIRECT , Binary: win32k!SetRedirectionBitmap
 Uswd        0        0         2      576 WINDOW , Binary: win32k!xxxCreateWindowEx
 Uscb        0        0         3      544 CLIPBOARD , Binary: win32k!_ConvertMemHandle
 Gcsl        0        0         1      496 Gdi string resource script names
 Ustx        0        0        10      496 TEXT , Binary: win32k!NtUserDrawCaptionTemp
 Ussw        0        0         1      496 SWP , Binary: win32k!_BeginDeferWindowPos
 Gdev        0        0         2      480 Gdi GDITAG_DEVMODE
 Usih        0        0        10      480 IMEHOTKEY , Binary: win32k!SetImeHotKey
 Gdrv        0        0         1      368 UNKNOWN pooltag ‘Gdrv’, please update pooltag.txt
 GVdv        0        0         1      320 UNKNOWN pooltag ‘GVdv’, please update pooltag.txt
 Gmap        0        0         1      320 Gdi font map signature table
 Uskb        0        0         2      288 KBDLAYOUT , Binary: win32k!xxxLoadKeyboardLayoutEx
 Uskf        0        0         2      288 KBDFILE , Binary: win32k!LoadKeyboardLayoutFile
 Uswe        0        0         2      224 WINEVENT , Binary: win32k!_SetWinEventHook
 Gddf        0        0         2      224 Gdi ddraw driver heaps
 Gddv        0        0         2      192 Gdi ddraw driver video memory list
 GFil        0        0         2      192 Gdi engine descriptor list
 Gdwd        0        0         2       96 Gdi watchdog support objects , Binary: win32k.sys
 Usd9        0        0         1       80 DDE9 , Binary: win32k!xxxCsDdeInitialize
 Gvds        0        0         1       64 UNKNOWN pooltag ‘Gvds’, please update pooltag.txt
 GreA        0        0         1       64 UNKNOWN pooltag ‘GreA’, please update pooltag.txt
 Usse        0        0         1       48 SECURITY , Binary: win32k!SetDisconnectDesktopSecu
 Usvl        0        0         1       48 VWPL , Binary: win32k!VWPLAdd
 Mdxg        1      112         0        0 UNKNOWN pooltag ‘Mdxg’, please update pooltag.txt
 Gini        3      128         0        0 Gdi fast mutex
 Usev        1       64         0        0 EVENT , Binary: win32k!xxxPollAndWaitForSingleO
 Gdde        3      240         0        0 Gdi ddraw event
 TOTAL           9      624      6256 24408704

The first application leaks fonts:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
     hdc = BeginPaint(hWnd, &ps);
     while (true)
     {
        HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
        SelectObject(ps.hdc, hf);
     }
     EndPaint(hWnd, &ps);
     break;

We clearly see the leak as the greatly increased number of allocations for ”Gla:” pool tag:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 Gla:        0        0     10194  6687264 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 BIG         0        0       248  3690496 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 NV          0        0       203  1347648 nVidia video driver
 Gh05        0        0       396  1057888 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ttfd        0        0       226  1043264 TrueType Font driver

The second application leaks fonts and pens:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
     hdc = BeginPaint(hWnd, &ps);
     while (true)
     {
       HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
       HPEN hp = CreatePen(0, 10, RGB(10, 20, 30));
       SelectObject(ps.hdc, hf);
       SelectObject(ps.hdc, hp);
     }
     EndPaint(hWnd, &ps);
     break;

We see that the roughly the same number of allocations is split between ”Gla:” and “Gh0@” pool tags:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       262  3874816 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 Gla:        0        0      5203  3413168 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 NV          0        0       203  1347648 nVidia video driver
 Gh0@        0        0      5077  1137248 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ttfd        0        0       233  1053152 TrueType Font driver

The third program leaks fonts, pens and brushes:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
   hdc = BeginPaint(hWnd, &ps);
   while (true)
   {
     HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
     HPEN hp = CreatePen(0, 10, RGB(10, 20, 30));
     HBRUSH hb = CreateSolidBrush(RGB(10, 20, 30));
     SelectObject(ps.hdc, hf);
     SelectObject(ps.hdc, hp);
     SelectObject(ps.hdc, hb);
   }
   EndPaint(hWnd, &ps);
   break;

Now we see that the same number of allocations is almost equally split between ”Gla:”, “Gh0@” and “Gla@” pool tags:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       262  3874816 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 Gla:        0        0      3539  2321584 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 NV          0        0       203  1347648 nVidia video driver
 Ttfd        0        0       233  1053152 TrueType Font driver
 Gh05        0        0       392  1052768 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla1        0        0       353   796368 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gh0@        0        0      3414   764736 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla@        0        0      3665   762320 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gcac        0        0        25   640880 Gdi glyph cache

When the certain amount of handles is reached all subsequent GDI Create calls fail and other applications start showing various visual defects. Print screen operation also fails with insufficient memory message.
                                                                                                                                                                                                        
- Dmitry Vostokov @ DumpAnalysis.org -

Another pattern that occurs frequently is Hardware Error. This can be internal CPU malfunction due to overheating, RAM or hard disk I/O problem. It usually results in the appropriate bugcheck and the most frequent one is the 6th from the top of Bug Check Frequency Table:

• BugCheck 9C: MACHINE_CHECK_EXCEPTION

Other relevant bugchecks include:

• BugCheck 7B: INACCESSIBLE_BOOT_DEVICE

• BugCheck 77: KERNEL_STACK_INPAGE_ERROR

• BugCheck 7A: KERNEL_DATA_INPAGE_ERROR

Another bugcheck from this category can also be triggered on purpose to get a crash dump of a hanging or slow system:

• BugCheck 80: NMI_HARDWARE_FAILURE

Please also note that other popular bugchecks like  

• BugCheck 7F: UNEXPECTED_KERNEL_MODE_TRAP

• BugCheck 50: PAGE_FAULT_IN_NONPAGED_AREA

can result from RAM problems but we should try to find a software cause first.

Sometimes the following bugchecks like

• BugCheck 7E: SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

report EXCEPTION_DOESNOT_MATCH_CODE where read or write address doesn’t correspond to faulted instruction at EIP:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: bf802671, The address that the exception occurred at
Arg3: f10b8c74, Exception Record Address
Arg4: f10b88c4, Context Record Address

FAULTING_IP:
driver!AcquireSemaphoreShared+4
bf802671 90 nop

EXCEPTION_RECORD: f10b8c74 -- (.exr fffffffff10b8c74)
ExceptionAddress: bf802671 (driver!AcquireSemaphoreShared+0x00000004)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 0000000c
Attempt to write to address 0000000c

CONTEXT: f10b88c4 -- (.cxr fffffffff10b88c4)
eax=884d2d01 ebx=0000000c ecx=00000000 edx=80010031 esi=8851ef60 edi=bc3846d4
eip=bf802671 esp=f10b8d3c ebp=f10b8d70 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206
driver!AcquireSemaphoreShared+0x4:
bf802671 90 nop
Resetting default scope

WRITE_ADDRESS: 0000000c

EXCEPTION_DOESNOT_MATCH_CODE: This indicates a hardware error.
Instruction at bf802671 does not read/write to 0000000c

Code mismatch can also happen in user mode but from my experience it usually results from improper Hooked Function or similar corruption: 

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c848768 (ntdll!_LdrpInitialize+0x00000184)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 0

DEFAULT_BUCKET_ID: CODE_ADDRESS_MISMATCH

WRITE_ADDRESS: f774f120

FAULTING_IP:
ntdll!_LdrpInitialize+184
7c848768 cc int 3

EXCEPTION_DOESNOT_MATCH_CODE: This indicates a hardware error.
Instruction at 7c848768 does not read/write to f774f120

STACK_TEXT:
0012fd14 7c8284c5 0012fd28 7c800000 00000000 ntdll!_LdrpInitialize+0x184
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x25

In such cases EIP might point to the middle of the expected instruction (Wild Code):

FAULTING_IP:
+59c3659
059c3659 86990508f09b xchg bl,byte ptr [ecx-640FF7FBh]

Here is an example of the real hardware error (note the concatenated error code for bugcheck 0×9C):

MACHINE_CHECK_EXCEPTION (9c)
A fatal Machine Check Exception has occurred.
KeBugCheckEx parameters;
    x86 Processors
        If the processor has ONLY MCE feature available (For example Intel
        Pentium), the parameters are:
        1 - Low  32 bits of P5_MC_TYPE MSR
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of P5_MC_ADDR MSR
        4 - Low  32 bits of P5_MC_ADDR MSR
        If the processor also has MCA feature available (For example Intel
        Pentium Pro), the parameters are:
        1 - Bank number
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
        4 - Low  32 bits of MCi_STATUS MSR for the MCA bank that had the error
    IA64 Processors
        1 - Bugcheck Type
            1 - MCA_ASSERT
            2 - MCA_GET_STATEINFO
                SAL returned an error for SAL_GET_STATEINFO while processing MCA.
            3 - MCA_CLEAR_STATEINFO
                SAL returned an error for SAL_CLEAR_STATEINFO while processing MCA.
            4 - MCA_FATAL
                FW reported a fatal MCA.
            5 - MCA_NONFATAL
                SAL reported a recoverable MCA and we don't support currently
                support recovery or SAL generated an MCA and then couldn't
                produce an error record.
            0xB - INIT_ASSERT
            0xC - INIT_GET_STATEINFO
                  SAL returned an error for SAL_GET_STATEINFO while processing INIT event.
            0xD - INIT_CLEAR_STATEINFO
                  SAL returned an error for SAL_CLEAR_STATEINFO while processing INIT event.
            0xE - INIT_FATAL
                  Not used.
        2 - Address of log
        3 - Size of log
        4 - Error code in the case of x_GET_STATEINFO or x_CLEAR_STATEINFO
    AMD64 Processors
        1 - Bank number
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
        4 - Low  32 bits of MCi_STATUS MSR for the MCA bank that had the error
Arguments:
Arg1: 00000000
Arg2: 808a07a0
Arg3: be000300
Arg4: 1008081f

Debugging Details:
------------------

   NOTE:  This is a hardware error.  This error was reported by the CPU
   via Interrupt 18.  This analysis will provide more information about
   the specific error.  Please contact the manufacturer for additional
   information about this error and troubleshooting assistance.

   This error is documented in the following publication:

      - IA-32 Intel(r) Architecture Software Developer's Manual
        Volume 3: System Programming Guide

   Bit Mask:

    MA                           Model Specific       MCA
 O  ID      Other Information      Error Code     Error Code
VV  SDP ___________|____________ _______|_______ _______|______
AEUECRC|                        |               |             
LRCNVVC|                        |               |             
^^^^^^^|                        |               |              
   6         5         4         3         2         1
3210987654321098765432109876543210987654321098765432109876543210
----------------------------------------------------------------
1011111000000000000000110000000000010000000010000000100000011111 

VAL   - MCi_STATUS register is valid
        Indicates that the information contained within the IA32_MCi_STATUS
        register is valid.  When this flag is set, the processor follows the
        rules given for the OVER flag in the IA32_MCi_STATUS register when
        overwriting previously valid entries.  The processor sets the VAL
        flag and software is responsible for clearing it.

UC    - Error Uncorrected
        Indicates that the processor did not or was not able to correct the
        error condition.  When clear, this flag indicates that the processor
        was able to correct the error condition.

EN    - Error Enabled
        Indicates that the error was enabled by the associated EEj bit of the
        IA32_MCi_CTL register.

MISCV - IA32_MCi_MISC Register Valid
        Indicates that the IA32_MCi_MISC register contains additional
        information regarding the error.  When clear, this flag indicates
        that the IA32_MCi_MISC register is either not implemented or does
        not contain additional information regarding the error.

ADDRV - IA32_MCi_ADDR register valid
        Indicates that the IA32_MCi_ADDR register contains the address where
        the error occurred.

PCC   - Processor Context Corrupt
        Indicates that the state of the processor might have been corrupted
        by the error condition detected and that reliable restarting of the
        processor may not be possible.

BUSCONNERR - Bus and Interconnect Error   BUS{LL}_{PP}_{RRRR}_{II}_{T}_err
        These errors match the format 0000 1PPT RRRR IILL

   Concatenated Error Code:
   --------------------------
   _VAL_UC_EN_MISCV_ADDRV_PCC_BUSCONNERR_1F

   This error code can be reported back to the manufacturer.
   They may be able to provide additional information based upon
   this error.  All questions regarding STOP 0x9C should be
   directed to the hardware manufacturer.

BUGCHECK_STR:  0x9C_IA32_GenuineIntel

DEFAULT_BUCKET_ID:  DRIVER_FAULT

PROCESS_NAME:  Idle

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 80a7fbd8 to 8087b6be

STACK_TEXT: 
f773d280 80a7fbd8 0000009c 00000000 f773d2b0 nt!KeBugCheckEx+0x1b
f773d3b4 80a7786f f7737fe0 00000000 00000000 hal!HalpMcaExceptionHandler+0x11e
f773d3b4 f75a9ca2 f7737fe0 00000000 00000000 hal!HalpMcaExceptionHandlerWrapper+0x77
f78c6d50 8083abf2 00000000 0000000e 00000000 intelppm!AcpiC1Idle+0x12
f78c6d54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa

- Dmitry Vostokov @ DumpAnalysis.org -