Due to a request I opened AERA subforum:
http://www.dumpanalysis.org/forum/viewforum.php?f=29
Everyone is welcome to comment and add thoughts.
IBCDAS code name has been changed to AERA 
- Dmitry Vostokov @ DumpAnalysis.org -
I decided to post selected crash dumps for some case studies and patterns after getting feedback from readers of my MDAA book. They are custom made from my toy programs and preprocessed to clear sensitive information that might have leaked from my home computers otherwise. They can be found at the following public FTP address:
The first example was published today for Missing Component (static linking) pattern. I also created the following page to list them all including the ones that I add for some previously published patterns:
- Dmitry Vostokov @ DumpAnalysis.org -
Previously I introduced Missing Component pattern where the example and emphasis was on dynamically loaded modules. In this part I cover statically linked modules. Failure for a loader to find one of them results in a software exception. The most frequent of them are (numbers were taken from Google search):
C0000142 918
C0000143 919
C0000145 1,530
C0000135 24,900
0:001> !error c0000142
Error code: (NTSTATUS) 0xc0000142 (3221225794) - {DLL Initialization Failed} Initialization of the dynamic link library %hs failed. The process is terminating abnormally.
0:001> !error c0000143
Error code: (NTSTATUS) 0xc0000143 (3221225795) - {Missing System File} The required system file %hs is bad or missing.
0:001> !error c0000145
Error code: (NTSTATUS) 0xc0000145 (3221225797) - {Application Error} The application failed to initialize properly (0x%lx). Click on OK to terminate the application.
0:000> !error c0000135
Error code: (NTSTATUS) 0xc0000135 (3221225781) - {Unable To Locate Component} This application has failed to start because %hs was not found. Re-installing the application may fix this problem.
In this part I only consider user mode exceptions. If we have a default debugger configured it will usually save a crash dump. To model this problem I modified one of my applications by changing all occurrences of KERNEL32.DLL to KERNEL32.DL using Visual Studio Binary Editor. CDB was configured as a default postmortem debugger (see Custom postmortem debuggers on Vista). When the application was launched CDB attached to it and saved a crash dump. If we open it in WinDbg we get characteristic Special Stack Trace involving loader functions:
Loading Dump File [C:\UserDumps\CDAPatternMissingComponent.dmp]
User Mini Dump File with Full Memory: Only application data is available
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Vista Version 6000 MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Debug session time: Thu Jun 12 12:03:28.000 2008 (GMT+1)
System Uptime: 1 days 8:46:23.167
Process Uptime: 0 days 0:00:48.000
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(da4.f60): Wake debugger - code 80000007 (first/second chance not available)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!_LdrpInitialize+0x6d:
77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000
0:000> kL
ChildEBP RetAddr
0017fd00 77b937ea ntdll!_LdrpInitialize+0×6d
0017fd10 00000000 ntdll!LdrInitializeThunk+0×10
Verbose analysis command doesn’t give us an indication of what had happened so we need to dig further:
0:000> !analyze -v
[...]
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000007 (Wake debugger)
ExceptionFlags: 00000000
NumberParameters: 0
BUGCHECK_STR: 80000007
PROCESS_NAME: StackOverflow.exe
ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened} the system debugger was awakened by an interrupt.
NTGLOBALFLAG: 400
APPLICATION_VERIFIER_FLAGS: 0
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 da4.f60 Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00000f60
DEFAULT_BUCKET_ID: APPLICATION_HANG_BusyHang
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BusyHang
LAST_CONTROL_TRANSFER: from 77b937ea to 77bcf1d1
FAULTING_THREAD: 00000000
STACK_TEXT:
0017fd00 77b937ea 0017fd24 77b60000 00000000 ntdll!_LdrpInitialize+0x6d
0017fd10 00000000 0017fd24 77b60000 00000000 ntdll!LdrInitializeThunk+0x10
FOLLOWUP_IP:
ntdll!_LdrpInitialize+6d
77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h]
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!_LdrpInitialize+6d
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4549bdf8
STACK_COMMAND: ~0s ; kb
BUCKET_ID: 80000007_ntdll!_LdrpInitialize+6d
FAILURE_BUCKET_ID: ntdll.dll!_LdrpInitialize_80000007_APPLICATION_HANG_BusyHang
Followup: MachineOwner
Last event and error code are not helpful too:
0:000> .lastevent
Last event: da4.f60: Wake debugger - code 80000007 (first/second chance not available)
debugger time: Thu Jun 12 15:04:38.917 2008 (GMT+1)
0:000> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0
Let’s search thread raw stack data for any signs of exceptions:
0:000> !teb
TEB at 7efdd000
ExceptionList: 0017fcf0
StackBase: 00180000
StackLimit: 0017e000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7efdd000
EnvironmentPointer: 00000000
ClientId: 00000da4 . 00000f60
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7efde000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 0017e000 00180000
[...]
0017f8d8 7efdd000
0017f8dc 0017f964
0017f8e0 77c11c78 ntdll!_except_handler4
0017f8e4 00000000
0017f8e8 0017f988
0017f8ec 0017f900
0017f8f0 77ba1ddd ntdll!RtlCallVectoredContinueHandlers+0x15
0017f8f4 0017f988
0017f8f8 0017f9d8
0017f8fc 77c40370 ntdll!RtlpCallbackEntryList
0017f900 0017f970
0017f904 77ba1db5 ntdll!RtlDispatchException+0×11f
0017f908 0017f988
0017f90c 0017f9d8
0017f910 7efde028
0017f914 00000001
0017f918 77630000 kernel32!_imp___aullrem <PERF> (kernel32+0×0)
0017f91c 00000001
0017f920 776ced81 kernel32!_DllMainCRTStartupForGS2+0×10
0017f924 0017f938
0017f928 7765d4d9 kernel32!BaseDllInitialize+0×18
0017f92c 76042340 user32!$$VProc_ImageExportDirectory
0017f930 00000001
0017f934 00000000
0017f938 0017f9e0
0017f93c 77b8f890 ntdll!LdrpSnapThunk+0xc9
0017f940 0040977a StackOverflow+0×977a
0017f944 0000030b
0017f948 76030000 user32!_imp__RegSetValueExW <PERF> (user32+0×0)
0017f94c 76042f94 user32!$$VProc_ImageExportDirectory+0xc54
0017f950 77bb8881 ntdll!LdrpSnapThunk+0×40d
0017f954 0017bb30
0017f958 00409770 StackOverflow+0×9770
0017f95c 00881a50
0017f960 004098b2 StackOverflow+0×98b2
0017f964 77bac282 ntdll!ZwRaiseException+0×12
0017f968 00180000
0017f96c 0017fc48
0017f970 0017fd00
0017f974 77bac282 ntdll!ZwRaiseException+0×12
0017f978 77b7ee72 ntdll!KiUserExceptionDispatcher+0×2a
0017f97c 0017f988 ; exception record
0017f980 0017f9d8 ; exception context
0017f984 00000000
0017f988 c0000135
0017f98c 00000001
0017f990 00000000
0017f994 77bcf1d1 ntdll!_LdrpInitialize+0×6d
0017f998 00000000
0017f99c 77c11c78 ntdll!_except_handler4
0017f9a0 77b8dab8 ntdll!RtlpRunTable+0×218
0017f9a4 fffffffe
0017f9a8 77ba2515 ntdll!vDbgPrintExWithPrefixInternal+0×214
0017f9ac 77ba253b ntdll!DbgPrintEx+0×1e
0017f9b0 77b7f356 ntdll! ?? ::FNODOBFM::`string’
0017f9b4 00000055
0017f9b8 00000003
0017f9bc 77b809c2 ntdll! ?? ::FNODOBFM::`string’
0017f9c0 0017fc9c
0017f9c4 00000001
0017f9c8 0017fd00
0017f9cc 77bcf28e ntdll!_LdrpInitialize+0×12a
0017f9d0 00000055
0017f9d4 75ce3cf9
0017f9d8 0001003f
0017f9dc 00000000
0017f9e0 00000000
0017f9e4 00000000
0017f9e8 00000000
0017f9ec 00000000
0017f9f0 00000000
0017f9f4 0000027f
[…]
We see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get one:
0:000> .exr 0017f988
ExceptionAddress: 77bcf1d1 (ntdll!_LdrpInitialize+0x0000006d)
ExceptionCode: c0000135
ExceptionFlags: 00000001
NumberParameters: 0
Error c0000135 means that the loader was unable to locate a component. Now we try to examine the same raw stack data for any string patterns. For example, the following UNICODE pattern is clearly visible:
0017f2fc 00000000
0017f300 00880ec4
0017f304 77b910d7 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x344
0017f308 00000000
0017f30c 43000043
0017f310 0042002a
0017f314 0017f33c
0017f318 00000000
0017f31c 00000002
0017f320 00000008
0017f324 00000000
0017f328 0000008c
0017f32c 000a0008
0017f330 77b91670 ntdll!`string'
0017f334 00b92bd6
0017f338 0017f5d4
0017f33c 003a0043
0017f340 0050005c
0017f344 006f0072
0017f348 00720067
0017f34c 006d0061
0017f350 00460020
0017f354 006c0069
0017f358 00730065
0017f35c 00280020
0017f360 00380078
0017f364 00290036
0017f368 0043005c
0017f36c 006d006f
0017f370 006f006d
0017f374 0020006e
0017f378 00690046
0017f37c 0065006c
0017f380 005c0073
0017f384 006f0052
0017f388 00690078
0017f38c 0020006f
0017f390 00680053
0017f394 00720061
0017f398 00640065
0017f39c 0044005c
0017f3a0 004c004c
0017f3a4 00680053
0017f3a8 00720061
0017f3ac 00640065
0017f3b0 004b005c
0017f3b4 00520045
0017f3b8 0045004e
0017f3bc 0033004c
0017f3c0 002e0032
0017f3c4 006c0064
0017f3c8 00000000
0017f3cc 00000000
It is a path to DLL that was probably missing:
0:000> du 0017f33c
0017f33c "C:\Program Files (x86)\Common Fi"
0017f37c "les\Roxio Shared\DLLShared\KERNE"
0017f3bc "L32.dl"
I think the loader was trying to find KERNEL32.dl following the DLL search order and this was the last path element:
0:000> !peb
PEB at 7efde000
InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: Yes
ImageBaseAddress: 00400000
Ldr 77c40080
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 00881ad0 . 008831b8
Ldr.InLoadOrderModuleList: 00881a50 . 00883dc8
Ldr.InMemoryOrderModuleList: 00881a58 . 00883dd0
Base TimeStamp Module
[...]
Environment: 00881de8
[...]
Path=C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
[…]
In similar situations !dlls command might help that shows the load order (-l switch) and points to the last processed DLL:
0:001> !dlls -l
0x004740e8: C:\Program Files\Application\Application.exe
Base 0x012a0000 EntryPoint 0x012b0903 Size 0x00057000
Flags 0x00004010 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_ENTRY_PROCESSED
0x00474158: C:\Windows\SysWOW64\ntdll.dll
Base 0x77d00000 EntryPoint 0x00000000 Size 0x00160000
Flags 0x00004014 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_ENTRY_PROCESSED
0x00474440: C:\Windows\syswow64\kernel32.dll
Base 0x77590000 EntryPoint 0x775a1f3e Size 0x00110000
Flags 0x00084014 LoadCount 0x0000ffff TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_ENTRY_PROCESSED
LDRP_PROCESS_ATTACH_CALLED
[…]
0x00498ff8: C:\Windows\WinSxS\x86_microsoft.windows.common-controls_...\comctl32.dll
Base 0x74d90000 EntryPoint 0x74dc43e5 Size 0x0019e000
Flags 0x100c4014 LoadCount 0x00000003 TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_ENTRY_PROCESSED
LDRP_DONT_CALL_FOR_THREADS
LDRP_PROCESS_ATTACH_CALLED
0x004991b8: C:\Windows\WinSxS\x86_microsoft.vc80.mfcloc_...\MFC80ENU.DLL
Base 0x71b10000 EntryPoint 0x00000000 Size 0x0000e000
Flags 0x10004014 LoadCount 0x00000001 TlsIndex 0x00000000
LDRP_IMAGE_DLL
LDRP_ENTRY_PROCESSED
If it is difficult to identify what had really happened in crash dumps we can enable loader snaps using gflags and run the application under a debugger. For example, for notepad.exe we have:
Microsoft (R) Windows Debugger Version 6.8.0004.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: C:\Windows\notepad.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00000000`ffac0000 00000000`ffaef000 notepad.exe
ModLoad: 00000000`779b0000 00000000`77b2a000 ntdll.dll
LDR: NEW PROCESS
Image Path: C:\Windows\notepad.exe (notepad.exe)
Current Directory: C:\Program Files\Debugging Tools for Windows 64-bit\
Search Path: C:\Windows; C:\Windows\system32; C:\Windows\system; C:\Windows;.; C:\Program Files\Debugging Tools for Windows 64-bit\winext\arcade; C:\Windows\system32;C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
LDR: LdrLoadDll, loading kernel32.dll from
ModLoad: 00000000`777a0000 00000000`778d1000 C:\Windows\system32\kernel32.dll
LDR: kernel32.dll bound to ntdll.dll
LDR: kernel32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from kernel32.dll
LDR: LdrGetProcedureAddress by NAME - BaseThreadInitThunk
[3d8,1278] LDR: Real INIT LIST for process C:\Windows\notepad.exe pid 984 0x3d8
[3d8,1278] C:\Windows\system32\kernel32.dll init routine 00000000777DC960
[3d8,1278] LDR: kernel32.dll loaded - Calling init routine at 00000000777DC960
LDR: notepad.exe bound to ADVAPI32.dll
ModLoad: 000007fe`fe520000 000007fe`fe61f000 C:\Windows\system32\ADVAPI32.dll
LDR: ADVAPI32.dll bound to ntdll.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from ADVAPI32.dll
LDR: ADVAPI32.dll bound to KERNEL32.dll
LDR: ADVAPI32.dll has stale binding to KERNEL32.dll
LDR: ADVAPI32.dll bound to ntdll.dll via forwarder(s) from kernel32.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind KERNEL32.dll from ADVAPI32.dll
LDR: LdrGetProcedureAddress by NAME - RtlAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlReAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlEncodePointer
LDR: LdrGetProcedureAddress by NAME - RtlDecodePointer
LDR: LdrGetProcedureAddress by NAME - RtlSizeHeap
LDR: LdrGetProcedureAddress by NAME - RtlDeleteCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlEnterCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlLeaveCriticalSection
LDR: ADVAPI32.dll bound to RPCRT4.dll
[...]
This technique only works for native platform loader snaps. For example, it doesn’t show loader snaps for 32-bit modules loaded under WOW64 (to the best of my knowledge):
Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: C:\Apps\StackOverflow.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00418000 StackOverflow.exe
ModLoad: 77b60000 77cb0000 ntdll.dll
LDR: NEW PROCESS
Image Path: C:\Apps\StackOverflow.exe (StackOverflow.exe)
[...]
LDR: Loading (STATIC, NON_REDIRECTED) C:\Windows\system32\wow64cpu.dll
LDR: wow64cpu.dll bound to ntdll.dll
LDR: wow64cpu.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from wow64cpu.dll
LDR: wow64cpu.dll bound to wow64.dll
LDR: wow64cpu.dll has stale binding to wow64.dll
LDR: Stale Bind wow64.dll from wow64cpu.dll
LDR: wow64.dll has stale binding to wow64cpu.dll
LDR: Stale Bind wow64cpu.dll from wow64.dll
LDR: Refcount wow64cpu.dll (1)
LDR: Refcount wow64.dll (2)
LDR: Refcount wow64win.dll (1)
LDR: Refcount wow64.dll (3)
LDR: LdrGetProcedureAddress by NAME - Wow64LdrpInitialize
[...]
ModLoad: 77630000 77740000 C:\Windows\syswow64\kernel32.dll
ModLoad: 76030000 76100000 C:\Windows\syswow64\USER32.dll
ModLoad: 775a0000 77630000 C:\Windows\syswow64\GDI32.dll
ModLoad: 76d00000 76dbf000 C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 76df0000 76ee0000 C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75d60000 75dc0000 C:\Windows\syswow64\Secur32.dll
(1ec.1290): Unknown exception - code c0000135 (first chance)
(1ec.1290): Unknown exception - code c0000135 (!!! second chance !!!)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!_LdrpInitialize+0×6d:
77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000
The dump file that I used was modified to remove sensitive information (see Data Hiding in Crash Dumps for this technique). It can be downloaded from FTP to play with:
ftp://dumpanalysis.org/pub/CDAPatternMissingComponent.zip
- Dmitry Vostokov @ DumpAnalysis.org -
Suppose we want to send a complete memory dump to a vendor but want to remove certain sensitive details or perhaps the whole process or image from it. In this case we can use f WinDbg command (virtual addresses) or fp (physical addresses) to fill pages with zeroes. Let’s open a complete memory dump and erase environment variables for a process:
kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 0014a000 ObjectTable: fffffa8000000c10 HandleCount: 730.
Image: System
PROCESS fffffadfe6edc040
SessionId: none Cid: 0130 Peb: 7fffffdf000 ParentCid: 0004
DirBase: 34142000 ObjectTable: fffffa80009056d0 HandleCount: 19.
Image: smss.exe
[...]
PROCESS fffffadfe67905a0
SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc
DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55.
Image: SystemDump.exe
kd> .process /r /p fffffadfe7287610
Implicit process is now fffffadf`e7287610
Loading User Symbols
kd> !peb
PEB at 000007fffffd4000
[...]
Environment: 0000000000010000
kd> dd 10000
00000000`00010000 004c0041 0055004c 00450053 00530052
00000000`00010010 00520050 0046004f 004c0049 003d0045
00000000`00010020 003a0043 0044005c 0063006f 006d0075
00000000`00010030 006e0065 00730074 00610020 0064006e
00000000`00010040 00530020 00740065 00690074 0067006e
00000000`00010050 005c0073 006c0041 0020006c 00730055
00000000`00010060 00720065 002e0073 00320057 0033004b
00000000`00010070 00410000 00500050 00410044 00410054
kd> f 10000 10000+1000 0
Filled 0x1000 bytes
kd> dd 10000
00000000`00010000 00000000 00000000 00000000 00000000
00000000`00010010 00000000 00000000 00000000 00000000
00000000`00010020 00000000 00000000 00000000 00000000
00000000`00010030 00000000 00000000 00000000 00000000
00000000`00010040 00000000 00000000 00000000 00000000
00000000`00010050 00000000 00000000 00000000 00000000
00000000`00010060 00000000 00000000 00000000 00000000
00000000`00010070 00000000 00000000 00000000 00000000
Now we can save the modified complete dump file:
kd> .dump /f c:\Dumps\SecuredDump.dmp
If we want to find and erase read-write pages, for example, we can use !vad WinDbg command to get the description of virtual address ranges:
kd> !process
PROCESS fffffadfe67905a0
SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc
DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55.
Image: SystemDump.exe
VadRoot fffffadfe6f293e0 Vads 65 Clone 0 Private 388. Modified 84. Locked 0.
DeviceMap fffffa80020777c0
Token fffffa80008e5b50
ElapsedTime 00:00:06.265
UserTime 00:00:00.031
KernelTime 00:00:00.062
QuotaPoolUsage[PagedPool] 113464
QuotaPoolUsage[NonPagedPool] 5152
Working Set Sizes (now,min,max) (1429, 50, 345) (5716KB, 200KB, 1380KB)
PeakWorkingSetSize 1429
VirtualSize 61 Mb
PeakVirtualSize 63 Mb
PageFaultCount 1555
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 471
kd> !vad fffffadfe6f293e0
VAD level start end commit
fffffadfe682bdf0 ( 6) 10 10 1 Private READWRITE
fffffadfe73a0e10 ( 5) 20 20 1 Private READWRITE
fffffadfe73a0dd0 ( 4) 30 12f 8 Private READWRITE
fffffadfe71a4770 ( 5) 130 134 0 Mapped READONLY
fffffadfe781bbe0 ( 3) 140 141 0 Mapped READONLY
[…]
fffffadfe772d630 (-2) 7fffffdc 7fffffdd 2 Private READWRITE
fffffadfe788e180 (-1) 7fffffde 7fffffdf 2 Private READWRITE
Total VADs: 65 average level: 66076419 maximum depth: -1
In the output start and end columns refer to virtual page numbers (VPN). To get an address we need to multiply by 0×1000, for example, 7fffffde000.
Filling memory with zeroes to hide data with subsequent saving of a modified crash dump is applicable to user dumps too. Please also check for additional security-related flags in .dump command:
Another application for data hiding and modification could be the creation of the customized crash dumps for digital forensics exercises and contests.
- Dmitry Vostokov @ DumpAnalysis.org -
I’ve just found that although I covered Stack Overflow in kernel mode I didn’t do this for user mode. In fact this is one of the simplest patterns to see in crash dumps. It has its own characteristic exception code and stack trace:
FAULTING_IP:
StackOverflow!SoFunction+27
00401317 6a00 push 0
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00401300 (StackOverflow!SoFunction+0x00000010)
ExceptionCode: c00000fd (Stack overflow)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 00082ffc
0:000> kL
ChildEBP RetAddr
00083000 00401317 StackOverflow!SoFunction+0x10
00083010 00401317 StackOverflow!SoFunction+0×27
00083020 00401317 StackOverflow!SoFunction+0×27
00083030 00401317 StackOverflow!SoFunction+0×27
00083040 00401317 StackOverflow!SoFunction+0×27
00083050 00401317 StackOverflow!SoFunction+0×27
00083060 00401317 StackOverflow!SoFunction+0×27
00083070 00401317 StackOverflow!SoFunction+0×27
00083080 00401317 StackOverflow!SoFunction+0×27
00083090 00401317 StackOverflow!SoFunction+0×27
000830a0 00401317 StackOverflow!SoFunction+0×27
000830b0 00401317 StackOverflow!SoFunction+0×27
000830c0 00401317 StackOverflow!SoFunction+0×27
000830d0 00401317 StackOverflow!SoFunction+0×27
000830e0 00401317 StackOverflow!SoFunction+0×27
000830f0 00401317 StackOverflow!SoFunction+0×27
00083100 00401317 StackOverflow!SoFunction+0×27
00083110 00401317 StackOverflow!SoFunction+0×27
00083120 00401317 StackOverflow!SoFunction+0×27
00083130 00401317 StackOverflow!SoFunction+0×27
There could be thousands of stack frames:
0:000> kL 2000
[...]
000a2fa0 00401317 StackOverflow!SoFunction+0x27
000a2fb0 00401317 StackOverflow!SoFunction+0x27
000a2fc0 00401317 StackOverflow!SoFunction+0x27
000a2fd0 00401317 StackOverflow!SoFunction+0x27
000a2fe0 00401317 StackOverflow!SoFunction+0x27
000a2ff0 00401317 StackOverflow!SoFunction+0x27
To reach the bottom and avoid over scrolling we can dump the raw stack data, search for the end of the repeating pattern of StackOverflow!SoFunction+0×27 and try to manually reconstruct the bottom of the stack trace:
0:000> !teb
TEB at 7efdd000
ExceptionList: 0017fdf0
StackBase: 00180000
StackLimit: 00081000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7efdd000
EnvironmentPointer: 00000000
ClientId: 00001dc4 . 00001b74
RpcHandle: 00000000
Tls Storage: 7efdd02c
PEB Address: 7efde000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 00081000 00180000
[...]
0017fc74 00401317 StackOverflow!SoFunction+0×27
0017fc78 00000000
0017fc7c a3a8ea65
0017fc80 0017fc90
0017fc84 00401317 StackOverflow!SoFunction+0×27
0017fc88 10001843
0017fc8c a3a8ea95
0017fc90 0017fca0
0017fc94 00401317 StackOverflow!SoFunction+0×27
0017fc98 0017fcb8
0017fc9c a3a8ea85
0017fca0 0017fcb0
0017fca4 00401317 StackOverflow!SoFunction+0×27
0017fca8 00000003
0017fcac a3a8eab5
0017fcb0 0017fcc0
0017fcb4 00401317 StackOverflow!SoFunction+0×27
0017fcb8 76c68738 user32!_EndUserApiHook+0×11
0017fcbc a3a8eaa5
0017fcc0 0017fcd0
0017fcc4 00401317 StackOverflow!SoFunction+0×27
0017fcc8 76c6a6cc user32!DefWindowProcW+0×94
0017fccc a3a8ead5
0017fcd0 0017fce0
0017fcd4 00401317 StackOverflow!SoFunction+0×27
0017fcd8 0037311e
0017fcdc a3a8eac5
0017fce0 0017fcf0
0017fce4 00401317 StackOverflow!SoFunction+0×27
0017fce8 0017fcd0
0017fcec a3a8eaf5
0017fcf0 0017fd00
0017fcf4 00401317 StackOverflow!SoFunction+0×27
0017fcf8 76c6ad0f user32!NtUserBeginPaint+0×15
0017fcfc a3a8eae5
0017fd00 0017fd5c
0017fd04 00401272 StackOverflow!WndProc+0xe2
0017fd08 00401190 StackOverflow!WndProc
0017fd0c 00000003
0017fd10 cf017ada
[…]
We use the extended version of k WinDbg command and supply EBP, ESP and EIP to see in what function it started:
0:000> r
eax=a3b739e5 ebx=00000000 ecx=ac430000 edx=ffefd944 esi=0037311e edi=00000000
eip=00401300 esp=00082ff8 ebp=00083000 iopl=0 nv up ei ng nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010282
StackOverflow!SoFunction+0×10:
00401300 89442404 mov dword ptr [esp+4],eax ss:002b:00082ffc=00000000
0:000> k L=0017fcf0 00082ff8 00401300
ChildEBP RetAddr
0017fcb0 00401317 StackOverflow!SoFunction+0×10
0017fd00 00401272 StackOverflow!SoFunction+0×27
0017fd5c 76c687af StackOverflow!WndProc+0xe2
0017fd88 76c68936 user32!InternalCallWinProc+0×23
0017fe00 76c6a571 user32!UserCallWinProcCheckWow+0×109
0017fe5c 76c6a5dd user32!DispatchClientMessage+0xe0
0017fe98 77ccee2e user32!__fnDWORD+0×2b
0017fedc 0040107d ntdll!KiUserCallbackDispatcher+0×2e
0017ff08 0040151e StackOverflow!wWinMain+0×7d
00402ba0 20245c8b StackOverflow!__tmainCRTStartup+0×176
- Dmitry Vostokov @ DumpAnalysis.org -
If you remember I promised to review this CD:
The first Memory Analysis CD album
Finally it arrived by post and here it is:
I’ve just finished listening to it. It is wonderful! Clearly belongs to my Music for Debugging collection. Here is my commentary (italics) on track titles:
1. Chameleon
It crashes then hangs then spikes then leaks. You never know what happens next…
2. Silence Before The Storm
Waiting for the problem to start dumping memory
3. Appearances
Multiple occurrences of the issue ease crash dump collection
4. Memory Analysis
Memory dump analysis activity
5. Voices From Heaven
It’s my blog full of crash dump analysis patterns
6. Figure In The Fog
It’s that DLL! I see it clearly!!!
7. Diamond Tear
Customer is happy
8. The Toy Soldier
WinDbg
9. Tears In Dragon’s Eyes
Operating system vendor is happy too
10. Forgotten Song
No pasaran (bugs)
11. Skies
We are debugging gods!
Enjoy :-)
- Dmitry Vostokov @ DumpAnalysis.org -
Due to the request from blog readers I made my old Practical Foundations of Debugging lectures available in PDF format:
x64 version:
I keep both versions (HTMP and PDF) on the following pages where updates or corrections will be posted in the future:
Practical Foundations of Debugging (x86)
Practical Foundations of Debugging (x64)
- Dmitry Vostokov @ DumpAnalysis.org -
Just discovered that China has its own 1000 page Windows debugging book:
ISBN: 9787121064074 (June, 2008)
Author is Zhang Yinkui (Raymond Zhang).
Google translation is quite impressive:
Automatically translated Table of Contents
- Dmitry Vostokov @ DumpAnalysis.org -
Motivated by The Science of Sherlock Holmes I plan to write a book about the history of debugging with the following preliminary product details and tentative release date (which may come earlier if I have enough time):
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes certain files are opened but not closed when not needed and this prevents other applications and users from using, deleting or replacing them. Sometimes on behalf of certain API calls another process opens them. One of the questions I was asked recently is how to find that process. The answer that I found is very simple: just list all handles from all processes and search for that file name there. This works in kernel and complete memory dumps and also in live kernel debugging session including its local kernel debugging variant. The following WinDbg command lists all objects (we need to open a log because the output is usually of several megabytes):
3: kd> !for_each_process "!handle 0 3 @#Process"
!handle 0 3 @#Process
processor number 3, process 8a392818
PROCESS 8a392818 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: bfc51000 ObjectTable: e1001e00 HandleCount: 2650.
Image: System
Handle table at e16a3000 with 2650 Entries in use
0004: Object: 8a392818 GrantedAccess: 001f0fff Entry: e1004008
Object: 8a392818 Type: (8a392e70) Process
ObjectHeader: 8a392800 (old version)
HandleCount: 3 PointerCount: 235
0008: Object: 8a391db0 GrantedAccess: 00000000 Entry: e1004010
Object: 8a391db0 Type: (8a392ca0) Thread
ObjectHeader: 8a391d98 (old version)
HandleCount: 1 PointerCount: 1
000c: Object: e101bca0 GrantedAccess: 00000000 Entry: e1004018
Object: e101bca0 Type: (8a37db00) Key
ObjectHeader: e101bc88 (old version)
HandleCount: 1 PointerCount: 3
Directory Object: 00000000 Name: \REGISTRY
[...]
1fac: Object: 88ec72b0 GrantedAccess: 00000003 (Protected) Entry: e1ed7f58
Object: 88ec72b0 Type: (8a36f900) File
ObjectHeader: 88ec7298 (old version)
HandleCount: 1 PointerCount: 2
Directory Object: 00000000 Name: \Documents and Settings\MyUserName\NTUSER.DAT {HarddiskVolume1}
[...]
07fc: Object: e1000768 GrantedAccess: 00000003 Entry: e2fefff8
Object: e1000768 Type: (8a387e70) KeyedEvent
ObjectHeader: e1000750 (old version)
HandleCount: 273 PointerCount: 274
Directory Object: e1001a48 Name: CritSecOutOfMemoryEvent
processor number 3, process 8873f3b8
PROCESS 8873f3b8 SessionId: 6 Cid: 4c1c Peb: 7ffdf000 ParentCid: 42bc
DirBase: 49dbb000 ObjectTable: e325f788 HandleCount: 90.
Image: PROFLWIZ.EXE
Handle table at e36c3000 with 90 Entries in use
0004: Object: e1000768 GrantedAccess: 00000003 Entry: e36c3008
Object: e1000768 Type: (8a387e70) KeyedEvent
ObjectHeader: e1000750 (old version)
HandleCount: 273 PointerCount: 274
Directory Object: e1001a48 Name: CritSecOutOfMemoryEvent
[...]
- Dmitry Vostokov @ DumpAnalysis.org -
While working on WDN book I noticed that it is possible to analyze managed code in complete memory dumps. We just need to switch to the process in question and load SOS DLL (if memory dumps are from 64-bit Windows we need to run 64-bit WinDbg because it needs to load 64-bit SOS from Microsoft.NET \ Framework64 \ vX.X.XXXXX folder).
Here is some command output from a complete memory dump generated from SystemDump when running TestDefaultDebugger.NET application where we try to find and disassemble IL code for this function:
namespace WindowsApplication1
{
public partial class Form1 : Form
{
[...]
private void button1_Click(object sender, EventArgs e)
{
System.Collections.Stack stack = new System.Collections.Stack();stack.Pop();
}
}
}
Loading Dump File [C:\W2K3\MEMORY_NET.DMP]
Kernel Complete Dump File: Full address space is available
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x64
Product: Server, suite: Enterprise TerminalServer
Built by: 3790.srv03_sp2_gdr.070321-2337
Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`01198b00
kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
[...]
PROCESS fffffadfe7287610
SessionId: 0 Cid: 0ad8 Peb: 7fffffdf000 ParentCid: 0acc
DirBase: 12cd9000 ObjectTable: fffffa800163c260 HandleCount: 114.
Image: TestDefaultDebugger.NET.exe
PROCESS fffffadfe67905a0
SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc
DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55.
Image: SystemDump.exe
kd> .process /r /p fffffadfe7287610
Implicit process is now fffffadf`e7287610
Loading User Symbols
kd> .loadby sos mscorwks
kd> !threads
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive Lock
ID OSID ThreadOBJ State GC GC Alloc Context Domain Count APT Exception
1 a94 0000000000161150 6020 Enabled 0000000000000000:0000000000000000 000000000014ccb0 0 STA
2 604 00000000001688b0 b220 Enabled 0000000000000000:0000000000000000 000000000014ccb0 0 MTA (Finalizer)
kd> !process fffffadfe7287610 4
PROCESS fffffadfe7287610
SessionId: 0 Cid: 0ad8 Peb: 7fffffdf000 ParentCid: 0acc
DirBase: 12cd9000 ObjectTable: fffffa800163c260 HandleCount: 114.
Image: TestDefaultDebugger.NET.exe
THREAD fffffadfe668cbf0 Cid 0ad8.0a94 Teb: 000007fffffdd000 Win32Thread: fffff97ff4df2830 WAIT
THREAD fffffadfe727e6d0 Cid 0ad8.0f54 Teb: 000007fffffdb000 Win32Thread: 0000000000000000 WAIT
THREAD fffffadfe72d5bf0 Cid 0ad8.0604 Teb: 000007fffffd9000 Win32Thread: 0000000000000000 WAIT
THREAD fffffadfe679cbf0 Cid 0ad8.06b0 Teb: 000007fffffd7000 Win32Thread: 0000000000000000 WAIT
THREAD fffffadfe67d23d0 Cid 0ad8.0b74 Teb: 000007fffffd5000 Win32Thread: fffff97ff4b99010 WAIT
kd> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000000002a41030
generation 1 starts at 0x0000000002a41018
generation 2 starts at 0x0000000002a41000
ephemeral segment allocation context: (0x0000000002a8d528, 0x0000000002a8dfe8)
segment begin allocated size
00000000001a1260 0000064274e28f60 0000064274e5f610 0x00000000000366b0(222896)
00000000001a1070 000006427692ffe8 000006427695af20 0x000000000002af38(175928)
0000000000164f60 00000642787c7380 0000064278809150 0x0000000000041dd0(269776)
0000000002a40000 0000000002a41000 0000000002a8dfe8 0x000000000004cfe8(315368)
Large object heap starts at 0x0000000012a41000
segment begin allocated size
0000000012a40000 0000000012a41000 0000000012a4e738 0x000000000000d738(55096)
Total Size 0xfdad8(1039064)
------------------------------
GC Heap Size 0xfdad8(1039064)
kd> !gchandles
Bad MethodTable for Obj at 0000000002a7a7b8
Bad MethodTable for Obj at 0000000002a7a750
Bad MethodTable for Obj at 0000000002a445b0
GC Handle Statistics:
Strong Handles: 25
Pinned Handles: 7
Async Pinned Handles: 0
Ref Count Handles: 1
Weak Long Handles: 30
Weak Short Handles: 63
Other Handles: 0
Statistics:
MT Count TotalSize Class Name
[...]
0000064280016580 1 464 WindowsApplication1.Form1
[…]
kd> !dumpmt -md 0000064280016580
EEClass: 0000064280143578
Module: 0000064280012e00
Name: WindowsApplication1.Form1
mdToken: 02000002 (C:\TestDefaultDebugger.NET.exe)
BaseSize: 0×1d0
ComponentSize: 0×0
Number of IFaces in IFaceMap: 15
Slots in VTable: 375
————————————–
MethodDesc Table
Entry MethodDesc JIT Name
[…]
0000064280150208 00000642800164d0 JIT WindowsApplication1.Form1.InitializeComponent()
0000064280150210 00000642800164e0 JIT WindowsApplication1.Form1..ctor()
0000064280150218 00000642800164f0 NONE WindowsApplication1.Form1.button1_Click(System.Object, System.EventArgs)
kd> !dumpil 00000642800164f0
ilAddr = 00000000004021bc
IL_0000: newobj System.Collections.Stack::.ctor
IL_0005: stloc.0
IL_0006: ldloc.0
IL_0007: callvirt System.Collections.Stack::Pop
IL_000c: pop
IL_000d: ret
- Dmitry Vostokov @ DumpAnalysis.org -
Mistake #1 - Not looking at full stack traces
By default WinDbg cuts off stack traces after 20th line and an analyst misses essential information when looking at Stack Trace or Stack Trace Collection. Consider this thread stack trace taken from a user process dump where runaway information was not saved but customers reported CPU spikes:
0:000> ~3kvL
ChildEBP RetAddr
0290f864 773976f2 user32!_SEH_prolog+0xb
0290f86c 0047f9ec user32!EnableMenuItem+0xf
0290f884 00488f6d Application!Close+0x142c
0290f8a4 0047a9c6 Application!EnableMenu+0x5d
0290f8b8 0048890d Application!EnableWindows+0x106
0290f8d0 0048cc2b Application!SetHourGlass+0xbd
0290f8fc 0046327a Application!WriteDataStream+0x24b
0290f924 0048d8f9 Application!WriteDataStream+0x21a
0290fa68 00479811 Application!WriteDataStream+0xcb9
0290fadc 5b5e976c Application!OnWrite+0x3c1
0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2
0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22
0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91
0290fc10 00516454 mfc71!AfxWndProc+0x46
0290fc3c 7739c3b7 Application!ExitCheck+0x28f34
0290fc68 7739c484 user32!InternalCallWinProc+0x28
0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151
0290fd10 773ad03f user32!CallWindowProcAorW+0x98
0290fd30 0047a59a user32!CallWindowProcA+0x1b
We can see that it uses MFC libraries and window messaging API but was it caught accidentally? Is it a typical message loop like idle message loops in Passive Thread pattern using GetMessage or it is an active GUI message pump using PeekMessage? If we expand stack trace we would see that the thread is actually MFC GUI thread that spins according to MFC source code:
int CWinThread::Run()
{
for (;;)
{
while (bIdle &&
!::PeekMessage(&(pState->m_msgCur),
NULL, NULL, NULL, PM_NOREMOVE))
{
0:000> ~3kvL 100
ChildEBP RetAddr
0290f864 773976f2 user32!_SEH_prolog+0xb
0290f86c 0047f9ec user32!EnableMenuItem+0xf
0290f884 00488f6d Application!Close+0x142c
0290f8a4 0047a9c6 Application!EnableMenu+0x5d
0290f8b8 0048890d Application!EnableWindows+0x106
0290f8d0 0048cc2b Application!SetHourGlass+0xbd
0290f8fc 0046327a Application!WriteDataStream+0x24b
0290f924 0048d8f9 Application!WriteDataStream+0x21a
0290fa68 00479811 Application!WriteDataStream+0xcb9
0290fadc 5b5e976c Application!OnWrite+0x3c1
0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2
0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22
0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91
0290fc10 00516454 mfc71!AfxWndProc+0x46
0290fc3c 7739c3b7 Application!ExitCheck+0x28f34
0290fc68 7739c484 user32!InternalCallWinProc+0x28
0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151
0290fd10 773ad03f user32!CallWindowProcAorW+0x98
0290fd30 0047a59a user32!CallWindowProcA+0x1b
0290fdb0 7739c3b7 Application!OnOK+0x77a
0290fddc 7739c484 user32!InternalCallWinProc+0x28
0290fe54 7739c73c user32!UserCallWinProcCheckWow+0x151
0290febc 7738e406 user32!DispatchMessageWorker+0x327
0290fecc 5b609076 user32!DispatchMessageA+0xf
0290fedc 5b60913e mfc71!AfxInternalPumpMessage+0x3e
0290fef8 004ba7cf mfc71!CWinThread::Run+0×54
0290ff04 5b61b30c Application!CMyThread::Run+0xf
0290ff84 5b869565 mfc71!_AfxThreadEntry+0×100
0290ffb8 77e66063 msvcr71!_endthreadex+0xa0
0290ffec 00000000 kernel32!BaseThreadStart+0×34
There is also WinDbg .kframes meta-command that can change default stack trace depth:
2: kd> .kframes 0n100
Default stack trace depth is 0n100 frames
- Dmitry Vostokov @ DumpAnalysis.org -
It now becomes a tradition to publish a sample chapter about heap. As promised here are two draft pages from the forthcoming Windows® Debugging Notebook:
05 (0×05): Heap / Pool / Paged / NonPaged
- Dmitry Vostokov @ DumpAnalysis.org -
Recently being interested in forensic science and intelligence I decided to elaborate on memory analysis classification and came up with the following division:
- Memory Analysis Forensics
Answering questions related to a committed computer crime. The suspect may be a human or a software / hardware component. Incident response, troubleshooting and debugging belong to this category. Postmortem memory analysis is usually analysis of dump files saved and detached from the original system or operating conditions.
- Memory Analysis Intelligence
Monitoring memory state for behavioural and structural patterns to prevent certain events from occurring. Usually done in situ. However digital dumpster divers and spies may also collect and analyze memory data that was detached from the original computer system.
Each category can be further subdivided into:
– Functional Memory Analysis
Tracing of events.
Analysis of memory states and their evolution.
The latter can be subdivided into:
— Static Memory Analysis
Traditional memory dump analysis.
— Dynamic Memory Analysis
Live debugging.
- Dmitry Vostokov @ DumpAnalysis.org -
There is a bit of confusion around the division of exceptions between the first- and second-chance. For example, I am often asked whether saving crash dumps on first-chance exceptions should be disabled or enabled. So I decided to clarify this issue.
First, let me say that the concept of first-chance exceptions is purely a debugger-related. There is only one exception that happens when we access an invalid address, for example. However, that exception may be handled or may not be handled by exception handlers. Or it might be handled in peculiar way and terminate the thread that caused the exception, for example. And if it was not handled then an unhandled exception filter might be called. The default one might launch a postmortem debugger (or any process that can read process memory) to save a postmortem memory dump. Any thread can replace the default filter with a custom exception filter that might also do peculiar things and quietly terminate or exit. Even the properly configured postmortem debugger can fail to save a dump file. Therefore we have this question: how can we catch the exception and examine the process state as earlier as possible, before the execution flow goes through the exception handling mechanism?
Here we have the concept of the first chance exception dispatched to the attached user-mode debugger. if it wasn’t handled we have the same exception but called the second chance that is dispatched to the same debugger again. We see that it has nothing to do with the postmortem debugger although the attached live debugger can save crash dump files too, which what ADPlus does, for example.
- Dmitry Vostokov @ DumpAnalysis.org -
The book has finally made it through Google Search program and is available for search here:
http://www.google.com/books?id=RR5whfK1BYsC
Also, yesterday I approved the hardcover version for distribution on Amazon and it should be available there in a week or so.
- Dmitry Vostokov @ DumpAnalysis.org -
Invalid Handle exception (0xC0000008) can frequently be seen in crash dumps. It results from an invalid handle value passed to CloseHandle and other Win32 API or when a handle or return status is checked manually for validity and the same exception is raised via RaiseException or internally via RtlRaiseStatus. Therefore it merits its own separate crash dump analysis pattern with the same name.
For example, critical sections are implemented using events and invalid event handle can result in this exception:
STACK_TEXT:
025bff00 7c94243c c0000008 7c9010ed 00231af0 ntdll!RtlRaiseStatus+0×26
025bff80 7c90104b 0015b4ac 77e76a6f 0015b4ac ntdll!RtlpWaitForCriticalSection+0×204
025bff88 77e76a6f 0015b4ac 010d2040 00000000 ntdll!RtlEnterCriticalSection+0×46
025bffa8 77e76c0a 0015b420 025bffec 7c80b683 rpcrt4!BaseCachedThreadRoutine+0xad
025bffb4 7c80b683 001feae8 010d2040 00000000 rpcrt4!ThreadStartRoutine+0×1a
025bffec 00000000 77e76bf0 001feae8 00000000 kernel32!BaseThreadStart+0×37
By default, unless raised manually, this exception doesn’t result in a default postmortem debugger called to save a crash dump. In order to do this we need to run the application under a debugger and save a crash dump upon this exception or use exception monitoring tools that save first-chance exceptions like Debug Diagnostics, ADPlus or Exception Monitor (see Early Crash Dump pattern):
0:002> g
(7b0.d1c): Invalid handle - code c0000008 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400 ret 4
0:000> g
(7b0.d1c): Invalid handle - code c0000008 (!!! second chance !!!)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400 ret 4
In order to catch it using postmortem debuggers we can use Application Verifier and configure its basic checks to include invalid handles. Then we will have crash dumps if a postmortem debugger or WER is properly configured. The typical stack might look like this and pointing straight to the problem component:
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 6b006369
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 00000000
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT
0:000> kL
ChildEBP RetAddr
0301ff44 0489a480 ntdll!NtClose+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0301ff54 7d4d8e4f vfbasics+0xa480
0301ff60 04894df9 kernel32!CloseHandle+0×59
0301ff70 00401022 vfbasics+0×4df9
0301ffc0 7d4e7d2a BadHandle+0×1022
0301fff0 00000000 kernel32!BaseProcessStart+0×28
or like this:
0:000> kL
Child-SP RetAddr Call Site
00000000`0012ed58 00000000`01f9395a ntdll!DbgBreakPoint
00000000`0012ed60 00000000`023e29a7 vrfcore!VerifierStopMessageEx+0×846
00000000`0012f090 00000000`023d9384 vfbasics+0×129a7
00000000`0012f0f0 00000000`77f251ec vfbasics+0×9384
00000000`0012f180 00000000`77ee5f36 ntdll!RtlpCallVectoredHandlers+0×26f
00000000`0012f210 00000000`77ee6812 ntdll!RtlDispatchException+0×46
00000000`0012f8c0 00000000`77ef325a ntdll!RtlRaiseException+0xae
00000000`0012fe00 00000000`77d6e314 ntdll!KiRaiseUserExceptionDispatcher+0×3a
00000000`0012fed0 00000001`40001028 kernel32!CloseHandle+0×5f
00000000`0012ff00 00000001`40001294 BadHandle+0×1028
00000000`0012ff30 00000000`77d5964c BadHandle+0×1294
00000000`0012ff80 00000000`00000000 kernel32!BaseProcessStart+0×29
vfbasics and vrfcore are Application Verifier DLLs that might translate an invalid handle exception to a breakpoint exception and therefore trigger the launch of a postmortem debugger from an unhandled exception filter. Application Verifier version (x64 or x86) must match the application platform (64-bit or 32-bit).
If invalid handle exception is raised manually we get the status code and possibly problem component immediately from !analyze command:
FAULTING_IP:
kernel32!RaiseException+53
7d4e2366 5e pop esi
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7d4e2366 (kernel32!RaiseException+0x00000053)
ExceptionCode: c0000008 (Invalid handle)
ExceptionFlags: 00000000
NumberParameters: 0
Thread tried to close a handle that was invalid or illegal to close
DEFAULT_BUCKET_ID: STATUS_INVALID_HANDLE
PROCESS_NAME: BadHandle.exe
ERROR_CODE: (NTSTATUS) 0xc0000008 - An invalid HANDLE was specified.
STACK_TEXT:
0012ff64 00401043 c0000008 00000000 00000000 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 BadHandle+0×1043
0012fff0 00000000 004012f9 00000000 00000000 kernel32!BaseProcessStart+0×28
FAULTING_THREAD: 00000b64
PRIMARY_PROBLEM_CLASS: STATUS_INVALID_HANDLE
BUGCHECK_STR: APPLICATION_FAULT_STATUS_INVALID_HANDLE
Because we have WinDbg warning about stack unwind we can double check the disassembly of RaiseException return address:
0:000> ub 00401043
BadHandle+0×1029:
00401029 push offset BadHandle+0×212c (0040212c)
0040102e push 0
00401030 call esi
00401032 push 0
00401034 push 0
00401036 push 0
00401038 push 0C0000008h
0040103d call dword ptr [BadHandle+0×2004 (00402004)]
0:000> dps 00402004 l1
00402004 7d4e2318 kernel32!RaiseException
Beware that in such cases the real problem might have been memory corruption overwriting stored valid handle values.
- Dmitry Vostokov @ DumpAnalysis.org -
After Bug Check Frequencies post I was curious enough to do the same Google counting procedure for exceptions. Here are my results for exceptions listed in Visual C++ Debug \ Exceptions dialog:
|
Control-C |
40010005 |
43 |
|
Control-Break |
40010008 |
7 |
|
Datatype misalignment |
80000002 |
27300 |
|
Breakpoint |
80000003 |
36400 |
|
Access violation |
C0000005 |
164000 |
|
In page error |
C0000006 |
1210 |
|
Invalid handle |
C0000008 |
1670 |
|
Not enough quota |
C0000017 |
176 |
|
Illigal instruction |
C000001D |
3400 |
|
Cannot continue |
C0000025 |
804 |
|
Invalid exception disposition |
C0000026 |
121 |
|
Array bounds exceeded |
C000008C |
100 |
|
Floating-point denormal operand |
C000008D |
84 |
|
Floating-point division by zero |
C000008E |
523 |
|
Floating-point inexact result |
C000008F |
401 |
|
Floating-point invalid operation |
C0000090 |
509 |
|
Floating-point overflow |
C0000091 |
121 |
|
Floating-point stack check |
C0000092 |
102 |
|
Floating-point underflow |
C0000093 |
138 |
|
Integer division by zero |
C0000094 |
1610 |
|
Integer overflow |
C0000095 |
99 |
|
Stack overflow |
C00000FD |
3110 |
|
Unable to locate component |
C0000135 |
3970 |
|
Ordinal not found |
C0000138 |
43 |
|
Entry point not found |
C0000139 |
724 |
|
DLL initialization failed |
C0000142 |
918 |
|
Module not found |
C06D007E |
171 |
|
Procedure not found |
C06D007F |
248 |
The corresponding graph:
- Dmitry Vostokov @ DumpAnalysis.org -
Continuing Music for Debugging post I would like to recommend the following album that I used to listen to long time ago and have just found buried in the old corner. In it various movements correspond to different debugging sessions, some glorious and some filled with tension and worries, gradually building up the ultimate problem resolution. I think almost all Vangelis music is very suitable to accompany debugging.
- Dmitry Vostokov @ DumpAnalysis.org -
A small note from the field. Sometimes on x64 Windows platform we set a default postmortem debugger or configure WER and then install Microsoft Application Verifier to do some checks. However no crash dump files are saved. The reason for this might be that we installed and configured amd64 bit version of Application Verifier but the problem application was 32-bit. For this application we need to install and configure x86 version of Application Verifier.
- Dmitry Vostokov @ DumpAnalysis.org -
