Software Diagnostics Library

Crash Dump File Examples

Thursday, June 12th, 2008

I decided to post selected crash dumps for some case studies and patterns after getting feedback from readers of my MDAA book. They are custom made from my toy programs and preprocessed to clear sensitive information that might have leaked from my home computers otherwise. They can be found at the following public FTP address:

ftp://dumpanalysis.org/pub/

The first example was published today for Missing Component (static linking) pattern. I also created the following page to list them all including the ones that I add for some previously published patterns:

Memory Dump File Samples

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Crash Dump Analysis, Debugging, Training and Seminars | 2 Comments »

Crash Dump Analysis Patterns (Part 59b)

Thursday, June 12th, 2008

Previously I introduced Missing Component pattern where the example and emphasis was on dynamically loaded modules. In this part I cover statically linked modules. Failure for a loader to find one of them results in a software exception. The most frequent of them are (numbers were taken from Google search):

C0000142 918 C0000143 919 C0000145 1,530 C0000135 24,900

0:001> !error c0000142 Error code: (NTSTATUS) 0xc0000142 (3221225794) - {DLL Initialization Failed} Initialization of the dynamic link library %hs failed. The process is terminating abnormally.

0:001> !error c0000143 Error code: (NTSTATUS) 0xc0000143 (3221225795) - {Missing System File} The required system file %hs is bad or missing.

0:001> !error c0000145 Error code: (NTSTATUS) 0xc0000145 (3221225797) - {Application Error} The application failed to initialize properly (0x%lx). Click on OK to terminate the application.

0:000> !error c0000135 Error code: (NTSTATUS) 0xc0000135 (3221225781) - {Unable To Locate Component} This application has failed to start because %hs was not found. Re-installing the application may fix this problem.

In this part I only consider user mode exceptions. If we have a default debugger configured it will usually save a crash dump. To model this problem I modified one of my applications by changing all occurrences of KERNEL32.DLL to KERNEL32.DL using Visual Studio Binary Editor. CDB was configured as a default postmortem debugger (see Custom postmortem debuggers on Vista). When the application was launched CDB attached to it and saved a crash dump. If we open it in WinDbg we get characteristic Special Stack Trace involving loader functions:

Loading Dump File [C:\UserDumps\CDAPatternMissingComponent.dmp] User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS Debug session time: Thu Jun 12 12:03:28.000 2008 (GMT+1) System Uptime: 1 days 8:46:23.167 Process Uptime: 0 days 0:00:48.000

This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (da4.f60): Wake debugger - code 80000007 (first/second chance not available) eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000 eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 ntdll!_LdrpInitialize+0x6d: 77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

0:000> kL ChildEBP RetAddr 0017fd00 77b937ea ntdll!_LdrpInitialize+0×6d 0017fd10 00000000 ntdll!LdrInitializeThunk+0×10

Verbose analysis command doesn’t give us an indication of what had happened so we need to dig further:

0:000> !analyze -v [...]

FAULTING_IP: +0 00000000 ?? ???

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 00000000 ExceptionCode: 80000007 (Wake debugger) ExceptionFlags: 00000000 NumberParameters: 0

BUGCHECK_STR: 80000007

PROCESS_NAME: StackOverflow.exe

ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened} the system debugger was awakened by an interrupt.

NTGLOBALFLAG: 400

APPLICATION_VERIFIER_FLAGS: 0

DERIVED_WAIT_CHAIN:

Dl Eid Cid WaitType -- --- ------- -------------------------- 0 da4.f60 Unknown

WAIT_CHAIN_COMMAND: ~0s;k;;

BLOCKING_THREAD: 00000f60

DEFAULT_BUCKET_ID: APPLICATION_HANG_BusyHang

PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BusyHang

LAST_CONTROL_TRANSFER: from 77b937ea to 77bcf1d1

FAULTING_THREAD: 00000000

STACK_TEXT: 0017fd00 77b937ea 0017fd24 77b60000 00000000 ntdll!_LdrpInitialize+0x6d 0017fd10 00000000 0017fd24 77b60000 00000000 ntdll!LdrInitializeThunk+0x10

FOLLOWUP_IP: ntdll!_LdrpInitialize+6d 77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h]

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: ntdll!_LdrpInitialize+6d

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME: ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 4549bdf8

STACK_COMMAND: ~0s ; kb

BUCKET_ID: 80000007_ntdll!_LdrpInitialize+6d

FAILURE_BUCKET_ID: ntdll.dll!_LdrpInitialize_80000007_APPLICATION_HANG_BusyHang

Followup: MachineOwner

Last event and error code are not helpful too:

0:000> .lastevent Last event: da4.f60: Wake debugger - code 80000007 (first/second chance not available) debugger time: Thu Jun 12 15:04:38.917 2008 (GMT+1)

0:000> !gle LastErrorValue: (Win32) 0 (0) - The operation completed successfully. LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0

Let’s search thread raw stack data for any signs of exceptions:

0:000> !teb TEB at 7efdd000 ExceptionList: 0017fcf0 StackBase: 00180000 StackLimit: 0017e000 SubSystemTib: 00000000 FiberData: 00001e00 ArbitraryUserPointer: 00000000 Self: 7efdd000 EnvironmentPointer: 00000000 ClientId: 00000da4 . 00000f60 RpcHandle: 00000000 Tls Storage: 00000000 PEB Address: 7efde000 LastErrorValue: 0 LastStatusValue: 0 Count Owned Locks: 0 HardErrorMode: 0

0:000> dds 0017e000 00180000 [...] 0017f8d8 7efdd000 0017f8dc 0017f964 0017f8e0 77c11c78 ntdll!_except_handler4 0017f8e4 00000000 0017f8e8 0017f988 0017f8ec 0017f900 0017f8f0 77ba1ddd ntdll!RtlCallVectoredContinueHandlers+0x15 0017f8f4 0017f988 0017f8f8 0017f9d8 0017f8fc 77c40370 ntdll!RtlpCallbackEntryList 0017f900 0017f970 0017f904 77ba1db5 ntdll!RtlDispatchException+0×11f 0017f908 0017f988 0017f90c 0017f9d8 0017f910 7efde028 0017f914 00000001 0017f918 77630000 kernel32!_imp___aullrem <PERF> (kernel32+0×0) 0017f91c 00000001 0017f920 776ced81 kernel32!_DllMainCRTStartupForGS2+0×10 0017f924 0017f938 0017f928 7765d4d9 kernel32!BaseDllInitialize+0×18 0017f92c 76042340 user32!$$VProc_ImageExportDirectory 0017f930 00000001 0017f934 00000000 0017f938 0017f9e0 0017f93c 77b8f890 ntdll!LdrpSnapThunk+0xc9 0017f940 0040977a StackOverflow+0×977a 0017f944 0000030b 0017f948 76030000 user32!_imp__RegSetValueExW <PERF> (user32+0×0) 0017f94c 76042f94 user32!$$VProc_ImageExportDirectory+0xc54 0017f950 77bb8881 ntdll!LdrpSnapThunk+0×40d 0017f954 0017bb30 0017f958 00409770 StackOverflow+0×9770 0017f95c 00881a50 0017f960 004098b2 StackOverflow+0×98b2 0017f964 77bac282 ntdll!ZwRaiseException+0×12 0017f968 00180000 0017f96c 0017fc48 0017f970 0017fd00 0017f974 77bac282 ntdll!ZwRaiseException+0×12 0017f978 77b7ee72 ntdll!KiUserExceptionDispatcher+0×2a 0017f97c 0017f988 ; exception record 0017f980 0017f9d8 ; exception context 0017f984 00000000 0017f988 c0000135 0017f98c 00000001 0017f990 00000000 0017f994 77bcf1d1 ntdll!_LdrpInitialize+0×6d 0017f998 00000000 0017f99c 77c11c78 ntdll!_except_handler4 0017f9a0 77b8dab8 ntdll!RtlpRunTable+0×218 0017f9a4 fffffffe 0017f9a8 77ba2515 ntdll!vDbgPrintExWithPrefixInternal+0×214 0017f9ac 77ba253b ntdll!DbgPrintEx+0×1e 0017f9b0 77b7f356 ntdll! ?? ::FNODOBFM::`string’ 0017f9b4 00000055 0017f9b8 00000003 0017f9bc 77b809c2 ntdll! ?? ::FNODOBFM::`string’ 0017f9c0 0017fc9c 0017f9c4 00000001 0017f9c8 0017fd00 0017f9cc 77bcf28e ntdll!_LdrpInitialize+0×12a 0017f9d0 00000055 0017f9d4 75ce3cf9 0017f9d8 0001003f 0017f9dc 00000000 0017f9e0 00000000 0017f9e4 00000000 0017f9e8 00000000 0017f9ec 00000000 0017f9f0 00000000 0017f9f4 0000027f […]

We see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get one:

0:000> .exr 0017f988 ExceptionAddress: 77bcf1d1 (ntdll!_LdrpInitialize+0x0000006d) ExceptionCode: c0000135 ExceptionFlags: 00000001 NumberParameters: 0

Error c0000135 means that the loader was unable to locate a component. Now we try to examine the same raw stack data for any string patterns. For example, the following UNICODE pattern is clearly visible:

0017f2fc 00000000 0017f300 00880ec4 0017f304 77b910d7 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x344 0017f308 00000000 0017f30c 43000043 0017f310 0042002a 0017f314 0017f33c 0017f318 00000000 0017f31c 00000002 0017f320 00000008 0017f324 00000000 0017f328 0000008c 0017f32c 000a0008 0017f330 77b91670 ntdll!`string' 0017f334 00b92bd6 0017f338 0017f5d4 0017f33c 003a0043 0017f340 0050005c 0017f344 006f0072 0017f348 00720067 0017f34c 006d0061 0017f350 00460020 0017f354 006c0069 0017f358 00730065 0017f35c 00280020 0017f360 00380078 0017f364 00290036 0017f368 0043005c 0017f36c 006d006f 0017f370 006f006d 0017f374 0020006e 0017f378 00690046 0017f37c 0065006c 0017f380 005c0073 0017f384 006f0052 0017f388 00690078 0017f38c 0020006f 0017f390 00680053 0017f394 00720061 0017f398 00640065 0017f39c 0044005c 0017f3a0 004c004c 0017f3a4 00680053 0017f3a8 00720061 0017f3ac 00640065 0017f3b0 004b005c 0017f3b4 00520045 0017f3b8 0045004e 0017f3bc 0033004c 0017f3c0 002e0032 0017f3c4 006c0064 0017f3c8 00000000 0017f3cc 00000000

It is a path to DLL that was probably missing:

0:000> du 0017f33c 0017f33c "C:\Program Files (x86)\Common Fi" 0017f37c "les\Roxio Shared\DLLShared\KERNE" 0017f3bc "L32.dl"

I think the loader was trying to find KERNEL32.dl following the DLL search order and this was the last path element:

0:000> !peb PEB at 7efde000 InheritedAddressSpace: No ReadImageFileExecOptions: No BeingDebugged: Yes ImageBaseAddress: 00400000 Ldr 77c40080 Ldr.Initialized: Yes Ldr.InInitializationOrderModuleList: 00881ad0 . 008831b8 Ldr.InLoadOrderModuleList: 00881a50 . 00883dc8 Ldr.InMemoryOrderModuleList: 00881a58 . 00883dd0 Base TimeStamp Module [...] Environment: 00881de8 [...] Path=C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\ […]

In similar situations !dlls command might help that shows the load order (-l switch) and points to the last processed DLL:

0:001> !dlls -l

0x004740e8: C:\Program Files\Application\Application.exe Base 0x012a0000 EntryPoint 0x012b0903 Size 0x00057000 Flags 0x00004010 LoadCount 0x0000ffff TlsIndex 0x00000000 LDRP_ENTRY_PROCESSED

0x00474158: C:\Windows\SysWOW64\ntdll.dll Base 0x77d00000 EntryPoint 0x00000000 Size 0x00160000 Flags 0x00004014 LoadCount 0x0000ffff TlsIndex 0x00000000 LDRP_IMAGE_DLL LDRP_ENTRY_PROCESSED

0x00474440: C:\Windows\syswow64\kernel32.dll Base 0x77590000 EntryPoint 0x775a1f3e Size 0x00110000 Flags 0x00084014 LoadCount 0x0000ffff TlsIndex 0x00000000 LDRP_IMAGE_DLL LDRP_ENTRY_PROCESSED LDRP_PROCESS_ATTACH_CALLED

[…]

0x00498ff8: C:\Windows\WinSxS\x86_microsoft.windows.common-controls_...\comctl32.dll Base 0x74d90000 EntryPoint 0x74dc43e5 Size 0x0019e000 Flags 0x100c4014 LoadCount 0x00000003 TlsIndex 0x00000000 LDRP_IMAGE_DLL LDRP_ENTRY_PROCESSED LDRP_DONT_CALL_FOR_THREADS LDRP_PROCESS_ATTACH_CALLED

0x004991b8: C:\Windows\WinSxS\x86_microsoft.vc80.mfcloc_...\MFC80ENU.DLL Base 0x71b10000 EntryPoint 0x00000000 Size 0x0000e000 Flags 0x10004014 LoadCount 0x00000001 TlsIndex 0x00000000 LDRP_IMAGE_DLL LDRP_ENTRY_PROCESSED

If it is difficult to identify what had really happened in crash dumps we can enable loader snaps using gflags and run the application under a debugger. For example, for notepad.exe we have:

CommandLine: C:\Windows\notepad.exe Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: ModLoad: 00000000`ffac0000 00000000`ffaef000 notepad.exe ModLoad: 00000000`779b0000 00000000`77b2a000 ntdll.dll LDR: NEW PROCESS Image Path: C:\Windows\notepad.exe (notepad.exe) Current Directory: C:\Program Files\Debugging Tools for Windows 64-bit\ Search Path: C:\Windows; C:\Windows\system32; C:\Windows\system; C:\Windows;.; C:\Program Files\Debugging Tools for Windows 64-bit\winext\arcade; C:\Windows\system32;C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\ LDR: LdrLoadDll, loading kernel32.dll from ModLoad: 00000000`777a0000 00000000`778d1000 C:\Windows\system32\kernel32.dll LDR: kernel32.dll bound to ntdll.dll LDR: kernel32.dll has stale binding to ntdll.dll LDR: Stale Bind ntdll.dll from kernel32.dll LDR: LdrGetProcedureAddress by NAME - BaseThreadInitThunk [3d8,1278] LDR: Real INIT LIST for process C:\Windows\notepad.exe pid 984 0x3d8 [3d8,1278] C:\Windows\system32\kernel32.dll init routine 00000000777DC960 [3d8,1278] LDR: kernel32.dll loaded - Calling init routine at 00000000777DC960 LDR: notepad.exe bound to ADVAPI32.dll ModLoad: 000007fe`fe520000 000007fe`fe61f000 C:\Windows\system32\ADVAPI32.dll LDR: ADVAPI32.dll bound to ntdll.dll LDR: ADVAPI32.dll has stale binding to ntdll.dll LDR: Stale Bind ntdll.dll from ADVAPI32.dll LDR: ADVAPI32.dll bound to KERNEL32.dll LDR: ADVAPI32.dll has stale binding to KERNEL32.dll LDR: ADVAPI32.dll bound to ntdll.dll via forwarder(s) from kernel32.dll LDR: ADVAPI32.dll has stale binding to ntdll.dll LDR: Stale Bind KERNEL32.dll from ADVAPI32.dll LDR: LdrGetProcedureAddress by NAME - RtlAllocateHeap LDR: LdrGetProcedureAddress by NAME - RtlReAllocateHeap LDR: LdrGetProcedureAddress by NAME - RtlEncodePointer LDR: LdrGetProcedureAddress by NAME - RtlDecodePointer LDR: LdrGetProcedureAddress by NAME - RtlSizeHeap LDR: LdrGetProcedureAddress by NAME - RtlDeleteCriticalSection LDR: LdrGetProcedureAddress by NAME - RtlEnterCriticalSection LDR: LdrGetProcedureAddress by NAME - RtlLeaveCriticalSection LDR: ADVAPI32.dll bound to RPCRT4.dll [...]

This technique only works for native platform loader snaps. For example, it doesn’t show loader snaps for 32-bit modules loaded under WOW64 (to the best of my knowledge):

CommandLine: C:\Apps\StackOverflow.exe Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: ModLoad: 00400000 00418000 StackOverflow.exe ModLoad: 77b60000 77cb0000 ntdll.dll LDR: NEW PROCESS Image Path: C:\Apps\StackOverflow.exe (StackOverflow.exe) [...] LDR: Loading (STATIC, NON_REDIRECTED) C:\Windows\system32\wow64cpu.dll LDR: wow64cpu.dll bound to ntdll.dll LDR: wow64cpu.dll has stale binding to ntdll.dll LDR: Stale Bind ntdll.dll from wow64cpu.dll LDR: wow64cpu.dll bound to wow64.dll LDR: wow64cpu.dll has stale binding to wow64.dll LDR: Stale Bind wow64.dll from wow64cpu.dll LDR: wow64.dll has stale binding to wow64cpu.dll LDR: Stale Bind wow64cpu.dll from wow64.dll LDR: Refcount wow64cpu.dll (1) LDR: Refcount wow64.dll (2) LDR: Refcount wow64win.dll (1) LDR: Refcount wow64.dll (3) LDR: LdrGetProcedureAddress by NAME - Wow64LdrpInitialize [...] ModLoad: 77630000 77740000 C:\Windows\syswow64\kernel32.dll ModLoad: 76030000 76100000 C:\Windows\syswow64\USER32.dll ModLoad: 775a0000 77630000 C:\Windows\syswow64\GDI32.dll ModLoad: 76d00000 76dbf000 C:\Windows\syswow64\ADVAPI32.dll ModLoad: 76df0000 76ee0000 C:\Windows\syswow64\RPCRT4.dll ModLoad: 75d60000 75dc0000 C:\Windows\syswow64\Secur32.dll (1ec.1290): Unknown exception - code c0000135 (first chance) (1ec.1290): Unknown exception - code c0000135 (!!! second chance !!!) eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000 eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 ntdll!_LdrpInitialize+0×6d: 77bcf1d1 8b45b8 mov eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

The dump file that I used was modified to remove sensitive information (see Data Hiding in Crash Dumps for this technique). It can be downloaded from FTP to play with:

ftp://dumpanalysis.org/pub/CDAPatternMissingComponent.zip

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | 2 Comments »

Data Hiding in Crash Dumps

Tuesday, June 10th, 2008

Suppose we want to send a complete memory dump to a vendor but want to remove certain sensitive details or perhaps the whole process or image from it. In this case we can use f WinDbg command (virtual addresses) or fp (physical addresses) to fill pages with zeroes. Let’s open a complete memory dump and erase environment variables for a process:

kd> !process 0 0 **** NT ACTIVE PROCESS DUMP **** PROCESS fffffadfe7afd8e0 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000 DirBase: 0014a000 ObjectTable: fffffa8000000c10 HandleCount: 730. Image: System

PROCESS fffffadfe6edc040 SessionId: none Cid: 0130 Peb: 7fffffdf000 ParentCid: 0004 DirBase: 34142000 ObjectTable: fffffa80009056d0 HandleCount: 19. Image: smss.exe

[...]

PROCESS fffffadfe67905a0 SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55. Image: SystemDump.exe

kd> .process /r /p fffffadfe7287610 Implicit process is now fffffadf`e7287610 Loading User Symbols

kd> !peb PEB at 000007fffffd4000 [...] Environment: 0000000000010000

kd> dd 10000 00000000`00010000 004c0041 0055004c 00450053 00530052 00000000`00010010 00520050 0046004f 004c0049 003d0045 00000000`00010020 003a0043 0044005c 0063006f 006d0075 00000000`00010030 006e0065 00730074 00610020 0064006e 00000000`00010040 00530020 00740065 00690074 0067006e 00000000`00010050 005c0073 006c0041 0020006c 00730055 00000000`00010060 00720065 002e0073 00320057 0033004b 00000000`00010070 00410000 00500050 00410044 00410054

kd> f 10000 10000+1000 0 Filled 0x1000 bytes

kd> dd 10000 00000000`00010000 00000000 00000000 00000000 00000000 00000000`00010010 00000000 00000000 00000000 00000000 00000000`00010020 00000000 00000000 00000000 00000000 00000000`00010030 00000000 00000000 00000000 00000000 00000000`00010040 00000000 00000000 00000000 00000000 00000000`00010050 00000000 00000000 00000000 00000000 00000000`00010060 00000000 00000000 00000000 00000000 00000000`00010070 00000000 00000000 00000000 00000000

Now we can save the modified complete dump file:

kd> .dump /f c:\Dumps\SecuredDump.dmp

If we want to find and erase read-write pages, for example, we can use !vad WinDbg command to get the description of virtual address ranges:

kd> !process PROCESS fffffadfe67905a0 SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55. Image: SystemDump.exe VadRoot fffffadfe6f293e0 Vads 65 Clone 0 Private 388. Modified 84. Locked 0. DeviceMap fffffa80020777c0 Token fffffa80008e5b50 ElapsedTime 00:00:06.265 UserTime 00:00:00.031 KernelTime 00:00:00.062 QuotaPoolUsage[PagedPool] 113464 QuotaPoolUsage[NonPagedPool] 5152 Working Set Sizes (now,min,max) (1429, 50, 345) (5716KB, 200KB, 1380KB) PeakWorkingSetSize 1429 VirtualSize 61 Mb PeakVirtualSize 63 Mb PageFaultCount 1555 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 471

kd> !vad fffffadfe6f293e0 VAD level start end commit fffffadfe682bdf0 ( 6) 10 10 1 Private READWRITE fffffadfe73a0e10 ( 5) 20 20 1 Private READWRITE fffffadfe73a0dd0 ( 4) 30 12f 8 Private READWRITE fffffadfe71a4770 ( 5) 130 134 0 Mapped READONLY fffffadfe781bbe0 ( 3) 140 141 0 Mapped READONLY […] fffffadfe772d630 (-2) 7fffffdc 7fffffdd 2 Private READWRITE fffffadfe788e180 (-1) 7fffffde 7fffffdf 2 Private READWRITE

Total VADs: 65 average level: 66076419 maximum depth: -1

In the output start and end columns refer to virtual page numbers (VPN). To get an address we need to multiply by 0×1000, for example, 7fffffde000.

Filling memory with zeroes to hide data with subsequent saving of a modified crash dump is applicable to user dumps too. Please also check for additional security-related flags in .dump command:

WinDbg is privacy-aware

Another application for data hiding and modification could be the creation of the customized crash dumps for digital forensics exercises and contests.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, Memory Analysis Forensics and Intelligence, Security | 5 Comments »

Crash Dump Analysis Patterns (Part 16b)

Tuesday, June 10th, 2008

I’ve just found that although I covered Stack Overflow in kernel mode I didn’t do this for user mode. In fact this is one of the simplest patterns to see in crash dumps. It has its own characteristic exception code and stack trace:

FAULTING_IP: StackOverflow!SoFunction+27 00401317 6a00 push 0

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 00401300 (StackOverflow!SoFunction+0x00000010) ExceptionCode: c00000fd (Stack overflow) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000001 Parameter[1]: 00082ffc

0:000> kL ChildEBP RetAddr 00083000 00401317 StackOverflow!SoFunction+0x10 00083010 00401317 StackOverflow!SoFunction+0×27 00083020 00401317 StackOverflow!SoFunction+0×27 00083030 00401317 StackOverflow!SoFunction+0×27 00083040 00401317 StackOverflow!SoFunction+0×27 00083050 00401317 StackOverflow!SoFunction+0×27 00083060 00401317 StackOverflow!SoFunction+0×27 00083070 00401317 StackOverflow!SoFunction+0×27 00083080 00401317 StackOverflow!SoFunction+0×27 00083090 00401317 StackOverflow!SoFunction+0×27 000830a0 00401317 StackOverflow!SoFunction+0×27 000830b0 00401317 StackOverflow!SoFunction+0×27 000830c0 00401317 StackOverflow!SoFunction+0×27 000830d0 00401317 StackOverflow!SoFunction+0×27 000830e0 00401317 StackOverflow!SoFunction+0×27 000830f0 00401317 StackOverflow!SoFunction+0×27 00083100 00401317 StackOverflow!SoFunction+0×27 00083110 00401317 StackOverflow!SoFunction+0×27 00083120 00401317 StackOverflow!SoFunction+0×27 00083130 00401317 StackOverflow!SoFunction+0×27

There could be thousands of stack frames:

0:000> kL 2000 [...] 000a2fa0 00401317 StackOverflow!SoFunction+0x27 000a2fb0 00401317 StackOverflow!SoFunction+0x27 000a2fc0 00401317 StackOverflow!SoFunction+0x27 000a2fd0 00401317 StackOverflow!SoFunction+0x27 000a2fe0 00401317 StackOverflow!SoFunction+0x27 000a2ff0 00401317 StackOverflow!SoFunction+0x27

To reach the bottom and avoid over scrolling we can dump the raw stack data, search for the end of the repeating pattern of StackOverflow!SoFunction+0×27 and try to manually reconstruct the bottom of the stack trace:

0:000> !teb TEB at 7efdd000 ExceptionList: 0017fdf0 StackBase: 00180000 StackLimit: 00081000 SubSystemTib: 00000000 FiberData: 00001e00 ArbitraryUserPointer: 00000000 Self: 7efdd000 EnvironmentPointer: 00000000 ClientId: 00001dc4 . 00001b74 RpcHandle: 00000000 Tls Storage: 7efdd02c PEB Address: 7efde000 LastErrorValue: 0 LastStatusValue: c0000034 Count Owned Locks: 0 HardErrorMode: 0

0:000> dds 00081000 00180000 [...] 0017fc74 00401317 StackOverflow!SoFunction+0×27 0017fc78 00000000 0017fc7c a3a8ea65 0017fc80 0017fc90 0017fc84 00401317 StackOverflow!SoFunction+0×27 0017fc88 10001843 0017fc8c a3a8ea95 0017fc90 0017fca0 0017fc94 00401317 StackOverflow!SoFunction+0×27 0017fc98 0017fcb8 0017fc9c a3a8ea85 0017fca0 0017fcb0 0017fca4 00401317 StackOverflow!SoFunction+0×27 0017fca8 00000003 0017fcac a3a8eab5 0017fcb0 0017fcc0 0017fcb4 00401317 StackOverflow!SoFunction+0×27 0017fcb8 76c68738 user32!_EndUserApiHook+0×11 0017fcbc a3a8eaa5 0017fcc0 0017fcd0 0017fcc4 00401317 StackOverflow!SoFunction+0×27 0017fcc8 76c6a6cc user32!DefWindowProcW+0×94 0017fccc a3a8ead5 0017fcd0 0017fce0 0017fcd4 00401317 StackOverflow!SoFunction+0×27 0017fcd8 0037311e 0017fcdc a3a8eac5 0017fce0 0017fcf0 0017fce4 00401317 StackOverflow!SoFunction+0×27 0017fce8 0017fcd0 0017fcec a3a8eaf5 0017fcf0 0017fd00 0017fcf4 00401317 StackOverflow!SoFunction+0×27 0017fcf8 76c6ad0f user32!NtUserBeginPaint+0×15 0017fcfc a3a8eae5 0017fd00 0017fd5c 0017fd04 00401272 StackOverflow!WndProc+0xe2 0017fd08 00401190 StackOverflow!WndProc 0017fd0c 00000003 0017fd10 cf017ada […]

We use the extended version of k WinDbg command and supply EBP, ESP and EIP to see in what function it started:

0:000> r eax=a3b739e5 ebx=00000000 ecx=ac430000 edx=ffefd944 esi=0037311e edi=00000000 eip=00401300 esp=00082ff8 ebp=00083000 iopl=0 nv up ei ng nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010282 StackOverflow!SoFunction+0×10: 00401300 89442404 mov dword ptr [esp+4],eax ss:002b:00082ffc=00000000

0:000> k L=0017fcf0 00082ff8 00401300 ChildEBP RetAddr 0017fcb0 00401317 StackOverflow!SoFunction+0×10 0017fd00 00401272 StackOverflow!SoFunction+0×27 0017fd5c 76c687af StackOverflow!WndProc+0xe2 0017fd88 76c68936 user32!InternalCallWinProc+0×23 0017fe00 76c6a571 user32!UserCallWinProcCheckWow+0×109 0017fe5c 76c6a5dd user32!DispatchClientMessage+0xe0 0017fe98 77ccee2e user32!__fnDWORD+0×2b 0017fedc 0040107d ntdll!KiUserCallbackDispatcher+0×2e 0017ff08 0040151e StackOverflow!wWinMain+0×7d 00402ba0 20245c8b StackOverflow!__tmainCRTStartup+0×176

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | 4 Comments »

Review of Memory Analysis album

Friday, June 6th, 2008

If you remember I promised to review this CD:

The first Memory Analysis CD album

Finally it arrived by post and here it is:

I’ve just finished listening to it. It is wonderful! Clearly belongs to my Music for Debugging collection. Here is my commentary (italics) on track titles:

1. Chameleon
   It crashes then hangs then spikes then leaks. You never know what happens next…
2. Silence Before The Storm
   Waiting for the problem to start dumping memory
3. Appearances
   Multiple occurrences of the issue ease crash dump collection
4. Memory Analysis
   Memory dump analysis activity
5. Voices From Heaven
   It’s my blog full of crash dump analysis patterns
6. Figure In The Fog
   It’s that DLL! I see it clearly!!!
7. Diamond Tear
   Customer is happy
8. The Toy Soldier
WinDbg
9. Tears In Dragon’s Eyes
   Operating system vendor is happy too
10. Forgotten Song
   No pasaran (bugs)
11. Skies
   We are debugging gods!

Enjoy :-)

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, Fun with Crash Dumps, Music for Debugging | 2 Comments »

PFD lectures are available in PDF format

Thursday, June 5th, 2008

Due to the request from blog readers I made my old Practical Foundations of Debugging lectures available in PDF format:

x64 version:

I keep both versions (HTMP and PDF) on the following pages where updates or corrections will be posted in the future:

Practical Foundations of Debugging (x86)
Practical Foundations of Debugging (x64)

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Assembly Language, Debugging, Software Technical Support | 4 Comments »

Software Debugging book from China

Thursday, June 5th, 2008

Just discovered that China has its own 1000 page Windows debugging book:

ISBN: 9787121064074 (June, 2008)

Author is Zhang Yinkui (Raymond Zhang).

Google translation is quite impressive:

Automatically translated

Automatically translated Table of Contents

Original

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Books, Debugging | 10 Comments »

The Science of Dr. Watson

Tuesday, June 3rd, 2008

Motivated by The Science of Sherlock Holmes I plan to write a book about the history of debugging with the following preliminary product details and tentative release date (which may come earlier if I have enough time):

Title: The Science of Dr. Watson: An Illustrated History of Debugging
Author: Dmitry Vostokov
Publisher: Opentask (01 September 2010)
Language: English
Product Dimensions: 22.86 x 15.24
ISBN-13: 978-1-906717-07-0
Paperback: 256 pages

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Books, Crash Dump Analysis, Debugging, History, Publishing, Science of Memory Dump Analysis | No Comments »

Who opened that file?

Friday, May 30th, 2008

Sometimes certain files are opened but not closed when not needed and this prevents other applications and users from using, deleting or replacing them. Sometimes on behalf of certain API calls another process opens them. One of the questions I was asked recently is how to find that process. The answer that I found is very simple: just list all handles from all processes and search for that file name there. This works in kernel and complete memory dumps and also in live kernel debugging session including its local kernel debugging variant. The following WinDbg command lists all objects (we need to open a log because the output is usually of several megabytes):

3: kd> !for_each_process "!handle 0 3 @#Process"

!handle 0 3 @#Process processor number 3, process 8a392818 PROCESS 8a392818 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000 DirBase: bfc51000 ObjectTable: e1001e00 HandleCount: 2650. Image: System

Handle table at e16a3000 with 2650 Entries in use 0004: Object: 8a392818 GrantedAccess: 001f0fff Entry: e1004008 Object: 8a392818 Type: (8a392e70) Process ObjectHeader: 8a392800 (old version) HandleCount: 3 PointerCount: 235

0008: Object: 8a391db0 GrantedAccess: 00000000 Entry: e1004010 Object: 8a391db0 Type: (8a392ca0) Thread ObjectHeader: 8a391d98 (old version) HandleCount: 1 PointerCount: 1

000c: Object: e101bca0 GrantedAccess: 00000000 Entry: e1004018 Object: e101bca0 Type: (8a37db00) Key ObjectHeader: e101bc88 (old version) HandleCount: 1 PointerCount: 3 Directory Object: 00000000 Name: \REGISTRY

[...]

1fac: Object: 88ec72b0 GrantedAccess: 00000003 (Protected) Entry: e1ed7f58 Object: 88ec72b0 Type: (8a36f900) File ObjectHeader: 88ec7298 (old version) HandleCount: 1 PointerCount: 2 Directory Object: 00000000 Name: \Documents and Settings\MyUserName\NTUSER.DAT {HarddiskVolume1}

[...]

07fc: Object: e1000768 GrantedAccess: 00000003 Entry: e2fefff8 Object: e1000768 Type: (8a387e70) KeyedEvent ObjectHeader: e1000750 (old version) HandleCount: 273 PointerCount: 274 Directory Object: e1001a48 Name: CritSecOutOfMemoryEvent

processor number 3, process 8873f3b8 PROCESS 8873f3b8 SessionId: 6 Cid: 4c1c Peb: 7ffdf000 ParentCid: 42bc DirBase: 49dbb000 ObjectTable: e325f788 HandleCount: 90. Image: PROFLWIZ.EXE

Handle table at e36c3000 with 90 Entries in use 0004: Object: e1000768 GrantedAccess: 00000003 Entry: e36c3008 Object: e1000768 Type: (8a387e70) KeyedEvent ObjectHeader: e1000750 (old version) HandleCount: 273 PointerCount: 274 Directory Object: e1001a48 Name: CritSecOutOfMemoryEvent

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, WinDbg Tips and Tricks | 3 Comments »

.NET Managed Code Analysis in Complete Memory Dumps

Wednesday, May 28th, 2008

While working on WDN book I noticed that it is possible to analyze managed code in complete memory dumps. We just need to switch to the process in question and load SOS DLL (if memory dumps are from 64-bit Windows we need to run 64-bit WinDbg because it needs to load 64-bit SOS from Microsoft.NET \ Framework64 \ vX.X.XXXXX folder).

Here is some command output from a complete memory dump generated from SystemDump when running TestDefaultDebugger.NET application where we try to find and disassemble IL code for this function:

namespace WindowsApplication1 { public partial class Form1 : Form { [...] private void button1_Click(object sender, EventArgs e) { System.Collections.Stack stack = new System.Collections.Stack();stack.Pop(); } } }

Loading Dump File [C:\W2K3\MEMORY_NET.DMP] Kernel Complete Dump File: Full address space is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x64 Product: Server, suite: Enterprise TerminalServer Built by: 3790.srv03_sp2_gdr.070321-2337 Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`01198b00

kd> !process 0 0 **** NT ACTIVE PROCESS DUMP **** [...] PROCESS fffffadfe7287610 SessionId: 0 Cid: 0ad8 Peb: 7fffffdf000 ParentCid: 0acc DirBase: 12cd9000 ObjectTable: fffffa800163c260 HandleCount: 114. Image: TestDefaultDebugger.NET.exe

PROCESS fffffadfe67905a0 SessionId: 0 Cid: 085c Peb: 7fffffd4000 ParentCid: 0acc DirBase: 232e2000 ObjectTable: fffffa8000917e10 HandleCount: 55. Image: SystemDump.exe

kd> .process /r /p fffffadfe7287610 Implicit process is now fffffadf`e7287610 Loading User Symbols

kd> .loadby sos mscorwks

kd> !threads

ThreadCount: 2 UnstartedThread: 0 BackgroundThread: 1 PendingThread: 0 DeadThread: 0 Hosted Runtime: no PreEmptive Lock ID OSID ThreadOBJ State GC GC Alloc Context Domain Count APT Exception 1 a94 0000000000161150 6020 Enabled 0000000000000000:0000000000000000 000000000014ccb0 0 STA 2 604 00000000001688b0 b220 Enabled 0000000000000000:0000000000000000 000000000014ccb0 0 MTA (Finalizer)

kd> !process fffffadfe7287610 4 PROCESS fffffadfe7287610 SessionId: 0 Cid: 0ad8 Peb: 7fffffdf000 ParentCid: 0acc DirBase: 12cd9000 ObjectTable: fffffa800163c260 HandleCount: 114. Image: TestDefaultDebugger.NET.exe

THREAD fffffadfe668cbf0 Cid 0ad8.0a94 Teb: 000007fffffdd000 Win32Thread: fffff97ff4df2830 WAIT THREAD fffffadfe727e6d0 Cid 0ad8.0f54 Teb: 000007fffffdb000 Win32Thread: 0000000000000000 WAIT THREAD fffffadfe72d5bf0 Cid 0ad8.0604 Teb: 000007fffffd9000 Win32Thread: 0000000000000000 WAIT THREAD fffffadfe679cbf0 Cid 0ad8.06b0 Teb: 000007fffffd7000 Win32Thread: 0000000000000000 WAIT THREAD fffffadfe67d23d0 Cid 0ad8.0b74 Teb: 000007fffffd5000 Win32Thread: fffff97ff4b99010 WAIT

kd> !EEHeap -gc Number of GC Heaps: 1 generation 0 starts at 0x0000000002a41030 generation 1 starts at 0x0000000002a41018 generation 2 starts at 0x0000000002a41000 ephemeral segment allocation context: (0x0000000002a8d528, 0x0000000002a8dfe8) segment begin allocated size 00000000001a1260 0000064274e28f60 0000064274e5f610 0x00000000000366b0(222896) 00000000001a1070 000006427692ffe8 000006427695af20 0x000000000002af38(175928) 0000000000164f60 00000642787c7380 0000064278809150 0x0000000000041dd0(269776) 0000000002a40000 0000000002a41000 0000000002a8dfe8 0x000000000004cfe8(315368) Large object heap starts at 0x0000000012a41000 segment begin allocated size 0000000012a40000 0000000012a41000 0000000012a4e738 0x000000000000d738(55096) Total Size 0xfdad8(1039064) ------------------------------ GC Heap Size 0xfdad8(1039064)

kd> !gchandles Bad MethodTable for Obj at 0000000002a7a7b8 Bad MethodTable for Obj at 0000000002a7a750 Bad MethodTable for Obj at 0000000002a445b0 GC Handle Statistics: Strong Handles: 25 Pinned Handles: 7 Async Pinned Handles: 0 Ref Count Handles: 1 Weak Long Handles: 30 Weak Short Handles: 63 Other Handles: 0 Statistics: MT Count TotalSize Class Name [...] 0000064280016580 1 464 WindowsApplication1.Form1 […]

kd> !dumpmt -md 0000064280016580 EEClass: 0000064280143578 Module: 0000064280012e00 Name: WindowsApplication1.Form1 mdToken: 02000002 (C:\TestDefaultDebugger.NET.exe) BaseSize: 0×1d0 ComponentSize: 0×0 Number of IFaces in IFaceMap: 15 Slots in VTable: 375 ————————————– MethodDesc Table Entry MethodDesc JIT Name […] 0000064280150208 00000642800164d0 JIT WindowsApplication1.Form1.InitializeComponent() 0000064280150210 00000642800164e0 JIT WindowsApplication1.Form1..ctor() 0000064280150218 00000642800164f0 NONE WindowsApplication1.Form1.button1_Click(System.Object, System.EventArgs)

kd> !dumpil 00000642800164f0 ilAddr = 00000000004021bc IL_0000: newobj System.Collections.Stack::.ctor IL_0005: stloc.0 IL_0006: ldloc.0 IL_0007: callvirt System.Collections.Stack::Pop IL_000c: pop IL_000d: ret

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Debugging, WinDbg Tips and Tricks | No Comments »

10 Common Mistakes in Memory Analysis (Part 1)

Tuesday, May 27th, 2008

Mistake #1 - Not looking at full stack traces

By default WinDbg cuts off stack traces after 20th line and an analyst misses essential information when looking at Stack Trace or Stack Trace Collection. Consider this thread stack trace taken from a user process dump where runaway information was not saved but customers reported CPU spikes:

0:000> ~3kvL ChildEBP RetAddr 0290f864 773976f2 user32!_SEH_prolog+0xb 0290f86c 0047f9ec user32!EnableMenuItem+0xf 0290f884 00488f6d Application!Close+0x142c 0290f8a4 0047a9c6 Application!EnableMenu+0x5d 0290f8b8 0048890d Application!EnableWindows+0x106 0290f8d0 0048cc2b Application!SetHourGlass+0xbd 0290f8fc 0046327a Application!WriteDataStream+0x24b 0290f924 0048d8f9 Application!WriteDataStream+0x21a 0290fa68 00479811 Application!WriteDataStream+0xcb9 0290fadc 5b5e976c Application!OnWrite+0x3c1 0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2 0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22 0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91 0290fc10 00516454 mfc71!AfxWndProc+0x46 0290fc3c 7739c3b7 Application!ExitCheck+0x28f34 0290fc68 7739c484 user32!InternalCallWinProc+0x28 0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151 0290fd10 773ad03f user32!CallWindowProcAorW+0x98 0290fd30 0047a59a user32!CallWindowProcA+0x1b

We can see that it uses MFC libraries and window messaging API but was it caught accidentally? Is it a typical message loop like idle message loops in Passive Thread pattern using GetMessage or it is an active GUI message pump using PeekMessage? If we expand stack trace we would see that the thread is actually MFC GUI thread that spins according to MFC source code:

int CWinThread::Run() { for (;;) { while (bIdle && !::PeekMessage(&(pState->m_msgCur), NULL, NULL, NULL, PM_NOREMOVE)) {

0:000> ~3kvL 100 ChildEBP RetAddr 0290f864 773976f2 user32!_SEH_prolog+0xb 0290f86c 0047f9ec user32!EnableMenuItem+0xf 0290f884 00488f6d Application!Close+0x142c 0290f8a4 0047a9c6 Application!EnableMenu+0x5d 0290f8b8 0048890d Application!EnableWindows+0x106 0290f8d0 0048cc2b Application!SetHourGlass+0xbd 0290f8fc 0046327a Application!WriteDataStream+0x24b 0290f924 0048d8f9 Application!WriteDataStream+0x21a 0290fa68 00479811 Application!WriteDataStream+0xcb9 0290fadc 5b5e976c Application!OnWrite+0x3c1 0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2 0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22 0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91 0290fc10 00516454 mfc71!AfxWndProc+0x46 0290fc3c 7739c3b7 Application!ExitCheck+0x28f34 0290fc68 7739c484 user32!InternalCallWinProc+0x28 0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151 0290fd10 773ad03f user32!CallWindowProcAorW+0x98 0290fd30 0047a59a user32!CallWindowProcA+0x1b 0290fdb0 7739c3b7 Application!OnOK+0x77a 0290fddc 7739c484 user32!InternalCallWinProc+0x28 0290fe54 7739c73c user32!UserCallWinProcCheckWow+0x151 0290febc 7738e406 user32!DispatchMessageWorker+0x327 0290fecc 5b609076 user32!DispatchMessageA+0xf 0290fedc 5b60913e mfc71!AfxInternalPumpMessage+0x3e 0290fef8 004ba7cf mfc71!CWinThread::Run+0×54 0290ff04 5b61b30c Application!CMyThread::Run+0xf 0290ff84 5b869565 mfc71!_AfxThreadEntry+0×100 0290ffb8 77e66063 msvcr71!_endthreadex+0xa0 0290ffec 00000000 kernel32!BaseThreadStart+0×34

There is also WinDbg .kframes meta-command that can change default stack trace depth:

2: kd> .kframes 0n100 Default stack trace depth is 0n100 frames

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Common Mistakes, Crash Dump Analysis, Debugging | 4 Comments »

WDN Sample Chapter: Process Heap in UML

Sunday, May 25th, 2008

It now becomes a tradition to publish a sample chapter about heap. As promised here are two draft pages from the forthcoming Windows® Debugging Notebook:

05 (0×05): Heap / Pool / Paged / NonPaged

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Books, Crash Dump Analysis, Debugging | No Comments »

Memory Analysis (Part 1)

Friday, May 23rd, 2008

Recently being interested in forensic science and intelligence I decided to elaborate on memory analysis classification and came up with the following division:

- Memory Analysis Forensics

Answering questions related to a committed computer crime. The suspect may be a human or a software / hardware component. Incident response, troubleshooting and debugging belong to this category. Postmortem memory analysis is usually analysis of dump files saved and detached from the original system or operating conditions.

- Memory Analysis Intelligence

Monitoring memory state for behavioural and structural patterns to prevent certain events from occurring. Usually done in situ. However digital dumpster divers and spies may also collect and analyze memory data that was detached from the original computer system.

Each category can be further subdivided into:

– Functional Memory Analysis

Tracing of events.

– Memoretics

Analysis of memory states and their evolution.

The latter can be subdivided into:

— Static Memory Analysis

Traditional memory dump analysis.

— Dynamic Memory Analysis

Live debugging.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Memory Analysis Forensics and Intelligence, Science of Memory Dump Analysis, Security | 1 Comment »

Demystifying first-chance exceptions (Part 1)

Thursday, May 22nd, 2008

There is a bit of confusion around the division of exceptions between the first- and second-chance. For example, I am often asked whether saving crash dumps on first-chance exceptions should be disabled or enabled. So I decided to clarify this issue.

First, let me say that the concept of first-chance exceptions is purely a debugger-related. There is only one exception that happens when we access an invalid address, for example. However, that exception may be handled or may not be handled by exception handlers. Or it might be handled in peculiar way and terminate the thread that caused the exception, for example. And if it was not handled then an unhandled exception filter might be called. The default one might launch a postmortem debugger (or any process that can read process memory) to save a postmortem memory dump. Any thread can replace the default filter with a custom exception filter that might also do peculiar things and quietly terminate or exit. Even the properly configured postmortem debugger can fail to save a dump file. Therefore we have this question: how can we catch the exception and examine the process state as earlier as possible, before the execution flow goes through the exception handling mechanism?

Here we have the concept of the first chance exception dispatched to the attached user-mode debugger. if it wasn’t handled we have the same exception but called the second chance that is dispatched to the same debugger again. We see that it has nothing to do with the postmortem debugger although the attached live debugger can save crash dump files too, which what ADPlus does, for example.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging | 5 Comments »

MDAA Volume 1 is available on Google Search

Thursday, May 22nd, 2008

The book has finally made it through Google Search program and is available for search here:

http://www.google.com/books?id=RR5whfK1BYsC

Also, yesterday I approved the hardcover version for distribution on Amazon and it should be available there in a week or so.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Announcements, Books, Crash Dump Analysis, Debugging, Publishing, Software Technical Support | 1 Comment »

Crash Dump Analysis Patterns (Part 61)

Tuesday, May 20th, 2008

Invalid Handle exception (0xC0000008) can frequently be seen in crash dumps. It results from an invalid handle value passed to CloseHandle and other Win32 API or when a handle or return status is checked manually for validity and the same exception is raised via RaiseException or internally via RtlRaiseStatus. Therefore it merits its own separate crash dump analysis pattern with the same name.

For example, critical sections are implemented using events and invalid event handle can result in this exception:

STACK_TEXT: 025bff00 7c94243c c0000008 7c9010ed 00231af0 ntdll!RtlRaiseStatus+0×26 025bff80 7c90104b 0015b4ac 77e76a6f 0015b4ac ntdll!RtlpWaitForCriticalSection+0×204 025bff88 77e76a6f 0015b4ac 010d2040 00000000 ntdll!RtlEnterCriticalSection+0×46 025bffa8 77e76c0a 0015b420 025bffec 7c80b683 rpcrt4!BaseCachedThreadRoutine+0xad 025bffb4 7c80b683 001feae8 010d2040 00000000 rpcrt4!ThreadStartRoutine+0×1a 025bffec 00000000 77e76bf0 001feae8 00000000 kernel32!BaseThreadStart+0×37

By default, unless raised manually, this exception doesn’t result in a default postmortem debugger called to save a crash dump. In order to do this we need to run the application under a debugger and save a crash dump upon this exception or use exception monitoring tools that save first-chance exceptions like Debug Diagnostics, ADPlus or Exception Monitor (see Early Crash Dump pattern):

0:002> g (7b0.d1c): Invalid handle - code c0000008 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378 eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 ntdll!NtClose+0×12: 7d61c92d c20400 ret 4

0:000> g (7b0.d1c): Invalid handle - code c0000008 (!!! second chance !!!) eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378 eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 ntdll!NtClose+0×12: 7d61c92d c20400 ret 4

In order to catch it using postmortem debuggers we can use Application Verifier and configure its basic checks to include invalid handles. Then we will have crash dumps if a postmortem debugger or WER is properly configured. The typical stack might look like this and pointing straight to the problem component:

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 6b006369 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 1 Parameter[0]: 00000000

DEFAULT_BUCKET_ID: STATUS_BREAKPOINT

0:000> kL ChildEBP RetAddr 0301ff44 0489a480 ntdll!NtClose+0x12 WARNING: Stack unwind information not available. Following frames may be wrong. 0301ff54 7d4d8e4f vfbasics+0xa480 0301ff60 04894df9 kernel32!CloseHandle+0×59 0301ff70 00401022 vfbasics+0×4df9 0301ffc0 7d4e7d2a BadHandle+0×1022 0301fff0 00000000 kernel32!BaseProcessStart+0×28

or like this:

0:000> kL Child-SP RetAddr Call Site 00000000`0012ed58 00000000`01f9395a ntdll!DbgBreakPoint 00000000`0012ed60 00000000`023e29a7 vrfcore!VerifierStopMessageEx+0×846 00000000`0012f090 00000000`023d9384 vfbasics+0×129a7 00000000`0012f0f0 00000000`77f251ec vfbasics+0×9384 00000000`0012f180 00000000`77ee5f36 ntdll!RtlpCallVectoredHandlers+0×26f 00000000`0012f210 00000000`77ee6812 ntdll!RtlDispatchException+0×46 00000000`0012f8c0 00000000`77ef325a ntdll!RtlRaiseException+0xae 00000000`0012fe00 00000000`77d6e314 ntdll!KiRaiseUserExceptionDispatcher+0×3a 00000000`0012fed0 00000001`40001028 kernel32!CloseHandle+0×5f 00000000`0012ff00 00000001`40001294 BadHandle+0×1028 00000000`0012ff30 00000000`77d5964c BadHandle+0×1294 00000000`0012ff80 00000000`00000000 kernel32!BaseProcessStart+0×29

vfbasics and vrfcore are Application Verifier DLLs that might translate an invalid handle exception to a breakpoint exception and therefore trigger the launch of a postmortem debugger from an unhandled exception filter. Application Verifier version (x64 or x86) must match the application platform (64-bit or 32-bit).

If invalid handle exception is raised manually we get the status code and possibly problem component immediately from !analyze command:

FAULTING_IP: kernel32!RaiseException+53 7d4e2366 5e pop esi

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 7d4e2366 (kernel32!RaiseException+0x00000053) ExceptionCode: c0000008 (Invalid handle) ExceptionFlags: 00000000 NumberParameters: 0 Thread tried to close a handle that was invalid or illegal to close

DEFAULT_BUCKET_ID: STATUS_INVALID_HANDLE

PROCESS_NAME: BadHandle.exe

ERROR_CODE: (NTSTATUS) 0xc0000008 - An invalid HANDLE was specified.

STACK_TEXT: 0012ff64 00401043 c0000008 00000000 00000000 kernel32!RaiseException+0×53 WARNING: Stack unwind information not available. Following frames may be wrong. 0012ffc0 7d4e7d2a 00000000 00000000 7efde000 BadHandle+0×1043 0012fff0 00000000 004012f9 00000000 00000000 kernel32!BaseProcessStart+0×28

FAULTING_THREAD: 00000b64

PRIMARY_PROBLEM_CLASS: STATUS_INVALID_HANDLE

BUGCHECK_STR: APPLICATION_FAULT_STATUS_INVALID_HANDLE

Because we have WinDbg warning about stack unwind we can double check the disassembly of RaiseException return address:

0:000> ub 00401043 BadHandle+0×1029: 00401029 push offset BadHandle+0×212c (0040212c) 0040102e push 0 00401030 call esi 00401032 push 0 00401034 push 0 00401036 push 0 00401038 push 0C0000008h 0040103d call dword ptr [BadHandle+0×2004 (00402004)]

0:000> dps 00402004 l1 00402004 7d4e2318 kernel32!RaiseException

Beware that in such cases the real problem might have been memory corruption overwriting stored valid handle values.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | 3 Comments »

Win32 Exception Frequencies

Tuesday, May 20th, 2008

After Bug Check Frequencies post I was curious enough to do the same Google counting procedure for exceptions. Here are my results for exceptions listed in Visual C++ Debug \ Exceptions dialog:

Control-C	40010005	43
Control-Break	40010008	7
Datatype misalignment	80000002	27300
Breakpoint	80000003	36400
Access violation	C0000005	164000
In page error	C0000006	1210
Invalid handle	C0000008	1670
Not enough quota	C0000017	176
Illigal instruction	C000001D	3400
Cannot continue	C0000025	804
Invalid exception disposition	C0000026	121
Array bounds exceeded	C000008C	100
Floating-point denormal operand	C000008D	84
Floating-point division by zero	C000008E	523
Floating-point inexact result	C000008F	401
Floating-point invalid operation	C0000090	509
Floating-point overflow	C0000091	121
Floating-point stack check	C0000092	102
Floating-point underflow	C0000093	138
Integer division by zero	C0000094	1610
Integer overflow	C0000095	99
Stack overflow	C00000FD	3110
Unable to locate component	C0000135	3970
Ordinal not found	C0000138	43
Entry point not found	C0000139	724
DLL initialization failed	C0000142	918
Module not found	C06D007E	171
Procedure not found	C06D007F	248

The corresponding graph:

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Science of Memory Dump Analysis | 1 Comment »

The Glory of Debugging

Tuesday, May 20th, 2008

Continuing Music for Debugging post I would like to recommend the following album that I used to listen to long time ago and have just found buried in the old corner. In it various movements correspond to different debugging sessions, some glorious and some filled with tension and worries, gradually building up the ultimate problem resolution. I think almost all Vangelis music is very suitable to accompany debugging.

El Greco

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, Music for Debugging, Tools | No Comments »

Application Verifier on x64 platforms

Tuesday, May 20th, 2008

A small note from the field. Sometimes on x64 Windows platform we set a default postmortem debugger or configure WER and then install Microsoft Application Verifier to do some checks. However no crash dump files are saved. The reason for this might be that we installed and configured amd64 bit version of Application Verifier but the problem application was 32-bit. For this application we need to install and configure x86 version of Application Verifier.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, Tools | 2 Comments »

WinDbg 6.9.3.113 lost type discovery?

Sunday, May 18th, 2008

Seems dt command wildcards doesn’t work anymore for ntdll and nt modules (I haven’t tested other modules):

1: kd> dt nt!* 1: kd>

Previous version of WinDbg on the same crash dump works fine. I’m listing its sorted output to preserve types for future reference in case Microsoft decided to remove this feature (NTDLL types can be found here):

nt!LIST_ENTRY64 nt!_ACCESS_STATE nt!_ACL nt!_ACTIVATION_CONTEXT nt!_ACTIVATION_CONTEXT_DATA nt!_ACTIVATION_CONTEXT_STACK nt!_ADAPTER_OBJECT nt!_ALPC_COMMUNICATION_INFO nt!_ALPC_COMPLETION_LIST nt!_ALPC_COMPLETION_LIST_HEADER nt!_ALPC_COMPLETION_LIST_STATE nt!_ALPC_COMPLETION_PACKET_LOOKASIDE nt!_ALPC_COMPLETION_PACKET_LOOKASIDE_ENTRY nt!_ALPC_DISPATCH_CONTEXT nt!_ALPC_HANDLE_ENTRY nt!_ALPC_HANDLE_TABLE nt!_ALPC_MESSAGE_ATTRIBUTES nt!_ALPC_MESSAGE_ZONE nt!_ALPC_PORT nt!_ALPC_PORT_ATTRIBUTES nt!_ALPC_PROCESS_CONTEXT nt!_ALPHA_LOADER_BLOCK nt!_ALTERNATIVE_ARCHITECTURE_TYPE nt!_AMD64_DBGKD_CONTROL_SET nt!_ARBITER_ACTION nt!_ARBITER_ADD_RESERVED_PARAMETERS nt!_ARBITER_ALLOCATION_STATE nt!_ARBITER_ALTERNATIVE nt!_ARBITER_BOOT_ALLOCATION_PARAMETERS nt!_ARBITER_CONFLICT_INFO nt!_ARBITER_INSTANCE nt!_ARBITER_INTERFACE nt!_ARBITER_LIST_ENTRY nt!_ARBITER_ORDERING nt!_ARBITER_ORDERING_LIST nt!_ARBITER_PARAMETERS nt!_ARBITER_QUERY_ALLOCATED_RESOURCES_PARAMETERS nt!_ARBITER_QUERY_ARBITRATE_PARAMETERS nt!_ARBITER_QUERY_CONFLICT_PARAMETERS nt!_ARBITER_REQUEST_SOURCE nt!_ARBITER_RESULT nt!_ARBITER_RETEST_ALLOCATION_PARAMETERS nt!_ARBITER_TEST_ALLOCATION_PARAMETERS nt!_ARC_DISK_INFORMATION nt!_ARM_DBGKD_CONTROL_SET nt!_ASSEMBLY_STORAGE_MAP nt!_BITMAP_RANGE nt!_BLOB nt!_BLOB_ID nt!_BLOB_TYPE nt!_BUS_EXTENSION_LIST nt!_CACHE_DESCRIPTOR nt!_CACHE_MANAGER_CALLBACKS nt!_CACHE_UNINITIALIZE_EVENT nt!_CACHED_CHILD_LIST nt!_CACHED_KSTACK_LIST nt!_CALL_HASH_ENTRY nt!_CALL_PERFORMANCE_DATA nt!_CELL_DATA nt!_CHILD_LIST nt!_CLIENT_ID nt!_CLS_LSN nt!_CM_BIG_DATA nt!_CM_CACHED_VALUE_INDEX nt!_CM_CELL_REMAP_BLOCK nt!_CM_FULL_RESOURCE_DESCRIPTOR nt!_CM_INDEX_HINT_BLOCK nt!_CM_INTENT_LOCK nt!_CM_KCB_UOW nt!_CM_KEY_BODY nt!_CM_KEY_CONTROL_BLOCK nt!_CM_KEY_HASH nt!_CM_KEY_INDEX nt!_CM_KEY_NODE nt!_CM_KEY_REFERENCE nt!_CM_KEY_SECURITY nt!_CM_KEY_SECURITY_CACHE nt!_CM_KEY_SECURITY_CACHE_ENTRY nt!_CM_KEY_VALUE nt!_CM_NAME_CONTROL_BLOCK nt!_CM_NAME_HASH nt!_CM_NOTIFY_BLOCK nt!_CM_PARTIAL_RESOURCE_DESCRIPTOR nt!_CM_PARTIAL_RESOURCE_LIST nt!_CM_RESOURCE_LIST nt!_CM_RM nt!_CM_TRANS nt!_CM_VIEW_OF_FILE nt!_CM_WORKITEM nt!_CMHIVE nt!_COMPRESSED_DATA_INFO nt!_CONFIGURATION_CLASS nt!_CONFIGURATION_COMPONENT nt!_CONFIGURATION_COMPONENT_DATA nt!_CONFIGURATION_TYPE nt!_CONTEXT nt!_CONTROL_AREA nt!_CURDIR nt!_DBGKD_ANY_CONTROL_SET nt!_DBGKD_BREAKPOINTEX nt!_DBGKD_CONTINUE nt!_DBGKD_CONTINUE2 nt!_DBGKD_FILL_MEMORY nt!_DBGKD_GET_CONTEXT nt!_DBGKD_GET_INTERNAL_BREAKPOINT32 nt!_DBGKD_GET_INTERNAL_BREAKPOINT64 nt!_DBGKD_GET_SET_BUS_DATA nt!_DBGKD_GET_VERSION32 nt!_DBGKD_GET_VERSION64 nt!_DBGKD_LOAD_SYMBOLS32 nt!_DBGKD_LOAD_SYMBOLS64 nt!_DBGKD_MANIPULATE_STATE32 nt!_DBGKD_MANIPULATE_STATE64 nt!_DBGKD_QUERY_MEMORY nt!_DBGKD_QUERY_SPECIAL_CALLS nt!_DBGKD_READ_MEMORY32 nt!_DBGKD_READ_MEMORY64 nt!_DBGKD_READ_WRITE_IO_EXTENDED32 nt!_DBGKD_READ_WRITE_IO_EXTENDED64 nt!_DBGKD_READ_WRITE_IO32 nt!_DBGKD_READ_WRITE_IO64 nt!_DBGKD_READ_WRITE_MSR nt!_DBGKD_RESTORE_BREAKPOINT nt!_DBGKD_SEARCH_MEMORY nt!_DBGKD_SET_CONTEXT nt!_DBGKD_SET_INTERNAL_BREAKPOINT32 nt!_DBGKD_SET_INTERNAL_BREAKPOINT64 nt!_DBGKD_SET_SPECIAL_CALL32 nt!_DBGKD_SET_SPECIAL_CALL64 nt!_DBGKD_SWITCH_PARTITION nt!_DBGKD_WRITE_BREAKPOINT32 nt!_DBGKD_WRITE_BREAKPOINT64 nt!_DBGKD_WRITE_MEMORY32 nt!_DBGKD_WRITE_MEMORY64 nt!_DBGKM_EXCEPTION32 nt!_DBGKM_EXCEPTION64 nt!_DEFERRED_WRITE nt!_DESCRIPTOR nt!_DEVICE_CAPABILITIES nt!_DEVICE_FLAGS nt!_DEVICE_MAP nt!_DEVICE_NODE nt!_DEVICE_OBJECT nt!_DEVICE_OBJECT_POWER_EXTENSION nt!_DEVICE_POWER_STATE nt!_DEVICE_RELATION_TYPE nt!_DEVICE_RELATIONS nt!_DEVICE_USAGE_NOTIFICATION_TYPE nt!_DEVOBJ_EXTENSION nt!_DEVPROPKEY nt!_DISPATCHER_HEADER nt!_DPFLTR_TYPE nt!_DRIVER_EXTENSION nt!_DRIVER_OBJECT nt!_DUAL nt!_DUMP_INITIALIZATION_CONTEXT nt!_DUMP_STACK_CONTEXT nt!_ECP_LIST nt!_EFI_FIRMWARE_INFORMATION nt!_EJOB nt!_EPROCESS nt!_EPROCESS_QUOTA_BLOCK nt!_ERESOURCE nt!_ETHREAD nt!_ETIMER nt!_ETW_BUFFER_CONTEXT nt!_ETW_BUFFER_STATE nt!_ETW_GUID_ENTRY nt!_ETW_GUID_TYPE nt!_ETW_KERNEL_TRACE_TIMESTAMP nt!_ETW_LAST_ENABLE_INFO nt!_ETW_PROVIDER_STATE nt!_ETW_PROVIDER_TABLE_ENTRY nt!_ETW_REF_CLOCK nt!_ETW_REG_ENTRY nt!_ETW_REPLY_QUEUE nt!_ETW_RT_EVENT_LOSS nt!_ETW_SYSTEMTIME nt!_EVENT_DATA_DESCRIPTOR nt!_EVENT_DESCRIPTOR nt!_EVENT_TYPE nt!_EX_FAST_REF nt!_EX_PUSH_LOCK nt!_EX_PUSH_LOCK_CACHE_AWARE nt!_EX_PUSH_LOCK_WAIT_BLOCK nt!_EX_RUNDOWN_REF nt!_EX_WORK_QUEUE nt!_EXCEPTION_DISPOSITION nt!_EXCEPTION_POINTERS nt!_EXCEPTION_RECORD nt!_EXCEPTION_RECORD32 nt!_EXCEPTION_RECORD64 nt!_EXCEPTION_REGISTRATION_RECORD nt!_FAST_IO_DISPATCH nt!_FAST_MUTEX nt!_FILE_BASIC_INFORMATION nt!_FILE_GET_QUOTA_INFORMATION nt!_FILE_INFORMATION_CLASS nt!_FILE_NETWORK_OPEN_INFORMATION nt!_FILE_OBJECT nt!_FILE_STANDARD_INFORMATION nt!_FIRMWARE_INFORMATION_LOADER_BLOCK nt!_flags nt!_FLOATING_SAVE_AREA nt!_FLS_CALLBACK_INFO nt!_FNSAVE_FORMAT nt!_FREE_DISPLAY nt!_FS_FILTER_CALLBACK_DATA nt!_FS_FILTER_CALLBACKS nt!_FS_FILTER_PARAMETERS nt!_FS_FILTER_SECTION_SYNC_TYPE nt!_FS_FILTER_STREAM_FO_NOTIFICATION_TYPE nt!_FSINFOCLASS nt!_FX_SAVE_AREA nt!_FXSAVE_FORMAT nt!_GDI_TEB_BATCH nt!_GENERAL_LOOKASIDE nt!_GENERAL_LOOKASIDE_POOL nt!_GENERIC_MAPPING nt!_GUID nt!_HANDLE_TABLE nt!_HANDLE_TABLE_ENTRY nt!_HANDLE_TABLE_ENTRY_INFO nt!_HANDLE_TRACE_DB_ENTRY nt!_HANDLE_TRACE_DEBUG_INFO nt!_HARDWARE_PTE nt!_HBASE_BLOCK nt!_HEADLESS_LOADER_BLOCK nt!_HEAP nt!_HEAP_COUNTERS nt!_HEAP_DEBUGGING_INFORMATION nt!_HEAP_ENTRY nt!_HEAP_ENTRY_EXTRA nt!_HEAP_FAILURE_TYPE nt!_HEAP_FREE_ENTRY nt!_HEAP_FREE_ENTRY_EXTRA nt!_HEAP_LIST_LOOKUP nt!_HEAP_LOCAL_SEGMENT_INFO nt!_HEAP_LOCK nt!_HEAP_LOOKASIDE nt!_HEAP_PSEUDO_TAG_ENTRY nt!_HEAP_SEGMENT nt!_HEAP_STOP_ON_TAG nt!_HEAP_STOP_ON_VALUES nt!_HEAP_SUBSEGMENT nt!_HEAP_TAG_ENTRY nt!_HEAP_TUNING_PARAMETERS nt!_HEAP_UCR_DESCRIPTOR nt!_HEAP_USERDATA_HEADER nt!_HEAP_VIRTUAL_ALLOC_ENTRY nt!_HHIVE nt!_HIVE_LIST_ENTRY nt!_HMAP_DIRECTORY nt!_HMAP_ENTRY nt!_HMAP_TABLE nt!_I386_LOADER_BLOCK nt!_IA64_DBGKD_CONTROL_SET nt!_IA64_LOADER_BLOCK nt!_IMAGE_DATA_DIRECTORY nt!_IMAGE_DEBUG_DIRECTORY nt!_IMAGE_DOS_HEADER nt!_IMAGE_FILE_HEADER nt!_IMAGE_NT_HEADERS nt!_IMAGE_OPTIONAL_HEADER nt!_IMAGE_ROM_OPTIONAL_HEADER nt!_IMAGE_SECTION_HEADER nt!_IMAGE_SECURITY_CONTEXT nt!_INITIAL_PRIVILEGE_SET nt!_INTERFACE nt!_INTERFACE_TYPE nt!_INTERLOCK_SEQ nt!_IO_ALLOCATION_ACTION nt!_IO_CLIENT_EXTENSION nt!_IO_COMPLETION_CONTEXT nt!_IO_DRIVER_CREATE_CONTEXT nt!_IO_PAGING_PRIORITY nt!_IO_PRIORITY_HINT nt!_IO_PRIORITY_INFO nt!_IO_RESOURCE_DESCRIPTOR nt!_IO_RESOURCE_LIST nt!_IO_RESOURCE_REQUIREMENTS_LIST nt!_IO_SECURITY_CONTEXT nt!_IO_STACK_LOCATION nt!_IO_STATUS_BLOCK nt!_IO_TIMER nt!_iobuf nt!_IOV_FORCED_PENDING_TRACE nt!_IRP nt!_IRQ_DEVICE_POLICY nt!_IRQ_PRIORITY nt!_JOB_ACCESS_STATE nt!_KALPC_HANDLE_DATA nt!_KALPC_MESSAGE nt!_KALPC_MESSAGE_ATTRIBUTES nt!_KALPC_REGION nt!_KALPC_RESERVE nt!_KALPC_SECTION nt!_KALPC_SECURITY_DATA nt!_KALPC_VIEW nt!_KAPC nt!_KAPC_STATE nt!_KDEVICE_QUEUE nt!_KDEVICE_QUEUE_ENTRY nt!_KDPC nt!_KDPC_DATA nt!_KENLISTMENT nt!_KENLISTMENT_HISTORY nt!_KENLISTMENT_STATE nt!_KERNEL_STACK_CONTROL nt!_KERNEL_STACK_SEGMENT nt!_KEVENT nt!_KEXECUTE_OPTIONS nt!_KGATE nt!_KGDTENTRY nt!_KGUARDED_MUTEX nt!_KIDTENTRY nt!_KiIoAccessMap nt!_KINTERRUPT nt!_KINTERRUPT_MODE nt!_KINTERRUPT_POLARITY nt!_KLOCK_QUEUE_HANDLE nt!_KMUTANT nt!_KNODE nt!_KOBJECTS nt!_KPCR nt!_KPRCB nt!_KPROCESS nt!_KPROCESSOR_STATE nt!_KQUEUE nt!_KRESOURCEMANAGER nt!_KRESOURCEMANAGER_COMPLETION_BINDING nt!_KRESOURCEMANAGER_STATE nt!_KSEMAPHORE nt!_KSPECIAL_REGISTERS nt!_KSPIN_LOCK_QUEUE nt!_KSPIN_LOCK_QUEUE_NUMBER nt!_KSYSTEM_TIME nt!_KTHREAD nt!_KTHREAD_STATE nt!_KTIMER nt!_KTIMER_TABLE_ENTRY nt!_KTM nt!_KTMNOTIFICATION_PACKET nt!_KTMOBJECT_NAMESPACE nt!_KTMOBJECT_NAMESPACE_LINK nt!_KTRANSACTION nt!_KTRANSACTION_HISTORY nt!_KTRANSACTION_OUTCOME nt!_KTRANSACTION_STATE nt!_KTRAP_FRAME nt!_KTSS nt!_KUSER_SHARED_DATA nt!_KWAIT_BLOCK nt!_KWAIT_REASON nt!_LARGE_INTEGER nt!_LAZY_WRITER nt!_LDR_DATA_TABLE_ENTRY nt!_LIST_ENTRY nt!_LOADER_PARAMETER_BLOCK nt!_LOADER_PARAMETER_EXTENSION nt!_LOADER_PERFORMANCE_DATA nt!_LOOKASIDE_LIST_EX nt!_LPCP_MESSAGE nt!_LPCP_NONPAGED_PORT_QUEUE nt!_LPCP_PORT_OBJECT nt!_LPCP_PORT_QUEUE nt!_LUID nt!_LUID_AND_ATTRIBUTES nt!_MAILSLOT_CREATE_PARAMETERS nt!_MAPPED_FILE_SEGMENT nt!_MBCB nt!_MDL nt!_MEMORY_ALLOCATION_DESCRIPTOR nt!_MEMORY_CACHING_TYPE nt!_MEMORY_CACHING_TYPE_ORIG nt!_MI_COLOR_BASE nt!_MI_DYNAMIC_MEMORY_LOCKTYPE nt!_MI_EXTRA_IMAGE_INFORMATION nt!_MI_IMAGE_SECURITY_REFERENCE nt!_MI_PAGEFILE_TRACES nt!_MI_PER_SESSION_PROTOS nt!_MI_PFN_CACHE_ATTRIBUTE nt!_MI_SECTION_CREATION_GATE nt!_MI_SECTION_IMAGE_INFORMATION nt!_MI_SPECIAL_POOL nt!_MI_SYSTEM_PTE_TYPE nt!_MI_SYSTEM_VA_TYPE nt!_MI_VAD_TYPE nt!_MI_VERIFIER_DRIVER_ENTRY nt!_MI_VERIFIER_POOL_HEADER nt!_MM_AVL_TABLE nt!_MM_DRIVER_VERIFIER_DATA nt!_MM_PAGE_ACCESS_INFO nt!_MM_PAGE_ACCESS_INFO_FLAGS nt!_MM_PAGE_ACCESS_INFO_HEADER nt!_MM_PAGE_ACCESS_TYPE nt!_MM_PAGED_POOL_INFO nt!_MM_POOL_FAILURE_REASONS nt!_MM_POOL_PRIORITIES nt!_MM_POOL_TYPES nt!_MM_PREEMPTIVE_TRIMS nt!_MM_SESSION_SPACE nt!_MM_SESSION_SPACE_FLAGS nt!_MM_SUBSECTION_AVL_TABLE nt!_MMADDRESS_LIST nt!_MMADDRESS_NODE nt!_MMBANKED_SECTION nt!_MMEXTEND_INFO nt!_MMLISTS nt!_MMMOD_WRITER_MDL_ENTRY nt!_MMPAGING_FILE nt!_MMPAGING_FILE_FREE_ENTRY nt!_MMPFN nt!_MMPFNENTRY nt!_MMPFNLIST nt!_MMPTE nt!_MMPTE_FLUSH_LIST nt!_MMPTE_HARDWARE nt!_MMPTE_HIGHLOW nt!_MMPTE_LIST nt!_MMPTE_PROTOTYPE nt!_MMPTE_SOFTWARE nt!_MMPTE_SUBSECTION nt!_MMPTE_TIMESTAMP nt!_MMPTE_TRANSITION nt!_MMSECTION_FLAGS nt!_MMSECURE_FLAGS nt!_MMSESSION nt!_MMSUBSECTION_FLAGS nt!_MMSUBSECTION_NODE nt!_MMSUPPORT nt!_MMSUPPORT_FLAGS nt!_MMVAD nt!_MMVAD_FLAGS nt!_MMVAD_FLAGS2 nt!_MMVAD_FLAGS3 nt!_MMVAD_LONG nt!_MMVAD_SHORT nt!_MMVIEW nt!_MMWSL nt!_MMWSLE nt!_MMWSLE_FREE_ENTRY nt!_MMWSLE_HASH nt!_MMWSLE_NONDIRECT_HASH nt!_MMWSLENTRY nt!_MODE nt!_MSUBSECTION nt!_NAMED_PIPE_CREATE_PARAMETERS nt!_NETWORK_LOADER_BLOCK nt!_NLS_DATA_BLOCK nt!_NPAGED_LOOKASIDE_LIST nt!_NT_PRODUCT_TYPE nt!_NT_TIB nt!_OB_DUPLICATE_OBJECT_STATE nt!_OB_OPEN_REASON nt!_OBJECT_ATTRIBUTES nt!_OBJECT_CREATE_INFORMATION nt!_OBJECT_DIRECTORY nt!_OBJECT_DIRECTORY_ENTRY nt!_OBJECT_DUMP_CONTROL nt!_OBJECT_HANDLE_COUNT_DATABASE nt!_OBJECT_HANDLE_COUNT_ENTRY nt!_OBJECT_HANDLE_INFORMATION nt!_OBJECT_HEADER nt!_OBJECT_HEADER_CREATOR_INFO nt!_OBJECT_HEADER_HANDLE_INFO nt!_OBJECT_HEADER_NAME_INFO nt!_OBJECT_HEADER_QUOTA_INFO nt!_OBJECT_INFORMATION_CLASS nt!_OBJECT_NAME_INFORMATION nt!_OBJECT_REF_INFO nt!_OBJECT_REF_STACK_INFO nt!_OBJECT_REF_TRACE nt!_OBJECT_SYMBOLIC_LINK nt!_OBJECT_TYPE nt!_OBJECT_TYPE_INITIALIZER nt!_OBP_LOOKUP_CONTEXT nt!_OWNER_ENTRY nt!_PAGED_LOOKASIDE_LIST nt!_PAGEFAULT_HISTORY nt!_PCAT_FIRMWARE_INFORMATION nt!_PCI_HOTPLUG_SLOT_INTERRUPT nt!_PEB nt!_PEB_LDR_DATA nt!_PERFINFO_GROUPMASK nt!_PERFINFO_HARDPAGEFAULT_INFORMATION nt!_PERFINFO_TRACE_HEADER nt!_PF_FILE_ACCESS_TYPE nt!_PF_HARD_FAULT_INFO nt!_PF_KERNEL_GLOBALS nt!_PHYSICAL_MEMORY_DESCRIPTOR nt!_PHYSICAL_MEMORY_RUN nt!_PI_BUS_EXTENSION nt!_PI_RESOURCE_ARBITER_ENTRY nt!_PLUGPLAY_EVENT_BLOCK nt!_PLUGPLAY_EVENT_CATEGORY nt!_PNP_ASSIGN_RESOURCES_CONTEXT nt!_PNP_DEVICE_COMPLETION_QUEUE nt!_PNP_DEVICE_EVENT_ENTRY nt!_PNP_DEVICE_EVENT_LIST nt!_PNP_DEVNODE_STATE nt!_PNP_RESOURCE_REQUEST nt!_PNP_VETO_TYPE nt!_PO_DEVICE_NOTIFY nt!_PO_DEVICE_NOTIFY_ORDER nt!_PO_HIBER_PERF nt!_PO_IRP_MANAGER nt!_PO_IRP_QUEUE nt!_PO_MEMORY_RANGE_ARRAY nt!_PO_MEMORY_RANGE_ARRAY_LINK nt!_PO_MEMORY_RANGE_ARRAY_RANGE nt!_PO_NOTIFY_ORDER_LEVEL nt!_POLICY_AUDIT_EVENT_TYPE nt!_POOL_BLOCK_HEAD nt!_POOL_DESCRIPTOR nt!_POOL_HACKER nt!_POOL_HEADER nt!_POOL_TRACKER_BIG_PAGES nt!_POOL_TRACKER_TABLE nt!_POOL_TYPE nt!_POP_ACTION_TRIGGER nt!_POP_CPU_INFO nt!_POP_DEVICE_SYS_STATE nt!_POP_DISPLAY_RESUME_CONTEXT nt!_POP_HIBER_CONTEXT nt!_POP_POWER_ACTION nt!_POP_SHUTDOWN_BUG_CHECK nt!_POP_THERMAL_ZONE nt!_POP_TRIGGER_WAIT nt!_PORT_MESSAGE nt!_POWER_CHANNEL_SUMMARY nt!_POWER_SEQUENCE nt!_POWER_STATE nt!_POWER_STATE_TYPE nt!_PP_LOOKASIDE_LIST nt!_PP_NPAGED_LOOKASIDE_NUMBER nt!_PPM_DIA_STATS nt!_PPM_IDLE_STATE nt!_PPM_IDLE_STATES nt!_PPM_PERF_STATE nt!_PPM_PERF_STATES nt!_PRIVATE_CACHE_MAP nt!_PRIVATE_CACHE_MAP_FLAGS nt!_PRIVILEGE_SET nt!_PROCESSOR_CACHE_TYPE nt!_PROCESSOR_POWER_STATE nt!_PROFILE_PARAMETER_BLOCK nt!_PROXY_CLASS nt!_PS_CLIENT_SECURITY_CONTEXT nt!_PS_RESOURCE_TYPE nt!_PSP_RATE_APC nt!_PTE_TRACKER nt!_QUAD nt!_REG_NOTIFY_CLASS nt!_REMOTE_PORT_VIEW nt!_RTL_ACTIVATION_CONTEXT_STACK_FRAME nt!_RTL_ATOM_TABLE nt!_RTL_ATOM_TABLE_ENTRY nt!_RTL_AVL_TABLE nt!_RTL_BALANCED_LINKS nt!_RTL_BITMAP nt!_RTL_CRITICAL_SECTION nt!_RTL_CRITICAL_SECTION_DEBUG nt!_RTL_DRIVE_LETTER_CURDIR nt!_RTL_GENERIC_COMPARE_RESULTS nt!_RTL_HANDLE_TABLE nt!_RTL_HANDLE_TABLE_ENTRY nt!_RTL_RANGE nt!_RTL_RANGE_LIST nt!_RTL_SRWLOCK nt!_RTL_USER_PROCESS_PARAMETERS nt!_RTLP_RANGE_LIST_ENTRY nt!_SCSI_REQUEST_BLOCK nt!_SE_AUDIT_PROCESS_CREATION_INFO nt!_SECTION_IMAGE_INFORMATION nt!_SECTION_OBJECT nt!_SECTION_OBJECT_POINTERS nt!_SECURITY_CLIENT_CONTEXT nt!_SECURITY_DESCRIPTOR nt!_SECURITY_DESCRIPTOR_RELATIVE nt!_SECURITY_IMPERSONATION_LEVEL nt!_SECURITY_OPERATION_CODE nt!_SECURITY_QUALITY_OF_SERVICE nt!_SECURITY_SUBJECT_CONTEXT nt!_SECURITY_TOKEN_AUDIT_DATA nt!_SECURITY_TOKEN_PROXY_DATA nt!_SEGMENT nt!_SEGMENT_FLAGS nt!_SEGMENT_OBJECT nt!_SEP_AUDIT_POLICY nt!_SEP_LOGON_SESSION_REFERENCES nt!_SEP_TOKEN_PRIVILEGES nt!_SETUP_LOADER_BLOCK nt!_SHARED_CACHE_MAP nt!_SHARED_CACHE_MAP_LIST_CURSOR nt!_SID nt!_SID_AND_ATTRIBUTES nt!_SID_AND_ATTRIBUTES_HASH nt!_SID_IDENTIFIER_AUTHORITY nt!_SINGLE_LIST_ENTRY nt!_SLIST_HEADER nt!_SMBIOS_TABLE_HEADER nt!_STACK_TABLE nt!_STRING nt!_SUBSECTION nt!_SYSPTES_HEADER nt!_SYSTEM_POWER_POLICY nt!_SYSTEM_POWER_STATE nt!_SYSTEM_POWER_STATE_CONTEXT nt!_SYSTEM_TRACE_HEADER nt!_TEB nt!_TEB_ACTIVE_FRAME nt!_TEB_ACTIVE_FRAME_CONTEXT nt!_TERMINATION_PORT nt!_THERMAL_INFORMATION nt!_THERMAL_INFORMATION_EX nt!_TIME_FIELDS nt!_TOKEN nt!_TOKEN_AUDIT_POLICY nt!_TOKEN_CONTROL nt!_TOKEN_SOURCE nt!_TOKEN_TYPE nt!_TP_CALLBACK_ENVIRON nt!_TP_CALLBACK_INSTANCE nt!_TP_CLEANUP_GROUP nt!_TP_DIRECT nt!_TP_POOL nt!_TP_TASK nt!_TP_TASK_CALLBACKS nt!_TRACE_ENABLE_CONTEXT nt!_TRACE_ENABLE_CONTEXT_EX nt!_TRACE_ENABLE_INFO nt!_TXN_PARAMETER_BLOCK nt!_TYPE_OF_MEMORY nt!_u nt!_ULARGE_INTEGER nt!_UNICODE_STRING nt!_VACB nt!_VACB_ARRAY_HEADER nt!_VACB_LEVEL_REFERENCE nt!_VF_BTS_DATA_MANAGEMENT_AREA nt!_VF_BTS_RECORD nt!_VF_POOL_TRACE nt!_VF_TRACKER nt!_VF_TRACKER_STAMP nt!_VI_CANCEL_GLOBALS nt!_VI_CNT_INDEX_TYPE nt!_VI_DEADLOCK_ADDRESS_RANGE nt!_VI_DEADLOCK_GLOBALS nt!_VI_DEADLOCK_NODE nt!_VI_DEADLOCK_RESOURCE nt!_VI_DEADLOCK_RESOURCE_TYPE nt!_VI_DEADLOCK_THREAD nt!_VI_FAULT_TRACE nt!_VI_POOL_ENTRY nt!_VI_POOL_ENTRY_INUSE nt!_VI_POOL_PAGE_HEADER nt!_VI_TRACK_IRQL nt!_VI_VERIFIER_ISSUE nt!_VIRTUAL_EFI_RUNTIME_SERVICES nt!_VOLUME_CACHE_MAP nt!_VPB nt!_WAIT_CONTEXT_BLOCK nt!_WAIT_TYPE nt!_WHEA_ERROR_PACKET nt!_WHEA_ERROR_PACKET_FLAGS nt!_WHEA_ERROR_RECORD nt!_WHEA_ERROR_RECORD_HEADER nt!_WHEA_ERROR_RECORD_HEADER_FLAGS nt!_WHEA_ERROR_RECORD_HEADER_VALIDBITS nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR_FLAGS nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR_VALIDBITS nt!_WHEA_ERROR_SEVERITY nt!_WHEA_ERROR_SOURCE_TYPE nt!_WHEA_ERROR_STATUS nt!_WHEA_ERROR_TYPE nt!_WHEA_GENERIC_PROCESSOR_ERROR nt!_WHEA_GENERIC_PROCESSOR_ERROR_VALIDBITS nt!_WHEA_MEMORY_ERROR nt!_WHEA_MEMORY_ERROR_VALIDBITS nt!_WHEA_NMI_ERROR nt!_WHEA_NMI_ERROR_FLAGS nt!_WHEA_PCIEXPRESS_BRIDGE_CONTROL_STATUS nt!_WHEA_PCIEXPRESS_COMMAND_STATUS nt!_WHEA_PCIEXPRESS_DEVICE_ID nt!_WHEA_PCIEXPRESS_ERROR nt!_WHEA_PCIEXPRESS_ERROR_VALIDBITS nt!_WHEA_PCIEXPRESS_VERSION nt!_WHEA_PCIXBUS_COMMAND nt!_WHEA_PCIXBUS_ERROR nt!_WHEA_PCIXBUS_ERROR_VALIDBITS nt!_WHEA_PCIXBUS_ID nt!_WHEA_PCIXDEVICE_ERROR nt!_WHEA_PCIXDEVICE_ERROR_VALIDBITS nt!_WHEA_PCIXDEVICE_ID nt!_WHEA_PERSISTENCE_INFO nt!_WHEA_RAW_DATA_FORMAT nt!_WHEA_REVISION nt!_WHEA_TIMESTAMP nt!_WMI_BUFFER_HEADER nt!_WMI_LOGGER_CONTEXT nt!_WMI_TRACE_PACKET nt!_WNODE_HEADER nt!_WORK_QUEUE_ENTRY nt!_WORK_QUEUE_ITEM nt!_WOW64_SHARED_INFORMATION nt!_X86_DBGKD_CONTROL_SET nt!BATTERY_REPORTING_SCALE nt!BUS_QUERY_ID_TYPE nt!CMP_OFFSET_ARRAY nt!DEVICE_TEXT_TYPE nt!EX_QUEUE_WORKER_INFO nt!HSTORAGE_TYPE nt!KTM_STATE nt!LIST_ENTRY32 nt!LIST_ENTRY64 nt!LSA_FOREST_TRUST_RECORD_TYPE nt!PO_MEMORY_IMAGE nt!POP_POLICY_DEVICE_TYPE nt!POWER_ACTION nt!POWER_ACTION_POLICY nt!PPM_IDLE_ACCOUNTING nt!PPM_IDLE_STATE_ACCOUNTING nt!PROCESSOR_IDLESTATE_INFO nt!PROCESSOR_IDLESTATE_POLICY nt!PROCESSOR_PERFSTATE_POLICY nt!PROFILE_STATUS nt!ReplacesCorHdrNumericDefines nt!SYSTEM_POWER_CAPABILITIES nt!SYSTEM_POWER_CONDITION nt!SYSTEM_POWER_LEVEL nt!UoWActionType nt!WHEA_PCIEXPRESS_DEVICE_TYPE nt!WHEA_PCIXDEVICE_REGISTER_PAIR

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging | 3 Comments »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘Debugging’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta