Archive for December 12th, 2011

Crash Dump Analysis Patterns (Part 161)

Monday, December 12th, 2011

This is another stack trace related pattern that we call Empty Stack Trace. Here we might need to do manual stack trace reconstruction like in the following example:

0:002> ~2s
eax=00000070 ebx=0110fb94 ecx=00000010 edx=005725d8 esi=0110fe58 edi=00000d80
eip=7c82847c esp=0110efe0 ebp=0110eff0 iopl=0  nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000      efl=00000246
ntdll!KiFastSystemCallRet:
7c82847c c3              ret

0:002> kL
ChildEBP RetAddr
0110efdc 00000000 ntdll!KiFastSystemCallRet

0:002> !teb
TEB at 7ffdc000
ExceptionList:        0110f980
StackBase:            01110000
StackLimit:           0110d000
SubSystemTib:         00000000
FiberData:            00001e00
ArbitraryUserPointer: 00000000
Self:                 7ffdc000
EnvironmentPointer:   00000000
ClientId:             00000b04 . 00000bd0
RpcHandle:            00000000
Tls Storage:          00000000
PEB Address:          7ffda000
LastErrorValue:       87
LastStatusValue:      c000000d
Count Owned Locks:    0
HardErrorMode:        0

0:002> dps 0110d000 01110000
0110d000  00000000
0110d004  00000000
[...]
0110f63c  00001000
0110f640  0110f64c
0110f644  02b91ea8
0110f648  00001000
0110f64c  00000004
0110f650  0110f6f0
0110f654  0374669d DbgHelp!WriteFullMemory+0×3cd
0110f658  ffffffff
0110f65c  0110d000
0110f660  00000000
0110f664  0480f5c0
0110f668  00003000
0110f66c  0110f7b0
0110f670  0110d000
0110f674  00000000
0110f678  00000065
0110f67c  00003000
0110f680  0110d000
0110f684  00000000
0110f688  01010000
0110f68c  00000000
0110f690  00000004
0110f694  00060002
0110f698  00003000
0110f69c  00000000
0110f6a0  00001000
0110f6a4  00000004
0110f6a8  00020000
0110f6ac  00040004
0110f6b0  7ffe0000 SharedUserData
0110f6b4  00000000
0110f6b8  00001000
0110f6bc  00000000
0110f6c0  0480f5c0
0110f6c4  00000000
0110f6c8  04c4a000
0110f6cc  00000000
0110f6d0  000003c7
0110f6d4  00000000
0110f6d8  00023b17
0110f6dc  00000000
0110f6e0  01110000
0110f6e4  00000000
0110f6e8  0099f000
0110f6ec  00000000
0110f6f0  0110f704
0110f6f4  037469d6 DbgHelp!WriteDumpData+0×206
0110f6f8  0110f738
0110f6fc  0110f7b0
0110f700  00000000
0110f704  0110f868
0110f708  03747449 DbgHelp!MiniDumpProvideDump+0×359
0110f70c  0110f738
0110f710  0110f7b0
0110f714  02b91fb0
0110f718  00000000
0110f71c  00000000
0110f720  00000000
0110f724  02b91fb0
0110f728  00000000
0110f72c  00000000
[…]
0110ff1c  00000001
0110ff20  00000008
0110ff24  0000000a
0110ff28  33017f51 ModuleA!Run+0xde
0110ff2c  00000001
0110ff30  0110ff74
0110ff34  00f08898
0110ff38  00000000
0110ff3c  00f082a8
0110ff40  00000000
0110ff44  00000001
0110ff48  33017e33 ModuleA!ThreadProc+0×2c
0110ff4c  a9b21e1e
0110ff50  00000000
0110ff54  00000000
0110ff58  00f08898
0110ff5c  0110ff4c
0110ff60  0110ffac
0110ff64  0110ff9c
0110ff68  33054245
0110ff6c  9ba52ad2
0110ff70  00000000
0110ff74  0110ffac
0110ff78  78543433 msvcr90!_endthreadex+0×44
0110ff7c  00f082a8
0110ff80  a9b2b0d3
0110ff84  00000000
0110ff88  00000000
0110ff8c  00f08898
0110ff90  0110ff80
0110ff94  0110ff80
0110ff98  0110ffdc
0110ff9c  0110ffdc
0110ffa0  7858cf5e msvcr90!_except_handler4
0110ffa4  d0f887df
0110ffa8  00000000
0110ffac  0110ffb8
0110ffb0  785434c7 msvcr90!_endthreadex+0xd8
0110ffb4  00000000
0110ffb8  0110ffec
0110ffbc  77e6482f kernel32!BaseThreadStart+0×34
0110ffc0  00f08898
0110ffc4  00000000
0110ffc8  00000000
0110ffcc  00f08898
0110ffd0  00000000
0110ffd4  0110ffc4
0110ffd8  80833bcc
0110ffdc  ffffffff
0110ffe0  77e61a60 kernel32!_except_handler3
0110ffe4  77e64838 kernel32!`string’+0×98
0110ffe8  00000000
0110ffec  00000000
0110fff0  00000000
0110fff4  7854345e msvcr90!_endthreadex+0×6f
0110fff8  00f08898
0110fffc  00000000
01110000  00000130

0:002> k L=0110f650 0110f650  0110f650
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0110f650 0374669d 0x110f650
0110f6f0 037469d6 DbgHelp!WriteFullMemory+0x3cd
0110f704 03747449 DbgHelp!WriteDumpData+0x206
0110f868 03747662 DbgHelp!MiniDumpProvideDump+0x359
0110f8dc 33050dd9 DbgHelp!MiniDumpWriteDump+0x1b2
[...]
0110fdfc 33031726 ModuleA!WriteExceptionMiniDump+0x50
0110fea0 33018c81 ModuleA!ThreadHung+0x6c
[...]
0110ff44 33017e33 ModuleA!Run+0xde
00000000 00000000 ModuleA!ThreadProc+0x2c

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

2012

Monday, December 12th, 2011

This is a specially commissioned artwork for the first celebration of Memoristmas. Those in the know will instantly recognize processor timing diagram:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg shortcuts: .ecxr

Monday, December 12th, 2011

If you are impatient with !analyze -v you can always use a replacement command that shows and sets the context for the current exception so you can quickly get to the possible crashing point (signature):

0:000> .ecxr
eax=00000000 ebx=00000001 ecx=00000000 edx=0018fe40 esi=00426310 edi=00000111
eip=0041ff21 esp=0018f81c ebp=0018f850 iopl=0  nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00010246
*** ERROR: Module load completed but symbols could not be loaded for TestWER.exe
TestWER+0x1ff21:
0041ff21 c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0018f850 00403620 TestWER+0x1ff21
0018f860 0040382f TestWER+0x3620
0018f890 00402df6 TestWER+0x382f
0018f8b4 00409ef8 TestWER+0x2df6
0018f904 0040a792 TestWER+0x9ef8
0018f9a0 00406dea TestWER+0xa792
0018f9c0 00409713 TestWER+0x6dea
0018fa28 004097a2 TestWER+0x9713
0018fa48 76f66238 TestWER+0x97a2
0018fa74 76f668ea user32!InternalCallWinProc+0x23
0018faec 76f6cd1a user32!UserCallWinProcCheckWow+0x109
0018fb30 76f6cd81 user32!SendMessageWorker+0x581
0018fb54 74fb4e95 user32!SendMessageW+0x7f
0018fb74 74fb4ef7 comctl32!Button_NotifyParent+0x3d
0018fb90 74fb4d89 comctl32!Button_ReleaseCapture+0x113
0018fbf0 76f66238 comctl32!Button_WndProc+0xa18
0018fc1c 76f668ea user32!InternalCallWinProc+0x23
0018fc94 76f67d31 user32!UserCallWinProcCheckWow+0x109
0018fcf4 76f67dfa user32!DispatchMessageWorker+0x3bc
0018fd04 76f82292 user32!DispatchMessageW+0xf
0018fd30 0040618c user32!IsDialogMessageW+0x5f6
0018fd44 004071e2 TestWER+0x618c
0018fd50 00402dd3 TestWER+0x71e2
0018fd64 00408dc1 TestWER+0x2dd3
0018fd78 00403f35 TestWER+0x8dc1
0018fd90 00404090 TestWER+0x3f35
0018fd9c 00403f80 TestWER+0x4090
0018fda8 004040dd TestWER+0x3f80
0018fde0 00403440 TestWER+0x40dd
0018fe2c 004204ee TestWER+0x3440
0018fee4 0041fdf5 TestWER+0x204ee
0018fef8 0040fc3e TestWER+0x1fdf5
0018ff88 76ce3677 TestWER+0xfc3e
0018ff94 77b89f02 kernel32!BaseThreadInitThunk+0xe
0018ffd4 77b89ed5 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

However, in case of multiple exceptions you still need to do stack trace collection analysis:

0:000> .ecxr
eax=00000030 ebx=7efde000 ecx=750d2dd9 edx=00000000 esi=00000000 edi=00000000
eip=770d280c esp=0037f828 ebp=0037f870 iopl=0  nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00000202
KERNELBASE!DebugBreak+0x2:
770d280c cc              int     3

0:000> ~*k 6

.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0037f1a4 770d0bdd ntdll!NtWaitForMultipleObjects+0x15
0037f240 7529162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0037f288 75291921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0037f2a4 752b9b2d kernel32!WaitForMultipleObjects+0x18
0037f310 752b9bca kernel32!WerpReportFaultInternal+0x186
0037f324 752b98f8 kernel32!WerpReportFault+0×70

1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
0080f9ac 770d31bb ntdll!NtDelayExecution+0x15
0080fa14 770d3a8b KERNELBASE!SleepEx+0x65
0080fa24 752d28dd KERNELBASE!Sleep+0xf
0080fa38 752b98f8 kernel32!WerpReportFault+0×3f
0080fa48 752b9875 kernel32!BasepReportFault+0×20
0080fad4 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

2  Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
00abf640 770d31bb ntdll!NtDelayExecution+0x15
00abf6a8 770d3a8b KERNELBASE!SleepEx+0x65
00abf6b8 752d28dd KERNELBASE!Sleep+0xf
00abf6cc 752b98f8 kernel32!WerpReportFault+0×3f
00abf6dc 752b9875 kernel32!BasepReportFault+0×20
00abf768 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

What is Software Trace and Memory Dump Analysis? A One Sentence Definition

Monday, December 12th, 2011

More than 4 years passed since I provided a longer structuralist definition. Recently I came to recognize a pattern-driven iterative and incremental nature of memory and software trace analysis and post-construction software problem solving in general and therefore a one sentence definition became necessary:

“Recognition and interpretation of patterns of software behavior”

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memoristmas

Monday, December 12th, 2011

This is an annual celebration at the overflow boundary 31 - 32 [1] (December - January). Its date is kept coincidental with The New Year to allow backward and legacy compatibility. It is an official celebration in memory religion, Memorianity, but it is also an open one and not particularly tied to it similar to other religious celebrations that became secular holidays. A series of special artistic images and pictures have been commissioned for the first Memoristmas, so stay tuned (listen to memory for news). If you are curious about etymology of this new word please take a note that -mas suffix denotes memory analysis service.

Dmitry Vostokov,
Memoriarch

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -