Archive for the ‘WinDbg Tips and Tricks’ Category

Crash Dump Analysis Patterns (Part 162)

Wednesday, December 14th, 2011

Sometimes Problem Module pattern can help in troubleshooting. Problem modules (including process names) are components that due to their value adding behaviour might break normal software behaviour and therefore require some troubleshooting workarounds from minor configuration changes to complete removal. Typical examples include memory optimization services for terminal services environments or hooksware. Typically you can see main process modules in the output of !vm or !process 0 0 commands. lm command will list module names such as DLLs from a process memory dump, lmk command can give you the list of kernel space modules (for example, drivers) from kernel and complete memory dumps, and the following command lists all user space modules for each process in a complete memory dump:

!for_each_process ".process /r /p @#Process; lmu"

Of course you can also try various lm command variants if you are interested in timestamps and module information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg shortcuts: .ecxr

Monday, December 12th, 2011

If you are impatient with !analyze -v you can always use a replacement command that shows and sets the context for the current exception so you can quickly get to the possible crashing point (signature):

0:000> .ecxr
eax=00000000 ebx=00000001 ecx=00000000 edx=0018fe40 esi=00426310 edi=00000111
eip=0041ff21 esp=0018f81c ebp=0018f850 iopl=0  nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00010246
*** ERROR: Module load completed but symbols could not be loaded for TestWER.exe
TestWER+0x1ff21:
0041ff21 c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0018f850 00403620 TestWER+0x1ff21
0018f860 0040382f TestWER+0x3620
0018f890 00402df6 TestWER+0x382f
0018f8b4 00409ef8 TestWER+0x2df6
0018f904 0040a792 TestWER+0x9ef8
0018f9a0 00406dea TestWER+0xa792
0018f9c0 00409713 TestWER+0x6dea
0018fa28 004097a2 TestWER+0x9713
0018fa48 76f66238 TestWER+0x97a2
0018fa74 76f668ea user32!InternalCallWinProc+0x23
0018faec 76f6cd1a user32!UserCallWinProcCheckWow+0x109
0018fb30 76f6cd81 user32!SendMessageWorker+0x581
0018fb54 74fb4e95 user32!SendMessageW+0x7f
0018fb74 74fb4ef7 comctl32!Button_NotifyParent+0x3d
0018fb90 74fb4d89 comctl32!Button_ReleaseCapture+0x113
0018fbf0 76f66238 comctl32!Button_WndProc+0xa18
0018fc1c 76f668ea user32!InternalCallWinProc+0x23
0018fc94 76f67d31 user32!UserCallWinProcCheckWow+0x109
0018fcf4 76f67dfa user32!DispatchMessageWorker+0x3bc
0018fd04 76f82292 user32!DispatchMessageW+0xf
0018fd30 0040618c user32!IsDialogMessageW+0x5f6
0018fd44 004071e2 TestWER+0x618c
0018fd50 00402dd3 TestWER+0x71e2
0018fd64 00408dc1 TestWER+0x2dd3
0018fd78 00403f35 TestWER+0x8dc1
0018fd90 00404090 TestWER+0x3f35
0018fd9c 00403f80 TestWER+0x4090
0018fda8 004040dd TestWER+0x3f80
0018fde0 00403440 TestWER+0x40dd
0018fe2c 004204ee TestWER+0x3440
0018fee4 0041fdf5 TestWER+0x204ee
0018fef8 0040fc3e TestWER+0x1fdf5
0018ff88 76ce3677 TestWER+0xfc3e
0018ff94 77b89f02 kernel32!BaseThreadInitThunk+0xe
0018ffd4 77b89ed5 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

However, in case of multiple exceptions you still need to do stack trace collection analysis:

0:000> .ecxr
eax=00000030 ebx=7efde000 ecx=750d2dd9 edx=00000000 esi=00000000 edi=00000000
eip=770d280c esp=0037f828 ebp=0037f870 iopl=0  nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00000202
KERNELBASE!DebugBreak+0x2:
770d280c cc              int     3

0:000> ~*k 6

.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0037f1a4 770d0bdd ntdll!NtWaitForMultipleObjects+0x15
0037f240 7529162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0037f288 75291921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0037f2a4 752b9b2d kernel32!WaitForMultipleObjects+0x18
0037f310 752b9bca kernel32!WerpReportFaultInternal+0x186
0037f324 752b98f8 kernel32!WerpReportFault+0×70

1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
0080f9ac 770d31bb ntdll!NtDelayExecution+0x15
0080fa14 770d3a8b KERNELBASE!SleepEx+0x65
0080fa24 752d28dd KERNELBASE!Sleep+0xf
0080fa38 752b98f8 kernel32!WerpReportFault+0×3f
0080fa48 752b9875 kernel32!BasepReportFault+0×20
0080fad4 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

2  Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
00abf640 770d31bb ntdll!NtDelayExecution+0x15
00abf6a8 770d3a8b KERNELBASE!SleepEx+0x65
00abf6b8 752d28dd KERNELBASE!Sleep+0xf
00abf6cc 752b98f8 kernel32!WerpReportFault+0×3f
00abf6dc 752b9875 kernel32!BasepReportFault+0×20
00abf768 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 159)

Monday, December 5th, 2011

Sometimes we have a value or a pointer or a handle and would like to know all memory addresses that reference it. This can be done by virtual memory search (s WinDbg command). If you look for references in code (for example, or pool tags please see this case study) you can combine search with !for_each_module WinDbg extension command. There is also !search command for physical pages. We cover this Value References pattern in the forthcoming Advanced Windows Memory Dump Analysis training with a step-by-step complete memory dump analysis exercise. For object references there is also recently added !obtrace command with good examples in WinDbg help.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

2 WinDbg Scripts That Changed The World

Saturday, December 3rd, 2011

If not for you then definitely for me because I now diagnose Spiking Thread pattern much faster. One of the readers if this blog asked me whether there is !runaway command equivalent for kernel and complete memory dumps. So, after some thinking I gave it a try especially in the context of WinDbg scripting exercises designed for Advanced Windows Memory Dump Analysis training. As a result I wrote 2 scripts initially that you can try yourself. Their output here is taken from a complete memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation.

The first one dumps the most CPU consuming threads for user and kernel mode:

$$
$$ krunawaymost.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest UserTime value: ”
? @$t0
!thread @$t2 ff
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest KernelTime value: ”
? @$t0
!thread @$t2 ff

0: kd> $$><c:\Scripts\krunawaymost.wds
The largest UserTime value:
Evaluate expression: 5470 = 00000000`0000155e

THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

The largest KernelTime value:
Evaluate expression: 187 = 00000000`000000bb

THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The second script takes two arguments and shows all threads that have UserTime and KernelTime ticks value greater than (you can have the idea of the maximum from the previous script):

$$
$$ krunawaygt.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); r $t0 = $arg1; .if (@$t1 > @$t0) {!thread @#Thread ff}”
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); r $t0 = $arg2; .if (@$t1 > @$t0) {!thread @#Thread ff}”

Using hints from the previous script run (the largest UserTime ticks value is 0×155e) we now get threads that spent more than 0×100 ticks in user mode:

0: kd> $$>a<c:\Scripts\krunawaygt.wds 100 100
THREAD fffffa800843e060  Cid 03f4.0658  Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21211          Ticks: 1084 (0:00:00:16.910)
Context Switch Count      6028            
UserTime                  00:00:10.140
KernelTime                00:00:00.296
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d4bc70 Current fffff88009d4b660
Base fffff88009d4c000 Limit fffff88009d46000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d4b6a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d4b7e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d4b870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d4b930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d4b9c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d4ba70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d4bae0)
00000000`0209fb08 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`0209fb10 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`0209fb70 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`0209fbb0 00000000`0045cbc0 mprtp!MpPluginSignatureChange+0×3e170
00000000`0209fbb8 000007fe`fbac25ff 0×45cbc0
00000000`0209fbc0 00000000`00466610 FLTLIB!FilterGetMessage+0×2b
00000000`0209fc20 00000000`00000000 0×466610

THREAD fffffa800845c060  Cid 03f4.065c  Teb: 000007fffff8e000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa800843c2c0  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      21520          Ticks: 775 (0:00:00:12.090)
Context Switch Count      4979            
UserTime                  00:00:04.149
KernelTime                00:00:00.156
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d52c70 Current fffff88009d52660
Base fffff88009d53000 Limit fffff88009d4d000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mprtp.dll -
Child-SP          RetAddr           Call Site
fffff880`09d526a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d527e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d52870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d52930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d529c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d52a70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d52ae0)
00000000`01ccf498 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`01ccf4a0 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`01ccf500 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`01ccf540 00000000`0045d030 mprtp!MpPluginSignatureChange+0×3e170
00000000`01ccf548 000007fe`fbac25ff 0×45d030
00000000`01ccf550 00000000`004666b0 FLTLIB!FilterGetMessage+0×2b
00000000`01ccf5b0 00000000`00000000 0×4666b0

THREAD fffffa80092b7060  Cid 03f4.1268  Teb: 000007fffff6a000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
    fffffa8009299140  QueueObject
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800916b060       Image:         MsMpEng.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      7762           Ticks: 14533 (0:00:03:46.716)
Context Switch Count      3297            
UserTime                  00:00:06.489
KernelTime                00:00:00.499
Win32 Start Address ntdll!TppWorkerThread (0×00000000772bfbc0)
Stack Init fffff8800e620c70 Current fffff8800e620680
Base fffff8800e621000 Limit fffff8800e61b000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0e6206c0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0e620800 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`0e620890 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`0e620950 fffff800`02ecdb36 nt!IoRemoveIoCompletion+0×47
fffff880`0e6209e0 fffff800`02ee0ed3 nt!NtWaitForWorkViaWorkerFactory+0×285
fffff880`0e620ae0 00000000`772f2c1a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0e620ae0)
00000000`0540f998 00000000`772bfe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0540f9a0 00000000`7719652d ntdll!TppWorkerThread+0×2c9
00000000`0540fca0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0540fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa80098d7b60  Cid 07bc.0a14  Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
    fffffa8008a4a030  NotificationEvent
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa80096beb30       Image:         dwm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22294          Ticks: 1 (0:00:00:00.015)
Context Switch Count      15473                 LargeStack
UserTime                  00:00:06.801
KernelTime                00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

THREAD fffffa800451d720  Cid 1418.17fc  Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff8a001ce6b90
Owning Process            fffffa800442ab30       Image:         ApplicationE.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22295          Ticks: 0
Context Switch Count      27960            
UserTime                  00:01:25.332
KernelTime                00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3

Memory Dump Analysis Services is now working to incorporate client-side WinDbg scripting into their CARE2 architecture. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org

Crash Dump Analysis Patterns (Part 156)

Tuesday, November 29th, 2011

One pattern I missed is called FPU Exception and it sometimes happens where you least expect it. Here’s extract from one crash dump raw stack analysis showing exception context, record and the usage of r WinDbg command variant to display FPU registers:

0:002> dps 056c1000 057c0000  
[...]
057bdee0  00000008
057bdee4  00000000
057bdee8  057bed6c
057bdeec  0d6e3130
057bdef0  057c0000
057bdef4  057b9000
057bdef8  006e3138
057bdefc  057be200
057bdf00  7c90e48a ntdll!KiUserExceptionDispatcher+0xe
057bdf04  057bed6c
057bdf08  057bdf2c
057bdf0c  057bdf14
057bdf10  057bdf2c
057bdf14  c0000090
057bdf18  00000010
057bdf1c  00000000
057bdf20  79098cc0 mscorjit!Compiler::FlatFPIsSameAsFloat+0xd
057bdf24  00000001
057bdf28  00000000
057bdf2c  0001003f
057bdf30  00000000
057bdf34  00000000
057bdf38  00000000
057bdf3c  00000000
057bdf40  00000000
057bdf44  00000000
057bdf48  ffff1372
057bdf4c  fffffda1
057bdf50  ffffbfff 
[…]

0:002> .cxr 057bdf2c
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
mscorwks!SOTolerantBoundaryFilter+0x22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

0:002> .exr 057bdf14
ExceptionAddress: 79098cc0 (mscorjit!Compiler::FlatFPIsSameAsFloat+0x0000000d)
   ExceptionCode: c0000090
  ExceptionFlags: 00000010
NumberParameters: 1
   Parameter[0]: 00000000

0:002> !error c0000090
Error code: (NTSTATUS) 0xc0000090 (3221225616) - {EXCEPTION}  Floating-point invalid operation.

0:002> rMF
Last set context:
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
fpcw=1372: rn 64 pu–d-  fpsw=FDA1: top=7 cc=1101 b-p—-i  fptw=BFFF
fopcode=045D  fpip=001b:79098cc0  fpdp=0023:057bea7c
st0=-1.#IND00000000000000000e+0000  st1= 0.006980626232475338220e-4916
st2= 6.543831490564206840810e-4932  st3=-0.003025663186207448300e+2614
st4= 2.000000000000000000000e+0000  st5= 6.291456000000000000000e+0006
st6= 1.000000000000000000000e+0000  st7= 2.500000000000000000000e-0001
mscorwks!SOTolerantBoundaryFilter+0×22:
79f5236b d9059823f579    fld     dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

New Book: Accelerated .NET Memory Dump Analysis

Sunday, November 13th, 2011

During the previous several months some companies and individuals expressed their interest in the training (the next one is scheduled for January) but its time was not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

  • Title: Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
  • Description: The full transcript of Memory Dump Analysis Services Training with 7 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers 20 .NET memory dump analysis patterns plus additional unmanaged patterns.
  • Authors: Dmitry Vostokov, Memory Dump Analysis Services
  • Publisher: OpenTask (November 2011)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 204 pages
  • ISBN-13: 978-1908043320

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Public Preview of Accelerated .NET Memory Dump Analysis Training

Sunday, November 6th, 2011

The slides are available from Memory Dump Analysis Services:

http://www.dumpanalysis.com/Training/Accelerated-NET-Memory-Dump-Analysis-Public.pdf

It also offers the new training sessions in January, 2012:

http://www.dumpanalysis.com/accelerated-net-memory-dump-analysis

http://www.dumpanalysis.com/accelerated-windows-memory-dump-analysis

Also the registration for Advanced training session in December is still open:

http://www.dumpanalysis.com/advanced-windows-memory-dump-analysis

There are also new coming courses for 2012 so stay tuned!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Volume 6 of Memory Dump Analysis Anthology

Thursday, November 3rd, 2011

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

  • 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
  • 4 new pattern interaction case studies
  • 11 new trace analysis patterns
  • New Debugware pattern
  • Introduction to UI problem analysis patterns
  • Introduction to intelligence analysis patterns
  • Introduction to unified debugging pattern language
  • Introduction to generative debugging, metadefect template library and DNA of software behaviour
  • The new school of debugging and trends
  • .NET memory dump analysis checklist
  • Software trace analysis checklist
  • Introduction to close and deconstructive readings of a software trace
  • Memory dump analysis compass
  • Computical and Stack Trace Art
  • The abductive reasoning of Philip Marlowe
  • Orbifold memory space and cloud computing
  • Memory worldview
  • Interpretation of cyberspace
  • Relationship of memory dumps to religion
  • Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 6
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 300 pages
  • Publisher: Opentask (December 2011)
  • ISBN-13: 978-1-908043-19-1
  • Hardcover: 300 pages
  • Publisher: Opentask (January 2012)
  • ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 27b)

Monday, October 24th, 2011

Here’s a managed space counterpart to unmanaged Stack Trace Collection pattern. When looking at crash dumps from a different than a postmortem analysis machine we might need the appropriate version-specific SOS extension. To list all managed stack traces we use this combined command:

0:000> ~*e !CLRStack
OS Thread Id: 0×36f0 (0)
Child SP IP       Call Site
0031cf84 779b0f34 [HelperMethodFrame: 0031cf84]
0031cfd8 6b65665e System.Collections.ArrayList.GetEnumerator()
0031cfe4 059f4c92 ActiproSoftware.SyntaxEditor.EditorView.a(System.Windows.Forms.PaintEventArgs, System.Drawing.Rectangle, ActiproSoftware.SyntaxEditor.DocumentLine, ActiproSoftware.SyntaxEditor.DisplayLine, ActiproSoftware.SyntaxEditor.EditPositionRange, Int32 ByRef)
0031e158 05a01798 ActiproSoftware.SyntaxEditor.EditorView.OnRender(System.Windows.Forms.PaintEventArgs)
0031e748 04c5f888 ActiproSoftware.WinUICore.UIElement.Render(System.Windows.Forms.PaintEventArgs)
0031e758 04c5f602 ActiproSoftware.WinUICore.UIControl.OnRenderChildElements(System.Windows.Forms.PaintEventArgs)
0031e80c 04c5f1ac ActiproSoftware.WinUICore.UIControl.Render(System.Windows.Forms.PaintEventArgs)
0031e81c 04c5e6fe ActiproSoftware.WinUICore.UIControl.a(System.Windows.Forms.PaintEventArgs)
0031e9a4 04c5e415 ActiproSoftware.WinUICore.UIControl.OnPaint(System.Windows.Forms.PaintEventArgs)
0031e9b4 69f156f5 System.Windows.Forms.Control.PaintWithErrorHandling(System.Windows.Forms.PaintEventArgs, Int16)
0031e9e8 69f1809e System.Windows.Forms.Control.WmPaint(System.Windows.Forms.Message ByRef)
0031ead4 69f073b1 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0031ead8 69f102aa [InlinedCallFrame: 0031ead8]
0031eb2c 69f102aa System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message ByRef)
0031eb38 048269c3 ActiproSoftware.SyntaxEditor.SyntaxEditor.WndProc(System.Windows.Forms.Message ByRef)
0031ebe8 69f070f3 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0031ebf0 69f07071 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0031ec04 69f06fb6 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
0031ee44 00e71365 [InlinedCallFrame: 0031ee44]
0031ee40 69f22eec DomainNeutralILStubClass.IL_STUB_PInvoke(MSG ByRef)
0031ee44 69f171ff [InlinedCallFrame: 0031ee44] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0031ee88 69f171ff System.Windows.Forms.Application+ComponentManager.System.Windows.Forms. UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
0031ee8c 69f16e2c [InlinedCallFrame: 0031ee8c]
0031ef24 69f16e2c System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0031ef7c 69f16c81 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0031efac 69ea366d System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0031efc0 003c3f0d LINQPad.Program.Run(System.String, Boolean, System.String, Boolean, Boolean, System.String)
0031f0b4 003c1515 LINQPad.Program.Go(System.String[])
0031f2e4 003c0584 LINQPad.Program.Start(System.String[])
0031f324 003c034f LINQPad.ProgramStarter.Run(System.String[])
0031f330 003c00f5 LINQPad.Loader.Main(System.String[])
0031f570 6c1721db [GCFrame: 0031f570]
OS Thread Id: 0×36f8 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

OS Thread Id: 0×36fc (2)
Child SP IP       Call Site
03c2fb78 779b0f34 [DebuggerU2MCatchHandlerFrame: 03c2fb78]
OS Thread Id: 0×3700 (3)
Child SP IP       Call Site
0470f0cc 779b0f34 [InlinedCallFrame: 0470f0cc]
0470f0c8 699380b7 DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafePipeHandle, IntPtr)
0470f0cc 699cc07c [InlinedCallFrame: 0470f0cc] Microsoft.Win32.UnsafeNativeMethods.ConnectNamedPipe(Microsoft.Win32.SafeHandles.SafePipeHandle, IntPtr)
0470f130 699cc07c System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
0470f140 003c31ed LINQPad.Program.Listen()
0470f1d8 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0470f1e8 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0470f20c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0470f228 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0470f44c 6c1721db [GCFrame: 0470f44c]
0470f710 6c1721db [DebuggerU2MCatchHandlerFrame: 0470f710]
OS Thread Id: 0×3704 (4)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

OS Thread Id: 0×3710 (5)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

OS Thread Id: 0×3714 (6)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

OS Thread Id: 0×3718 (7)
Child SP IP       Call Site
0596ee40 779b0f34 [GCFrame: 0596ee40]
0596ef34 779b0f34 [HelperMethodFrame_1OBJ: 0596ef34] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
0596ef90 6b5d9140 System.Threading.Monitor.Wait(System.Object, Int32, Boolean)
0596efa0 6bb9428a System.Threading.Monitor.Wait(System.Object)
0596efa4 003cb55a ActiproSoftware.SyntaxEditor.SemanticParserService.c()
0596f068 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0596f078 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0596f09c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0596f0b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0596f2dc 6c1721db [GCFrame: 0596f2dc]
0596f5a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0596f5a0]
OS Thread Id: 0×3764 (8)
Child SP IP       Call Site
0564f148 779b0f34 [HelperMethodFrame_1OBJ: 0564f148] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
0564f1f0 6b64b5ef System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
0564f20c 6b64b5ad System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0564f224 6b64b570 System.Threading.WaitHandle.WaitOne()
0564f22c 04b607d4 ActiproBridge.ReferenceManager+?15?.<StartAdvanceFeeder>b__4()
0564f268 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0564f278 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0564f29c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0564f2b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0564f4dc 6c1721db [GCFrame: 0564f4dc]
0564f7a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0564f7a0]
OS Thread Id: 0×3768 (9)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×376c (10)
Child SP IP       Call Site
GetFrameContext failed: 1
OS Thread Id: 0×3798 (11)
Child SP IP       Call Site
GetFrameContext failed: 1
OS Thread Id: 0×37e0 (12)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

OS Thread Id: 0×37f8 (13)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 153)

Friday, October 21st, 2011

This pattern is a Duplicate Module equivalent for a debugger that uses loaded modules to extend its functionality. For example, in the case of WinDbg there is a possibility that two different Version-Specific Extensions are loaded wreaking havoc on debugging process (Debugger DLL Hell). For example, we loaded a specific version of SOS extension and successfully got a stack trace:

0:000> lmv m mscorwks
start    end        module name
79e70000 7a3ff000   mscorwks   (deferred)            
    Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
    Image name: mscorwks.dll
    Timestamp:        Wed Oct 24 08:41:29 2007 (471EF729)
    CheckSum:         00597AA8
    ImageSize:        0058F000
    File version:     2.0.50727.1433
    Product version:  2.0.50727.1433

    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     mscorwks.dll
    OriginalFilename: mscorwks.dll
    ProductVersion:   2.0.50727.1433
    FileVersion:      2.0.50727.1433 (REDBITS.050727-1400)
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
    dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\dbghelp.dll]
    ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\ext.dll]
    exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\exts.dll]
    uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\uext.dll]
    ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\ntsdexts.dll]

0:000> .load .load C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
    C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.1433, API 1.0.0, built Wed Oct 24 04:41:30 2007
        [path: C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos.dll]

    dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\dbghelp.dll]
    ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\ext.dll]
    exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\exts.dll]
    uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\uext.dll]
    ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\ntsdexts.dll]

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP       EIP    
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8] System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f System.Windows.Forms.Application+ComponentManager.System.Windows.Forms. UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean, Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

Then we tried the default analysis command !analyze -v -hang and continued using SOS commands. Unfortunately they no longer worked correctly:

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP       EIP    
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8]
002eeaa4 7b08374f
002eeb44 7b0831a5
002eebbc 7b082fe3
002eebec 7b0692c2
002eebfc 00833264
002eec50 008311dc
002eedac 00830545
002eede0 00830362
002eede8 008300e3

002ef00c 79e7c74b [GCFrame: 002ef00c]

Looking at loaded extensions list we see that an additional wrong version of sos.dll was loaded and that one gets all SOS commands:

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
    C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.4963, API 1.0.0, built Thu Jul 07 03:08:08 2011
        [path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll]

    C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos: image 2.0.50727.1433, API 1.0.0, built Wed Oct 24 04:41:30 2007
        [path: C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos.dll]

    dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\dbghelp.dll]
    ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\ext.dll]
    exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\exts.dll]
    uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\winext\uext.dll]
    ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
        [path: C:\Program Files (x86)\Debugging Tools for Windows (x86)\WINXP\ntsdexts.dll]

If we specify the full path to the correct extension we get right stack trace:

0:000> !C:\Frameworks\32-bit\Framework.Updates\Microsoft.NET\Framework\v2.0.50727\sos.CLRStack
OS Thread Id: 0xdd0 (0)
ESP       EIP    
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8] System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f System.Windows.Forms.Application+ComponentManager.System.Windows.Forms. UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean, Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

To avoid confusion we unload the last loaded extension:

0:000> .unload C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos
Unloading C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos extension DLL

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP       EIP    
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8] System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f System.Windows.Forms.Application+ComponentManager.System.Windows.Forms. UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean, Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg tips and tricks: getting the bottom of a stack trace

Tuesday, October 11th, 2011

Sometimes the number of frames for well-formed stack overflow stack trace is so high that k* frame count parameter is not enough:

0:000> kc 0xffff
ntdll!RtlpLocateActivationContextSection
ntdll!RtlpFindNextActivationContextSection
ntdll!RtlpFindFirstActivationContextSection
ntdll!RtlFindActivationContextSectionString
ntdll!AitFireParentUsageEvent
ntdll!RtlDosApplyFileIsolationRedirection_Ustr
ntdll!LdrpApplyFileNameRedirection
ntdll!LdrGetDllHandleEx
ntdll!LdrGetDllHandle
KERNELBASE!GetModuleHandleForUnicodeString
KERNELBASE!BasepGetModuleHandleExW
KERNELBASE!GetModuleHandleW
KERNELBASE!GetModuleHandleA
msvcr80!_decode_pointer
msvcr80!__set_flsgetvalue
msvcr80!_getptd_noexit
msvcr80!_errno
msvcr80!_get_winmajor
msvcr80!_beginthreadex
msvcr80!_decode_pointer
msvcr80!__set_flsgetvalue
msvcr80!_getptd_noexit
msvcr80!_errno
msvcr80!_get_winmajor
msvcr80!_beginthreadex
msvcr80!_decode_pointer
msvcr80!__set_flsgetvalue
msvcr80!_getptd_noexit
msvcr80!_errno
[...]
msvcr80!_get_winmajor
msvcr80!_beginthreadex
msvcr80!_decode_pointer
msvcr80!__set_flsgetvalue
msvcr80!_getptd_noexit
msvcr80!_errno
msvcr80!_get_winmajor
msvcr80!_beginthreadex
msvcr80!_decode_pointer
msvcr80!__set_flsgetvalue

Please not that the maximum number is 0xffff:

0:000> kc 0xfffff
Requested number of stack frames (0xfffff) is too large! The maximum number is 0xffff.
                ^ Range error in 'kc 0xfffff'

We specified 0xffff instead of ffff to avoid value truncation because the command would have been interpreted as kc f fff where the the first f parameters enables the output of the distance in bytes between frames:

0:000> kc ffff
  Memory 
          ntdll!RtlpLocateActivationContextSection
       30 ntdll!RtlpFindNextActivationContextSection
       18 ntdll!RtlpFindFirstActivationContextSection
       54 ntdll!RtlFindActivationContextSectionString
       bc ntdll!AitFireParentUsageEvent
      15c ntdll!RtlDosApplyFileIsolationRedirection_Ustr
       40 ntdll!LdrpApplyFileNameRedirection
      188 ntdll!LdrGetDllHandleEx
       1c ntdll!LdrGetDllHandle
       54 KERNELBASE!GetModuleHandleForUnicodeString
      478 KERNELBASE!BasepGetModuleHandleExW
       18 KERNELBASE!GetModuleHandleW
       18 KERNELBASE!GetModuleHandleA
        c msvcr80!_decode_pointer
        c msvcr80!__set_flsgetvalue
       10 msvcr80!_getptd_noexit
        4 msvcr80!_errno
        8 msvcr80!_get_winmajor
       1c msvcr80!_beginthreadex
        8 msvcr80!_decode_pointer
        c msvcr80!__set_flsgetvalue
       10 msvcr80!_getptd_noexit
        4 msvcr80!_errno
        8 msvcr80!_get_winmajor
       1c msvcr80!_beginthreadex
        8 msvcr80!_decode_pointer
        c msvcr80!__set_flsgetvalue
       10 msvcr80!_getptd_noexit
        4 msvcr80!_errno
[...]
        8 msvcr80!_get_winmajor
       1c msvcr80!_beginthreadex
        8 msvcr80!_decode_pointer
        c msvcr80!__set_flsgetvalue
       10 msvcr80!_getptd_noexit
        4 msvcr80!_errno
        8 msvcr80!_get_winmajor
       1c msvcr80!_beginthreadex

.kframes command helps here:

0:000> .kframes fffff
Default stack trace depth is 0n1048575 frames

0:000> .kframes ffffff
Default stack trace depth is 0n16777215 frames

0:000> .kframes fffffff
Default stack trace depth is 0n268435455 frames

0:000> .kframes ffffffff
Default stack trace depth is 0n-1 frames

0:000> k
Could not allocate memory for stack trace

0:000> .kframes fffffff
Default stack trace depth is 0n268435455 frames

0:000> k
Could not allocate memory for stack trace

0:000> .kframes ffffff
Default stack trace depth is 0n16777215 frames

0:000> k
Could not allocate memory for stack trace

0:000> .kframes fffff
Default stack trace depth is 0n1048575 frames

0:000> k
ChildEBP RetAddr
[...]
003efcd4 74b3182c msvcr80!_errno+0x5
003efcdc 74b32b11 msvcr80!_get_winmajor+0x10
003efcf8 74b32bac msvcr80!_beginthreadex+0xc9
003efd00 74b32bd7 msvcr80!_encode_pointer+0x4a
003efd08 74b31143 msvcr80!_encoded_null+0x7
003efd10 008b4d63 msvcr80!__set_app_type+0x6
003efd18 74b31762 iexplore!pre_c_init+0x6d
003efd20 008b4b4f msvcr80!_initterm_e+0x15
003efda8 770033ca iexplore!__tmainCRTStartup+0x94
003efdb4 775f9ed2 kernel32!BaseThreadInitThunk+0xe
003efdf4 775f9ea5 ntdll!__RtlUserThreadStart+0x70
003efe0c 00000000 ntdll!_RtlUserThreadStart+0x1b

Another approach is to use k 0ffff command first and then try k L=<ChildEBP> 0ffff several times taking EBP value from the last line:

0:000> k 0ffff
ChildEBP RetAddr
002f1024 775ee9d6 ntdll!RtlpLocateActivationContextSection+0×119
002f1054 775eeaf2 ntdll!RtlpFindNextActivationContextSection+0×64
002f106c 775eecf9 ntdll!RtlpFindFirstActivationContextSection+0×41
002f10c0 775ef3bf ntdll!RtlFindActivationContextSectionString+0×91
002f117c 775ef18a ntdll!AitFireParentUsageEvent+0×772
002f12d8 775efad6 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0×23e
002f1318 775efe0a ntdll!LdrpApplyFileNameRedirection+0×128
002f14a0 775efd0f ntdll!LdrGetDllHandleEx+0×139
002f14bc 75680dae ntdll!LdrGetDllHandle+0×18
002f1510 75680fc2 KERNELBASE!GetModuleHandleForUnicodeString+0×22
002f1988 756810bd KERNELBASE!BasepGetModuleHandleExW+0×181
002f19a0 75681f29 KERNELBASE!GetModuleHandleW+0×29
002f19b8 74b32c18 KERNELBASE!GetModuleHandleA+0×34
002f19c4 74b32c89 msvcr80!_decode_pointer+0×3f
002f19d0 74b32dc7 msvcr80!__set_flsgetvalue+0×1e
002f19e0 74b34351 msvcr80!_getptd_noexit+0×15
002f19e4 74b3182c msvcr80!_errno+0×5
002f19ec 74b32b11 msvcr80!_get_winmajor+0×10
002f1a08 74b32c23 msvcr80!_beginthreadex+0xc9
002f1a10 74b32c89 msvcr80!_decode_pointer+0×4a
002f1a1c 74b32dc7 msvcr80!__set_flsgetvalue+0×1e
002f1a2c 74b34351 msvcr80!_getptd_noexit+0×15
002f1a30 74b3182c msvcr80!_errno+0×5
[…]
003bd09c 74b32b11 msvcr80!_get_winmajor+0×10
003bd0b8 74b32c23 msvcr80!_beginthreadex+0xc9
003bd0c0 74b32c89 msvcr80!_decode_pointer+0×4a
003bd0cc 74b32dc7 msvcr80!__set_flsgetvalue+0×1e
003bd0dc 74b34351 msvcr80!_getptd_noexit+0×15
003bd0e0 74b3182c msvcr80!_errno+0×5
003bd0e8 74b32b11 msvcr80!_get_winmajor+0×10
003bd104 74b32c23 msvcr80!_beginthreadex+0xc9

0:000> k L=003bd104 0ffff
ChildEBP RetAddr 
003bd104 74b32c23 ntdll!RtlpLocateActivationContextSection+0x119
003bd158 74b32c89 msvcr80!_decode_pointer+0x4a
003bd164 74b32dc7 msvcr80!__set_flsgetvalue+0x1e
003bd174 74b34351 msvcr80!_getptd_noexit+0x15
003bd178 74b3182c msvcr80!_errno+0x5
003bd180 74b32b11 msvcr80!_get_winmajor+0x10
003bd19c 74b32c23 msvcr80!_beginthreadex+0xc9
003bd1a4 74b32c89 msvcr80!_decode_pointer+0x4a
[...]
003efcdc 74b32b11 msvcr80!_get_winmajor+0x10
003efcf8 74b32bac msvcr80!_beginthreadex+0xc9
003efd00 74b32bd7 msvcr80!_encode_pointer+0x4a
003efd08 74b31143 msvcr80!_encoded_null+0x7
003efd10 008b4d63 msvcr80!__set_app_type+0x6
003efd18 74b31762 iexplore!pre_c_init+0x6d
003efd20 008b4b4f msvcr80!_initterm_e+0x15
003efda8 770033ca iexplore!__tmainCRTStartup+0x94
003efdb4 775f9ed2 kernel32!BaseThreadInitThunk+0xe
003efdf4 775f9ea5 ntdll!__RtlUserThreadStart+0x70
003efe0c 00000000 ntdll!_RtlUserThreadStart+0x1b

Note: sometimes k 0fffff or 0cffff will work despite the limit of 0xffff.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 150)

Monday, October 10th, 2011

I noticed this pattern when analyzing the output of !DumpStack WinDbg SOS extension command:

0:011> !DumpStack
OS Thread Id: 0xac (11)
[...]
ChildEBP RetAddr  Caller, Callee
[…]
0b73f65c 77c416dc ntdll!RtlAllocateHeap+0×17c, calling ntdll!RtlpLowFragHeapAllocFromContext
0b73f688 77c486cd ntdll!RtlAllocateHeap+0×193, calling ntdll!memset
0b73f6b0 7653a467 kernel32!TlsSetValue+0×4c, calling ntdll!RtlAllocateHeap
0b73f6cc 77a01c48 urlmon!CUrlMkTls::TLSAllocData+0×3f, calling kernel32!TlsSetValue
0b73f6dc 77a0198d urlmon!CUrlMkTls::CUrlMkTls+0×29, calling urlmon!CUrlMkTls::TLSAllocData
0b73f6e8 77a01be5 urlmon!TlsDllMain+0×100, calling urlmon!EnsureFeatureCache
0b73f6f4 6d016a21 mshtml!DllMain+0×10, calling kernel32!GetCurrentThreadId
0b73f704 6d016b6c mshtml!_CRT_INIT+0×281, calling mshtml!DllMain
0b73f71c 7239133e msimtf!_CRT_INIT+0×281, calling msimtf!DllMain
0b73f728 72391375 msimtf!_CRT_INIT+0×3e7, calling msimtf!_SEH_epilog4
0b73f764 6d016ad0 mshtml!_DllMainStartup+0×56, calling mshtml!_DllMainCRTStartup
0b73f778 72391375 msimtf!_CRT_INIT+0×3e7, calling msimtf!_SEH_epilog4
0b73f77c 77c4a604 ntdll!LdrpCallInitRoutine+0×14
0b73f7a4 77c1ab6c ntdll!LdrpInitializeThread+0×1e9, calling ntdll!RtlLeaveCriticalSection
0b73f7ac 77c1a9ea ntdll!LdrpInitializeThread+0×1cd, calling ntdll!_SEH_epilog4
0b73f800 77c1ab15 ntdll!LdrpInitializeThread+0×11f, calling ntdll!RtlActivateActivationContextUnsafeFast
0b73f804 77c1ab53 ntdll!LdrpInitializeThread+0×167, calling ntdll!RtlDeactivateActivationContextUnsafeFast
0b73f838 77c1a9ea ntdll!LdrpInitializeThread+0×1cd, calling ntdll!_SEH_epilog4
0b73f83c 77c405a0 ntdll!NtTestAlert+0xc
0b73f840 77c1a968 ntdll!_LdrpInitialize+0×29c, calling ntdll!_SEH_epilog4
0b73f8a0 77c3f3d0 ntdll!NtContinue+0xc
0b73f8a4 77c1a98a ntdll!LdrInitializeThunk+0×1a, calling ntdll!NtContinue
0b73fb30 6afd59f6 clr!Thread::intermediateThreadProc+0×39, calling clr!_alloca_probe_16
0b73fb44 76573833 kernel32!BaseThreadInitThunk+0xe
0b73fb50 77c1a9bd ntdll!_RtlUserThreadStart+0×23

Obviously the command collected “call-type” execution residue from the raw stack. The “calling” part wasn’t found in the nearby region:

0:011> dps 0b73f7a4-20 0b73f7a4+20
0b73f784  72390000 msimtf!_imp__RegOpenKeyW <PERF> (msimtf+0×0)
0b73f788  00000002
0b73f78c  00000000
0b73f790  00000001
0b73f794  0b73f80c
0b73f798  0b73f80c
0b73f79c  00000001
0b73f7a0  05636578
0b73f7a4  0b73f83c
0b73f7a8  77c1ab6c ntdll!LdrpInitializeThread+0×1e9
0b73f7ac  77ca5340 ntdll!LdrpLoaderLock
0b73f7b0  77c1a9ea ntdll!LdrpInitializeThread+0×1cd
0b73f7b4  0b7321f2
0b73f7b8  7ff4e000
0b73f7bc  7ffdf000
0b73f7c0  77ca51f4 ntdll!LdrpProcessInitialized
0b73f7c4  00000000

I tried to disassemble backwards the addresses and found the callees:

0:011> ub 77c1ab6c
ntdll!LdrpInitializeThread+0×16b:
77c1ab57 90              nop
77c1ab58 90              nop
77c1ab59 90              nop
77c1ab5a 90              nop
77c1ab5b 90              nop
77c1ab5c ff054452ca77    inc     dword ptr [ntdll!LdrpActiveThreadCount (77ca5244)]
77c1ab62 684053ca77      push    offset ntdll!LdrpLoaderLock (77ca5340)
77c1ab67 e8bd820000      call    ntdll!RtlLeaveCriticalSection (77c22e29)

0:011> ub 77a01be5
urlmon!TlsDllMain+0×2f:
77a01bce 8d4510          lea     eax,[ebp+10h]
77a01bd1 50              push    eax
77a01bd2 8d4d0c          lea     ecx,[ebp+0Ch]
77a01bd5 e88efdffff      call    urlmon!CUrlMkTls::CUrlMkTls (77a01968)
77a01bda 397d10          cmp     dword ptr [ebp+10h],edi
77a01bdd 7c09            jl      urlmon!TlsDllMain+0×103 (77a01be8)
77a01bdf 56              push    esi
77a01be0 e887fcffff      call    urlmon!EnsureFeatureCache (77a0186c)

In the past I was frequently referencing this pattern especially when discussing coincidental symbolic information but didn’t name it. Now it’s time to do that: Caller-n-Callee.

We can also run !DumpStack command against every thread (including nonmanaged) to get the summary of the call-type execution residue:

0:011> ~4s
eax=76573821 ebx=00000002 ecx=00000000 edx=74d01909 esi=00000000 edi=00000000
eip=77c40f34 esp=0478f8a0 ebp=0478f93c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
77c40f34 c3 ret

0:004> k
ChildEBP RetAddr 
0478f89c 77c40690 ntdll!KiFastSystemCallRet
0478f8a0 76577e09 ntdll!ZwWaitForMultipleObjects+0xc
0478f93c 7674c4af kernel32!WaitForMultipleObjectsEx+0x11d
0478f990 76748b7b user32!RealMsgWaitForMultipleObjectsEx+0x13c
0478f9ac 74d01965 user32!MsgWaitForMultipleObjects+0x1f
0478f9f8 76573833 GdiPlus!BackgroundThreadProc+0x59
0478fa04 77c1a9bd kernel32!BaseThreadInitThunk+0xe
0478fa44 00000000 ntdll!_RtlUserThreadStart+0x23

0:004> !DumpStack
OS Thread Id: 0x950 (4)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr  Caller, Callee
0478f89c 77c40690 ntdll!ZwWaitForMultipleObjects+0xc
0478f8a0 76577e09 kernel32!WaitForMultipleObjectsEx+0x11d, calling ntdll!NtWaitForMultipleObjects
0478f914 76751a91 user32!UserCallWinProcCheckWow+0x5c, calling ntdll!RtlActivateActivationContextUnsafeFast
0478f918 76751b41 user32!UserCallWinProcCheckWow+0x16a, calling ntdll!RtlDeactivateActivationContextUnsafeFast
0478f93c 7674c4af user32!RealMsgWaitForMultipleObjectsEx+0x13c, calling kernel32!WaitForMultipleObjectsEx
0478f968 76752a65 user32!DispatchMessageWorker+0x396, calling user32!_SEH_epilog4
0478f980 76743c64 user32!PeekMessageA+0x129, calling user32!_PeekMessage
0478f990 76748b7b user32!MsgWaitForMultipleObjects+0x1f, calling user32!MsgWaitForMultipleObjectsEx
0478f9ac 74d01965 GdiPlus!BackgroundThreadProc+0x59, calling user32!MsgWaitForMultipleObjects
0478f9f8 76573833 kernel32!BaseThreadInitThunk+0xe
0478fa04 77c1a9bd ntdll!_RtlUserThreadStart+0x23

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 127c)

Monday, October 10th, 2011

When looking at process memory dumps and seeing CLR threads we can find fragments of JIT-ed code return addresses on the unmanaged stack trace:

0:011> kL
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b73e120 057223e2 0×572240f
0b73e134 6af44a2a 0×57223e2
0b73e1b0 6af44bcc clr!CallDescrWorkerWithHandler+0×8e
0b73e2f0 6af44c01 clr!MethodDesc::CallDescr+0×194
0b73e30c 6af44c21 clr!MethodDesc::CallTargetWorker+0×21
0b73e324 6afb7856 clr!MethodDescCallSite::Call+0×1c
0b73e4e8 6afb7ba3 clr!CallWithValueTypes_RetArgSlotWrapper+0×5c
0b73e7b4 6afb7d65 clr!InvokeImpl+0×621
0b73e880 6963d689 clr!RuntimeMethodHandle::InvokeMethodFast+0×180
0b73e8d4 6963d3d0 mscorlib_ni+0×2bd689
0b73e90c 6963bfed mscorlib_ni+0×2bd3d0
0b73e934 69643284 mscorlib_ni+0×2bbfed
0b73e958 6af3de7e mscorlib_ni+0×2c3284
0b73eb64 05720988 clr!ListLockEntry::Release+0×68
0b73ebc0 6962ae5b 0×5720988
0b73ebd0 695b7ff4 mscorlib_ni+0×2aae5b
0b73ebec 695b7f34 mscorlib_ni+0×237ff4
0b73ec0c 6962ade8 mscorlib_ni+0×237f34
0b73ec24 6af221db mscorlib_ni+0×2aade8
0b73ec34 6af44a2a clr!CallDescrWorker+0×33
0b73ecb0 6af44bcc clr!CallDescrWorkerWithHandler+0×8e
0b73ede8 6af44c01 clr!MethodDesc::CallDescr+0×194
0b73ee04 6b0bb512 clr!MethodDesc::CallTargetWorker+0×21
0b73f010 6afd5c05 clr!ThreadNative::KickOffThread_Worker+0×1e1
0b73f024 6afd5c87 clr!Thread::DoExtraWorkForFinalizer+0×114
0b73f0d4 6afd5d42 clr!Thread::ShouldChangeAbortToUnload+0×101
0b73f134 6afc37a2 clr!Thread::ShouldChangeAbortToUnload+0×399
0b73f140 6b0a6465 clr!Thread::RaiseCrossContextException+0×3f8
0b73f220 6afc37cf clr!Thread::DoADCallBack+0xf0
0b73f240 6afd5c87 clr!Thread::DoExtraWorkForFinalizer+0xfa
0b73f2f0 6afd5d42 clr!Thread::ShouldChangeAbortToUnload+0×101
0b73f350 6afd5dd9 clr!Thread::ShouldChangeAbortToUnload+0×399
0b73f374 6b0bb3e5 clr!Thread::ShouldChangeAbortToUnload+0×43a
0b73f38c 6b0bb2e0 clr!ManagedThreadBase::KickOff+0×15
0b73f424 6afd5a08 clr!ThreadNative::KickOffThread+0×23e
0b73fb44 76573833 clr!Thread::intermediateThreadProc+0×4b
0b73fb50 77c1a9bd kernel32!BaseThreadInitThunk+0xe

With the correct CLR version extension loaded we can inspect these addresses and get their method names, module and class addresses using !IP2MD WinDbg SOS extension command:

0:011> !IP2MD 0x572240f
MethodDesc:   057420e8
Method Name:  UserQuery+ClassMain.Main()
Class:        057341d8
MethodTable:  05742108
mdToken:      06000004
Module:       05741048
IsJitted:     yes
CodeAddr:     05722400
Transparency: Critical

0:011> !IP2MD 0x57223e2
MethodDesc:   0574204c
Method Name:  UserQuery.RunUserAuthoredQuery()
Class:        057340a4
MethodTable:  0574206c
mdToken:      06000001
Module:       05741048
IsJitted:     yes
CodeAddr:     057223d0
Transparency: Critical

0:011> !IP2MD 0x5720988
MethodDesc:   056e601c
Method Name:  LINQPad.ExecutionModel.Server.StartClrQuery()
Class:        0571f6e4
MethodTable:  056e60e4
mdToken:      06000c59
Module:       056e336c
IsJitted:     yes
CodeAddr:     05720910
Transparency: Critical

These method calls can also be seen on managed stack trace:

0:011> !CLRStack
OS Thread Id: 0xac (11)
Child SP IP       Call Site
0b73e120 0572240f UserQuery+ClassMain.Main()
0b73e128 057223e2 UserQuery.RunUserAuthoredQuery()
0b73e674 6af221db [DebuggerU2MCatchHandlerFrame: 0b73e674]
0b73e640 6af221db [CustomGCFrame: 0b73e640]
0b73e614 6af221db [GCFrame: 0b73e614]
0b73e5f8 6af221db [GCFrame: 0b73e5f8]
0b73e81c 6af221db [HelperMethodFrame_PROTECTOBJ: 0b73e81c] System.RuntimeMethodHandle._InvokeMethodFast(System.IRuntimeMethodInfo, System.Object, System.Object[], System.SignatureStruct ByRef, System.Reflection.MethodAttributes, System.RuntimeType)
0b73e898 6963d689 System.RuntimeMethodHandle.InvokeMethodFast(System.IRuntimeMethodInfo, System.Object, System.Object[], System.Signature, System.Reflection.MethodAttributes, System.RuntimeType)
0b73e8ec 6963d3d0 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)
0b73e928 6963bfed System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
0b73e94c 69643284 System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
0b73e958 0572134c LINQPad.ExecutionModel.Server.RunClrQuery()
0b73eb6c 05720988 LINQPad.ExecutionModel.Server.StartClrQuery()
0b73ebc8 6962ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0b73ebd8 695b7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0b73ebfc 695b7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0b73ec18 6962ade8 System.Threading.ThreadHelper.ThreadStart()
0b73ee30 6af221db [GCFrame: 0b73ee30]
0b73f0f4 6af221db [DebuggerU2MCatchHandlerFrame: 0b73f0f4]
0b73f18c 6af221db [ContextTransitionFrame: 0b73f18c]
0b73f310 6af221db [DebuggerU2MCatchHandlerFrame: 0b73f310]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Debugging TV Frames Programme

Friday, October 7th, 2011

After the launch of the first episode about symbols I decided to make it recurrent where registration will be needed only once. So I apologize to all who already registered for episode 0×01 that another registration well be required for episode 0×02. However, no registration will be necessary for episode 0×03 and so on. If anyone misses episode 0×02 they can still register for episode 0×03 and all subsequent episodes only once, and so on by induction.

The second episode is about symbol file troubleshooting. All about this topic in 8 slides in 8 minutes including live WinDbg demonstration plus extra 8 minutes for you to ask questions.

Register for Debugging TV Frame 0×02 and further weekly episodes
Date: Friday, October 14, 2011
Time: 5:45 PM - 6:01 PM BST

Space is limited.
Reserve your seat now at:
https://www3.gotomeeting.com/register/318613774

After registering you will receive a confirmation email containing information about joining the show.

Debugging TV Frame 0×01
Recording: https://www3.gotomeeting.com/register/640694470
Slides: DebuggingTV_Frame_0×01.pdf
WinDbg log: DebuggingTV_Frame_0×01.txt

More frames are coming and www.debugging.tv will host TV programme and recordings of past episodes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Recent News and Updates

Tuesday, October 4th, 2011

First, we announced Debugging TV and its first weekly program called Frames where each episode features some facet of debugging, memory dump, and software trace analysis in 8 minutes. The first episode is about symbol files plus extra 8 minutes to ask questions.

Debugging TV Frame 0×01
Date: Friday, October 7, 2011
Time: 5:45 PM - 6:01 PM BST

Space is limited.
Reserve your seat now at:
https://www3.gotomeeting.com/register/640694470

Second, Accelerated Windows Memory Dump Analysis book became available on Amazon and Barnes & Noble.

Third, a recording of Fundamentals of Complete Crash and Hang Memory Dump Analysis (Revision 2) Webinar was made available for viewing.

Fourth, I’m working now on the next 5 crash dump analysis patterns to be published this week.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 148)

Wednesday, September 7th, 2011

Whereas Stack Trace Collection pattern covers all thread stack traces from a memory dump Stack Trace Set pattern covers only unique non-duplicated thread stack traces differing for example, in stack frame modules and function names. In user process memory dumps it is !uniqstack WinDbg command (don’t forget that command has optional parameters, for example, -v to simulate verbose ~*kv output):

0:000> ~
.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
   1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
   2  Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen

0:000> ~*kc

.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen

ntdll!NtWaitForMultipleObjects
KERNELBASE!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjectsExImplementation
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
KERNELBASE!DebugBreak
ApplicationK!main
ApplicationK!__tmainCRTStartup
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen

ntdll!NtDelayExecution
KERNELBASE!SleepEx
KERNELBASE!Sleep
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
ApplicationK!thread_two
ApplicationK!_callthreadstart
ApplicationK!_threadstart
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   2  Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen

ntdll!NtDelayExecution
KERNELBASE!SleepEx
KERNELBASE!Sleep
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
ApplicationK!thread_two
ApplicationK!_callthreadstart
ApplicationK!_threadstart
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

0:000> !uniqstack
Processing 3 threads, please wait

.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
      Start: ApplicationK!mainCRTStartup (013a137c)
      Priority: 0  Priority class: 32  Affinity: 3
ChildEBP RetAddr 
0037f1a4 770d0bdd ntdll!NtWaitForMultipleObjects+0x15
0037f240 7529162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0037f288 75291921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0037f2a4 752b9b2d kernel32!WaitForMultipleObjects+0x18
0037f310 752b9bca kernel32!WerpReportFaultInternal+0x186
0037f324 752b98f8 kernel32!WerpReportFault+0x70
0037f334 752b9875 kernel32!BasepReportFault+0x20
0037f3c0 77b10df7 kernel32!UnhandledExceptionFilter+0x1af
0037f3c8 77b10cd4 ntdll!__RtlUserThreadStart+0x62
0037f3dc 77b10b71 ntdll!_EH4_CallFilterFunc+0x12
0037f404 77ae6ac9 ntdll!_except_handler4+0x8e
0037f428 77ae6a9b ntdll!ExecuteHandler2+0x26
0037f4d8 77ab010f ntdll!ExecuteHandler+0x24
0037f4d8 770d280c ntdll!KiUserExceptionDispatcher+0xf
0037f824 013a1035 KERNELBASE!DebugBreak+0x2
0037f828 013a1325 ApplicationK!main+0x25
0037f870 75293677 ApplicationK!__tmainCRTStartup+0xfb
0037f87c 77ad9f02 kernel32!BaseThreadInitThunk+0xe
0037f8bc 77ad9ed5 ntdll!__RtlUserThreadStart+0x70
0037f8d4 00000000 ntdll!_RtlUserThreadStart+0x1b

.  1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
      Start: ApplicationK!_threadstart (013a10d1)
      Priority: 0  Priority class: 32  Affinity: 3
ChildEBP RetAddr 
0080f9ac 770d31bb ntdll!NtDelayExecution+0x15
0080fa14 770d3a8b KERNELBASE!SleepEx+0x65
0080fa24 752d28dd KERNELBASE!Sleep+0xf
0080fa38 752b98f8 kernel32!WerpReportFault+0x3f
0080fa48 752b9875 kernel32!BasepReportFault+0x20
0080fad4 77b10df7 kernel32!UnhandledExceptionFilter+0x1af
0080fadc 77b10cd4 ntdll!__RtlUserThreadStart+0x62
0080faf0 77b10b71 ntdll!_EH4_CallFilterFunc+0x12
0080fb18 77ae6ac9 ntdll!_except_handler4+0x8e
0080fb3c 77ae6a9b ntdll!ExecuteHandler2+0x26
0080fbec 77ab010f ntdll!ExecuteHandler+0x24
0080fbec 013a1000 ntdll!KiUserExceptionDispatcher+0xf
0080ff38 013a10ab ApplicationK!thread_two
0080ff70 013a1147 ApplicationK!_callthreadstart+0x1b
0080ff78 75293677 ApplicationK!_threadstart+0x76
0080ff84 77ad9f02 kernel32!BaseThreadInitThunk+0xe
0080ffc4 77ad9ed5 ntdll!__RtlUserThreadStart+0x70
0080ffdc 00000000 ntdll!_RtlUserThreadStart+0x1b

Total threads: 3
Duplicate callstacks: 1(windbg thread #s follow):
2

Generally, any property can be chosen to form such a set from a collection of stack traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced Windows Memory Dump Analysis with Data Structures Training Course

Sunday, August 14th, 2011

Due to the need to extend existing basic and intermediate Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises advanced training course. Here is the description and registration information:

Learn how to navigate through memory dump space and Windows data structures to troubleshoot and debug complex software incidents. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose structural and behavioral patterns in 32-bit and 64-bit process, kernel and complete memory dumps.

Advanced Windows Memory Dump Analysis Logo

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

  1. A full transcript in PDF format (retail price $200)
  2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
  3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic and intermediate level Windows memory dump analysis: ability to list processors, processes, threads, modules, apply symbols, walk through stack traces and raw stack data, diagnose patterns such as heap corruption, CPU spike, memory and handle leaks, access violation, stack overflow, critical section and resource wait chains and deadlocks. If you don’t feel comfortable with prerequisites then Accelerated Windows Memory Dump Analysis training is recommended to take (or purchase a corresponding book) before attending this course.

Audience: Software developers, software technical support and escalation engineers.

Session: December 9, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/3788047691824598784

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg Command on Certificate

Saturday, August 13th, 2011

Finally you can even learn a WinDbg command from a certificate. Memory Dump Analysis Services has created a certificate with dv WinDbg command on it:

Source: http://www.dumpanalysis.com/sample-certificate

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Pleasing the WinDbg SOS Extension

Wednesday, August 10th, 2011

Recently when looking at one of process memory dumps with CLR runtime I got this message when trying to load SOS extension from the appropriate Framework location:

0:022> .load [..]\Microsoft.NET\Framework\v2.0.50727\sos.dll
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------

So I literally did what was asked by saving a new dump from the dump:

0:022> .dump /ma c:\userdumps\dump-as-asked.dmp
Creating c:\userdumps\dump-as-asked.dmp - mini user dump
Dump successfully written

I got a slightly bigger dump than the original and after that SOS didn’t complain when I opened the new dump and loaded the extension.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

New Book: Accelerated Windows Memory Dump Analysis

Sunday, August 7th, 2011

During the previous several months many people expressed their interest in the training (the next one is scheduled for November) but its time was not suitable due to the very different geographic time zones. So I have decided to publish this training in book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

  • Title: Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
  • Description: The full transcript of Memory Dump Analysis Services Training with 21 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers about 50 crash dump analysis patterns from process, kernel and complete memory dumps.
  • Authors: Dmitry Vostokov, Memory Dump Analysis Services
  • Publisher: OpenTask (August 2011)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 360 pages
  • ISBN-13: 978-1908043290

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -