Archive for the ‘Crash Dump Analysis’ Category
Sunday, December 11th, 2011
When doing software behavior artifact collection, live debugging or postmortem memory dump analysis we must also take into consideration the possibility of Debugger Bugs. I classify them into hard and soft bugs. The former are those software defects and behavioral problems that result in further abnormal software behavior incidents like crashes and hangs. One example is this Microsoft KB article about DebugDiag. Soft debugger bugs usually manifest themselves as glitches in data output, nonsense or false positive diagnostics, for example, this excessive non-paged pool usage message in the output from !vm WinDbg command (see the corresponding MS KB article):
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 1031581 ( 4126324 Kb)
Page File: \??\C:\pagefile.sys
Current: 4433524 Kb Free Space: 4433520 Kb
Minimum: 4433524 Kb Maximum: 12378972 Kb
Unimplemented error for MiSystemVaTypeCount
Available Pages: 817652 ( 3270608 Kb)
ResAvail Pages: 965229 ( 3860916 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33555714 ( 134222856 Kb)
Modified Pages: 15794 ( 63176 Kb)
Modified PF Pages: 15793 ( 63172 Kb)
NonPagedPool Usage: 88079121 ( 352316484 Kb)
NonPagedPoolNx Usage: 12885 ( 51540 Kb)
NonPagedPool Max: 764094 ( 3056376 Kb)
********** Excessive NonPaged Pool Usage *****
PagedPool 0 Usage: 35435 ( 141740 Kb)
PagedPool 1 Usage: 3620 ( 14480 Kb)
PagedPool 2 Usage: 573 ( 2292 Kb)
PagedPool 3 Usage: 535 ( 2140 Kb)
PagedPool 4 Usage: 538 ( 2152 Kb)
PagedPool Usage: 40701 ( 162804 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 9309 ( 37236 Kb)
Shared Commit: 6460 ( 25840 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 5760 ( 23040 Kb)
PagedPool Commit: 40765 ( 163060 Kb)
Driver Commit: 2805 ( 11220 Kb)
Committed pages: 212472 ( 849888 Kb)
Commit limit: 2139487 ( 8557948 Kb)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Live Debugging, Software Behavior Patterns, Tools | 7 Comments »
Monday, December 5th, 2011
Sometimes we have a value or a pointer or a handle and would like to know all memory addresses that reference it. This can be done by virtual memory search (s WinDbg command). If you look for references in code (for example, or pool tags please see this case study) you can combine search with !for_each_module WinDbg extension command. There is also !search command for physical pages. We cover this Value References pattern in the forthcoming Advanced Windows Memory Dump Analysis training with a step-by-step complete memory dump analysis exercise. For object references there is also recently added !obtrace command with good examples in WinDbg help.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, WinDbg Tips and Tricks | 4 Comments »
Sunday, December 4th, 2011
This is a variant of Self-Diagnosis (kernel mode) pattern for system configuration database (registry). Sometimes it is possible to see which part of it (hive) caused the problem. Here’s an example involving possibly corrupt user profiles:
REGISTRY_ERROR (51)
Something has gone badly wrong with the registry. If a kernel debugger is available, get a stack trace. It can also indicate that the registry got an I/O error while trying to read one of its files, so it can be caused by hardware problems or filesystem corruption. It may occur due to a failure in a refresh operation, which is used only in by the security system, and then only when resource limits are encountered.
Arguments:
Arg1: 00000003, (reserved)
Arg2: 00000004, (reserved)
Arg3: e82372f8, depends on where Windows bugchecked, may be pointer to hive
Arg4: 00000000, depends on where Windows bugchecked, may be return code of HvCheckHive if the hive is corrupt.
0: kd> !reg hivelist
-------------------------------------------------------------------------------------------------------------
| HiveAddr |Stable Length|Stable Map|Volatile Length|Volatile Map|MappedViews|PinnedViews|U(Cnt)| BaseBlock | FileName
-------------------------------------------------------------------------------------------------------------
| e1008a68 | 13000 | e1008ac8 | 1000 | e1008c04 | 0 | 0 | 0| e1015000 | <NONAME>
| e101a4e0 | 901000 | e1023000 | 40000 | e101a67c | 202 | 0 | 0| e101e000 | SYSTEM
| e1938188 | d000 | e19381e8 | 4000 | e1938324 | 0 | 0 | 0| e193a000 | <NONAME>
| e1968290 | 8000 | e19682f0 | 0 | 00000000 | 3 | 0 | 0| e1d39000 | \SystemRoot\System32\Config\SAM
| e1cab270 | 3d000 | e1cab2d0 | 1000 | e1cab40c | 16 | 0 | 0| e1d32000 | emRoot\System32\Config\SECURITY
| e1c9f448 | 3f70000 | e1e37000 | 1000 | e1c9f5e4 | 256 | 0 | 0| e1d71000 | temRoot\System32\Config\DEFAULT
| e1d75a80 | 7d5d000 | e1ee3000 | 23000 | e1d75c1c | 254 | 12 | 0| e1d37000 | emRoot\System32\Config\SOFTWARE
| e1ba30d0 | 37000 | e1ba3130 | 1000 | e1ba326c | 17 | 0 | 0| e1b9e000 | tings\NetworkService\ntuser.dat
| e1ba8060 | 1000 | e1ba80c0 | 0 | 00000000 | 1 | 0 | 0| e1b8e000 | \Microsoft\Windows\UsrClass.dat
| e1afc068 | 3b000 | e1afc0c8 | 1000 | e1afc204 | 17 | 0 | 0| e1b3d000 | ettings\LocalService\ntuser.dat
| e1d6e2a0 | 1000 | e1d6e300 | 0 | 00000000 | 1 | 0 | 0| e1b39000 | \Microsoft\Windows\UsrClass.dat
[...]
| e82372f8 | 106000 | e8237358 | 0 | 00000000 | 55 | 4 | 0| e514c000 | ings\User123\NTUSER.DAT
[…]
————————————————————————————————————-
0: kd> dt _CMHIVE e82372f8
nt!_CMHIVE
+0x000 Hive : _HHIVE
+0x2d0 FileHandles : [3] 0x80002234 Void
+0x2dc NotifyList : _LIST_ENTRY [ 0x0 - 0x0 ]
+0x2e4 HiveList : _LIST_ENTRY [ 0xe7a38d64 - 0xe4d9fc9c ]
+0x2ec HiveLock : _EX_PUSH_LOCK
+0x2f0 ViewLock : 0x877b0120 _KGUARDED_MUTEX
+0x2f4 WriterLock : _EX_PUSH_LOCK
+0x2f8 FlusherLock : _EX_PUSH_LOCK
+0x2fc SecurityLock : _EX_PUSH_LOCK
+0x300 LRUViewListHead : _LIST_ENTRY [ 0xe6160170 - 0xe3d71978 ]
+0x308 PinViewListHead : _LIST_ENTRY [ 0xe2714fe0 - 0xe108d9e0 ]
+0x310 FileObject : 0x89ecf310 _FILE_OBJECT
+0x314 FileFullPath : _UNICODE_STRING "\Device\HarddiskVolumeX\Documents and Settings\User123\NTUSER.DAT"
+0×31c FileUserName : _UNICODE_STRING “\??\E:\Documents and Settings\User123\NTUSER.DAT”
+0×324 MappedViews : 0×37
+0×326 PinnedViews : 4
+0×328 UseCount : 0
+0×32c SecurityCount : 9
+0×330 SecurityCacheSize : 9
+0×334 SecurityHitHint : 0n0
+0×338 SecurityCache : 0xe74d5008 _CM_KEY_SECURITY_CACHE_ENTRY
+0×33c SecurityHash : [64] _LIST_ENTRY [ 0xe3f80228 - 0xe5901ef0 ]
+0×53c UnloadEvent : (null)
+0×540 RootKcb : (null)
+0×544 Frozen : 0 ”
+0×548 UnloadWorkItem : (null)
+0×54c GrowOnlyMode : 0 ”
+0×550 GrowOffset : 0
+0×554 KcbConvertListHead : _LIST_ENTRY [ 0xe823784c - 0xe823784c ]
+0×55c KnodeConvertListHead : _LIST_ENTRY [ 0xe8237854 - 0xe8237854 ]
+0×564 CellRemapArray : (null)
+0×568 Flags : 1
+0×56c TrustClassEntry : _LIST_ENTRY [ 0xe8237864 - 0xe8237864 ]
+0×574 FlushCount : 0
+0×578 CreatorOwner : (null)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Bugchecks Depicted, Crash Dump Analysis, Crash Dump Patterns | No Comments »
Sunday, December 4th, 2011
Certain System Objects can be found in object directory and can be useful to see additional system and other product activity. For example, in a complete memory dump from Accelerated .NET Memory Dump Analysis training we see that LowCommitCondition event is signalled:
1: kd> !object \KernelObjects
Object: 85a08030 Type: (82b38ed0) Directory
ObjectHeader: 85a08018 (old version)
HandleCount: 0 PointerCount: 19
Directory Object: 85a074c0 Name: KernelObjects
Hash Address Type Name
---- ------- ---- ----
02 82b7b0b8 Event HighCommitCondition
04 82b7b780 Event HighMemoryCondition
10 82b7b178 Event LowNonPagedPoolCondition
11 82b7b138 Event HighNonPagedPoolCondition
17 82b7b0f8 Event LowCommitCondition
20 82b78d08 Event SuperfetchParametersChanged
82b6eb58 Event BootLoaderTraceReady
23 84bfdd58 Session Session0
82b78c88 Event PrefetchTracesReady
24 84b7d1f8 Session Session1
25 82b78cc8 Event SuperfetchScenarioNotify
82b7b740 Event LowPagedPoolCondition
26 82b7b1b8 Event HighPagedPoolCondition
82b7a030 Event MemoryErrors
28 82b78c48 Event SuperfetchTracesReady
32 82b7b7c0 Event LowMemoryCondition
85a09d00 KeyedEvent CritSecOutOfMemoryEvent
34 82b7b078 Event MaximumCommitCondition
1: kd> dt _DISPATCHER_HEADER 82b7b0f8
ntdll!_DISPATCHER_HEADER
+0x000 Type : 0 ''
+0x001 Abandoned : 0 ''
+0x001 Absolute : 0 ''
+0x001 NpxIrql : 0 ''
+0x001 Signalling : 0 ''
+0x002 Size : 0x4 ''
+0x002 Hand : 0x4 ''
+0x003 Inserted : 0 ''
+0x003 DebugActive : 0 ''
+0x003 DpcActive : 0 ''
+0x000 Lock : 0n262144
+0×004 SignalState : 0n1
+0×008 WaitListHead : _LIST_ENTRY [ 0×82b7b100 - 0×82b7b100 ]
If we check virtual memory statistics we see lots of free space for the currrent physical memory and pagefile:
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 261872 ( 1047488 Kb)
Page File: \??\C:\pagefile.sys
Current: 1354688 Kb Free Space: 53120 Kb
Minimum: 1354688 Kb Maximum: 4194304 Kb
Available Pages: 180984 ( 723936 Kb)
ResAvail Pages: 216475 ( 865900 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 352925 ( 1411700 Kb)
Modified Pages: 129 ( 516 Kb)
Modified PF Pages: 94 ( 376 Kb)
NonPagedPool Usage: 0 ( 0 Kb)
NonPagedPoolNx Usage: 16894 ( 67576 Kb)
NonPagedPool Max: 192350 ( 769400 Kb)
PagedPool 0 Usage: 5957 ( 23828 Kb)
PagedPool 1 Usage: 3218 ( 12872 Kb)
PagedPool 2 Usage: 965 ( 3860 Kb)
PagedPool 3 Usage: 1311 ( 5244 Kb)
PagedPool 4 Usage: 1064 ( 4256 Kb)
PagedPool Usage: 12515 ( 50060 Kb)
PagedPool Maximum: 523264 ( 2093056 Kb)
Session Commit: 5021 ( 20084 Kb)
Shared Commit: 15023 ( 60092 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 1938 ( 7752 Kb)
PagedPool Commit: 12523 ( 50092 Kb)
Driver Commit: 2592 ( 10368 Kb)
Committed pages: 402494 ( 1609976 Kb)
Commit limit: 589254 ( 2357016 Kb)
[...]
Another example is from Windows 7 memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation. Here we can find WER reporting mutant in session 1 object directory and get problem PID from its name:
0: kd> !object \Sessions\1\BaseNamedObjects\
Object: fffff8a0016eb290 Type: (fffffa800426df30) Directory
ObjectHeader: fffff8a0016eb260 (new version)
HandleCount: 57 PointerCount: 217
Directory Object: fffff8a0016e9220 Name: BaseNamedObjects
Hash Address Type Name
---- ------- ---- ----
00 fffffa8008437670 Event STOP_HOOKING64
[...]
08 fffffa80044baa40 Mutant WERReportingForProcess1788
[…]
0: kd> !process 0n1788 1
Searching for Process with Cid == 6fc
Cid handle table at fffff8a00180b000 with 21248 entries in use
PROCESS fffffa8004364060
SessionId: 1 Cid: 06fc Peb: 7fffffd4000 ParentCid: 0840
DirBase: 5fbc2000 ObjectTable: fffff8a004c8e930 HandleCount: 16.
Image: ApplicationD.exe
VadRoot fffffa8009d85170 Vads 34 Clone 0 Private 206. Modified 0. Locked 0.
DeviceMap fffff8a001ce6b90
Token fffff8a003eab060
ElapsedTime 00:01:51.543
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (483, 50, 345) (1932KB, 200KB, 1380KB)
PeakWorkingSetSize 483
VirtualSize 13 Mb
PeakVirtualSize 13 Mb
PageFaultCount 481
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 231
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Memory Dump Analysis Services, Training and Seminars, Windows 7 | No Comments »
Saturday, December 3rd, 2011
In addition to functions we also have module variables like nt!MmPagedPoolCommit in Windows 7:
0: kd> x nt!MmPagedPool*
fffff800`031148d0 nt!MmPagedPoolInfo = <no type information>
fffff800`03092d20 nt!MmPagedPoolCommit = <no type information>
fffff800`031141a0 nt!MmPagedPoolEnd = <no type information>
fffff800`031175c0 nt!MmPagedPoolWs = <no type information>
If we are not sure whether we have a function or Module Variable we can try to disassemble:
0: kd> u nt!MmPagedPoolCommit
nt!MmPagedPoolCommit:
fffff800`03092d20 e3b2 jrcxz nt!MmTotalNonPagedPoolQuota+0x4 (fffff800`03092cd4)
fffff800`03092d22 0000 add byte ptr [rax],al
fffff800`03092d24 0000 add byte ptr [rax],al
fffff800`03092d26 0000 add byte ptr [rax],al
fffff800`03092d28 0000 add byte ptr [rax],al
fffff800`03092d2a 0000 add byte ptr [rax],al
fffff800`03092d2c 0000 add byte ptr [rax],al
fffff800`03092d2e 0000 add byte ptr [rax],al
Here the value is probably in pages so we multiply by 4 to get Kb value and compare to the output of !vm command:
0: kd> dp nt!MmPagedPoolCommit
fffff800`03092d20 00000000`0000b2e3 00000000`00000000
fffff800`03092d30 00000000`00000000 00000000`00000000
fffff800`03092d40 00000000`00000001 00000000`00000000
fffff800`03092d50 00000000`00000000 00000000`00060107
fffff800`03092d60 fffff800`03092d60 fffff800`03092d60
fffff800`03092d70 00000000`00000000 00000000`0001e972
fffff800`03092d80 fffff900`c0000000 00000000`00000002
fffff800`03092d90 fffff880`071dc0a8 fffff880`057340a8
0: kd> ? b2e3 * 4
Evaluate expression: 183180 = 00000000`0002cb8c
0: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 1035228 ( 4140912 Kb)
Page File: \??\C:\pagefile.sys
Current: 4448112 Kb Free Space: 4448108 Kb
Minimum: 4448112 Kb Maximum: 12422736 Kb
Unimplemented error for MiSystemVaTypeCount
Available Pages: 594029 ( 2376116 Kb)
ResAvail Pages: 889795 ( 3559180 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33556870 ( 134227480 Kb)
Modified Pages: 20079 ( 80316 Kb)
Modified PF Pages: 19441 ( 77764 Kb)
NonPagedPool Usage: 50865104 ( 203460416 Kb)
NonPagedPoolNx Usage: 28163 ( 112652 Kb)
NonPagedPool Max: 763396 ( 3053584 Kb)
********** Excessive NonPaged Pool Usage *****
PagedPool 0 Usage: 39420 ( 157680 Kb)
PagedPool 1 Usage: 5194 ( 20776 Kb)
PagedPool 2 Usage: 367 ( 1468 Kb)
PagedPool 3 Usage: 338 ( 1352 Kb)
PagedPool 4 Usage: 440 ( 1760 Kb)
PagedPool Usage: 45759 ( 183036 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 8112 ( 32448 Kb)
Shared Commit: 31802 ( 127208 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 10765 ( 43060 Kb)
PagedPool Commit: 45795 ( 183180 Kb)
Driver Commit: 13773 ( 55092 Kb)
Committed pages: 540998 ( 2163992 Kb)
Commit limit: 2146794 ( 8587176 Kb)
[…]
Knowledge of available module variables is useful because some of them are not included in WinDbg extension command output. For their list please consult Windows Internals book. Also useful variables can be found in other modules as well, for example, srv!srvcomputername:
0: kd> dS srv!srvcomputername
fffff8a0`0344b090 "MYNOTEBOOK"
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns, Debugging, Windows 7 | 1 Comment »
Saturday, December 3rd, 2011
Sometimes we need to narrow general stack trace collection to a few threads that satisfy some predicate, for example, all threads with kernel time spent greater than some value or all suspended threads or all threads that wait for a specific synchronization object type. We call this pattern variant Stack Trace Collection (predicate). This can be implemented using WinDbg scripts and / or debugger extensions.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, WinDbg Scripts | 3 Comments »
Saturday, December 3rd, 2011
If not for you then definitely for me because I now diagnose Spiking Thread pattern much faster. One of the readers if this blog asked me whether there is !runaway command equivalent for kernel and complete memory dumps. So, after some thinking I gave it a try especially in the context of WinDbg scripting exercises designed for Advanced Windows Memory Dump Analysis training. As a result I wrote 2 scripts initially that you can try yourself. Their output here is taken from a complete memory dump I used for Fundamentals of Complete Crash and Hang Memory Dump Analysis presentation.
The first one dumps the most CPU consuming threads for user and kernel mode:
$$
$$ krunawaymost.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest UserTime value: ”
? @$t0
!thread @$t2 ff
r $t0 = 0
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); .if (@$t1 > @$t0) {r $t0 = @$t1; r $t2 = @#Thread}”
.echo “The largest KernelTime value: ”
? @$t0
!thread @$t2 ff
0: kd> $$><c:\Scripts\krunawaymost.wds
The largest UserTime value:
Evaluate expression: 5470 = 00000000`0000155e
THREAD fffffa800451d720 Cid 1418.17fc Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap fffff8a001ce6b90
Owning Process fffffa800442ab30 Image: ApplicationE.exe
Attached Process N/A Image: N/A
Wait Start TickCount 22295 Ticks: 0
Context Switch Count 27960
UserTime 00:01:25.332
KernelTime 00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3
The largest KernelTime value:
Evaluate expression: 187 = 00000000`000000bb
THREAD fffffa80098d7b60 Cid 07bc.0a14 Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
fffffa8008a4a030 NotificationEvent
Not impersonating
DeviceMap fffff8a001ce6b90
Owning Process fffffa80096beb30 Image: dwm.exe
Attached Process N/A Image: N/A
Wait Start TickCount 22294 Ticks: 1 (0:00:00:00.015)
Context Switch Count 15473 LargeStack
UserTime 00:00:06.801
KernelTime 00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
The second script takes two arguments and shows all threads that have UserTime and KernelTime ticks value greater than (you can have the idea of the maximum from the previous script):
$$
$$ krunawaygt.wds
$$ Copyright (c) 2011 Memory Dump Analysis Services
$$ GNU GENERAL PUBLIC LICENSE
$$ http://www.gnu.org/licenses/gpl-3.0.txt
$$
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, UserTime)) ); r $t0 = $arg1; .if (@$t1 > @$t0) {!thread @#Thread ff}”
!for_each_thread “r $t1 = dwo( @#Thread + @@c++(#FIELD_OFFSET(nt!_KTHREAD, KernelTime)) ); r $t0 = $arg2; .if (@$t1 > @$t0) {!thread @#Thread ff}”
Using hints from the previous script run (the largest UserTime ticks value is 0×155e) we now get threads that spent more than 0×100 ticks in user mode:
0: kd> $$>a<c:\Scripts\krunawaygt.wds 100 100
THREAD fffffa800843e060 Cid 03f4.0658 Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
fffffa800843c2c0 QueueObject
Not impersonating
DeviceMap fffff8a000008aa0
Owning Process fffffa800916b060 Image: MsMpEng.exe
Attached Process N/A Image: N/A
Wait Start TickCount 21211 Ticks: 1084 (0:00:00:16.910)
Context Switch Count 6028
UserTime 00:00:10.140
KernelTime 00:00:00.296
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d4bc70 Current fffff88009d4b660
Base fffff88009d4c000 Limit fffff88009d46000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mprtp.dll -
Child-SP RetAddr Call Site
fffff880`09d4b6a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d4b7e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d4b870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d4b930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d4b9c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d4ba70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d4bae0)
00000000`0209fb08 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`0209fb10 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`0209fb70 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`0209fbb0 00000000`0045cbc0 mprtp!MpPluginSignatureChange+0×3e170
00000000`0209fbb8 000007fe`fbac25ff 0×45cbc0
00000000`0209fbc0 00000000`00466610 FLTLIB!FilterGetMessage+0×2b
00000000`0209fc20 00000000`00000000 0×466610
THREAD fffffa800845c060 Cid 03f4.065c Teb: 000007fffff8e000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
fffffa800843c2c0 QueueObject
Not impersonating
DeviceMap fffff8a000008aa0
Owning Process fffffa800916b060 Image: MsMpEng.exe
Attached Process N/A Image: N/A
Wait Start TickCount 21520 Ticks: 775 (0:00:00:12.090)
Context Switch Count 4979
UserTime 00:00:04.149
KernelTime 00:00:00.156
Win32 Start Address msvcrt!endthreadex (0×000007feff5173fc)
Stack Init fffff88009d52c70 Current fffff88009d52660
Base fffff88009d53000 Limit fffff88009d4d000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mprtp.dll -
Child-SP RetAddr Call Site
fffff880`09d526a0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`09d527e0 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`09d52870 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`09d52930 fffff800`0319cae5 nt!IoRemoveIoCompletion+0×47
fffff880`09d529c0 fffff800`02ee0ed3 nt!NtRemoveIoCompletion+0×145
fffff880`09d52a70 00000000`772f13aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`09d52ae0)
00000000`01ccf498 000007fe`fd9e169d ntdll!ZwRemoveIoCompletion+0xa
00000000`01ccf4a0 00000000`7718a4e1 KERNELBASE!GetQueuedCompletionStatus+0×39
00000000`01ccf500 00000000`748f2c74 kernel32!GetQueuedCompletionStatusStub+0×11
00000000`01ccf540 00000000`0045d030 mprtp!MpPluginSignatureChange+0×3e170
00000000`01ccf548 000007fe`fbac25ff 0×45d030
00000000`01ccf550 00000000`004666b0 FLTLIB!FilterGetMessage+0×2b
00000000`01ccf5b0 00000000`00000000 0×4666b0
THREAD fffffa80092b7060 Cid 03f4.1268 Teb: 000007fffff6a000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
fffffa8009299140 QueueObject
Not impersonating
DeviceMap fffff8a000008aa0
Owning Process fffffa800916b060 Image: MsMpEng.exe
Attached Process N/A Image: N/A
Wait Start TickCount 7762 Ticks: 14533 (0:00:03:46.716)
Context Switch Count 3297
UserTime 00:00:06.489
KernelTime 00:00:00.499
Win32 Start Address ntdll!TppWorkerThread (0×00000000772bfbc0)
Stack Init fffff8800e620c70 Current fffff8800e620680
Base fffff8800e621000 Limit fffff8800e61b000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`0e6206c0 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0e620800 fffff800`02ee9f93 nt!KiCommitThreadWait+0×1d2
fffff880`0e620890 fffff800`031ca647 nt!KeRemoveQueueEx+0×323
fffff880`0e620950 fffff800`02ecdb36 nt!IoRemoveIoCompletion+0×47
fffff880`0e6209e0 fffff800`02ee0ed3 nt!NtWaitForWorkViaWorkerFactory+0×285
fffff880`0e620ae0 00000000`772f2c1a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0e620ae0)
00000000`0540f998 00000000`772bfe0b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0540f9a0 00000000`7719652d ntdll!TppWorkerThread+0×2c9
00000000`0540fca0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0540fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
THREAD fffffa80098d7b60 Cid 07bc.0a14 Teb: 000007fffffd7000 Win32Thread: fffff900c2ca0c20 WAIT: (UserRequest) KernelMode Non-Alertable
fffffa8008a4a030 NotificationEvent
Not impersonating
DeviceMap fffff8a001ce6b90
Owning Process fffffa80096beb30 Image: dwm.exe
Attached Process N/A Image: N/A
Wait Start TickCount 22294 Ticks: 1 (0:00:00:00.015)
Context Switch Count 15473 LargeStack
UserTime 00:00:06.801
KernelTime 00:00:02.917
Win32 Start Address dwmcore!CPartitionThread::ThreadMain (0×000007fef8a1f0d8)
Stack Init fffff8800d3d5c70 Current fffff8800d3d5740
Base fffff8800d3d6000 Limit fffff8800d3cf000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`0d3d5780 fffff800`02ee6f32 nt!KiSwapContext+0×7a
fffff880`0d3d58c0 fffff800`02ee974f nt!KiCommitThreadWait+0×1d2
fffff880`0d3d5950 fffff880`0fef65b3 nt!KeWaitForSingleObject+0×19f
fffff880`0d3d59f0 fffff960`001fedea dxgkrnl!DxgkWaitForVerticalBlankEvent+0×53f
fffff880`0d3d5ab0 fffff800`02ee0ed3 win32k!NtGdiDdDDIWaitForVerticalBlankEvent+0×12
fffff880`0d3d5ae0 000007fe`ff1d143a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0d3d5ae0)
00000000`0287f778 000007fe`f8791da1 GDI32!NtGdiDdDDIWaitForVerticalBlankEvent+0xa
00000000`0287f780 000007fe`f89e1b6e dxgi!CDXGIOutput::WaitForVBlank+0×51
00000000`0287f7c0 000007fe`f89e1ae9 dwmcore!CD3DDeviceLevel1::WaitForVBlank+0×1f9
00000000`0287f810 000007fe`f89e1a9d dwmcore!CHwDisplayRenderTarget::WaitForVBlank+0×39
00000000`0287f850 000007fe`f89e1a4c dwmcore!CDesktopRenderTarget::WaitForVBlank+0×40
00000000`0287f880 000007fe`f89d3513 dwmcore!CSlaveHWndRenderTarget::WaitForVBlank+0×2c
00000000`0287f8c0 000007fe`f89d3584 dwmcore!CRenderTargetManager::WaitForVBlank+0×7d
00000000`0287f900 000007fe`f89d2661 dwmcore!CPartitionVerticalBlankScheduler::WaitForVBlank+0×7c
00000000`0287f950 000007fe`f8a1f0f4 dwmcore!CPartitionVerticalBlankScheduler::Run+0xe5
00000000`0287f9b0 00000000`7719652d dwmcore!CPartitionThread::ThreadMain+0×1c
00000000`0287f9e0 00000000`772cc521 kernel32!BaseThreadInitThunk+0xd
00000000`0287fa10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
THREAD fffffa800451d720 Cid 1418.17fc Teb: 000007fffffdc000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap fffff8a001ce6b90
Owning Process fffffa800442ab30 Image: ApplicationE.exe
Attached Process N/A Image: N/A
Wait Start TickCount 22295 Ticks: 0
Context Switch Count 27960
UserTime 00:01:25.332
KernelTime 00:00:00.015
*** ERROR: Module load completed but symbols could not be loaded for ApplicationE.exe
Win32 Start Address ApplicationE (0×000000013f0f1578)
Stack Init fffff8800723cc70 Current fffff8800723c960
Base fffff8800723d000 Limit fffff88007237000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
00000000`0021f9e0 00000000`00000000 ApplicationE+0×6cd3
Memory Dump Analysis Services is now working to incorporate client-side WinDbg scripting into their CARE2 architecture.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Complete Memory Dump Analysis, Crash Analysis Report Environment (CARE), Crash Dump Analysis, Crash Dump Patterns, Debugging, Memory Dump Analysis Services, Training and Seminars, WinDbg Scripts, WinDbg Tips and Tricks, x64 Windows | No Comments »
Tuesday, November 29th, 2011
A page to reference all different kinds of exception related patterns is necessary, so I created this post:
I’ll update it as soon as I add more similar patterns.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »
Tuesday, November 29th, 2011
One pattern I missed is called FPU Exception and it sometimes happens where you least expect it. Here’s extract from one crash dump raw stack analysis showing exception context, record and the usage of r WinDbg command variant to display FPU registers:
0:002> dps 056c1000 057c0000
[...]
057bdee0 00000008
057bdee4 00000000
057bdee8 057bed6c
057bdeec 0d6e3130
057bdef0 057c0000
057bdef4 057b9000
057bdef8 006e3138
057bdefc 057be200
057bdf00 7c90e48a ntdll!KiUserExceptionDispatcher+0xe
057bdf04 057bed6c
057bdf08 057bdf2c
057bdf0c 057bdf14
057bdf10 057bdf2c
057bdf14 c0000090
057bdf18 00000010
057bdf1c 00000000
057bdf20 79098cc0 mscorjit!Compiler::FlatFPIsSameAsFloat+0xd
057bdf24 00000001
057bdf28 00000000
057bdf2c 0001003f
057bdf30 00000000
057bdf34 00000000
057bdf38 00000000
057bdf3c 00000000
057bdf40 00000000
057bdf44 00000000
057bdf48 ffff1372
057bdf4c fffffda1
057bdf50 ffffbfff
[…]
0:002> .cxr 057bdf2c
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010297
mscorwks!SOTolerantBoundaryFilter+0x22:
79f5236b d9059823f579 fld dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000
0:002> .exr 057bdf14
ExceptionAddress: 79098cc0 (mscorjit!Compiler::FlatFPIsSameAsFloat+0x0000000d)
ExceptionCode: c0000090
ExceptionFlags: 00000010
NumberParameters: 1
Parameter[0]: 00000000
0:002> !error c0000090
Error code: (NTSTATUS) 0xc0000090 (3221225616) - {EXCEPTION} Floating-point invalid operation.
0:002> rMF
Last set context:
eax=c0000090 ebx=00000000 ecx=c0000090 edx=00000000 esi=057be244 edi=001d4388
eip=79f5236b esp=057be1f8 ebp=057be200 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010297
fpcw=1372: rn 64 pu–d- fpsw=FDA1: top=7 cc=1101 b-p—-i fptw=BFFF
fopcode=045D fpip=001b:79098cc0 fpdp=0023:057bea7c
st0=-1.#IND00000000000000000e+0000 st1= 0.006980626232475338220e-4916
st2= 6.543831490564206840810e-4932 st3=-0.003025663186207448300e+2614
st4= 2.000000000000000000000e+0000 st5= 6.291456000000000000000e+0006
st6= 1.000000000000000000000e+0000 st7= 2.500000000000000000000e-0001
mscorwks!SOTolerantBoundaryFilter+0×22:
79f5236b d9059823f579 fld dword ptr [mscorwks!_real (79f52398)] ds:0023:79f52398=40800000
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, WinDbg Tips and Tricks | No Comments »
Sunday, November 13th, 2011
During the previous several months some companies and individuals expressed their interest in the training (the next one is scheduled for January) but its time was not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:
- Title: Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
- Description: The full transcript of Memory Dump Analysis Services Training with 7 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers 20 .NET memory dump analysis patterns plus additional unmanaged patterns.
- Authors: Dmitry Vostokov, Memory Dump Analysis Services
- Publisher: OpenTask (November 2011)
- Language: English
- Product Dimensions: 28.0 x 21.6
- Paperback: 204 pages
- ISBN-13: 978-1908043320
Table of Contents
Now available for sale in PDF format from Memory Dump Analysis Services.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Announcements, Books, Crash Dump Analysis, Crash Dump Patterns, Debugging, Escalation Engineering, Medicine, Memory Dump Analysis Services, Publishing, Software Engineering, Software Technical Support, Testing, Tools, Training and Seminars, WinDbg Tips and Tricks | No Comments »
Friday, November 11th, 2011
One of the new initiatives for 2012 is the development of SPDL (Software Problem Description Language). Its purpose is automatic generation of a software troubleshooting tool(s) based on the description of a problem. Here software problem means a post-construction problem as outlined in Introduction to Pattern-Driven Software Problem Solving. The tool construction will utilize an expanded set of DebugWare and Workaround patterns together with the refind version of RADII software development process. This will also provide necessary effectiveness, efficiency and enhanced problem solving capabilities to existing TaaS (Tools as a Service) implementations that are limited in the number of tools they offer.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Announcements, Computer Science, Crash Dump Analysis, DebugWare Patterns, Debugging, Generative Debugging, New Acronyms, New Debugging School, SPDL, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Engineering, Software Problem Solving, Software Technical Support, Software Trace Analysis, Software Troubleshooting Patterns, Software and Modeling, TaaS, Testing, Tool Objects, Tools, Troubleshooting Methodology, Unified Debugging Patterns, Windows System Administration | No Comments »
Thursday, November 10th, 2011
Hidden Parameter pattern is a variant of Execution Residue and String Parameter where we have parameters left out from stack trace due to register calling conventions and compiler optimizations. However, raw stack analysis in a region around stack frames of interest we find what we are looking for. Here’s an example from an x64 system blocked thread waiting for data from a named pipe:
0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
fffffa60`2c3627d0 fffff800`018b90fa nt!KiSwapContext+0x7f
fffffa60`2c362910 fffff800`018add3b nt!KiSwapThread+0x13a
fffffa60`2c362980 fffff800`01b2121f nt!KeWaitForSingleObject+0x2cb
fffffa60`2c362a10 fffff800`01b319b6 nt!IopXxxControlFile+0xdeb
fffffa60`2c362b40 fffff800`018b68f3 nt!NtFsControlFile+0x56
fffffa60`2c362bb0 00000000`778d6eaa nt!KiSystemServiceCopyEnd+0x13
00000000`11f4da68 00000000`77767b6e ntdll!ZwFsControlFile+0xa
00000000`11f4da70 000007fe`ff94abc8 kernel32!WaitNamedPipeW+0×22f
00000000`11f4db60 000007fe`ff98a32d RPCRT4!NdrProxyForwardingFunction255+0×814d
00000000`11f4dc30 000007fe`ff98918b RPCRT4!OSF_CCONNECTION::TransOpen+0xcd
00000000`11f4dcc0 000007fe`ff988f9b RPCRT4!OSF_CCONNECTION::OpenConnectionAndBind+0×17b
00000000`11f4dd90 000007fe`ff988ec6 RPCRT4!OSF_CCALL::BindToServer+0xbb
00000000`11f4de40 000007fe`ff983368 RPCRT4!OSF_BINDING_HANDLE::InitCCallWithAssociation+0xa5
00000000`11f4dea0 000007fe`ff983220 RPCRT4!OSF_BINDING_HANDLE::AllocateCCall+0×118
00000000`11f4dfd0 000007fe`ffa1f740 RPCRT4!OSF_BINDING_HANDLE::NegotiateTransferSyntax+0×30
00000000`11f4e020 000007fe`ffa1fecb RPCRT4!Ndr64pClientSetupTransferSyntax+0×200
00000000`11f4e080 000007fe`ffa20281 RPCRT4!NdrpClientCall3+0×6b
00000000`11f4e2d0 000007fe`fe087c8c RPCRT4!NdrClientCall3+0xdd
[…]
Even if we disassemble the return address of a caller of WaitNamedPipeW function we won’t easily find the passed first string parameter (named pipe name) unless we do a substancial reverse engineering and data flow analysis:
0: kd> ub 000007fe`ff94abc8
RPCRT4!_imp_load_getaddrinfo+0×7:
000007fe`ff94ab9f jmp RPCRT4!_tailMerge_WS2_32_dll (000007fe`ff94cef8)
000007fe`ff94aba4 call qword ptr [RPCRT4!_imp_GetLastError (000007fe`ffa2d528)]
000007fe`ff94abaa mov r12d,eax
000007fe`ff94abad cmp r12d,0E7h
000007fe`ff94abb4 jne RPCRT4!NdrProxyForwardingFunction255+0×8193 (000007fe`ff99c8fb)
000007fe`ff94abba mov edx,3E8h
000007fe`ff94abbf mov rcx,rsi
000007fe`ff94abc2 call qword ptr [RPCRT4!_imp_WaitNamedPipeW (000007fe`ffa2d3f8)]
So dumping raw stack date around corresponding frames give us pipe name clue and possible service to look further:
0: kd> dpu 00000000`11f4da70
00000000`11f4da70 00000000`11f4dba8 “\\.\PIPE\ServiceArpc”
00000000`11f4da78 00000000`00000000
00000000`11f4da80 00000000`00000000
00000000`11f4da88 00000000`000003e8
00000000`11f4da90 00000000`11f4db30
00000000`11f4da98 00000000`00110018
00000000`11f4daa0 00000000`0d9001a0
00000000`11f4daa8 00000000`0000001a
00000000`11f4dab0 00000000`00000000
00000000`11f4dab8 00000000`00000000
00000000`11f4dac0 00000000`0020000c
00000000`11f4dac8 00000000`0d9001e2 “ServiceArpc”
00000000`11f4dad0 00000000`00000000
00000000`11f4dad8 00000000`00000000
00000000`11f4dae0 00000000`00240022
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, x64 Windows | 1 Comment »
Sunday, November 6th, 2011
Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, Debugging, Escalation Engineering, Memory Dump Analysis Services, Software Engineering, Software Technical Support, Testing, Training and Seminars, WinDbg Tips and Tricks | 2 Comments »
Thursday, November 3rd, 2011
The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:
- 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
- 4 new pattern interaction case studies
- 11 new trace analysis patterns
- New Debugware pattern
- Introduction to UI problem analysis patterns
- Introduction to intelligence analysis patterns
- Introduction to unified debugging pattern language
- Introduction to generative debugging, metadefect template library and DNA of software behaviour
- The new school of debugging and trends
- .NET memory dump analysis checklist
- Software trace analysis checklist
- Introduction to close and deconstructive readings of a software trace
- Memory dump analysis compass
- Computical and Stack Trace Art
- The abductive reasoning of Philip Marlowe
- Orbifold memory space and cloud computing
- Memory worldview
- Interpretation of cyberspace
- Relationship of memory dumps to religion
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5
Product information:
- Title: Memory Dump Analysis Anthology, Volume 6
- Author: Dmitry Vostokov
- Language: English
- Product Dimensions: 22.86 x 15.24
- Paperback: 300 pages
- Publisher: Opentask (December 2011)
- ISBN-13: 978-1-908043-19-1
- Hardcover: 300 pages
- Publisher: Opentask (January 2012)
- ISBN-13: 978-1-908043-20-7
Back cover features 3d memory space visualization image created with ParaView.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Announcements, Art, Books, Cloud Computing, Cloud Memory Dump Analysis, Common Mistakes, Complete Memory Dump Analysis, Computer Science, Computicart (Computical Art), Crash Dump Analysis, Crash Dump Patterns, Cyber Intelligence, Cyber Problems, Cyber Security, Cyber Space, Cyber Warfare, DebugWare Patterns, Debugging, Debugging Industry, Debugging Methodology, Debugging Slang, Debugging Trends, Escalation Engineering, Generative Debugging, Intelligence Analysis Patterns, Kernel Development, Memoidealism, Memoretics, Memory Visualization, Metadefect Template Library, New Debugging School, Philosophy, Physicalist Art, Publishing, Root Cause Analysis, Science of Memory Dump Analysis, Science of Software Tracing, Security, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Engineering, Software Narratology, Software Technical Support, Software Trace Analysis, Software Trace Deconstruction, Software Trace Reading, Software Victimology, Testing, The Way of Philip Marlowe, Tools, Trace Analysis Patterns, Training and Seminars, Troubleshooting Methodology, UI Problem Analysis Patterns, Unified Debugging Patterns, Victimware, WinDbg Tips and Tricks, Windows 7, Windows Azure, Windows Data Structures, Windows Server 2008, Windows System Administration, x64 Windows | No Comments »
Thursday, October 27th, 2011
When analyzing memory dumps from specific application platforms we see threads having definite purpose as part of that specific platform architecture, design and implementation. For example, in applications and services involving .NET CLR we see the following Special Threads:
0:000> !Threads -special
ThreadCount: 9
UnstartedThread: 0
BackgroundThread: 7
PendingThread: 0
DeadThread: 1
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 b10 002fbe88 6020 Enabled 0acbdebc:0acbf5a4 002f17d0 0 STA
2 2 bf0 00306b18 b220 Enabled 00000000:00000000 002f17d0 0 MTA (Finalizer)
3 3 b34 0034c188 b220 Enabled 00000000:00000000 002f17d0 0 MTA
XXXX 5 0037e3e0 19820 Enabled 00000000:00000000 002f17d0 0 Ukn
5 7 700 04b606c8 200b220 Enabled 00000000:00000000 002f17d0 0 MTA
6 4 ec4 04baffa0 200b220 Enabled 00000000:00000000 002f17d0 0 MTA
8 8 10c 04bf19b8 8009220 Enabled 00000000:00000000 002f17d0 0 MTA (Threadpool Completion Port)
9 11 464 0be106d8 1220 Enabled 00000000:00000000 002f17d0 0 Ukn
10 10 da0 003c7958 7220 Disabled 00000000:00000000 0be1dd00 0 STA
OSID Special thread type
1 c08 DbgHelper
2 bf0 Finalizer
7 f54 Gate
8 10c IOCompletion
9 464 ADUnloadHelper
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, Debugging, Stack Trace Collection | No Comments »
Monday, October 24th, 2011
Here’s a managed heap counterpart to process heap and kernel pool corruption patterns. It is usually detected by CLR during GC phases. Here is a typical stack from CLR 2 (CLR 4 is similar):
0:000> kL
ChildEBP RetAddr
002baae0 779b06a0 ntdll!KiFastSystemCallRet
002baae4 772c77d4 ntdll!NtWaitForSingleObject+0xc
002bab54 772c7742 kernel32!WaitForSingleObjectEx+0xbe
002bab68 7a0c0a43 kernel32!WaitForSingleObject+0x12
002bab98 7a0c0e89 mscorwks!ClrWaitForSingleObject+0x24
002bb054 7a0c2bfd mscorwks!RunWatson+0x1df
002bb798 7a0c3171 mscorwks!DoFaultReportWorker+0xb62
002bb7d4 7a106b2d mscorwks!DoFaultReport+0xc3
002bb7fc 7a1061ac mscorwks!WatsonLastChance+0x43
002bbcb8 7a10624f mscorwks!EEPolicy::LogFatalError+0x3ae
002bbcd0 79ffee2f mscorwks!EEPolicy::HandleFatalError+0x36
002bbcf4 79f04f1f mscorwks!CLRVectoredExceptionHandlerPhase3+0xc1
002bbd28 79f04e98 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
002bbd5c 79f9149e mscorwks!CLRVectoredExceptionHandler+0x10a
002bbd70 779b1039 mscorwks!CLRVectoredExceptionHandlerShimX86+0x27
002bbd94 779b100b ntdll!ExecuteHandler2+0x26
002bbe3c 779b0e97 ntdll!ExecuteHandler+0x24
002bbe3c 79f69360 ntdll!KiUserExceptionDispatcher+0xf
002bc13c 79f663f1 mscorwks!SVR::heap_segment_next_rw+0xf
002bc228 79f65d63 mscorwks!WKS::gc_heap::plan_phase+0×37c
002bc248 79f6614c mscorwks!WKS::gc_heap::gc1+0×6e
002bc25c 79f65f5d mscorwks!WKS::gc_heap::garbage_collect+0×261
002bc288 79f663c2 mscorwks!WKS::GCHeap::GarbageCollectGeneration+0×1a9
002bc314 79ef1566 mscorwks!WKS::gc_heap::try_allocate_more_space+0×12e
002bc328 79ef1801 mscorwks!WKS::gc_heap::allocate_more_space+0×11
002bc348 79e7510e mscorwks!WKS::GCHeap::Alloc+0×3b
002bc364 79e86713 mscorwks!Alloc+0×60
002bc3a0 79e86753 mscorwks!SlowAllocateString+0×29
002bc3ac 79eb4efb mscorwks!UnframedAllocateString+0xc
002bc3e0 79e91f58 mscorwks!AllocateStringObject+0×2e
002bc424 79e82892 mscorwks!GlobalStringLiteralMap::AddStringLiteral+0×3f
002bc438 79e82810 mscorwks!GlobalStringLiteralMap::GetStringLiteral+0×43
002bc47c 79e82956 mscorwks!AppDomainStringLiteralMap::GetStringLiteral+0×72
002bc494 79e81b6f mscorwks!BaseDomain::GetStringObjRefPtrFromUnicodeString+0×31
002bc4cc 79ef4704 mscorwks!Module::ResolveStringRef+0×88
002bc4e4 79f23132 mscorwks!ConstructStringLiteral+0×39
002bc558 7908c351 mscorwks!CEEInfo::constructStringLiteral+0×108
002bc57c 7906276d mscorjit!Compiler::fgMorphConst+0xa3
002bc598 79065ea0 mscorjit!Compiler::fgMorphTree+0×63
002bc610 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc63c 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc658 79065ea0 mscorjit!Compiler::fgMorphTree+0xa3
002bc6d0 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc6fc 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc718 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bc738 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bc774 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bc788 79064e63 mscorjit!Compiler::fgMorph+0×60
002bc798 790614e6 mscorjit!Compiler::compCompile+0×5f
002bc7f0 79061236 mscorjit!Compiler::compCompile+0×2df
002bc884 7906118c mscorjit!jitNativeCode+0xb8
002bc8bc 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002bc928 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002bc96c 79f0f8da mscorwks!invokeCompileMethod+0×31
002bc9c4 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002bcd7c 79f0e795 mscorwks!UnsafeJitFunction+0×230
002bce20 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002bce78 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002bcec8 00330836 mscorwks!PreStubWorker+0xeb
WARNING: Frame IP not in any known module. Following frames may be wrong.
002bcee0 79e7c74b 0×330836
002bcf10 79e7c6cc mscorwks!CallDescrWorker+0×33
002bcf90 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
002bd0d0 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
002bd0ec 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
002bd100 79e8b983 mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
002bd1d8 79e8b8e6 mscorwks!MethodTable::RunClassInitWorker+0×8b
002bd260 79e8b7fa mscorwks!MethodTable::RunClassInitEx+0×11e
002bd724 79ebcee6 mscorwks!MethodTable::DoRunClassInitThrowing+0×2f0
002bd79c 79fc49db mscorwks!MethodTable::CheckRunClassInitNT+0×8c
002bd82c 790a2801 mscorwks!CEEInfo::initClass+0×19b
002bddcc 79062cdc mscorjit!Compiler::impExpandInline+0×2aaa
002bde24 79062b7c mscorjit!Compiler::fgMorphCallInline+0xf8
002bde50 7906311f mscorjit!Compiler::fgMorphCall+0×27b
002bde6c 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bde8c 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bdec8 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bdedc 79064e63 mscorjit!Compiler::fgMorph+0×60
002bdeec 790614e6 mscorjit!Compiler::compCompile+0×5f
002bdf44 79061236 mscorjit!Compiler::compCompile+0×2df
002bdfd8 7906118c mscorjit!jitNativeCode+0xb8
002be010 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002be07c 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002be0c0 79f0f8da mscorwks!invokeCompileMethod+0×31
002be118 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002be4d0 79f0e795 mscorwks!UnsafeJitFunction+0×230
002be574 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002be5cc 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002be61c 00330836 mscorwks!PreStubWorker+0xeb
002be634 0570c859 0×330836
002be69c 0595bcc1 0×570c859
002be700 0595b954 0×595bcc1
002be704 099b66e0 0×595b954
002be708 002be728 0×99b66e0
002be70c 09589c90 0×2be728
002be728 099b67b8 0×9589c90
002be72c 00000000 0×99b67b8
Usually !VerifyHeap SOS command helps to find the first invalid object on managed heap and shows the last valid one. Sometimes the corruption can deeply affect heap or when a crash happens during traversal GC state might not be valid for analysis:
0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state
0:000> !DumpHeap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state
In such cases it is recommended to collect several dumps to catch more consistent heap state:
0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
object 0981f024: does not have valid MT
curr_object: 0981f024
Last good object: 0981f010
—————-
Then we can use !DumpObj (!do) command to check objects and d* WinDbg commands to inspect raw memory.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »
Monday, October 24th, 2011
Here’s a managed space counterpart to unmanaged Stack Trace Collection pattern. When looking at crash dumps from a different than a postmortem analysis machine we might need the appropriate version-specific SOS extension. To list all managed stack traces we use this combined command:
0:000> ~*e !CLRStack
OS Thread Id: 0×36f0 (0)
Child SP IP Call Site
0031cf84 779b0f34 [HelperMethodFrame: 0031cf84]
0031cfd8 6b65665e System.Collections.ArrayList.GetEnumerator()
0031cfe4 059f4c92 ActiproSoftware.SyntaxEditor.EditorView.a(System.Windows.Forms.PaintEventArgs, System.Drawing.Rectangle, ActiproSoftware.SyntaxEditor.DocumentLine, ActiproSoftware.SyntaxEditor.DisplayLine, ActiproSoftware.SyntaxEditor.EditPositionRange, Int32 ByRef)
0031e158 05a01798 ActiproSoftware.SyntaxEditor.EditorView.OnRender(System.Windows.Forms.PaintEventArgs)
0031e748 04c5f888 ActiproSoftware.WinUICore.UIElement.Render(System.Windows.Forms.PaintEventArgs)
0031e758 04c5f602 ActiproSoftware.WinUICore.UIControl.OnRenderChildElements(System.Windows.Forms.PaintEventArgs)
0031e80c 04c5f1ac ActiproSoftware.WinUICore.UIControl.Render(System.Windows.Forms.PaintEventArgs)
0031e81c 04c5e6fe ActiproSoftware.WinUICore.UIControl.a(System.Windows.Forms.PaintEventArgs)
0031e9a4 04c5e415 ActiproSoftware.WinUICore.UIControl.OnPaint(System.Windows.Forms.PaintEventArgs)
0031e9b4 69f156f5 System.Windows.Forms.Control.PaintWithErrorHandling(System.Windows.Forms.PaintEventArgs, Int16)
0031e9e8 69f1809e System.Windows.Forms.Control.WmPaint(System.Windows.Forms.Message ByRef)
0031ead4 69f073b1 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0031ead8 69f102aa [InlinedCallFrame: 0031ead8]
0031eb2c 69f102aa System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message ByRef)
0031eb38 048269c3 ActiproSoftware.SyntaxEditor.SyntaxEditor.WndProc(System.Windows.Forms.Message ByRef)
0031ebe8 69f070f3 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0031ebf0 69f07071 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0031ec04 69f06fb6 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
0031ee44 00e71365 [InlinedCallFrame: 0031ee44]
0031ee40 69f22eec DomainNeutralILStubClass.IL_STUB_PInvoke(MSG ByRef)
0031ee44 69f171ff [InlinedCallFrame: 0031ee44] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0031ee88 69f171ff System.Windows.Forms.Application+ComponentManager.System.Windows.Forms. UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
0031ee8c 69f16e2c [InlinedCallFrame: 0031ee8c]
0031ef24 69f16e2c System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0031ef7c 69f16c81 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0031efac 69ea366d System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0031efc0 003c3f0d LINQPad.Program.Run(System.String, Boolean, System.String, Boolean, Boolean, System.String)
0031f0b4 003c1515 LINQPad.Program.Go(System.String[])
0031f2e4 003c0584 LINQPad.Program.Start(System.String[])
0031f324 003c034f LINQPad.ProgramStarter.Run(System.String[])
0031f330 003c00f5 LINQPad.Loader.Main(System.String[])
0031f570 6c1721db [GCFrame: 0031f570]
OS Thread Id: 0×36f8 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×36fc (2)
Child SP IP Call Site
03c2fb78 779b0f34 [DebuggerU2MCatchHandlerFrame: 03c2fb78]
OS Thread Id: 0×3700 (3)
Child SP IP Call Site
0470f0cc 779b0f34 [InlinedCallFrame: 0470f0cc]
0470f0c8 699380b7 DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafePipeHandle, IntPtr)
0470f0cc 699cc07c [InlinedCallFrame: 0470f0cc] Microsoft.Win32.UnsafeNativeMethods.ConnectNamedPipe(Microsoft.Win32.SafeHandles.SafePipeHandle, IntPtr)
0470f130 699cc07c System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
0470f140 003c31ed LINQPad.Program.Listen()
0470f1d8 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0470f1e8 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0470f20c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0470f228 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0470f44c 6c1721db [GCFrame: 0470f44c]
0470f710 6c1721db [DebuggerU2MCatchHandlerFrame: 0470f710]
OS Thread Id: 0×3704 (4)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3710 (5)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3714 (6)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3718 (7)
Child SP IP Call Site
0596ee40 779b0f34 [GCFrame: 0596ee40]
0596ef34 779b0f34 [HelperMethodFrame_1OBJ: 0596ef34] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
0596ef90 6b5d9140 System.Threading.Monitor.Wait(System.Object, Int32, Boolean)
0596efa0 6bb9428a System.Threading.Monitor.Wait(System.Object)
0596efa4 003cb55a ActiproSoftware.SyntaxEditor.SemanticParserService.c()
0596f068 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0596f078 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0596f09c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0596f0b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0596f2dc 6c1721db [GCFrame: 0596f2dc]
0596f5a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0596f5a0]
OS Thread Id: 0×3764 (8)
Child SP IP Call Site
0564f148 779b0f34 [HelperMethodFrame_1OBJ: 0564f148] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
0564f1f0 6b64b5ef System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
0564f20c 6b64b5ad System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0564f224 6b64b570 System.Threading.WaitHandle.WaitOne()
0564f22c 04b607d4 ActiproBridge.ReferenceManager+?15?.<StartAdvanceFeeder>b__4()
0564f268 6b64ae5b System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0564f278 6b5d7ff4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0564f29c 6b5d7f34 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0564f2b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0564f4dc 6c1721db [GCFrame: 0564f4dc]
0564f7a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0564f7a0]
OS Thread Id: 0×3768 (9)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×376c (10)
Child SP IP Call Site
GetFrameContext failed: 1
OS Thread Id: 0×3798 (11)
Child SP IP Call Site
GetFrameContext failed: 1
OS Thread Id: 0×37e0 (12)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×37f8 (13)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns, Debugging, Stack Trace Collection, WinDbg Tips and Tricks | 1 Comment »