Crash Dump Analysis Patterns (Part 160)
When doing software behavior artifact collection, live debugging or postmortem memory dump analysis we must also take into consideration the possibility of Debugger Bugs. I classify them into hard and soft bugs. The former are those software defects and behavioral problems that result in further abnormal software behavior incidents like crashes and hangs. One example is this Microsoft KB article about DebugDiag. Soft debugger bugs usually manifest themselves as glitches in data output, nonsense or false positive diagnostics, for example, this excessive non-paged pool usage message in the output from !vm WinDbg command (see the corresponding MS KB article):
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 1031581 ( 4126324 Kb)
Page File: \??\C:\pagefile.sys
Current: 4433524 Kb Free Space: 4433520 Kb
Minimum: 4433524 Kb Maximum: 12378972 Kb
Unimplemented error for MiSystemVaTypeCount
Available Pages: 817652 ( 3270608 Kb)
ResAvail Pages: 965229 ( 3860916 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33555714 ( 134222856 Kb)
Modified Pages: 15794 ( 63176 Kb)
Modified PF Pages: 15793 ( 63172 Kb)
NonPagedPool Usage: 88079121 ( 352316484 Kb)
NonPagedPoolNx Usage: 12885 ( 51540 Kb)
NonPagedPool Max: 764094 ( 3056376 Kb)
********** Excessive NonPaged Pool Usage *****
PagedPool 0 Usage: 35435 ( 141740 Kb)
PagedPool 1 Usage: 3620 ( 14480 Kb)
PagedPool 2 Usage: 573 ( 2292 Kb)
PagedPool 3 Usage: 535 ( 2140 Kb)
PagedPool 4 Usage: 538 ( 2152 Kb)
PagedPool Usage: 40701 ( 162804 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 9309 ( 37236 Kb)
Shared Commit: 6460 ( 25840 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 5760 ( 23040 Kb)
PagedPool Commit: 40765 ( 163060 Kb)
Driver Commit: 2805 ( 11220 Kb)
Committed pages: 212472 ( 849888 Kb)
Commit limit: 2139487 ( 8557948 Kb)
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
December 3rd, 2020 at 11:30 am
This is the example of a soft variant of Debugger Bug crash dump analysis pattern () but in a different binary media and OS https://arxiv.org/abs/2011.13994
November 24th, 2021 at 9:47 am
A perennial WinDbg feature: to double each line of output after closing the debugging session (q command) and opening a new dump file.
November 24th, 2021 at 9:48 am
WinDbg 10.0.22000.194 (and 10.0.22473.1005) cannot do “.thread /w” command correctly for complete dumps even from Windows 11, I have to revert back to 10.0.20348.1 for my forthcoming training…+0×43
; 10.0.22000.194
0: kd> .thread /w ffffc38c2ecd4080
0: kd:x86> k
00 95d80609 00000000 0×6221d056
0: kd:x86> .reload
0: kd:x86> k
00 95d80609 00000000 0×6221d056
; 10.0.20348.1
0: kd> .thread /w ffffc38c2ecd4080
0: kd:x86> k
00 032bfaa0 74e7e768 0×7657112c
01 032bfad0 74ecb218 0×74e7e768
02 032bfb9c 74ec9f80 0×74ecb218
03 032bfcf8 74ecaa87 0×74ec9f80
04 032bfd80 74ecaaf5 0×74ecaa87
05 032bfda0 000211e4 0×74ecaaf5
06 032bfdd8 0002181a 0×211e4
07 032bfdf4 75bb6823 0×2181a
08 032bfe2c 75ca6739 0×75bb6823
09 032bfe3c 77058aff 0×75ca6739
0a 032bfe94 77058acd 0×77058aff
0b 032bfea4 00000000 0×77058acd
0: kd:x86> .reload
0: kd:x86> k
00 032bfa58 74e7e889 win32u!NtUserWaitMessage+0xc
01 032bfaa0 74e7e768 USER32!DialogBox2+0×10a
02 032bfad0 74ecb218 USER32!InternalDialogBox+0xd7
03 032bfb9c 74ec9f80 USER32!SoftModalMessageBox+0×718
04 032bfcf8 74ecaa87 USER32!MessageBoxWorker+0×314
05 032bfd80 74ecaaf5 USER32!MessageBoxTimeoutW+0×187
06 032bfda0 000211e4 USER32!MessageBoxW+0×45
07 032bfdd8 0002181a AppA+0×11e4
08 032bfdf4 75bb6823 AppA+0×181a
09 032bfe2c 75ca6739 ucrtbase!thread_start
0a 032bfe3c 77058aff KERNEL32!BaseThreadInitThunk+0×19
0b 032bfe94 77058acd ntdll_76ff0000!__RtlUserThreadStart+0×2b
0c 032bfea4 00000000 ntdll_76ff0000!_RtlUserThreadStart+0×1b
November 24th, 2021 at 9:48 am
Another command !running is broken in WinDbg 10.0.22000.194 from SDK (but works in 10.0.22473.1005 version from WinDbg Preview):
0: kd> !running
GetGlobalValue: unable to get nt!KiSubNodes type size
Could not get address of nt!KiSubNodes
Could not get idle processor summary.
May 28th, 2022 at 9:03 pm
The recent bug is an inability to reconstruct 32-bit stack traces from the WOW64 process in both process and complete memory dumps.
January 7th, 2025 at 7:36 pm
When the Linux crash tool crashes when asked to reconstruct the stack trace from the given RSP, we can dump raw stack data instead and reconstruct the stack trace ourselves.
November 15th, 2025 at 7:23 pm
Another example from Windows ARM64: aligned local variable:
_Alignas(4096) long long ll = 1;
0:000> dv /V
00000037`30b8e690 @x27+0×1000 ll = 0n-3689348814741910324
00000037`30b8d8c0 @x27+0×0230 align = 8
They are stored indirectly instead:
00007ff6`e7afafdc f9533f69 ldr x9,[x27,#0×2678]
00007ff6`e7afafe0 d2800028 mov x8,#1
00007ff6`e7afafe4 f9000128 str x8,[x9]
0:000> ? @x27+0×2678
Evaluate expression: 237040631048 = 00000037`30b8fd08
0:000> dp 00000037`30b8fd08 L1
00000037`30b8fd08 00000037`30b8d000
0:000> dpp 00000037`30b8fd08 L1
00000037`30b8fd08 00000037`30b8d000 00000000`00000001