Crash Dump Analysis Patterns (Part 233)

With the possibility of process cloning (reflection) starting from Windows 7 it is possible to get memory snapshots (Clone Dump) from a process clone (similar to fork API in Unix). Procdump tool has -r switch for that purpose. We checked this with x64 Windows 7 notepad.exe. We got two memory dumps: one is a clone with this stack trace:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dbgcfg.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: '
*** procdump -ma -r Notepad.exe
*** Manual dump'

0:000> ~*k

. 0 Id: 25ec.147c Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`02c8fd38 00000000`7733aae7 ntdll!NtSuspendThread+0xa
01 00000000`02c8fd40 00000000`77165a4d ntdll!RtlpProcessReflectionStartup+0×2e7
02 00000000`02c8fe30 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd
03 00000000`02c8fe60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The process memory has all address space of the original process including module list and heap structure:

0:000> lmn
start end module name
00000000`77050000 00000000`7714a000 user32 user32.dll
00000000`77150000 00000000`77270000 kernel32 kernel32.dll
00000000`77270000 00000000`77419000 ntdll ntdll.dll
00000000`ff030000 00000000`ff065000 notepad notepad.exe
000007fe`f57d0000 000007fe`f5841000 winspool winspool.drv
000007fe`fb730000 000007fe`fb786000 uxtheme uxtheme.dll
000007fe`fb910000 000007fe`fbb04000 comctl32 comctl32.dll
000007fe`fbf00000 000007fe`fbf0c000 version version.dll
000007fe`fceb0000 000007fe`fcebf000 CRYPTBASE CRYPTBASE.dll
000007fe`fd310000 000007fe`fd37c000 KERNELBASE KERNELBASE.dll
000007fe`fd3d0000 000007fe`fd499000 usp10 usp10.dll
000007fe`fd4a0000 000007fe`fd511000 shlwapi shlwapi.dll
000007fe`fd520000 000007fe`fd64d000 rpcrt4 rpcrt4.dll
000007fe`fd650000 000007fe`fd66f000 sechost sechost.dll
000007fe`fd680000 000007fe`fd6ae000 imm32 imm32.dll
000007fe`fd6b0000 000007fe`fd78b000 advapi32 advapi32.dll
000007fe`fd810000 000007fe`fd8a7000 comdlg32 comdlg32.dll
000007fe`fdd60000 000007fe`fddff000 msvcrt msvcrt.dll
000007fe`fdfe0000 000007fe`fed69000 shell32 shell32.dll
000007fe`fed70000 000007fe`fedd7000 gdi32 gdi32.dll
000007fe`ff0b0000 000007fe`ff1b9000 msctf msctf.dll
000007fe`ff1c0000 000007fe`ff1ce000 lpk lpk.dll
000007fe`ff1d0000 000007fe`ff2a7000 oleaut32 oleaut32.dll
000007fe`ff2b0000 000007fe`ff349000 clbcatq clbcatq.dll
000007fe`ff350000 000007fe`ff553000 ole32 ole32.dll

0:000> !address -summary

Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 48 7ff`faa06000 ( 8.000 TB) 100.00%
Image 129 0`01e79000 ( 30.473 MB) 35.47% 0.00%
21 0`01d10000 ( 29.063 MB) 33.83% 0.00%
Other 9 0`016be000 ( 22.742 MB) 26.47% 0.00%
Heap 26 0`00320000 ( 3.125 MB) 3.64% 0.00%
Stack 3 0`00080000 ( 512.000 kB) 0.58% 0.00%
TEB 1 0`00002000 ( 8.000 kB) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_IMAGE 130 0`01e7a000 ( 30.477 MB) 35.47% 0.00%
MEM_PRIVATE 47 0`01449000 ( 20.285 MB) 23.61% 0.00%
MEM_MAPPED 11 0`00c97000 ( 12.590 MB) 14.65% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 50 7ff`fc096000 ( 8.000 TB) 100.00%
MEM_COMMIT 176 0`02c10000 ( 44.063 MB) 51.29% 0.00%
MEM_RESERVE 12 0`0134a000 ( 19.289 MB) 22.45% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READONLY 83 0`01b46000 ( 27.273 MB) 31.75% 0.00%
PAGE_EXECUTE_READ 25 0`00f6d000 ( 15.426 MB) 17.95% 0.00%
PAGE_WRITECOPY 48 0`00126000 ( 1.148 MB) 1.34% 0.00%
PAGE_READWRITE 18 0`00032000 ( 200.000 kB) 0.23% 0.00%
PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kB) 0.02% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`ff065000 7fd`f676b000 ( 7.992 TB)
Image 7fe`fe4cb000 0`0089e000 ( 8.617 MB)
0`7f0e0000 0`00f00000 ( 15.000 MB)
Other 0`00610000 0`01590000 ( 21.563 MB)
Heap 0`003b8000 0`000c8000 ( 800.000 kB)
Stack 0`02c10000 0`0006c000 ( 432.000 kB)
TEB 7ff`fffdb000 0`00002000 ( 8.000 kB)
PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

0:000> !heap -s

******************************************************
NT HEAP STATS BELOW
******************************************************
LFH Key : 0x000000381021167d
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000000280000 00000002 1024 412 1024 14 4 1 0 0 LFH
0000000000250000 00001002 1088 256 1088 5 2 2 0 0 LFH
0000000001cc0000 00001002 64 8 64 3 1 1 0 0
0000000001e60000 00001002 512 120 512 49 3 1 0 0
0000000001dc0000 00001002 512 8 512 2 1 1 0 0
-------------------------------------------------------------------------------------

The other dump saved is a minidump from which we can get thread information for Execution Residue (raw stack data) and reconstruct stack traces in Clone Dump:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dmp]
Comment: '
*** procdump -ma -r Notepad.exe
*** Manual dump'
User Mini Dump File: Only registers, stack and portions of memory are available

0:000> ~
. 0 Id: 87c.27f4 Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen

0:000> k
# Child-SP RetAddr Call Site
00 00000000`0016fac8 00000000`77069e9e 0x77069e6a
01 00000000`0016fad0 00000000`00000000 0x77069e9e

0:000> r
rax=0000000000000000 rbx=000000000016fb40 rcx=0000000000280000
rdx=0000000000000000 rsi=0000000000000001 rdi=0000000000000000
rip=0000000077069e6a rsp=000000000016fac8 rbp=00000000ff030000
r8=000000000016f8e8 r9=00000000000a0cdc r10=0000000000000000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
00000000`77069e6a c3 ret

Now we can see the original stack trace in Clone Dump:

0:000> k =000000000016fac8 ff
# Child-SP RetAddr Call Site
00 00000000`0016fac8 00000000`77069e9e ntdll!NtSuspendThread+0xa
01 00000000`0016fad0 00000000`ff031064 user32!GetMessageW+0x34
02 00000000`0016fb00 00000000`ff03133c notepad!WinMain+0x182
03 00000000`0016fb80 00000000`77165a4d notepad!DisplayNonGenuineDlgWorker+0x2da
04 00000000`0016fc40 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd
05 00000000`0016fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Since we know TEB address from the minidump we can get stack region boundaries in Clone Dump:

0:000> dt _NT_TIB 000007fffffdd000
ntdll!_NT_TIB
+0x000 ExceptionList : (null)
+0x008 StackBase : 0x00000000`00170000 Void
+0x010 StackLimit : 0x00000000`0015b000 Void
+0x018 SubSystemTib : (null)
+0x020 FiberData : 0x00000000`00001e00 Void
+0x020 Version : 0x1e00
+0x028 ArbitraryUserPointer : (null)
+0x030 Self : 0x000007ff`fffdd000 _NT_TIB

Now we can check Execution Residue (for example, for signs of Hidden Exceptions):

0:000> dpS 0x00000000`0015b000 0x00000000`00170000
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply