Crash Dump Analysis Patterns (Part 254)

Sometimes we can diagnose potential data races by looking at Stack Trace Collection (from all processes and threads or from CPUs). For example, if we have one Exception Stack Trace and a very similar stack trace, for example, with Constant Subtrace and the same Stack Trace Motif, it may point to such a situation with potential diagnosis of related components. Here’s an example from a kernel memory dump where it was suggested as a workaround to close ProcessA before hibernating a system (the problem repeatedly happened on wake up):

0: kd> !analyze -v

[...]

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure. The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000003, A LIST_ENTRY has been corrupted (i.e. double remove).
Arg2: ffff8900ea8fae40, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffff8900ea8fad98, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

[...]

EXCEPTION_RECORD: ffff8900ea8fad98 -- (.exr 0xffff8900ea8fad98)
ExceptionAddress: fffff8034a9e2e93 (nt!CmpFreeKeyControlBlock+0x0000000000106a73)
ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 0000000000000003
Subcode: 0x3 FAST_FAIL_CORRUPT_LIST_ENTRY

[...]

0: kd> k
# Child-SP RetAddr Call Site
00 ffff8900`ea8fab18 fffff803`4a576ba9 nt!KeBugCheckEx
01 ffff8900`ea8fab20 fffff803`4a576f50 nt!KiBugCheckDispatch+0x69
02 ffff8900`ea8fac60 fffff803`4a575286 nt!KiFastFailDispatch+0xd0
03 ffff8900`ea8fae40 fffff803`4a9e2e93 nt!KiRaiseSecurityCheckFailure+0x3c6
04 ffff8900`ea8fafd0 fffff803`4a8c444b nt!CmpFreeKeyControlBlock+0×106a73
05 ffff8900`ea8fb000 fffff803`4a8e1392 nt!CmpDoParseKey+0×2adb
06 ffff8900`ea8fb3d0 fffff803`4a8bdd61 nt!CmpParseKey+0×302
07 ffff8900`ea8fb570 fffff803`4a8d3a2d nt!ObpLookupObjectName+0xb71
08 ffff8900`ea8fb740 fffff803`4a8d370d nt!ObOpenObjectByNameEx+0×1dd
09 ffff8900`ea8fb880 fffff803`4a8cfc5f nt!CmOpenKey+0×29d
0a ffff8900`ea8fba40 fffff803`4a576683 nt!NtOpenKeyEx+0xf
0b ffff8900`ea8fba80 00007ffc`ac3b82b4 nt!KiSystemServiceCopyEnd+0×13
0c 00000001`2c0feb28 00000000`00000000 0×00007ffc`ac3b82b4

0: kd> !stacks 2 nt!Cmp

[...]

[ffffaa0a35ccb2c0 winlogon.exe]
13dc.002668 ffffaa0a37e02080 fff4e8f4 RUNNING nt!KeBugCheckEx
nt!KiBugCheckDispatch+0×69
nt!KiFastFailDispatch+0xd0
nt!KiRaiseSecurityCheckFailure+0×3c6

nt!CmpFreeKeyControlBlock+0×106a73
nt!CmpDoParseKey+0×2adb
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bda780 ProcessA.exe]
1794.0017e4 ffffaa0a35c06080 fff4e8f4 RUNNING nt!CmpLockKcbStackShared+0×2
nt!CmpWalkOneLevel+0×93
nt!CmpDoParseKey+0×1c36
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bb8780 svchost.exe]
1754.001fec ffffaa0a3636a7c0 fff4e8f4 READY nt!KxDispatchInterrupt+0x122
nt!KiDpcInterrupt+0x3a6
nt!KeAbPreAcquire+0xd7
nt!ExfAcquirePushLockExclusiveEx+0x10b
nt!CmpDoParseKey+0×294e
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

We call this pattern Stack Trace Race.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply