Crash Dump Analysis Patterns (Part 88)
Some modules like drivers or runtime DLLs are always present after some action has happened. I call them Effect Components. It is the last thing to assume them to be the “Cause” components” or “Root Cause” or the so so called “culprit” components. Typical example, is dump disk driver symbolic references found in execution residue on the raw stack of a running bugchecking thread:
0: kd> !thread
THREAD fffffa8002bdebb0 Cid 03c4.03f0 Teb: 000007fffffde000 Win32Thread: fffff900c20f9810 RUNNING on processor 0
IRP List:
fffffa8002b986f0: (0006,0118) Flags: 00060000 Mdl: 00000000
Not impersonating
DeviceMap fffff88005346920
Owning Process fffffa80035bec10 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 35246 Ticks: 7 (0:00:00:00.109)
Context Switch Count 1595 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address Application (0x0000000140002708)
Stack Init fffffa600495ddb0 Current fffffa600495d720
Base fffffa600495e000 Limit fffffa6004955000 Call 0
Priority 11 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Child-SP RetAddr : Call Site
fffffa60`0495d558 fffff800`0186e3ee : nt!KeBugCheckEx
fffffa60`0495d560 fffff800`0186d2cb : nt!KiBugCheckDispatch+0×6e
fffffa60`0495d6a0 fffffa60`03d5917a : nt!KiPageFault+0×20b (TrapFrame @ fffffa60`0495d6a0)
[…]
0: kd> dps fffffa6004955000 fffffa600495e000
fffffa60`04955000 00d4d0c8`00d4d0c8
fffffa60`04955008 00d4d0c8`00d4d0c8
fffffa60`04955010 00d4d0c8`00d4d0c8
[…]
fffffa60`0495c7e0 00000000`00000001
fffffa60`0495c7e8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c7f0 fffffa80`024c05a8
fffffa60`0495c7f8 fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495c800 fffffa60`0495cb00
fffffa60`0495c808 fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c810 fffffa60`02860110 crashdmp!StrBeginningDump
fffffa60`0495c818 fffffa60`0495cb00
fffffa60`0495c820 00000000`00000000
fffffa60`0495c828 fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c830 00000000`00000000
fffffa60`0495c838 fffffa60`0495c8c0
fffffa60`0495c840 00000000`00000000
fffffa60`0495c848 fffffa60`00000024
fffffa60`0495c850 00000000`ffffffff
fffffa60`0495c858 00000000`00000000
fffffa60`0495c860 00000000`00000000
fffffa60`0495c868 fffffa60`0495cb00
fffffa60`0495c870 fffffa80`00000000
fffffa60`0495c878 00000000`00000000
fffffa60`0495c880 00000000`00000101
fffffa60`0495c888 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c890 fffffa60`0495cb0f
fffffa60`0495c898 fffff800`0182ff34 nt!output_l+0×6c0
fffffa60`0495c8a0 fffffa60`0495cb0f
fffffa60`0495c8a8 fffffa60`0495cb90
fffffa60`0495c8b0 00000000`00000040
fffffa60`0495c8b8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c8c0 fffffa80`024c0728
fffffa60`0495c8c8 fffffa80`024c0728
fffffa60`0495c8d0 00000001`00000000
fffffa60`0495c8d8 fffffa60`00000026
fffffa60`0495c8e0 00000000`ffffffff
fffffa60`0495c8e8 00000000`00000000
fffffa60`0495c8f0 fffffa80`00000000
fffffa60`0495c8f8 fffffa60`0495cb90
fffffa60`0495c900 00000000`00000000
fffffa60`0495c908 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c910 00000000`00000000
fffffa60`0495c918 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c920 fffff880`05311010
fffffa60`0495c928 00000000`00000002
fffffa60`0495c930 fffffa60`02875094 dump_SATA_Driver!AhciAdapterControl
fffffa60`0495c938 fffffa80`024c6018
fffffa60`0495c940 fffffa80`024c0728
fffffa60`0495c948 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495c950 fffffa80`024c0728
fffffa60`0495c958 00000000`00000000
fffffa60`0495c960 fffffa60`0495ca18
fffffa60`0495c968 00000000`00000000
fffffa60`0495c970 fffffa80`024c0728
fffffa60`0495c978 fffffa60`02876427 dump_SATA_Driver!AhciHwInitialize+0×337
fffffa60`0495c980 fffffa80`024c0be6
fffffa60`0495c988 fffffa60`0286a459 dump_dumpata!IdeDumpWaitOnRequest+0×79
fffffa60`0495c990 00000000`00000000
fffffa60`0495c998 00000000`0000023a
fffffa60`0495c9a0 20474e55`534d4153
fffffa60`0495c9a8 204a4831`36314448
fffffa60`0495c9b0 20202020`20202020
fffffa60`0495c9b8 20202020`20202020
fffffa60`0495c9c0 fffffa80`024c05a8
fffffa60`0495c9c8 fffffa60`02869b18 dump_dumpata!IdeDumpNotification+0×1e8
fffffa60`0495c9d0 00000000`00000000
fffffa60`0495c9d8 fffffa60`0495ca60
fffffa60`0495c9e0 00000000`00000001
fffffa60`0495c9e8 fffffa60`02869396 dump_dumpata!IdeDumpMiniportChannelInitialize+0×236
fffffa60`0495c9f0 fffffa80`024c05a8
fffffa60`0495c9f8 fffffa60`02869ad4 dump_dumpata!IdeDumpNotification+0×1a4
fffffa60`0495ca00 00000000`00000000
fffffa60`0495ca08 fffffa60`0495ca90
fffffa60`0495ca10 00000000`00000001
fffffa60`0495ca18 00000001`00000038
fffffa60`0495ca20 00000000`10010000
fffffa60`0495ca28 00000000`00000003
fffffa60`0495ca30 fffffa80`024c05a8
fffffa60`0495ca38 fffffa60`0286a954 dump_dumpata!AtaPortGetPhysicalAddress+0×2c
fffffa60`0495ca40 fffffa80`024c0728
fffffa60`0495ca48 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca50 00000000`00000001
fffffa60`0495ca58 0000003f`022a8856
fffffa60`0495ca60 fffffa80`0000000c
fffffa60`0495ca68 fffffa80`024c0728
fffffa60`0495ca70 00000000`00000200
fffffa60`0495ca78 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495ca80 fffffa80`024c0728
fffffa60`0495ca88 ffff6226`4f5f3eb8
fffffa60`0495ca90 00000000`00000010
fffffa60`0495ca98 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495caa0 fffffa80`024c05a8
fffffa60`0495caa8 fffffa60`02875a0d dump_SATA_Driver!AhciHwStartIo+0×69d
fffffa60`0495cab0 fffffa80`024c0728
fffffa60`0495cab8 00000000`00000000
fffffa60`0495cac0 00000000`00000001
fffffa60`0495cac8 fffff800`018f3dfc nt!DisplayCharacter+0×5c
fffffa60`0495cad0 00000000`00000000
fffffa60`0495cad8 fffffa60`02877f6f dump_SATA_Driver!RecordExecutionHistory+0xcf
fffffa60`0495cae0 00000000`00010000
fffffa60`0495cae8 00000000`00000000
fffffa60`0495caf0 fffffa60`0495cd10
fffffa60`0495caf8 fffffa60`0495cc00
fffffa60`0495cb00 fffffa80`024c01c0
fffffa60`0495cb08 fffffa60`02875c3f dump_SATA_Driver!AhciHwInterrupt+0×2b
fffffa60`0495cb10 fffffa80`024c05a8
fffffa60`0495cb18 00000000`00000000
fffffa60`0495cb20 00000000`00000000
fffffa60`0495cb28 fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495cb30 00000000`00010000
fffffa60`0495cb38 00000000`00000000
fffffa60`0495cb40 fffffa60`0495cd10
fffffa60`0495cb48 fffffa60`0495cc00
fffffa60`0495cb50 00000000`00000000
fffffa60`0495cb58 fffffa60`0286a429 dump_dumpata!IdeDumpWaitOnRequest+0×49
fffffa60`0495cb60 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cb68 00000000`d8bda325
fffffa60`0495cb70 00000000`00000000
fffffa60`0495cb78 00000000`0000033e
fffffa60`0495cb80 00000000`00000000
fffffa60`0495cb88 fffffa60`028694d2 dump_dumpata!IdeDumpWritePending+0xee
fffffa60`0495cb90 fffffa80`024c0000
fffffa60`0495cb98 fffffa80`024c01c0
fffffa60`0495cba0 00000000`00000000
fffffa60`0495cba8 00000000`00000000
fffffa60`0495cbb0 fffffa80`024c01c0
fffffa60`0495cbb8 fffffa80`01e3c740
fffffa60`0495cbc0 00000000`00010000
fffffa60`0495cbc8 00000000`00000000
fffffa60`0495cbd0 00000000`0c01f000
fffffa60`0495cbd8 fffffa60`0285bca9 crashdmp!WritePageSpanToDisk+0×181
fffffa60`0495cbe0 00000000`83d81000
fffffa60`0495cbe8 00000000`00000000
fffffa60`0495cbf0 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cbf8 00000000`00000002
fffffa60`0495cc00 00000000`00000000
fffffa60`0495cc08 00000000`00030000
fffffa60`0495cc10 00000000`00000000
fffffa60`0495cc18 fffffa60`00441000
fffffa60`0495cc20 fffffa60`00441000
fffffa60`0495cc28 00000000`00010000
fffffa60`0495cc30 00000000`0000c080
fffffa60`0495cc38 00000000`0000c081
fffffa60`0495cc40 00000000`0000c082
fffffa60`0495cc48 00000000`0000c083
fffffa60`0495cc50 00000000`0000c084
fffffa60`0495cc58 00000000`0000c085
fffffa60`0495cc60 00000000`0000c086
fffffa60`0495cc68 00000000`0000c087
fffffa60`0495cc70 00000000`0000c088
fffffa60`0495cc78 00000000`0000c089
fffffa60`0495cc80 00000000`0000c08a
fffffa60`0495cc88 00000000`0000c08b
fffffa60`0495cc90 00000000`0000c08c
fffffa60`0495cc98 00000000`0000c08d
fffffa60`0495cca0 00000000`0000c08e
fffffa60`0495cca8 00000000`0000c08f
fffffa60`0495ccb0 00000000`00000000
fffffa60`0495ccb8 00000000`00000000
fffffa60`0495ccc0 00000000`00000000
fffffa60`0495ccc8 00000000`00000010
fffffa60`0495ccd0 00000000`0000c01d
fffffa60`0495ccd8 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cce0 00000000`0000bf80
fffffa60`0495cce8 00000000`00000001
fffffa60`0495ccf0 00000000`00000000
fffffa60`0495ccf8 fffffa80`01e353d0
fffffa60`0495cd00 fffffa80`01e353f8
fffffa60`0495cd08 fffffa60`0285bacc crashdmp!WriteFullDump+0×70
fffffa60`0495cd10 00000002`3a3d8000
fffffa60`0495cd18 00000000`0000c080
fffffa60`0495cd20 fffffa80`00000000
fffffa60`0495cd28 fffffa60`0285c9c0 crashdmp!CrashdmpWriteRoutine
fffffa60`0495cd30 fffff880`05311010
fffffa60`0495cd38 00000000`00000002
fffffa60`0495cd40 fffffa60`0495cf70
fffffa60`0495cd48 00000000`00000000
fffffa60`0495cd50 fffffa60`02860370 crashdmp!Context+0×30
fffffa60`0495cd58 fffffa60`0285b835 crashdmp!DumpWrite+0xc5
fffffa60`0495cd60 00000000`00000000
fffffa60`0495cd68 00000000`0000000f
fffffa60`0495cd70 00000000`00000001
fffffa60`0495cd78 fffffa60`00000001
fffffa60`0495cd80 fffffa80`02bdebb0
fffffa60`0495cd88 fffffa60`0285b153 crashdmp!CrashdmpWrite+0×57
fffffa60`0495cd90 00000000`00000000
fffffa60`0495cd98 fffffa60`028602f0 crashdmp!StrInitPortDriver
fffffa60`0495cda0 00000000`00000000
fffffa60`0495cda8 fffffa60`02860a00 crashdmp!ContextCopy
fffffa60`0495cdb0 00000000`00000000
fffffa60`0495cdb8 fffff800`01902764 nt!IoWriteCrashDump+0×3f4
fffffa60`0495cdc0 fffffa60`0495ce00
fffffa60`0495cdc8 00000028`00000025
fffffa60`0495cdd0 fffff800`018afd40 nt! ?? ::FNODOBFM::`string’
fffffa60`0495cdd8 00000000`000000d1
fffffa60`0495cde0 fffff880`05311010
fffffa60`0495cde8 00000000`00000002
fffffa60`0495cdf0 00000000`00000000
fffffa60`0495cdf8 fffffa60`03d5917a
fffffa60`0495ce00 202a2a2a`0a0d0a0d
fffffa60`0495ce08 7830203a`504f5453
fffffa60`0495ce10 31443030`30303030
fffffa60`0495ce18 46464646`78302820
fffffa60`0495ce20 31333530`30383846
fffffa60`0495ce28 fffff800`018f5f83 nt!VidDisplayString+0×143
fffffa60`0495ce30 30303030`30300030
fffffa60`0495ce38 2c323030`30303030
fffffa60`0495ce40 30303030`30307830
fffffa60`0495ce48 30303030`30303030
fffffa60`0495ce50 46464678`302c3030
fffffa60`0495ce58 fffff800`018fe040 nt!KiInvokeBugCheckEntryCallbacks+0×80
fffffa60`0495ce60 fffffa80`02bdebb0
fffffa60`0495ce68 fffff800`01921d52 nt!InbvDisplayString+0×72
fffffa60`0495ce70 fffff880`05311000
fffffa60`0495ce78 fffff800`01d406c9 hal!KeStallExecutionProcessor+0×25
fffffa60`0495ce80 00000000`00000001
fffffa60`0495ce88 00000000`0000000a
fffffa60`0495ce90 fffffa60`03d5917a
fffffa60`0495ce98 00000000`40000082
fffffa60`0495cea0 00000000`00000001
fffffa60`0495cea8 fffff800`01922c3e nt!KeBugCheck2+0×92e
fffffa60`0495ceb0 fffff800`000000d1
fffffa60`0495ceb8 00000000`000004d0
fffffa60`0495cec0 fffff800`01a43640 nt!KiProcessorBlock
fffffa60`0495cec8 00000000`0000000a
fffffa60`0495ced0 fffffa60`03d5917a
fffffa60`0495ced8 fffffa60`0495cf70
fffffa60`0495cee0 fffffa80`02bdebb0
fffffa60`0495cee8 00000000`00000000
fffffa60`0495cef0 00000000`00000000
fffffa60`0495cef8 fffffa80`02bdebb0
fffffa60`0495cf00 00000000`c21a6d00
fffffa60`0495cf08 00000000`00000000
fffffa60`0495cf10 fffff800`0198e7a0 nt!KiInitialPCR+0×2a0
fffffa60`0495cf18 fffff800`0198e680 nt!KiInitialPCR+0×180
fffffa60`0495cf20 fffffa80`02bb7320
fffffa60`0495cf28 00000000`00000000
fffffa60`0495cf30 00000000`00000000
fffffa60`0495cf38 fffff960`00000003
fffffa60`0495cf40 fffffa60`0495e000
fffffa60`0495cf48 fffffa60`04955000
fffffa60`0495cf50 00000001`c0643000
fffffa60`0495cf58 00000000`00000000
fffffa60`0495cf60 fffff900`c06ca53c
fffffa60`0495cf68 fffffa60`0495d090
fffffa60`0495cf70 00000000`00000000
fffffa60`0495cf78 00000000`00000000
fffffa60`0495cf80 00000000`00000000
fffffa60`0495cf88 00000000`00000000
fffffa60`0495cf90 00000000`00000000
fffffa60`0495cf98 00000000`00000000
fffffa60`0495cfa0 00001f80`0010000f
fffffa60`0495cfa8 0053002b`002b0010
fffffa60`0495cfb0 00000286`0018002b
fffffa60`0495cfb8 00000000`00000000
fffffa60`0495cfc0 00000000`00000000
fffffa60`0495cfc8 00000000`00000000
fffffa60`0495cfd0 00000000`00000000
fffffa60`0495cfd8 00000000`00000000
fffffa60`0495cfe0 00000000`00000000
fffffa60`0495cfe8 fffffa60`0495d660
fffffa60`0495cff0 00000000`0000000a
fffffa60`0495cff8 fffff880`05311010
fffffa60`0495d000 fffff880`05311010
fffffa60`0495d008 fffffa60`0495d558
fffffa60`0495d010 fffffa60`0495d720
fffffa60`0495d018 fffffa80`02b986f0
fffffa60`0495d020 fffffa80`02b98720
fffffa60`0495d028 00000000`00000002
fffffa60`0495d030 00000000`00000000
fffffa60`0495d038 fffffa60`03d5917a
fffffa60`0495d040 00000000`000001f1
fffffa60`0495d048 fffffa80`026a9df0
fffffa60`0495d050 00000000`00000001
fffffa60`0495d058 00000000`83360018
fffffa60`0495d060 fffffa80`02b3ee40
fffffa60`0495d068 fffff800`0186e650 nt!KeBugCheckEx
fffffa60`0495d070 00000000`00000000
fffffa60`0495d078 00000000`00000000
fffffa60`0495d080 00000000`00000000
fffffa60`0495d088 00000000`00000000
fffffa60`0495d090 00000000`00000000
fffffa60`0495d098 00000000`00000000
fffffa60`0495d0a0 00000000`00000000
[…]
If a BSOD was reported after installing new drivers we shouldn’t suspect SATA_Driver package here because its components would almost always be present on any bugcheck thread as referenced after a bugcheck cause. There presence is the “effect”. This example might seem trivial and pointless but I’ve seen some memory dump analysis conclusions based on the reversal of causes and effects.
- Dmitry Vostokov @ DumpAnalysis.org -
March 11th, 2010 at 12:29 am
[…] !thread output fields (p. 376) - Stack Base and Limit fields can be useful to dump raw stack data via dps command to see execution residue or when reconstructing stack trace, see, for example, this pattern: http://www.dumpanalysis.org/blog/index.php/2009/10/23/crash-dump-analysis-patterns-part-88/ […]