Minidump Analysis (Part 3)
Part 2 dealt with stack traces. Unfortunately stack traces reported by WinDbg, especially involving 3rd-party components, are usually incomplete and sometimes not even correct. They can also point to stable drivers when the system failure happened after slowly accumulated corruption caused by some intermediate driver or a combination of drivers.
Sometimes there are other 3rd-party drivers involved before the system crash that are not visible in the output of !analyze -v command and simply removing them, disabling or upgrading software they are part from make the system stable. To see them we can look at the so called raw stack data. Because kernel mode thread stack size is small (12Kb or 0×3000) we can simply dump memory range between ESP-3000 and ESP+3000. We can use RSP register for x64 dumps but the output will be the same.
Let’s look at our minidump from the previous part. The stack trace is small, incomplete and points to DisplayDriver. This is because we don’t have symbol information for DisplayDriver.dll. Could it be the case that DisplayDriver.dll was used incorrectly by another driver or operating system component? What are other components that might have been used prior to BSOD? Raw stack dump shows additional symbols like DisplayDriver_mini, win32k and dxg:
0: kd> dps esp-3000 esp+3000
b4f4f8b4 ????????
b4f4f8b8 ????????
b4f4f8bc ????????
b4f4f8c0 ????????
...
...
...
b4f51ffc ????????
b4f52000 00001000
b4f52004 00006000
b4f52008 b4f5204c
b4f5200c 89025978
b4f52010 89139000
b4f52014 00000000
b4f52018 b4f527ec
b4f5201c b4f52840
b4f52020 bfbf0ca6 DisplayDriver+0x21bca6
b4f52024 00000000
b4f52028 89025978
...
...
...
b4f52100 e24079e0
b4f52104 bfbf0ca6 DisplayDriver+0x21bca6
b4f52108 00000008
...
...
...
b4f52364 b4f52414
b4f52368 804dc0b2 nt!ExecuteHandler+0x24
b4f5236c b4f527ec
b4f52370 b4f52d40
b4f52374 b4f524e8
b4f52378 b4f52400
b4f5237c bf9d2132 dxg!_except_handler3
b4f52380 2a2a2a0a
...
...
...
b4f523e8 b4f52408
b4f523ec 8053738a nt!KeBugCheckEx+0x1b
b4f523f0 0000008e
b4f523f4 c0000005
b4f523f8 bfbf0ca6 DisplayDriver+0x21bca6
b4f523fc b4f52840
b4f52400 00000000
b4f52404 00000000
b4f52408 b4f527d0
b4f5240c 80521fed nt!KiDispatchException+0x3b1
b4f52410 0000008e
b4f52414 c0000005
b4f52418 bfbf0ca6 DisplayDriver+0x21bca6
b4f5241c b4f52840
b4f52420 00000000
b4f52424 03a3fb4c
b4f52428 03a3fb4c
b4f5242c b4f52800
b4f52430 00000000
b4f52434 00000000
b4f52438 00000000
b4f5243c b9deffc6 DisplayDriver_mini+0x4bfc6
b4f52440 897c621c
b4f52444 00000086
b4f52448 0000003c
b4f5244c b9f3af5a DisplayDriver_mini+0x196f5a
b4f52450 897c6200
b4f52454 00000086
b4f52458 897c6200
b4f5245c 00000000
b4f52460 00000000
b4f52464 00000000
b4f52468 b9f38b4e DisplayDriver_mini+0x194b4e
b4f5246c 00000000
...
...
...
b4f5250c 00002800
b4f52510 b9f3ac10 DisplayDriver_mini+0x196c10
b4f52514 897c6200
b4f52518 00002504
b4f5251c 00000010
b4f52520 897c6200
b4f52524 b9f2d194 DisplayDriver_mini+0x189194
b4f52528 897c6200
b4f5252c 00002504
b4f52530 00000010
b4f52534 897c6200
b4f52538 898cca80
b4f5253c 00000080
b4f52540 89654008
b4f52544 b9f358e2 DisplayDriver_mini+0x1918e2
b4f52548 897c6200
...
...
...
b4f5256c 00000000
b4f52570 b9deff5c DisplayDriver_mini+0x4bf5c
b4f52574 00000000
...
...
...
b4f5259c e24079e0
b4f525a0 bfbf0ca6 DisplayDriver+0x21bca6
b4f525a4 00000008
b4f525a8 00010246
b4f525ac b4f528b4
b4f525b0 00000010
b4f525b4 0000003c
b4f525b8 b9f3af5a DisplayDriver_mini+0x196f5a
b4f525bc 897c6200
b4f525c0 00000086
b4f525c4 89b81008
b4f525c8 897c6200
b4f525cc 00000000
b4f525d0 00007c00
b4f525d4 b9deff5c DisplayDriver_mini+0x4bf5c
b4f525d8 b9deff5c DisplayDriver_mini+0x4bf5c
b4f525dc 8988d7d8
b4f525e0 b9deff66 DisplayDriver_mini+0x4bf66
b4f525e4 b9deff5c DisplayDriver_mini+0x4bf5c
b4f525e8 8961c288
b4f525ec b9deff66 DisplayDriver_mini+0x4bf66
b4f525f0 8961c288
b4f525f4 00000000
b4f525f8 00000046
b4f525fc 00000000
b4f52600 89903000
b4f52604 b9e625a9 DisplayDriver_mini+0xbe5a9
b4f52608 8961c288
b4f5260c 00000046
b4f52610 00000000
b4f52614 b9deff5c DisplayDriver_mini+0x4bf5c
b4f52618 896ac008
...
...
...
b4f52630 898a8000
b4f52634 b9e9f220 DisplayDriver_mini+0xfb220
b4f52638 89941400
b4f5263c b9e2ffec DisplayDriver_mini+0x8bfec
b4f52640 00000000
b4f52644 00000000
b4f52648 00000050
b4f5264c b9e790d3 DisplayDriver_mini+0xd50d3
b4f52650 897c6200
...
...
...
b4f5266c 89bf6200
b4f52670 805502fa nt!ExFreePoolWithTag+0x664
b4f52674 00000000
b4f52678 88f322e0
b4f5267c 88c9d708
b4f52680 00000001
b4f52684 898cf918
b4f52688 ffdff538
b4f5268c 804dc766 nt!KiUnlockDispatcherDatabase+0x1c
b4f52690 b4f52901
b4f52694 b4f526ac
b4f52698 00000001
b4f5269c 804eaf06 nt!IopFreeIrp+0xed
b4f526a0 00000000
b4f526a4 00000000
b4f526a8 88c9d708
b4f526ac b4f52700
b4f526b0 804f2b9f nt!IopCompleteRequest+0x319
b4f526b4 804f2bb5 nt!IopCompleteRequest+0x32f
b4f526b8 88c9d748
b4f526bc 89025978
b4f526c0 890259ac
b4f526c4 897752e8
b4f526c8 89025978
b4f526cc b4f52910
b4f526d0 b4f527c8
b4f526d4 00000000
b4f526d8 b9e0d300 DisplayDriver_mini+0x69300
b4f526dc 88c9d708
b4f526e0 00000000
b4f526e4 00000086
b4f526e8 b4f526b8
b4f526ec b9f3ad28 DisplayDriver_mini+0x196d28
b4f526f0 ffffffff
b4f526f4 804e2ed8 nt!_except_handler3
b4f526f8 804f2bb8 nt!GUID_DOCK_INTERFACE+0x424
b4f526fc ffffffff
b4f52700 804f2bb5 nt!IopCompleteRequest+0x32f
b4f52704 804f2db5 nt!KiDeliverApc+0xb3
b4f52708 88c9d748
b4f5270c b4f5274c
b4f52710 b4f52728
b4f52714 890259ac
b4f52718 804dce74 nt!KiDeliverApc+0x1e0
b4f5271c 806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52720 89025978
b4f52724 b4f527f8
b4f52728 00000000
b4f5272c 89025a60
b4f52730 00000001
b4f52734 b4f52d64
b4f52738 88e775c8
b4f5273c 804f2a72 nt!IopCompleteRequest
b4f52740 00000000
b4f52744 00000000
b4f52748 00000000
b4f5274c 00000000
b4f52750 b4f52768
b4f52754 806ffef2 hal!HalpApcInterrupt+0xc6
b4f52758 00000000
b4f5275c 00000000
b4f52760 b4f52768
b4f52764 00000000
b4f52768 b4f527f8
b4f5276c 806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52770 badb0d00
b4f52774 00000000
b4f52778 00000000
b4f5277c 806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52780 00000008
b4f52784 00000246
b4f52788 804e5d2c nt!KeInsertQueueApc+0x6d
b4f5278c 88c9d748
...
...
...
b4f527c0 b4f52c10
b4f527c4 804e2ed8 nt!_except_handler3
b4f527c8 804faca0 nt!KiFindFirstSetLeft+0x120
b4f527cc ffffffff
b4f527d0 b4f52840
b4f527d4 804de403 nt!CommonDispatchException+0x4d
b4f527d8 b4f527ec
...
...
...
b4f527f4 00000000
b4f527f8 bfbf0ca6 DisplayDriver+0x21bca6
b4f527fc 00000002
...
...
...
b4f52828 b4f52840
b4f5282c 804e0944 nt!KiTrap0E+0xd0
b4f52830 00000000
b4f52834 03a3fb4c
b4f52838 00000000
b4f5283c 804de3b4 nt!Kei386EoiHelper+0x18a
b4f52840 e24079e0
b4f52844 bfbf0ca6 DisplayDriver+0x21bca6
b4f52848 badb0d00
...
...
...
b4f52884 00000000
b4f52888 bfdba6c7 DisplayDriver+0x3e56c7
b4f5288c b4f52c10
...
...
...
b4f528a4 00000000
b4f528a8 bfbf0ca6 DisplayDriver+0x21bca6
b4f528ac 00000008
...
...
...
b4f528d8 000000f3
b4f528dc bfb6269f DisplayDriver+0x18d69f
b4f528e0 9745d083
b4f528e4 00000001
b4f528e8 e9a18d4c
b4f528ec ffffffff
b4f528f0 bfb268e7 DisplayDriver+0x1518e7
b4f528f4 000000ab
...
...
...
b4f52960 0000027a
b4f52964 bfb2696c DisplayDriver+0x15196c
b4f52968 00000000
...
...
...
b4f5298c e2004308
b4f52990 bfab8ce4 DisplayDriver+0xe3ce4
b4f52994 000000ab
...
...
...
b4f52bd0 00000000
b4f52bd4 bf804779 win32k!GreReleaseFastMutex+0x14
b4f52bd8 b4f52be8
b4f52bdc bf8a04e3 win32k!dhpdevRetrieveNode+0x32
b4f52be0 89b20128
b4f52be4 b4f52c50
b4f52be8 b4f52c20
b4f52bec bf907d15 win32k!WatchdogDdBlt+0x38
b4f52bf0 b4f52c50
...
...
...
b4f52c10 b4f52d40
b4f52c14 bf9877ae win32k!_except_handler3
b4f52c18 bf995380 win32k!`string'+0x2b4
b4f52c1c 00000000
b4f52c20 b4f52d50
b4f52c24 bf9cdd78 dxg!DxDdBlt+0x374
b4f52c28 b4f52c50
b4f52c2c b4f52d64
b4f52c30 038dfaf4
b4f52c34 bf907ca3 win32k!NtGdiDdBlt
b4f52c38 00000001
...
...
...
b4f52c90 000000b0
b4f52c94 bf805b42 win32k!AllocateObject+0xaa
b4f52c98 00000001
b4f52c9c 00000006
b4f52ca0 b4f52cb0
b4f52ca4 32040ddf
b4f52ca8 bf805734 win32k!HANDLELOCK::vLockHandle+0x75
b4f52cac 00000ff4
b4f52cb0 00000000
b4f52cb4 bc40ddf0
b4f52cb8 b4f52cd0
b4f52cbc 00000001
b4f52cc0 804da3ee nt!ExAcquireResourceExclusiveLite+0x67
b4f52cc4 00000008
...
...
...
b4f52ce8 80004005
b4f52cec 804dc605 nt!ExReleaseResourceLite+0x8d
b4f52cf0 00000000
...
...
...
b4f52d08 b4f52d18
b4f52d0c bf8018bf win32k!GreReleaseSemaphore+0xa
b4f52d10 bf803d1e win32k!GreUnlockDisplay+0x24
b4f52d14 00000000
...
...
...
b4f52d40 ffffffff
b4f52d44 bf9d2132 dxg!_except_handler3
b4f52d48 bf9d2928 dxg!GUID_MiscellaneousCallbacks+0x42c
b4f52d4c ffffffff
b4f52d50 b4f52d64
b4f52d54 804dd99f nt!KiFastCallEntry+0xfc
b4f52d58 02400002
...
...
...
b4f52ddc 00000023
b4f52de0 804ec781 nt!KiThreadStartup+0x16
b4f52de4 f7849b85 NDIS!ndisWorkerThread
b4f52de8 88c9d4d0
b4f52dec 00000000
b4f52df0 0020027f
b4f52df4 011c0000
b4f52df8 bfdb97b7 DisplayDriver+0x3e47b7
b4f52dfc 00000008
...
...
...
b4f52e70 00000000
b4f52e74 f7800000 InCDPass+0x1000
b4f52e78 00004026
...
...
...
b4f52ff8 00000000
b4f52ffc 00000000
b4f53000 ????????
b4f53004 ????????
Some are coincidental like InCDPass and NDIS. Obviously DisplayDriver, DisplayDriver_mini, dxg and win32k are related due to their functions: Display, DirectX, GDI (Graphics Device Interface).
Now we can check their module information:
0: kd> lmv m DisplayDriver
start end module name
bf9d5000 bff42500 DisplayDriver T (no symbols)
Loaded symbol image file: DisplayDriver.dll
Image path: DisplayDriver.dll
Image name: DisplayDriver.dll
Timestamp: Fri Jun 29 09:13:08 2007 (4684BF14)
CheckSum: 00570500
ImageSize: 0056D500
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
0: kd> lmv m DisplayDriver_mini
start end module name
b9da4000 ba421f20 DisplayDriver_mini T (no symbols)
Loaded symbol image file: DisplayDriver_mini.sys
Image path: DisplayDriver_mini.sys
Image name: DisplayDriver_mini.sys
Timestamp: Fri Jun 29 09:16:41 2007 (4684BFE9)
CheckSum: 00680F20
ImageSize: 0067DF20
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
0: kd> lmv m dxg
start end module name
bf9c3000 bf9d4580 dxg (pdb symbols)
Loaded symbol image file: dxg.sys
Mapped memory image file: c:\websymbols\dxg.sys\41107B9311580\dxg.sys
Image path: dxg.sys
Image name: dxg.sys
Timestamp: Wed Aug 04 07:00:51 2004 (41107B93)
CheckSum: 0001D181
ImageSize: 00011580
File version: 5.1.2600.2180
Product version: 5.1.2600.2180
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.7 Driver
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: dxg.sys
OriginalFilename: dxg.sys
ProductVersion: 5.1.2600.2180
FileVersion: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)
FileDescription: DirectX Graphics Driver
LegalCopyright: © Microsoft Corporation. All rights reserved.
0: kd> lmv m win32k
start end module name
bf800000 bf9c2180 win32k # (pdb symbols)
Loaded symbol image file: win32k.sys
Mapped memory image file: c:\websymbols\win32k.sys\45F013F61c2180\win32k.sys
Image path: win32k.sys
Image name: win32k.sys
Timestamp: Thu Mar 08 13:47:34 2007 (45F013F6)
CheckSum: 001C4886
ImageSize: 001C2180
File version: 5.1.2600.3099
Product version: 5.1.2600.3099
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.7 Driver
File date: 00000000.00000000
Translations: 0406.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operativsystem
InternalName: win32k.sys
OriginalFilename: win32k.sys
ProductVersion: 5.1.2600.3099
FileVersion: 5.1.2600.3099 (xpsp_sp2_gdr.070308-0222)
FileDescription: Win32-flerbrugerdriver
LegalCopyright: © Microsoft Corporation. Alle rettigheder forbeholdes.
- Dmitry Vostokov @ DumpAnalysis.org -
October 11th, 2007 at 9:03 am
Part 4:
http://www.dumpanalysis.org/blog/index.php/2007/10/11/minidump-analysis-part-4/
March 24th, 2010 at 7:04 pm
[…] Minidump Analysis (Part 3) […]