Software Diagnostics Library

Part 2 dealt with stack traces. Unfortunately stack traces reported by WinDbg, especially involving 3rd-party components, are usually incomplete and sometimes not even correct. They can also point to stable drivers when the system failure happened after slowly accumulated corruption caused by some intermediate driver or a combination of drivers.

Sometimes there are other 3rd-party drivers involved before the system crash that are not visible in the output of !analyze -v command and simply removing them, disabling or upgrading software they are part from make the system stable. To see them we can look at the so called raw stack data. Because kernel mode thread stack size is small (12Kb or 0×3000) we can simply dump memory range between ESP-3000 and ESP+3000. We can use RSP register for x64 dumps but the output will be the same.

Let’s look at our minidump from the previous part. The stack trace is small, incomplete and points to DisplayDriver. This is because we don’t have symbol information for DisplayDriver.dll. Could it be the case that DisplayDriver.dll was used incorrectly by another driver or operating system component? What are other components that might have been used prior to BSOD? Raw stack dump shows additional symbols like DisplayDriver_mini, win32k and dxg:

0: kd> dps esp-3000 esp+3000
b4f4f8b4  ????????
b4f4f8b8  ????????
b4f4f8bc  ????????
b4f4f8c0  ????????
...
...
...
b4f51ffc  ????????
b4f52000  00001000
b4f52004  00006000
b4f52008  b4f5204c
b4f5200c  89025978
b4f52010  89139000
b4f52014  00000000
b4f52018  b4f527ec
b4f5201c  b4f52840
b4f52020  bfbf0ca6 DisplayDriver+0x21bca6
b4f52024  00000000
b4f52028  89025978
...
...
...
b4f52100  e24079e0
b4f52104  bfbf0ca6 DisplayDriver+0x21bca6
b4f52108  00000008
...
...
...
b4f52364  b4f52414
b4f52368  804dc0b2 nt!ExecuteHandler+0x24
b4f5236c  b4f527ec
b4f52370  b4f52d40
b4f52374  b4f524e8
b4f52378  b4f52400
b4f5237c  bf9d2132 dxg!_except_handler3
b4f52380  2a2a2a0a
...
...
...
b4f523e8  b4f52408
b4f523ec  8053738a nt!KeBugCheckEx+0x1b
b4f523f0  0000008e
b4f523f4  c0000005
b4f523f8  bfbf0ca6 DisplayDriver+0x21bca6
b4f523fc  b4f52840
b4f52400  00000000
b4f52404  00000000
b4f52408  b4f527d0
b4f5240c  80521fed nt!KiDispatchException+0x3b1
b4f52410  0000008e
b4f52414  c0000005
b4f52418  bfbf0ca6 DisplayDriver+0x21bca6
b4f5241c  b4f52840
b4f52420  00000000
b4f52424  03a3fb4c
b4f52428  03a3fb4c
b4f5242c  b4f52800
b4f52430  00000000
b4f52434  00000000
b4f52438  00000000
b4f5243c  b9deffc6 DisplayDriver_mini+0x4bfc6
b4f52440  897c621c
b4f52444  00000086
b4f52448  0000003c
b4f5244c  b9f3af5a DisplayDriver_mini+0x196f5a
b4f52450  897c6200
b4f52454  00000086
b4f52458  897c6200
b4f5245c  00000000
b4f52460  00000000
b4f52464  00000000
b4f52468  b9f38b4e DisplayDriver_mini+0x194b4e
b4f5246c  00000000
...
...
...
b4f5250c  00002800
b4f52510  b9f3ac10 DisplayDriver_mini+0x196c10
b4f52514  897c6200
b4f52518  00002504
b4f5251c  00000010
b4f52520  897c6200
b4f52524  b9f2d194 DisplayDriver_mini+0x189194
b4f52528  897c6200
b4f5252c  00002504
b4f52530  00000010
b4f52534  897c6200
b4f52538  898cca80
b4f5253c  00000080
b4f52540  89654008
b4f52544  b9f358e2 DisplayDriver_mini+0x1918e2
b4f52548  897c6200
...
...
...
b4f5256c  00000000
b4f52570  b9deff5c DisplayDriver_mini+0x4bf5c
b4f52574  00000000
...
...
...
b4f5259c  e24079e0
b4f525a0  bfbf0ca6 DisplayDriver+0x21bca6
b4f525a4  00000008
b4f525a8  00010246
b4f525ac  b4f528b4
b4f525b0  00000010
b4f525b4  0000003c
b4f525b8  b9f3af5a DisplayDriver_mini+0x196f5a
b4f525bc  897c6200
b4f525c0  00000086
b4f525c4  89b81008
b4f525c8  897c6200
b4f525cc  00000000
b4f525d0  00007c00
b4f525d4  b9deff5c DisplayDriver_mini+0x4bf5c
b4f525d8  b9deff5c DisplayDriver_mini+0x4bf5c
b4f525dc  8988d7d8
b4f525e0  b9deff66 DisplayDriver_mini+0x4bf66
b4f525e4  b9deff5c DisplayDriver_mini+0x4bf5c
b4f525e8  8961c288
b4f525ec  b9deff66 DisplayDriver_mini+0x4bf66
b4f525f0  8961c288
b4f525f4  00000000
b4f525f8  00000046
b4f525fc  00000000
b4f52600  89903000
b4f52604  b9e625a9 DisplayDriver_mini+0xbe5a9
b4f52608  8961c288
b4f5260c  00000046
b4f52610  00000000
b4f52614  b9deff5c DisplayDriver_mini+0x4bf5c
b4f52618  896ac008
...
...
...
b4f52630  898a8000
b4f52634  b9e9f220 DisplayDriver_mini+0xfb220
b4f52638  89941400
b4f5263c  b9e2ffec DisplayDriver_mini+0x8bfec
b4f52640  00000000
b4f52644  00000000
b4f52648  00000050
b4f5264c  b9e790d3 DisplayDriver_mini+0xd50d3
b4f52650  897c6200
...
...
...
b4f5266c  89bf6200
b4f52670  805502fa nt!ExFreePoolWithTag+0x664
b4f52674  00000000
b4f52678  88f322e0
b4f5267c  88c9d708
b4f52680  00000001
b4f52684  898cf918
b4f52688  ffdff538
b4f5268c  804dc766 nt!KiUnlockDispatcherDatabase+0x1c
b4f52690  b4f52901
b4f52694  b4f526ac
b4f52698  00000001
b4f5269c  804eaf06 nt!IopFreeIrp+0xed
b4f526a0  00000000
b4f526a4  00000000
b4f526a8  88c9d708
b4f526ac  b4f52700
b4f526b0  804f2b9f nt!IopCompleteRequest+0x319
b4f526b4  804f2bb5 nt!IopCompleteRequest+0x32f
b4f526b8  88c9d748
b4f526bc  89025978
b4f526c0  890259ac
b4f526c4  897752e8
b4f526c8  89025978
b4f526cc  b4f52910
b4f526d0  b4f527c8
b4f526d4  00000000
b4f526d8  b9e0d300 DisplayDriver_mini+0x69300
b4f526dc  88c9d708
b4f526e0  00000000
b4f526e4  00000086
b4f526e8  b4f526b8
b4f526ec  b9f3ad28 DisplayDriver_mini+0x196d28
b4f526f0  ffffffff
b4f526f4  804e2ed8 nt!_except_handler3
b4f526f8  804f2bb8 nt!GUID_DOCK_INTERFACE+0x424
b4f526fc  ffffffff
b4f52700  804f2bb5 nt!IopCompleteRequest+0x32f
b4f52704  804f2db5 nt!KiDeliverApc+0xb3
b4f52708  88c9d748
b4f5270c  b4f5274c
b4f52710  b4f52728
b4f52714  890259ac
b4f52718  804dce74 nt!KiDeliverApc+0x1e0
b4f5271c  806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52720  89025978
b4f52724  b4f527f8
b4f52728  00000000
b4f5272c  89025a60
b4f52730  00000001
b4f52734  b4f52d64
b4f52738  88e775c8
b4f5273c  804f2a72 nt!IopCompleteRequest
b4f52740  00000000
b4f52744  00000000
b4f52748  00000000
b4f5274c  00000000
b4f52750  b4f52768
b4f52754  806ffef2 hal!HalpApcInterrupt+0xc6
b4f52758  00000000
b4f5275c  00000000
b4f52760  b4f52768
b4f52764  00000000
b4f52768  b4f527f8
b4f5276c  806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52770  badb0d00
b4f52774  00000000
b4f52778  00000000
b4f5277c  806ffae4 hal!KeReleaseQueuedSpinLock+0x3c
b4f52780  00000008
b4f52784  00000246
b4f52788  804e5d2c nt!KeInsertQueueApc+0x6d
b4f5278c  88c9d748
...
...
...
b4f527c0  b4f52c10
b4f527c4  804e2ed8 nt!_except_handler3
b4f527c8  804faca0 nt!KiFindFirstSetLeft+0x120
b4f527cc  ffffffff
b4f527d0  b4f52840
b4f527d4  804de403 nt!CommonDispatchException+0x4d
b4f527d8  b4f527ec
...
...
...
b4f527f4  00000000
b4f527f8  bfbf0ca6 DisplayDriver+0x21bca6
b4f527fc  00000002
...
...
...
b4f52828  b4f52840
b4f5282c  804e0944 nt!KiTrap0E+0xd0
b4f52830  00000000
b4f52834  03a3fb4c
b4f52838  00000000
b4f5283c  804de3b4 nt!Kei386EoiHelper+0x18a
b4f52840  e24079e0
b4f52844  bfbf0ca6 DisplayDriver+0x21bca6
b4f52848  badb0d00
...
...
...
b4f52884  00000000
b4f52888  bfdba6c7 DisplayDriver+0x3e56c7
b4f5288c  b4f52c10
...
...
...
b4f528a4  00000000
b4f528a8  bfbf0ca6 DisplayDriver+0x21bca6
b4f528ac  00000008
...
...
...
b4f528d8  000000f3
b4f528dc  bfb6269f DisplayDriver+0x18d69f
b4f528e0  9745d083
b4f528e4  00000001
b4f528e8  e9a18d4c
b4f528ec  ffffffff
b4f528f0  bfb268e7 DisplayDriver+0x1518e7
b4f528f4  000000ab
...
...
...
b4f52960  0000027a
b4f52964  bfb2696c DisplayDriver+0x15196c
b4f52968  00000000
...
...
...
b4f5298c  e2004308
b4f52990  bfab8ce4 DisplayDriver+0xe3ce4
b4f52994  000000ab
...
...
...
b4f52bd0  00000000
b4f52bd4  bf804779 win32k!GreReleaseFastMutex+0x14
b4f52bd8  b4f52be8
b4f52bdc  bf8a04e3 win32k!dhpdevRetrieveNode+0x32
b4f52be0  89b20128
b4f52be4  b4f52c50
b4f52be8  b4f52c20
b4f52bec  bf907d15 win32k!WatchdogDdBlt+0x38
b4f52bf0  b4f52c50
...
...
...
b4f52c10  b4f52d40
b4f52c14  bf9877ae win32k!_except_handler3
b4f52c18  bf995380 win32k!`string'+0x2b4
b4f52c1c  00000000
b4f52c20  b4f52d50
b4f52c24  bf9cdd78 dxg!DxDdBlt+0x374
b4f52c28  b4f52c50
b4f52c2c  b4f52d64
b4f52c30  038dfaf4
b4f52c34  bf907ca3 win32k!NtGdiDdBlt
b4f52c38  00000001
...
...
...
b4f52c90  000000b0
b4f52c94  bf805b42 win32k!AllocateObject+0xaa
b4f52c98  00000001
b4f52c9c  00000006
b4f52ca0  b4f52cb0
b4f52ca4  32040ddf
b4f52ca8  bf805734 win32k!HANDLELOCK::vLockHandle+0x75
b4f52cac  00000ff4
b4f52cb0  00000000
b4f52cb4  bc40ddf0
b4f52cb8  b4f52cd0
b4f52cbc  00000001
b4f52cc0  804da3ee nt!ExAcquireResourceExclusiveLite+0x67
b4f52cc4  00000008
...
...
...
b4f52ce8  80004005
b4f52cec  804dc605 nt!ExReleaseResourceLite+0x8d
b4f52cf0  00000000
...
...
...
b4f52d08  b4f52d18
b4f52d0c  bf8018bf win32k!GreReleaseSemaphore+0xa
b4f52d10  bf803d1e win32k!GreUnlockDisplay+0x24
b4f52d14  00000000
...
...
...
b4f52d40  ffffffff
b4f52d44  bf9d2132 dxg!_except_handler3
b4f52d48  bf9d2928 dxg!GUID_MiscellaneousCallbacks+0x42c
b4f52d4c  ffffffff
b4f52d50  b4f52d64
b4f52d54  804dd99f nt!KiFastCallEntry+0xfc
b4f52d58  02400002
...
...
...
b4f52ddc  00000023
b4f52de0  804ec781 nt!KiThreadStartup+0x16
b4f52de4  f7849b85 NDIS!ndisWorkerThread
b4f52de8  88c9d4d0
b4f52dec  00000000
b4f52df0  0020027f
b4f52df4  011c0000
b4f52df8  bfdb97b7 DisplayDriver+0x3e47b7
b4f52dfc  00000008
...
...
...
b4f52e70  00000000
b4f52e74  f7800000 InCDPass+0x1000
b4f52e78  00004026
...
...
...
b4f52ff8  00000000
b4f52ffc  00000000
b4f53000  ????????
b4f53004  ????????

Some are coincidental like InCDPass and NDIS. Obviously DisplayDriver, DisplayDriver_mini, dxg and win32k are related due to their functions: Display, DirectX, GDI (Graphics Device Interface). 

Now we can check their module information:

0: kd> lmv m DisplayDriver
start    end        module name
bf9d5000 bff42500   DisplayDriver T (no symbols)
    Loaded symbol image file: DisplayDriver.dll
    Image path: DisplayDriver.dll
    Image name: DisplayDriver.dll
    Timestamp:        Fri Jun 29 09:13:08 2007 (4684BF14)
    CheckSum:         00570500
    ImageSize:        0056D500
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

0: kd> lmv m DisplayDriver_mini
start    end        module name
b9da4000 ba421f20   DisplayDriver_mini T (no symbols)
    Loaded symbol image file: DisplayDriver_mini.sys
    Image path: DisplayDriver_mini.sys
    Image name: DisplayDriver_mini.sys
    Timestamp:        Fri Jun 29 09:16:41 2007 (4684BFE9)
    CheckSum:         00680F20
    ImageSize:        0067DF20
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

0: kd> lmv m dxg
start    end        module name
bf9c3000 bf9d4580   dxg        (pdb symbols)
    Loaded symbol image file: dxg.sys
    Mapped memory image file: c:\websymbols\dxg.sys\41107B9311580\dxg.sys
    Image path: dxg.sys
    Image name: dxg.sys
    Timestamp:        Wed Aug 04 07:00:51 2004 (41107B93)
    CheckSum:         0001D181
    ImageSize:        00011580
    File version:     5.1.2600.2180
    Product version:  5.1.2600.2180
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        3.7 Driver
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     dxg.sys
    OriginalFilename: dxg.sys
    ProductVersion:   5.1.2600.2180
    FileVersion:      5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)
    FileDescription:  DirectX Graphics Driver
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

0: kd> lmv m win32k
start    end        module name
bf800000 bf9c2180   win32k   # (pdb symbols)
    Loaded symbol image file: win32k.sys
    Mapped memory image file: c:\websymbols\win32k.sys\45F013F61c2180\win32k.sys
    Image path: win32k.sys
    Image name: win32k.sys
    Timestamp:        Thu Mar 08 13:47:34 2007 (45F013F6)
    CheckSum:         001C4886
    ImageSize:        001C2180
    File version:     5.1.2600.3099
    Product version:  5.1.2600.3099
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        3.7 Driver
    File date:        00000000.00000000
    Translations:     0406.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operativsystem
    InternalName:     win32k.sys
    OriginalFilename: win32k.sys
    ProductVersion:   5.1.2600.3099
    FileVersion:      5.1.2600.3099 (xpsp_sp2_gdr.070308-0222)
    FileDescription:  Win32-flerbrugerdriver
    LegalCopyright:   © Microsoft Corporation. Alle rettigheder forbeholdes.

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Wednesday, September 12th, 2007 at 3:30 pm and is filed under Crash Dump Analysis, Minidump Analysis, WinDbg Tips and Tricks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.