Crash Dump Analysis Patterns (Part 165)

Sometimes debugging information is absent from module info in memory dumps and a debugger can’t recognize and automatically load symbol files. For example, we see this stack trace without loaded component symbols:

THREAD 8a17c6d8  Cid 02ec.02f0  Teb: 7ffdf000 Win32Thread: e17b4420 WAIT: (UserRequest) UserMode Non-Alertable
89873d00  SynchronizationEvent
IRP List:
89d9fd20: (0006,0094) Flags: 00000800  Mdl: 00000000
Not impersonating
DeviceMap                 e10086c8
Owning Process            0       Image:         <Unknown>
Attached Process          8a17cda0       Image:         ApplicationA.exe
Wait Start TickCount      8164394        Ticks: 2884 (0:00:00:45.062)
Context Switch Count      1769160                 LargeStack
UserTime                  00:00:55.250
KernelTime                00:01:56.109
Start Address 0×0103e5e1
Stack Init ba390000 Current ba38fca0 Base ba390000 Limit ba38b000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
*** ERROR: Module load completed but symbols could not be loaded for ModuleA.dll
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0×2f
ba38fcc4 804fb068 nt!KiSwapThread+0×8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0×1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0×9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc (TrapFrame @ ba38fd64)
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0×132
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0×46
WARNING: Stack unwind information not available. Following frames may be wrong.
0006f7a4 0132b785 ModuleA+0×53df7
0006f7cc 0132c728 ModuleA+0xb785
0006f7e4 01346426 ModuleA+0xc728
0006f848 7e418734 ModuleA+0×26426

0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf

0: kd> .process /r /p 8a17cda0
Implicit process is now 8a17cda0
Loading User Symbols

0: kd> lmv m ModuleA
start    end        module name
01320000 013bb000   ModuleA   (deferred)
Image path: C:\Program Files\VendorA\ModuleA.dll
Image name: ModuleA.dll
Timestamp:        Thu Aug 11 21:42:08 2011 (4E4484F0)
CheckSum:         000A9C8B
ImageSize:        0009B000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

0: kd> !lmi ModuleA
Loaded Module Info: [ModuleA]
Module: ModuleA
Base Address: 01320000
Image Name: ModuleA.dll
Machine Type: 332 (I386)
Time Stamp: 4e4484f0 Thu Aug 11 21:42:08 2011
Size: 9b000
CheckSum: a9c8b
Characteristics: 2102
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    5e, 830a0,   830a0 [Debug data not mapped] - can’t validate symbols, if present.
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

However, in a stack trace collection (!process 0 ff WinDbg command) we find another stack trace from a different process but with loaded symbol files for ModuleA:

THREAD 89703020  Cid 1068.1430  Teb: 7ffdf000 Win32Thread: e34d43a8 WAIT: (UserRequest) UserMode Non-Alertable
89a3ac58  NotificationEvent
89703110  NotificationTimer
IRP List:
899ab488: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e10086c8
Owning Process            0       Image:         <Unknown>
Attached Process          89825020       Image:         ApplicationB.exe
Wait Start TickCount      8164457        Ticks: 2821 (0:00:00:44.078)
Context Switch Count      552                 LargeStack
UserTime                  00:00:00.296
KernelTime                00:00:00.890
Start Address 0×0103e5e1
Stack Init b8796000 Current b8795ca0 Base b8796000 Limit b8791000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr
b8795cb8 80503836 nt!KiSwapContext+0×2f
b8795cc4 804fb068 nt!KiSwapThread+0×8a
b8795cec 805c0750 nt!KeWaitForSingleObject+0×1c2
b8795d50 8054161c nt!NtWaitForSingleObject+0×9a
b8795d50 7c90e4f4 nt!KiFastCallEntry+0xfc (TrapFrame @ b8795d64)
0006fa1c 7c90df3c ntdll!KiFastSystemCallRet
0006fa20 7c8025db ntdll!NtWaitForSingleObject+0xc
0006fa84 010ae96a kernel32!WaitForSingleObjectEx+0xa8
0006fafc 010aeaaf ModuleA!Wait+0xaa
0006fb38 010b84ce ModuleA!Read+0×6f

[…]

0: kd> !lmi ModuleA
Loaded Module Info: [ModuleA]
Module: ModuleA
Base Address: 01090000
Image Name: ModuleA.dll
Machine Type: 332 (I386)
Time Stamp: 4e4484f0 Thu Aug 11 21:42:08 2011
Size: 9b000
CheckSum: a9c8b
Characteristics: 2102
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    5e, 830a0,   830a0 RSDS - GUID: {C14E734A-367F-4DD0-974D-FA47C1194F28}
Age: 1, Pdb: Y:\src\…\ModuleA.pdb
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

0: kd> lmv m ModuleA
start    end        module name
01090000 0112b000   ModuleA   (deferred)
Image path: C:\Program Files\VendorA\ModuleA.dll
Image name: ModuleA.dll
Timestamp:        Thu Aug 11 21:42:08 2011 (4E4484F0)
CheckSum:         000A9C8B
ImageSize:        0009B000
File version:     1.3.0.0
Product version:  1.3.0.0
File flags:       8 (Mask 3F) Private
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0409.04b0
CompanyName:      VendorA
ProductName:      VendorA
InternalName:     ModuleA.dll
OriginalFilename: ModuleA.dll
ProductVersion:   1.3
FileVersion:      1.3.0.0
FileDescription:  ModuleA GUI
LegalCopyright:   Copyright VendorA

So we switch to that thread (with the new process context) to get the needed symbol path:

0: kd> .thread /r /p 89703020
Implicit thread is now 89703020
Implicit process is now 89825020
Loading User Symbols

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
b8795cb8 80503836 nt!KiSwapContext+0x2f
b8795cc4 804fb068 nt!KiSwapThread+0x8a
b8795cec 805c0750 nt!KeWaitForSingleObject+0x1c2
b8795d50 8054161c nt!NtWaitForSingleObject+0x9a
b8795d50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006fa1c 7c90df3c ntdll!KiFastSystemCallRet
0006fa20 7c8025db ntdll!NtWaitForSingleObject+0xc
0006fa84 010ae96a kernel32!WaitForSingleObjectEx+0xa8
0006fafc 010aeaaf ModuleA!Wait+0xaa
0006fb38 010b84ce ModuleA!Read+0×6f

[…]

0: kd> lmv m ModuleA
start    end        module name
01090000 0112b000   ModuleA   (private pdb symbols)  c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281\ModuleA.pdb
Loaded symbol image file: ModuleA.dll
[…]

Now we switch back to our problem stack trace and set the found symbol path explicitly:

0: kd> .thread /r /p 8a17c6d8
Implicit thread is now 8a17c6d8
Implicit process is now 8a17cda0
Loading User Symbols

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0x2f
ba38fcc4 804fb068 nt!KiSwapThread+0x8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0x1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0x9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
*** ERROR: Module load completed but symbols could not be loaded for ModuleA.dll
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0x46
WARNING: Stack unwind information not available. Following frames may be wrong.
0006f7a4 0132b785 ModuleA+0×53df7
0006f7cc 0132c728 ModuleA+0xb785
0006f7e4 01346426 ModuleA+0xc728
0006f848 7e418734 ModuleA+0×26426

0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf
[…]

0: kd> .sympath+ c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281
Symbol search path is: SRV*c:\mss*http://msdl.microsoft.com/download/symbols; c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281
[…]

0: kd> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0x2f
ba38fcc4 804fb068 nt!KiSwapThread+0x8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0x1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0x9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0x46
0006f6e4 0132b22e ModuleA!CSLock+0×7
0006f7a4 0132b785 ModuleA!SignalEvent+0×5e
[…]
0006f848 7e418734 ModuleA!WindowProc+0×136

0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply