Crash Dump Analysis Patterns (Part 12)
Friday, April 20th, 2007Another pattern that happens so often in crash dumps: No Component Symbols. In this case we can guess what a component does by looking at its name, overall thread stack where it is called and also its import table. Here is an example. We have component.sys driver visible on some thread stack in a kernel dump but we don’t know what that component can potentially do. Because we don’t have symbols we cannot see its imported functions:
kd> x component!*
kd>
We use !dh command to dump its image headers:
kd> lmv m component
start end module name
fffffadf`e0eb5000 fffffadf`e0ebc000 component (no symbols)
Loaded symbol image file: component.sys
Image path: \??\C:\Component\x64\component.sys
Image name: component.sys
Timestamp: Sat Jul 01 19:06:16 2006 (44A6B998)
CheckSum: 000074EF
ImageSize: 00007000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
kd> !dh fffffadf`e0eb5000
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
8664 machine (X64)
6 number of sections
44A6B998 time date stamp Sat Jul 01 19:06:16 2006
0 file pointer to symbol table
0 number of symbols
F0 size of optional header
22 characteristics
Executable
App can handle >2gb addresses
OPTIONAL HEADER VALUES
20B magic #
8.00 linker version
C00 size of code
A00 size of initialized data
0 size of uninitialized data
5100 address of entry point
1000 base of code
----- new -----
0000000000010000 image base
1000 section alignment
200 file alignment
1 subsystem (Native)
5.02 operating system version
5.02 image version
5.02 subsystem version
7000 size of image
400 size of headers
74EF checksum
0000000000040000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
0 [ 0] address [size] of Export Directory
51B0 [ 28] address [size] of Import Directory
6000 [ 3B8] address [size] of Resource Directory
4000 [ 6C] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
0 [ 0] address [size] of Base Relocation Directory
2090 [ 1C] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
0 [ 0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
2000 [ 88] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
…
…
…
Then we display the contents of Import Address Table Directory using dps command:
kd> dps fffffadf`e0eb5000+2000 fffffadf`e0eb5000+2000+88
fffffadf`e0eb7000 fffff800`01044370 nt!IoCompleteRequest
fffffadf`e0eb7008 fffff800`01019700 nt!IoDeleteDevice
fffffadf`e0eb7010 fffff800`012551a0 nt!IoDeleteSymbolicLink
fffffadf`e0eb7018 fffff800`01056a90 nt!MiResolveTransitionFault+0x7c2
fffffadf`e0eb7020 fffff800`0103a380 nt!ObDereferenceObject
fffffadf`e0eb7028 fffff800`0103ace0 nt!KeWaitForSingleObject
fffffadf`e0eb7030 fffff800`0103c570 nt!KeSetTimer
fffffadf`e0eb7038 fffff800`0102d070 nt!IoBuildPartialMdl+0x3
fffffadf`e0eb7040 fffff800`012d4480 nt!PsTerminateSystemThread
fffffadf`e0eb7048 fffff800`01041690 nt!KeBugCheckEx
fffffadf`e0eb7050 fffff800`010381b0 nt!KeInitializeTimer
fffffadf`e0eb7058 fffff800`0103ceb0 nt!ZwClose
fffffadf`e0eb7060 fffff800`012b39f0 nt!ObReferenceObjectByHandle
fffffadf`e0eb7068 fffff800`012b7380 nt!PsCreateSystemThread
fffffadf`e0eb7070 fffff800`01251f90 nt!FsRtlpIsDfsEnabled+0x114
fffffadf`e0eb7078 fffff800`01275160 nt!IoCreateDevice
fffffadf`e0eb7080 00000000`00000000
fffffadf`e0eb7088 00000000`00000000
We see that this driver under certain circumstances could bugcheck the system using KeBugCheckEx, it creates system thread(s) (PsCreateSystemThread) and uses timer(s) (KeInitializeTimer, KeSetTimer).
If you see name+offset in import table (I think this is an effect of OMAP code optimization) you can get the function by using ln command (list nearest symbols):
kd> ln fffff800`01056a90
(fffff800`01056760) nt!MiResolveTransitionFault+0x7c2 | (fffff800`01056a92) nt!RtlInitUnicodeString
kd> ln fffff800`01251f90
(fffff800`01251e90) nt!FsRtlpIsDfsEnabled+0×114 | (fffff800`01251f92) nt!IoCreateSymbolicLink
This technique is useful if you have a bugcheck that happens when a driver calls certain functions or must call certain function in pairs, like bugcheck 0×20:
kd> !analyze -show 0x20
KERNEL_APC_PENDING_DURING_EXIT (20)
The key data item is the thread's APC disable count. If this is non-zero, then this is the source of the problem. The APC disable count is decremented each time a driver calls KeEnterCriticalRegion, KeInitializeMutex, or FsRtlEnterFileSystem. The APC disable count is incremented each time a driver calls KeLeaveCriticalRegion, KeReleaseMutex, or FsRtlExitFileSystem. Since these calls should always be in pairs, this value should be zero when a thread exits. A negative value indicates that a driver has disabled APC calls without re-enabling them. A positive value indicates that the reverse is true. If you ever see this error, be very suspicious of all drivers installed on the machine — especially unusual or non-standard drivers. Third party file system redirectors are especially suspicious since they do not generally receive the heavy duty testing that NTFS, FAT, RDR, etc receive. This current IRQL should also be 0. If it is not, that a driver’s cancelation routine can cause this bugcheck by returning at an elevated IRQL. Always attempt to note what you were doing/closing at the time of the crash, and note all of the installed drivers at the time of the crash. This symptom is usually a severe bug in a third party driver.
Then you can see at least whether the suspicious driver could have potentially used those functions and if it imports one of them you can see whether it imports the corresponding counterpart function.
- Dmitry Vostokov @ DumpAnalysis.org -
It is so ubiquitous. And its manifestations are random and usually crashes happen far away from the original corruption point. In your user mode and space part of exception threads (don’t forget about 
