Crash Dump Analysis Patterns (Part 99a)

Most of the time Incorrect Symbolic Information is associated with function names and offsets, for example, module!foo vs. module!foo+100. In some cases the module name is incorrect itself or absent altogether. This can happen in complete memory dumps when we forget to reload user space symbols after changing the process context, for example:

; previous process context of firefox.exe
; switching to winlogon.exe context

kd> .process fffffadfe718c040
Implicit process is now fffffadf`e718c040

kd> !process fffffadfe718c040
PROCESS fffffadfe718c040
    SessionId: 0  Cid: 017c    Peb: 7fffffd9000  ParentCid: 0130
    DirBase: 01916000  ObjectTable: fffffa800099a890  HandleCount: 754.
    Image: winlogon.exe
    VadRoot fffffadfe75e91f0 Vads 190 Clone 0 Private 2905. Modified 10047. Locked 0.
    DeviceMap fffffa8000004950
    Token                             fffffa800122a060
    ElapsedTime                       77 Days 02:14:26.109
    UserTime                          00:00:04.156
    KernelTime                        00:00:02.359
    QuotaPoolUsage[PagedPool]         143128
    QuotaPoolUsage[NonPagedPool]      191072
    Working Set Sizes (now,min,max)  (541, 50, 345) (2164KB, 200KB, 1380KB)
    PeakWorkingSetSize                6323
    VirtualSize                       108 Mb
    PeakVirtualSize                   118 Mb
    PageFaultCount                    212547
    MemoryPriority                    BACKGROUND
    BasePriority                      13
    CommitCharge                      3733

[...]

THREAD fffffadfe68f2040  Cid 017c.0198  Teb: 000007fffffd7000 Win32Thread: fffff97ff4a09010 WAIT: (Unknown) UserMode Non-Alertable
    fffffadfe7133160  Semaphore Limit 0x7fffffff
    fffffadfe68f20f8  NotificationTimer
Not impersonating
DeviceMap                 fffffa8000004950
Owning Process            fffffadfe718c040       Image:         winlogon.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      426298731      Ticks: 51 (0:00:00:00.796)
Context Switch Count      2215076                 LargeStack
UserTime                  00:00:00.187
KernelTime                00:00:00.468
Start Address 0×0000000077d6b6e0
Stack Init fffffadfe4481e00 Current fffffadfe4481860
Base fffffadfe4482000 Limit fffffadfe447a000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Child-SP          RetAddr           Call Site
fffffadf`e44818a0 fffff800`0103b093 nt!KiSwapContext+0×85
fffffadf`e4481a20 fffff800`0103c433 nt!KiSwapThread+0xc3
fffffadf`e4481a60 fffff800`012d25ae nt!KeWaitForSingleObject+0×528
fffffadf`e4481af0 fffff800`0104113d nt!NtReplyWaitReceivePortEx+0×8c8
fffffadf`e4481c00 00000000`77ef0caa nt!KiSystemServiceCopyEnd+0×3 (TrapFrame @ fffffadf`e4481c70)
00000000`00bcfb98 000007ff`7fd6ff61 ntdll!NtReplyWaitReceivePortEx+0xa
00000000`00bcfba0 00000000`000d2340 0×7ff`7fd6ff61
00000000`00bcfba8 00000000`00bcfde0 0xd2340
00000000`00bcfbb0 00000000`014cd220 0xbcfde0
00000000`00bcfbb8 00000000`000c1d30 0×14cd220
00000000`00bcfbc0 00000000`00bcfe18 0xc1d30
00000000`00bcfbc8 0000ffff`00001f80 0xbcfe18
00000000`00bcfbd0 00000000`006c0044 0xffff`00001f80
00000000`00bcfbd8 00000000`000006ec firefox+0×2c0044
00000000`00bcfbe0 00000000`000007b0 0×6ec
00000000`00bcfbe8 00000000`419b8385 0×7b0
00000000`00bcfbf0 00000000`00000000 0×419b8385

kd> lmu m firefox
start             end                 module name
00000000`00400000 00000000`00b67000   firefox  T (no symbols)

We have the return address 00000000`006c0044 which is just firefox+0×2c0044 (00000000`00400000 + 2c0044). We correct that by reloading user space symbols.

kd> .reload /user

kd> !process fffffadfe718c040
[...]
THREAD fffffadfe68f2040  Cid 017c.0198  Teb: 000007fffffd7000 Win32Thread: fffff97ff4a09010 WAIT: (Unknown) UserMode Non-Alertable
    fffffadfe7133160  Semaphore Limit 0x7fffffff
    fffffadfe68f20f8  NotificationTimer
Not impersonating
DeviceMap                 fffffa8000004950
Owning Process            fffffadfe718c040       Image:         winlogon.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      426298731      Ticks: 51 (0:00:00:00.796)
Context Switch Count      2215076                 LargeStack
UserTime                  00:00:00.187
KernelTime                00:00:00.468
Start Address kernel32!BaseThreadStart (0x0000000077d6b6e0)
Stack Init fffffadfe4481e00 Current fffffadfe4481860
Base fffffadfe4482000 Limit fffffadfe447a000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Child-SP          RetAddr           Call Site
fffffadf`e44818a0 fffff800`0103b093 nt!KiSwapContext+0x85
fffffadf`e4481a20 fffff800`0103c433 nt!KiSwapThread+0xc3
fffffadf`e4481a60 fffff800`012d25ae nt!KeWaitForSingleObject+0x528
fffffadf`e4481af0 fffff800`0104113d nt!NtReplyWaitReceivePortEx+0x8c8
fffffadf`e4481c00 00000000`77ef0caa nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`e4481c70)
00000000`00bcfb98 000007ff`7fd6ff61 ntdll!NtReplyWaitReceivePortEx+0xa
00000000`00bcfba0 000007ff`7fd45369 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x2a5
00000000`00bcfeb0 000007ff`7fd65996 RPCRT4!RecvLotsaCallsWrapper+0x9
00000000`00bcfee0 000007ff`7fd65d51 RPCRT4!BaseCachedThreadRoutine+0xde
00000000`00bcff50 00000000`77d6b71a RPCRT4!ThreadStartRoutine+0x21
00000000`00bcff80 00000000`00000000 kernel32!BaseThreadStart+0x3a

Commands like .process /r /p fffffadfe718c040 or !process fffffadfe718c040 ff do that automatically.

Another case for incorrect module names is malformed unloaded modules information. This can happen sometimes:

0:000> lmt
start    end        module name
[...]
7c800000 7c907000   kernel32  Mon Apr 16 16:53:05 2007 (46239BE1)
7c910000 7c9c7000   ntdll     Wed Aug 04 08:57:08 2004 (411096D4)
7c9d0000 7d1ef000   shell32   Tue Dec 19 21:49:37 2006 (45885E71)
7df20000 7dfc0000   urlmon    Wed Aug 22 14:13:03 2007 (46CC365F)
7e360000 7e3f0000   user32    Thu Mar 08 15:36:30 2007 (45F02D7E)
Missing image name, possible paged-out or corrupt data.

Unloaded modules:
00410053 008a00a3   Unknown_Module_00410053
    Timestamp: Tue Mar 17 20:27:26 1970 (0064002E)
    Checksum:  006C006C
00010755 007407c5   l      
    Timestamp: Wed Feb 04 21:26:01 1970 (002E0069)
    Checksum:  006C0064
00000011 411096d2   eme.dll
    Timestamp: Thu Apr 02 01:33:25 1970 (00780055)
    Checksum:  00680054
Missing image name, possible paged-out or corrupt data.
0064002e 00d0009a   Unknown_Module_0064002e
    Timestamp: unavailable (00000000)
    Checksum:  00000000

Here parts of UNICODE module names appear in checksums and timestamps as well. Such partial module names can appear on thread stacks and raw stack data, for example:

0:000> kL
ChildEBP RetAddr
[...]
0015ef3c 0366afc2 ModuleA!Validation+0x5b
WARNING: Frame IP not in any known module. Following frames may be wrong.
0015efcc 79e7c7a6 <Unloaded_ure.dll>+0x366afc1
03dc9b70 00000000 mscorwks!MethodDesc::CallDescr+0x1f

Default analysis falls victim too and suggests ure.dll you would try hard to find on your system:

MODULE_NAME: ure

IMAGE_NAME:  ure.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  750063

FAILURE_BUCKET_ID:  ure.dll!Unloaded_c0000005_APPLICATION_FAULT

Timestamp is suspiciously UNICODE-like. In such cases we can even reconstruct the module name:

00000011 411096d2   eme.dll
    Timestamp: Thu Apr 02 01:33:25 1970 (00780055)
    Checksum:  00680054

0:000> .formats 00780055
Evaluate expression:
  Hex:     00000000`00780055
  Decimal: 7864405
  Octal:   0000000000000036000125
  Binary:  00000000 00000000 00000000 00000000 00000000 01111000 00000000 01010101
  Chars:   …..x.U
  Time:    Thu Apr 02 01:33:25 1970
  Float:   low 1.10204e-038 high 0
  Double:  3.88553e-317

0:000> .formats 00680054
Evaluate expression:
  Hex:     00680054
  Decimal: 6815828
  Octal:   00032000124
  Binary:  00000000 01101000 00000000 01010100
  Chars:   .h.T
  Time:    Fri Mar 20 21:17:08 1970
  Float:   low 9.55101e-039 high 0
  Double:  3.36747e-317

We concatenate UNICODE Ux and Th with eme.dll to get UxTheme.dll which is a real DLL name we can find on a system.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply