Crash Dump Analysis Patterns (Part 99a)
Forthcoming Webinar on Pattern-Driven Software Diagnostics
2012 - The Year of Software Trace Analysis
Accelerated Memory Dump Analysis Training
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
Debugging Today Daily Newspaper
Most of the time Incorrect Symbolic Information is associated with function names and offsets, for example, module!foo vs. module!foo+100. In some cases the module name is incorrect itself or absent altogether. This can happen in complete memory dumps when we forget to reload user space symbols after changing the process context, for example:
; previous process context of firefox.exe
; switching to winlogon.exe context
kd> .process fffffadfe718c040
Implicit process is now fffffadf`e718c040
kd> !process fffffadfe718c040
PROCESS fffffadfe718c040
SessionId: 0 Cid: 017c Peb: 7fffffd9000 ParentCid: 0130
DirBase: 01916000 ObjectTable: fffffa800099a890 HandleCount: 754.
Image: winlogon.exe
VadRoot fffffadfe75e91f0 Vads 190 Clone 0 Private 2905. Modified 10047. Locked 0.
DeviceMap fffffa8000004950
Token fffffa800122a060
ElapsedTime 77 Days 02:14:26.109
UserTime 00:00:04.156
KernelTime 00:00:02.359
QuotaPoolUsage[PagedPool] 143128
QuotaPoolUsage[NonPagedPool] 191072
Working Set Sizes (now,min,max) (541, 50, 345) (2164KB, 200KB, 1380KB)
PeakWorkingSetSize 6323
VirtualSize 108 Mb
PeakVirtualSize 118 Mb
PageFaultCount 212547
MemoryPriority BACKGROUND
BasePriority 13
CommitCharge 3733
[...]
THREAD fffffadfe68f2040 Cid 017c.0198 Teb: 000007fffffd7000 Win32Thread: fffff97ff4a09010 WAIT: (Unknown) UserMode Non-Alertable
fffffadfe7133160 Semaphore Limit 0x7fffffff
fffffadfe68f20f8 NotificationTimer
Not impersonating
DeviceMap fffffa8000004950
Owning Process fffffadfe718c040 Image: winlogon.exe
Attached Process N/A Image: N/A
Wait Start TickCount 426298731 Ticks: 51 (0:00:00:00.796)
Context Switch Count 2215076 LargeStack
UserTime 00:00:00.187
KernelTime 00:00:00.468
Start Address 0×0000000077d6b6e0
Stack Init fffffadfe4481e00 Current fffffadfe4481860
Base fffffadfe4482000 Limit fffffadfe447a000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Child-SP RetAddr Call Site
fffffadf`e44818a0 fffff800`0103b093 nt!KiSwapContext+0×85
fffffadf`e4481a20 fffff800`0103c433 nt!KiSwapThread+0xc3
fffffadf`e4481a60 fffff800`012d25ae nt!KeWaitForSingleObject+0×528
fffffadf`e4481af0 fffff800`0104113d nt!NtReplyWaitReceivePortEx+0×8c8
fffffadf`e4481c00 00000000`77ef0caa nt!KiSystemServiceCopyEnd+0×3 (TrapFrame @ fffffadf`e4481c70)
00000000`00bcfb98 000007ff`7fd6ff61 ntdll!NtReplyWaitReceivePortEx+0xa
00000000`00bcfba0 00000000`000d2340 0×7ff`7fd6ff61
00000000`00bcfba8 00000000`00bcfde0 0xd2340
00000000`00bcfbb0 00000000`014cd220 0xbcfde0
00000000`00bcfbb8 00000000`000c1d30 0×14cd220
00000000`00bcfbc0 00000000`00bcfe18 0xc1d30
00000000`00bcfbc8 0000ffff`00001f80 0xbcfe18
00000000`00bcfbd0 00000000`006c0044 0xffff`00001f80
00000000`00bcfbd8 00000000`000006ec firefox+0×2c0044
00000000`00bcfbe0 00000000`000007b0 0×6ec
00000000`00bcfbe8 00000000`419b8385 0×7b0
00000000`00bcfbf0 00000000`00000000 0×419b8385
kd> lmu m firefox
start end module name
00000000`00400000 00000000`00b67000 firefox T (no symbols)
We have the return address 00000000`006c0044 which is just firefox+0×2c0044 (00000000`00400000 + 2c0044). We correct that by reloading user space symbols.
kd> .reload /user
kd> !process fffffadfe718c040
[...]
THREAD fffffadfe68f2040 Cid 017c.0198 Teb: 000007fffffd7000 Win32Thread: fffff97ff4a09010 WAIT: (Unknown) UserMode Non-Alertable
fffffadfe7133160 Semaphore Limit 0x7fffffff
fffffadfe68f20f8 NotificationTimer
Not impersonating
DeviceMap fffffa8000004950
Owning Process fffffadfe718c040 Image: winlogon.exe
Attached Process N/A Image: N/A
Wait Start TickCount 426298731 Ticks: 51 (0:00:00:00.796)
Context Switch Count 2215076 LargeStack
UserTime 00:00:00.187
KernelTime 00:00:00.468
Start Address kernel32!BaseThreadStart (0x0000000077d6b6e0)
Stack Init fffffadfe4481e00 Current fffffadfe4481860
Base fffffadfe4482000 Limit fffffadfe447a000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Child-SP RetAddr Call Site
fffffadf`e44818a0 fffff800`0103b093 nt!KiSwapContext+0x85
fffffadf`e4481a20 fffff800`0103c433 nt!KiSwapThread+0xc3
fffffadf`e4481a60 fffff800`012d25ae nt!KeWaitForSingleObject+0x528
fffffadf`e4481af0 fffff800`0104113d nt!NtReplyWaitReceivePortEx+0x8c8
fffffadf`e4481c00 00000000`77ef0caa nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`e4481c70)
00000000`00bcfb98 000007ff`7fd6ff61 ntdll!NtReplyWaitReceivePortEx+0xa
00000000`00bcfba0 000007ff`7fd45369 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x2a5
00000000`00bcfeb0 000007ff`7fd65996 RPCRT4!RecvLotsaCallsWrapper+0x9
00000000`00bcfee0 000007ff`7fd65d51 RPCRT4!BaseCachedThreadRoutine+0xde
00000000`00bcff50 00000000`77d6b71a RPCRT4!ThreadStartRoutine+0x21
00000000`00bcff80 00000000`00000000 kernel32!BaseThreadStart+0x3a
Commands like .process /r /p fffffadfe718c040 or !process fffffadfe718c040 ff do that automatically.
Another case for incorrect module names is malformed unloaded modules information. This can happen sometimes:
0:000> lmt
start end module name
[...]
7c800000 7c907000 kernel32 Mon Apr 16 16:53:05 2007 (46239BE1)
7c910000 7c9c7000 ntdll Wed Aug 04 08:57:08 2004 (411096D4)
7c9d0000 7d1ef000 shell32 Tue Dec 19 21:49:37 2006 (45885E71)
7df20000 7dfc0000 urlmon Wed Aug 22 14:13:03 2007 (46CC365F)
7e360000 7e3f0000 user32 Thu Mar 08 15:36:30 2007 (45F02D7E)
Missing image name, possible paged-out or corrupt data.
Unloaded modules:
00410053 008a00a3 Unknown_Module_00410053
Timestamp: Tue Mar 17 20:27:26 1970 (0064002E)
Checksum: 006C006C
00010755 007407c5 l
Timestamp: Wed Feb 04 21:26:01 1970 (002E0069)
Checksum: 006C0064
00000011 411096d2 eme.dll
Timestamp: Thu Apr 02 01:33:25 1970 (00780055)
Checksum: 00680054
Missing image name, possible paged-out or corrupt data.
0064002e 00d0009a Unknown_Module_0064002e
Timestamp: unavailable (00000000)
Checksum: 00000000
Here parts of UNICODE module names appear in checksums and timestamps as well. Such partial module names can appear on thread stacks and raw stack data, for example:
0:000> kL
ChildEBP RetAddr
[...]
0015ef3c 0366afc2 ModuleA!Validation+0x5b
WARNING: Frame IP not in any known module. Following frames may be wrong.
0015efcc 79e7c7a6 <Unloaded_ure.dll>+0x366afc1
03dc9b70 00000000 mscorwks!MethodDesc::CallDescr+0x1f
Default analysis falls victim too and suggests ure.dll you would try hard to find on your system:
MODULE_NAME: ure
IMAGE_NAME: ure.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 750063
FAILURE_BUCKET_ID: ure.dll!Unloaded_c0000005_APPLICATION_FAULT
Timestamp is suspiciously UNICODE-like. In such cases we can even reconstruct the module name:
00000011 411096d2 eme.dll
Timestamp: Thu Apr 02 01:33:25 1970 (00780055)
Checksum: 00680054
0:000> .formats 00780055
Evaluate expression:
Hex: 00000000`00780055
Decimal: 7864405
Octal: 0000000000000036000125
Binary: 00000000 00000000 00000000 00000000 00000000 01111000 00000000 01010101
Chars: …..x.U
Time: Thu Apr 02 01:33:25 1970
Float: low 1.10204e-038 high 0
Double: 3.88553e-317
0:000> .formats 00680054
Evaluate expression:
Hex: 00680054
Decimal: 6815828
Octal: 00032000124
Binary: 00000000 01101000 00000000 01010100
Chars: .h.T
Time: Fri Mar 20 21:17:08 1970
Float: low 9.55101e-039 high 0
Double: 3.36747e-317
We concatenate UNICODE Ux and Th with eme.dll to get UxTheme.dll which is a real DLL name we can find on a system.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sponsored link: Professional Software Debugging Services
/* Malware and Software Defects -> Victimware.org */
Copyright © 2006 - 2012. This is a non-profit research and scientific project.
_1125.png)
Citrix and Microsoft Customer Forum
Museum of Debugging and Memory Dumps
7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Management Bits: An Anthology from Reductionist Manager
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Introduction to Pattern-Driven Software Problem Solving
Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5
Windows Debugging Notebook: Essential User Space WinDbg Commands
Memory Dump Analysis Anthology, Volume 5
Memory Dump Analysis Anthology, Volume 4
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
