Crash Dump Analysis Patterns (Part 81)
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
One analysis problem that happens frequently is the absence of stack traces due to kernel stack pages being paged out and therefore not present in a complete memory dump that only contains physical memory. This shouldn’t be a problem for kernel or process memory dumps because they contain virtual memory. The problem usually manifests itself either on busy systems utilizing almost all physical memory or on over-aged systems where certain processes hadn’t been used for a long time. It could also be the case when a problem happened some time ago and only diagnosed much later. For example, this LPC wait chain for coupled processes happened to be 2 days ago before the dump was saved:
0: kd> !thread fffffadfcf9e8bf0 1f
THREAD fffffadfcf9e8bf0 Cid 61f0.2c70 Teb: 000007fffffdd000 Win32Thread: fffff97ff381a480 WAIT: (Unknown) UserMode Non-Alertable
fffffadfcf9e8f58 Semaphore Limit 0x1
Waiting for reply to LPC MessageId 01e2cb39:
Current LPC port fffffa800e5a9d10
Impersonation token: fffffa80039cd050 (Level Impersonation)
Owning Process fffffadffc7c7c20
Image: applicationA.exe
Wait Start TickCount 12018444
Ticks: 11312740 (2:01:06:01.562)
Context Switch Count 456 LargeStack
UserTime 00:00:00.046
KernelTime 00:00:00.078
Start Address applicationA (0×0000000100061411)
Stack Init fffffadc125d4e00 Current fffffadc125d48e0
Base fffffadc125d5000 Limit fffffadc125cc000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
Kernel stack not resident.
0: kd> !lpc message 01e2cb39
Searching message 1e2cb39 in threads …
Server thread fffffadff93c5bf0 is working on message 1e2cb39
0: kd> !thread fffffadff93c5bf0 1f
THREAD fffffadff93c5bf0 Cid 0218.5130 Teb: 000007ffffcbc000 Win32Thread: 0000000000000000 WAIT: (Unknown) UserMode Non-Alertable
fffffadff6c71c70 SynchronizationEvent
Impersonation token: fffffa803bde5060 (Level Impersonation)
Owning Process fffffadcde439280
Image: applicationB.exe
Wait Start TickCount 12018444
Ticks: 11312740 (2:01:06:01.562)
Context Switch Count 12
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0×0000000001e2cb39
LPC Server thread working on message Id 1e2cb39
Start Address kernel32 (0×0000000077d6b6a0)
Stack Init fffffadc28b19e00 Current fffffadc28b19950
Base fffffadc28b1a000 Limit fffffadc28b14000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
Kernel stack not resident.
One of tricks I started to recommend in such cases is to save user dumps of processes that could possibly be paged out before forcing a complete memory dump but I haven’t yet got enough data to see whether this helps or not.
- Dmitry Vostokov @ DumpAnalysis.org -
_1125.png)
Museum of Debugging and Memory Dumps
7/7/2010 - 8/8/2010 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Management Bits: An Anthology from Reductionist Manager
Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
