Crash Dump Analysis Patterns (Part 94a)
Forthcoming Webinar on Pattern-Driven Software Diagnostics
2012 - The Year of Software Trace Analysis
Accelerated Memory Dump Analysis Training
Sponsored link: Memory Dump Analysis Services
Debugging Experts Magazine Online
Debugging Today Daily Newspaper
Memory dump analysis is all about deviations and of them is Value Deviation (a super pattern), be it a number of open handles, a heap size, a number of contended locks, time spent in kernel, and so on. Every system or process property has its average and mean values and large deviations are noticable as the so called anomalies. In this post we provide an example of a stack trace size (depth) deviation. The average number of frames for most stack traces is dependent on the type of a memory dump: user, kernel and complete but considerably longer or shorter stack traces are clearly visible in stack trace collections. I originally planned to call this pattern a Black Swan but after a moment of thought dismissed that idea because such deviations are not really rare after all. Here is an example of a stack trace collection from a CPU spiking process with a number of identical stack traces with just only 3 frames:
0:000> ~*kL
[...]
19 Id: 1054.1430 Suspend: 1 Teb: 7ff9c000 Unfrozen
ChildEBP RetAddr
1ac6ff50 7739bf53 ntdll!KiFastSystemCallRet
1ac6ffb8 77e6482f user32!NtUserWaitMessage+0xc
1ac6ffec 00000000 kernel32!BaseThreadStart+0x34
20 Id: 1054.c90 Suspend: 1 Teb: 7ffaf000 Unfrozen
ChildEBP RetAddr
1b30ff50 7739bf53 ntdll!KiFastSystemCallRet
1b30ffb8 77e6482f user32!NtUserWaitMessage+0xc
1b30ffec 00000000 kernel32!BaseThreadStart+0x34
21 Id: 1054.a34 Suspend: 1 Teb: 7ff9a000 Unfrozen
ChildEBP RetAddr
1b63ff50 7739bf53 ntdll!KiFastSystemCallRet
1b63ffb8 77e6482f user32!NtUserWaitMessage+0xc
1b63ffec 00000000 kernel32!BaseThreadStart+0×34
22 Id: 1054.1584 Suspend: 1 Teb: 7ff99000 Unfrozen
ChildEBP RetAddr
1ba9ff50 7739bf53 ntdll!KiFastSystemCallRet
1ba9ffb8 77e6482f user32!NtUserWaitMessage+0xc
1ba9ffec 00000000 kernel32!BaseThreadStart+0x34
[...]
These stack traces are correct from RetAddr analysis perspective:
0:000> ub 7739bf53
user32!PeekMessageW+0×11e:
7739bf42 nop
7739bf43 nop
7739bf44 nop
7739bf45 nop
7739bf46 nop
user32!NtUserWaitMessage:
7739bf47 mov eax,124Ah
7739bf4c mov edx,offset SharedUserData!SystemCallStub (7ffe0300)
7739bf51 call dword ptr [edx]
0:000> ub 77e6482f
kernel32!BaseThreadStart+0×10:
77e6480b mov eax,dword ptr fs:[00000018h]
77e64811 cmp dword ptr [eax+10h],1E00h
77e64818 jne kernel32!BaseThreadStart+0×2e (77e64829)
77e6481a cmp byte ptr [kernel32!BaseRunningInServerProcess (77ecb008)],0
77e64821 jne kernel32!BaseThreadStart+0×2e (77e64829)
77e64823 call dword ptr [kernel32!_imp__CsrNewThread (77e4132c)]
77e64829 push dword ptr [ebp+0Ch]
77e6482c call dword ptr [ebp+8]
Looking at their thread times reveals that they were the most spikers:
0:000> !runaway
User Mode Time
Thread Time
19:1430 0 days 0:01:34.109
22:1584 0 days 0:01:28.140
21:a34 0 days 0:01:26.765
20:c90 0 days 0:01:24.218
0:e78 0 days 0:00:01.687
10:398 0 days 0:00:01.062
7:14e8 0 days 0:00:00.250
4:1258 0 days 0:00:00.093
6:2e8 0 days 0:00:00.015
1:11c0 0 days 0:00:00.015
26:1328 0 days 0:00:00.000
25:7ec 0 days 0:00:00.000
[…]
In order to hypothesize about a possible culptit component we look at execution residue left on their raw stack data. Indeed, we see lots of non-coincidental symbolic references to 3rdPartyExtension module:
0:000> ~22s
eax=00000000 ebx=00000000 ecx=1ba9f488 edx=00000001 esi=1952bd40 edi=00000000
eip=7c82860c esp=1ba9ff54 ebp=1ba9ffb8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00240246
ntdll!KiFastSystemCallRet:
7c82860c ret
0:022> !teb
TEB at 7ff99000
ExceptionList: 1ba9ffdc
StackBase: 1baa0000
StackLimit: 1ba8f000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ff99000
EnvironmentPointer: 00000000
ClientId: 00001054 . 00001584
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffd5000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
0:022> dds 1ba8f000 1baa0000
1ba8f000 00000000
1ba8f004 00000000
[...]
1ba939e8 00000000
1ba939ec 00000000
1ba939f0 00000037
1ba939f4 1906e6c0
1ba939f8 064e1112 3rdPartyExtension!DllUnregisterServer+0xe1f1f
1ba939fc 1a042678
1ba93a00 034d2918
1ba93a04 00000000
1ba93a08 1a042660
1ba93a0c 00000008
1ba93a10 064e18ea 3rdPartyExtension!DllUnregisterServer+0xe26f7
1ba93a14 1a042678
1ba93a18 00000001
1ba93a1c 034d2870
1ba93a20 034d2b78
1ba93a24 0000001f
1ba93a28 00000007
1ba93a2c 034d2870
1ba93a30 1a01fc68
1ba93a34 00000001
1ba93a38 1ba93a54
1ba93a3c 064e1b45 3rdPartyExtension!DllUnregisterServer+0xe2952
1ba93a40 034d2b78
1ba93a44 00000000
1ba93a48 00000000
1ba93a4c 06e7b498
1ba93a50 00000212
1ba93a54 1ba93c00
1ba93a58 064e3bce 3rdPartyExtension!DllUnregisterServer+0xe49db
1ba93a5c 00000001
1ba93a60 00000001
1ba93a64 00000000
1ba93a68 115d7fbc
1ba93a6c 06e7b498
1ba93a70 062de91d 3rdPartyExtension+0xe91d
1ba93a74 0000020c
1ba93a78 1ba93b78
1ba93a7c 06363797 3rdPartyExtension+0×93797
1ba93a80 00000024
1ba93a84 00000000
1ba93a88 00000000
1ba93a8c 1ba93ee0
[…]
0:022> ub 064e1112
3rdPartyExtension!DllUnregisterServer+0xe1f0d:
064e1100 jge 3rdPartyExtension!DllUnregisterServer+0xe1f16 (064e1109)
064e1102 mov ecx,dword ptr [ecx+10h]
064e1105 cmp ecx,eax
064e1107 jne 3rdPartyExtension!DllUnregisterServer+0xe1f0a (064e10fd)
064e1109 push ecx
064e110a push ebx
064e110b mov ecx,edi
064e110d call 3rdPartyExtension!DllUnregisterServer+0xe1d17 (064e0f0a)
- Dmitry Vostokov @ DumpAnalysis.org -
Sponsored link: Professional Software Debugging Services
/* Malware and Software Defects -> Victimware.org */
Copyright © 2006 - 2012. This is a non-profit research and scientific project.
_1125.png)
Citrix and Microsoft Customer Forum
Museum of Debugging and Memory Dumps
7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story
Crash and Hang Analysis Audit Service
CARE: Crash Analysis Report Environment
Crash Dump and Software Trace Analysis Training and Seminars
Access OpenTask Titles on Safari Books Online
DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.
Twitter @ DumpAnalysis You can now follow portal and blog news at DumpAnalysis on Twitter
LinkedIn Group Dr. Watson Enthusiasts All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)
2010 (0x7DA) - The Year of Dump Analysis 2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade
International Memory Analysts and Debuggers Day: 07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA
AnnouncementsComing Soon:
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Management Bits: An Anthology from Reductionist Manager
Crash Dump Analysis for System Administrators and Support Engineers
New Magazines:
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
New Books:
Introduction to Pattern-Driven Software Problem Solving
Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5
Windows Debugging Notebook: Essential User Space WinDbg Commands
Memory Dump Analysis Anthology, Volume 5
Memory Dump Analysis Anthology, Volume 4
Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
Memory Dump Analysis Anthology, Volume 3
First Fault Software Problem Solving: A Guide for Engineers, Managers and Users
x64 Windows Debugging: Practical Foundations
Also available:
Windows Debugging: Practical Foundations
DLL List Landscape: The Art from Computer Memory Space
Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov
WinDbg: A Reference Poster and Learning Cards
Memory Dump Analysis Anthology, Volume 2
Memory Dump Analysis Anthology, Volume 1
New Children's Book:
