Universal memory dumps come from astronomical observations:
Dumps “are meaningless without” symbols “to interpret them.”
Raymond Arthur Lyttleton, Quoted in Astroparticle Physics, page 50
- Dmitry Vostokov @ DumpAnalysis.org -
“Oh, he is a good” troubleshooter, “but he has no power of” debugging!”
Charles Darwin, The life and letters of Charles Darwin, Volume 1
- Dmitry Vostokov @ DumpAnalysis.org -
Comparative analysis is based on saving several consecutive memory dump files to see similarities and differences. Most often this technique is used for memory leaks, for example, process heap leaks. Here we see another example related to CPU spikes.
One application was spiking CPU and its user dump file was saved manually. !runaway WinDbg command showed the following distribution of kernel and user thread times:
0:000> !runaway f
User Mode Time
Thread Time
14:f48 0 days 1:22:34.343
15:f4c 0 days 0:00:45.671
17:f54 0 days 0:00:41.359
18:f58 0 days 0:00:33.468
16:f50 0 days 0:00:24.171
12:f40 0 days 0:00:17.531
13:f44 0 days 0:00:07.109
0:f10 0 days 0:00:00.500
[…]
Kernel Mode Time
Thread Time
15:f4c 0 days 0:00:51.046
17:f54 0 days 0:00:41.125
18:f58 0 days 0:00:32.765
16:f50 0 days 0:00:23.500
12:f40 0 days 0:00:18.671
13:f44 0 days 0:00:07.453
14:f48 0 days 0:00:02.031
1:f14 0 days 0:00:00.343
[…]
Elapsed Time
Thread Time
1:f14 0 days 12:01:42.656
0:f10 0 days 12:01:42.656
24:f70 0 days 12:01:42.093
23:f6c 0 days 12:01:42.093
22:f68 0 days 12:01:42.093
21:f64 0 days 12:01:42.093
20:f60 0 days 12:01:42.093
19:f5c 0 days 12:01:42.093
18:f58 0 days 12:01:42.093
17:f54 0 days 12:01:42.093
16:f50 0 days 12:01:42.093
15:f4c 0 days 12:01:42.093
14:f48 0 days 12:01:42.093
13:f44 0 days 12:01:42.093
[…]
We see that thread #14 spent 1 hour and 22 minutes in user space and only 2 seconds in kernel space. Its thread stack showed DllA and DllB modules:
0:000> ~14s
*** ERROR: Symbol file could not be found. Defaulted to export symbols for DllA.dll -
eax=00010da2 ebx=09227bdc ecx=09236bcc edx=0000003a esi=006769e4 edi=00000335
eip=6120b231 esp=00fefd04 ebp=00fefd2c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3549:
6120b231 8b45ec mov eax,dword ptr [ebp-14h] ss:0023:00fefd18=09251a30
0:000> kv
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00fefd2c 6120be31 00000063 8633cdb9 006769e4 DllA!foo+0×3549
00fefd64 6120c6db 00fefe00 00006f07 0353f594 DllA!foo+0×4149
00fefdbc 6120d517 00fefe00 00fefde8 8633cd25 DllA!foo+0×49f3
00fefdf8 6120db4d 0353f594 09251848 00000000 DllA!foo+0×582f
00fefe14 6120f280 006769a0 00000000 0000020c DllA!foo+0×5e65
00fefe84 00933436 0000020c 0353f5ac 00000001 DllA!foo+0×7598
00fefec0 008414f1 0353f580 034db5d0 00000003 DllB!read+0xc6
[…]
00feffb8 77e64829 0078f290 00000000 00000000 msvcr71!_endthreadex+0xa0
00feffec 00000000 7c3494f6 0078f290 00000000 kernel32!BaseThreadStart+0×34
These modules belong to different vendors and the code could loop in DllB or in one of DllA functions. In order to show statistically which module is more suspect than the other, we requested 3 consecutive memory dumps of the same spiking process. They revealed the similar uniform pattern:
0:000> !runaway f
User Mode Time
Thread Time
16:e5c 0 days 1:37:24.640
0:e1c 0 days 0:00:00.640
17:e60 0 days 0:00:00.453
18:e64 0 days 0:00:00.328
15:e58 0 days 0:00:00.265
14:e54 0 days 0:00:00.203
[…]
Kernel Mode Time
Thread Time
16:e5c 0 days 0:00:00.687
15:e58 0 days 0:00:00.421
17:e60 0 days 0:00:00.375
14:e54 0 days 0:00:00.265
18:e64 0 days 0:00:00.250
0:e1c 0 days 0:00:00.046
[…]
Elapsed Time
Thread Time
1:e20 0 days 4:41:11.562
0:e1c 0 days 4:41:11.562
18:e64 0 days 4:41:10.875
17:e60 0 days 4:41:10.875
16:e5c 0 days 4:41:10.875
15:e58 0 days 4:41:10.875
14:e54 0 days 4:41:10.875
13:e50 0 days 4:41:10.875
12:e4c 0 days 4:41:10.875
11:e48 0 days 4:41:10.875
10:e44 0 days 4:41:10.875
9:e40 0 days 4:41:10.875
8:e3c 0 days 4:41:10.875
7:e38 0 days 4:41:10.875
6:e34 0 days 4:41:10.875
5:e30 0 days 4:41:10.875
4:e2c 0 days 4:41:10.875
3:e28 0 days 4:41:10.875
2:e24 0 days 4:41:10.875
[…]
1st dump file: Process Uptime: 0 days 4:41:12.000
0:000> ~16r
eax=07abbb6c ebx=07ab45d4 ecx=07ab4604 edx=00679344 esi=006769e4 edi=00000035
eip=6120b23e esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3556:
6120b23e 8bcb mov ecx,ebx
0:000> ~16kv
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3556
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598
0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34
2nd dump file: Process Uptime: 0 days 4:41:42.000
0:000> ~16r
eax=0002550b ebx=00679344 ecx=07ab45d4 edx=00000009 esi=006769e4 edi=00000035
eip=6120b231 esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
*** ERROR: Symbol file could not be found. Defaulted to export symbols for DllA.dll -
DllA!foo+0×3549:
6120b231 8b45ec mov eax,dword ptr [ebp-14h] ss:0023:0106fd44=07abbb48
0:000> ~16kv
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3549
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598
0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34
3rd dump file: Process Uptime: 0 days 4:41:59.000
0:000> ~16r
eax=07abbb6c ebx=07ab4604 ecx=07ab45d4 edx=07ab4604 esi=006769e4 edi=00000035
eip=6120b239 esp=0106fd30 ebp=0106fd58 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
DllA!foo+0×3551:
6120b239 895108 mov dword ptr [ecx+8],edx ds:0023:07ab45dc=07ab4604
0:000> ~16kv
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0106fd58 6120be31 0000000b 47d6689c 006769e4 DllA!foo+0×3551
0106fd90 6120c6db 0106fe2c 00007329 0342d14c DllA!foo+0×4149
0106fde8 6120d517 0106fe2c 0106fe14 47d66b28 DllA!foo+0×49f3
0106fe24 6120db4d 0342d14c 07abb960 00000000 DllA!foo+0×582f
0106fe40 6120f280 006769a0 00000000 000001d8 DllA!foo+0×5e65
0106feb0 00933436 000001d8 0342d164 00000001 DllA!foo+0×7598
0106feec 008414f1 0342d138 00000000 0340b290 DllB!read+0xc6
[…]
0106ffb8 77e64829 00784be0 00000000 00000000 msvcr71!_endthreadex+0xa0
0106ffec 00000000 7c3494f6 00784be0 00000000 kernel32!BaseThreadStart+0×34
We see exactly the same stack trace except the first (top) frame. The offset differs very little: +0×3556, +0×3549 (twice) and +0×3551. We can suspect the small loop here:
0:000> ub
DllA!foo+0x3549:
6120b231 8b45ec mov eax,dword ptr [ebp-14h]
6120b234 8d0490 lea eax,[eax+edx*4]
6120b237 8b10 mov edx,dword ptr [eax]
6120b239 895108 mov dword ptr [ecx+8],edx
6120b23c 8908 mov dword ptr [eax],ecx
6120b23e 8bcb mov ecx,ebx
6120b240 85db test ebx,ebx
6120b242 75e3 jne DllA!foo+0x353f (6120b227)
0:000> ub 6120b231
DllA!foo+0x3535:
6120b21d 8b4d08 mov ecx,dword ptr [ebp+8]
6120b220 8b0c88 mov ecx,dword ptr [eax+ecx*4]
6120b223 3bcb cmp ecx,ebx
6120b225 741d je DllA!foo+0x355c (6120b244)
6120b227 8b5908 mov ebx,dword ptr [ecx+8]
6120b22a 8b410c mov eax,dword ptr [ecx+0Ch]
6120b22d 33d2 xor edx,edx
6120b22f f7f7 div eax,edi
0:000> u
DllA!foo+0x3549:
6120b231 8b45ec mov eax,dword ptr [ebp-14h]
6120b234 8d0490 lea eax,[eax+edx*4]
6120b237 8b10 mov edx,dword ptr [eax]
6120b239 895108 mov dword ptr [ecx+8],edx
6120b23c 8908 mov dword ptr [eax],ecx
6120b23e 8bcb mov ecx,ebx
6120b240 85db test ebx,ebx
6120b242 75e3 jne DllA!foo+0x353f (6120b227)
0:000> u 6120b242
DllA!foo+0x355a:
6120b242 75e3 jne DllA!foo+0x353f (6120b227)
6120b244 ff4508 inc dword ptr [ebp+8]
6120b247 8b4508 mov eax,dword ptr [ebp+8]
6120b24a 3b4608 cmp eax,dword ptr [esi+8]
6120b24d 72cc jb DllA!foo+0x3533 (6120b21b)
6120b24f ff36 push dword ptr [esi]
6120b251 e88dc70000 call DllA!foo+0xfcfb (612179e3)
6120b256 59 pop ecx
All this strongly rules out DllB and points to DllA as the first vendor contact.
- Dmitry Vostokov @ DumpAnalysis.org -
This is a computational ghost that comes from CPU spiking tops and hills to bring sickness to small programs. Red color from RGB triplets provides early warning signs.
Inspired by non-computational Acheri.
PS. This category of computational ghosts and bug hauntings was also inspired by an e-mail conversation between two software engineers that I witnessed some years ago when one told another: “Is this your fix that is still haunting us?”. If you have ever visited heated code review debates you can imagine the provoked response.
- Dmitry Vostokov @ DumpAnalysis.org -
Here are some artistic images found in process memory dumps of VLC media player and IE. The latter was freezing after memory leaking, the size of its memory dump file is more than 1.5Gb.
Needle (VLC)
The tip of the same Needle magnified by x8:
The Image of Heap (from IE):
Red Zones (from IE):
- Dmitry Vostokov @ DumpAnalysis.org -
This is a forthcoming book to be released next year after we finally publish DebugWare book by the end of this summer:
Software Tracing and Logging: Architecture, Design, Implementation and Analysis Patterns (ISBN: 978-1906717728)
I have already begun working on it in the background. The scope of DebugWare book is too wide to cover tracing and logging in great detail not to mention the very important subject of software trace analysis.
- Dmitry Vostokov @ TraceAnalysis.org -
Last weekend I came across this book in a local bookshop:
I immediately recalled all beautiful patterns I have seen and collected from visualized computer memory maps. I plan to include many images in a full-color book this summer:
Patterns of Computer Memory (ISBN: 978-1906717711)
This book should be essential for creative computer engineers and inquiring debugging minds!
- Dmitry Vostokov @ DumpAnalysis.org -
This post number is 1,000. Wordpress admin panel reports that I have written 999 (excluding this one):
Of course, I wrote my 1,000th blog post some time ago if I count all my 7 blogs but this is 1,000th post for just this blog.
To celebrate this event, I’m posting a picture of troubleshooting and debugging labyrinth resting on a notion of universal memory dumps that are observational snapshots and include both memory and various traces we collect to resolve problems.
This picture shows how we arrive to problem resolution. For example:
I’m also working on another picture, called T&D Tangram 
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes the assembly code looks almost wild (not like generated by your favourite compiler). For example (that also shows .NET runtime native unhandled exception processing):
0:000> kL 100
ChildEBP RetAddr
0014dbb4 77189254 ntdll!KiFastSystemCallRet
0014dbb8 75fec244 ntdll!ZwWaitForSingleObject+0xc
0014dc28 75fec1b2 kernel32!WaitForSingleObjectEx+0xbe
0014dc3c 72605389 kernel32!WaitForSingleObject+0x12
0014dc6c 726058e7 mscorwks!ClrWaitForSingleObject+0x24
0014e128 72608084 mscorwks!RunWatson+0x1df
0014e86c 7260874a mscorwks!DoFaultReportWorker+0xb59
0014e8a8 72657452 mscorwks!DoFaultReport+0xc3
0014e8cc 7265c0c7 mscorwks!WatsonLastChance+0x3f
0014e924 7265c173 mscorwks!CLRAddVectoredHandlers+0x209
0014e92c 7603f4be mscorwks!InternalUnhandledExceptionFilter+0x22
0014e9e8 771a85b7 kernel32!UnhandledExceptionFilter+0×127
0014e9f0 77139a14 ntdll!__RtlUserThreadStart+0×6f
0014ea04 771340f4 ntdll!_EH4_CallFilterFunc+0×12
0014ea2c 77189b99 ntdll!_except_handler4+0×8e
0014ea50 77189b6b ntdll!ExecuteHandler2+0×26
0014eb00 771899f7 ntdll!ExecuteHandler+0×24
0014eb00 03ca0141 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0014ee28 634c2f42 0×3ca0141
0014ee34 67715e44 System_ni+0×132f42
0014ee70 72431b4c System_ServiceProcess_ni+0×25e44
0014ee80 724421f9 mscorwks!CallDescrWorker+0×33
0014ef00 72456571 mscorwks!CallDescrWorkerWithHandler+0xa3
0014f03c 724565a4 mscorwks!MethodDesc::CallDescr+0×19c
0014f058 724565c2 mscorwks!MethodDesc::CallTargetWorker+0×1f
0014f070 724afac5 mscorwks!MethodDescCallSite::CallWithValueTypes+0×1a
0014f1d4 724af9e5 mscorwks!ClassLoader::RunMain+0×223
0014f43c 724aff35 mscorwks!Assembly::ExecuteMainMethod+0xa6
0014f90c 724b011f mscorwks!SystemDomain::ExecuteMainMethod+0×456
0014f95c 724b004f mscorwks!ExecuteEXE+0×59
0014f9a4 72f57c24 mscorwks!_CorExeMain+0×15c
0014f9b4 75fe4911 mscoree!_CorExeMain+0×2c
0014f9c0 7716e4b6 kernel32!BaseThreadInitThunk+0xe
0014fa00 7716e489 ntdll!__RtlUserThreadStart+0×23
0014fa18 00000000 ntdll!_RtlUserThreadStart+0×1b
We set exception context:
0:000> kv 100
ChildEBP RetAddr Args to Child
[...]
0014e9e8 771a85b7 0014ea18 77139a14 00000000 kernel32!UnhandledExceptionFilter+0×127 (FPO: [SEH])
[…]
0:000> .exptr 0014ea18
----- Exception record at 0014eb18:
ExceptionAddress: 03ca0141
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000
----- Context record at 0014eb34:
eax=00000001 ebx=08394ff8 ecx=00000000 edx=00000001 esi=056a2a94 edi=00000000
eip=03ca0141 esp=0014ee00 ebp=0014ee28 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
03ca0141 3909 cmp dword ptr [ecx],ecx ds:0023:00000000=????????
Then we disassemble the code at crash point and it looks strange including calls through DS data segment:
0:000> .asm no_code_bytes
Assembly options: no_code_bytes
0:000> u 03ca0141
03ca0141 cmp dword ptr [ecx],ecx
03ca0143 call dword ptr ds:[36067C0h]
03ca0149 mov ecx,dword ptr [esi+5Ch]
03ca014c cmp dword ptr [ecx],ecx
03ca014e call dword ptr ds:[3606D10h]
03ca0154 mov dword ptr [ebp-1Ch],0
03ca015b mov dword ptr [ebp-18h],0FCh
03ca0162 push 3CA0180h
However further disassembly finally reaches RET instruction:
0:000> u
03ca0167 jmp 03ca0169
03ca0169 movzx edx,byte ptr [ebp-24h]
03ca016d mov ecx,dword ptr [ebp-28h]
03ca0170 call System_ServiceProcess_ni+0x25140 (67715140)
03ca0175 pop eax
03ca0176 jmp eax
03ca0178 lea esp,[ebp-0Ch]
03ca017b pop ebx
0:000> u
03ca017c pop esi
03ca017d pop edi
03ca017e pop ebp
03ca017f ret
03ca0180 mov dword ptr [ebp-18h],0
03ca0187 jmp 03ca0178
03ca0189 add byte ptr [eax],al
03ca018b add byte ptr [eax],al
and backward disassembling shows the matching function prolog code:
0:000> ub 03ca0141
03ca0127 movzx eax,byte ptr [ebp-24h]
03ca012b test eax,eax
03ca012d je 03ca0154
03ca012f cmp dword ptr [esi+60h],0
03ca0133 je 03ca013e
03ca0135 mov ecx,dword ptr [esi+60h]
03ca0138 call dword ptr ds:[3C20010h]
03ca013e mov ecx,dword ptr [esi+58h]
0:000> ub 03ca0127
03ca0114 push esi
03ca0115 push ebx
03ca0116 sub esp,1Ch
03ca0119 xor eax,eax
03ca011b mov dword ptr [ebp-18h],eax
03ca011e mov dword ptr [ebp-28h],ecx
03ca0121 mov dword ptr [ebp-24h],edx
03ca0124 mov esi,dword ptr [ebp-28h]
0:000> ub 03ca0114
03ca0102 retf
03ca0103 add eax,dword ptr [eax+36h]
03ca0106 retf
03ca0107 add ebx,dword ptr [esi+esi-35h]
03ca010b add esi,esp
03ca010d cmp eax,8B550360h
03ca0112 in al,dx
03ca0113 push edi
From stack trace I suspected this code as JIT-compiled .NET code of the the main assemebly method. And indeed I found the similar call signatures like mine
03ca0141 cmp dword ptr [ecx],ecx
03ca0143 call dword ptr ds:[36067C0h]
in the following MSDN article:
Drill Into .NET Framework Internals to See How the CLR Creates Runtime Objects
Hence the name of this pattern: JIT Code.
- Dmitry Vostokov @ DumpAnalysis.org -
OpenTask, iterative and incremental publisher, plans to release a guide for general computer users explaining crashes and hangs on personal computers ranging from mobile phones, netbooks and laptops to desktops and servers:
My Computer Crashes and Freezes: A Non-technical Guide to Software and Hardware Errors (ISBN: 978-19067171704)
- Dmitry Vostokov @ DumpAnalysis.org -
I recently became interested in who else has the same surname. Vostokov means Mr. East, east - ost (Ger.) - vostok (Rus.). Here is the brief list of people with links I’m going to expand periodically.
Alexander Vostokov - One of the first Russian philologists
Yevgeniy Vostokov - Composer
Sergei Vostokov - Mathematician
Viktor Vostokov - Author. “Secrets of East Healers”, “Lessons of Tibetan Medicine: The Art of Relaxation” and other books in Russian
Eugeny Vostokov - Artist
More to come.
- Dmitry Vostokov @ DumpAnalysis.org -
Part 2 briefly discussed debuggers and their commands. Debugger commands can be grouped together into scripts that can be run against memory dump files and their resulted textual output can be redirected to log files.
Therefore we start with the first step in our analysis process diagram introduced in Part 1:
Information Collection: Scripts
Here a script is a sequence of instructions to extract formatted textual information from a memory dump file (or a live system). There are many advantages of scripts and logs. Many companies, for example, banks, avoid sending plain memory dump files because of security considerations in order to prevent exposure of company or private information. After the advent of 64-bit personal computing complete memory dump files became larger and larger and it is now common to get 32Gb memory dumps. Processing multiple long-running commands can easily be done from textual log files. Additionally, scripts can be used to process hundreds of memory dumps in one go to look for similarities and differences. Many tools can be used for such purpose including built-in debugger scripting capabilities, shell scripting and regular expressions for intelligent search.
- Dmitry Vostokov @ DumpAnalysis.org -
“To” debug “is proper to man.”
- Dmitry Vostokov @ DumpAnalysis.org -
Consider this crash point:
0:000> r
eax=02d0f15c ebx=02a62918 ecx=77e41c30 edx=00000000 esi=ffffffff edi=02a8ed28
eip=76154193 esp=02d0f124 ebp=02d0f130 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
Application!GetData+0xb:
76154193 8b9eac000000 mov ebx,dword ptr [esi+0ACh] ds:0023:000000ab=????????
Seeing 000000ab address we can think that ESI was 0 but it was 0xFFFFFFFF. Adding 0xAC to it produced an effective NULL data pointer 0xAB through integer addition overflow if we consider addition as unsigned. It is easy to see the result if we consider 0xFFFFFFFF as signed -1. Looking at stack trace and function disassembly we see that 0xFFFFFFFF was passed as a parameter:
0:000> kv
ChildEBP RetAddr Args to Child
02d0f130 7616328d ffffffff 02d0f15c 02d0f150 Application!GetData+0xb
[…]
02d0ffec 00000000 740420d8 02a74070 00000000 kernel32!BaseThreadStart+0×34
0:000> u Application!GetData
Application!GetData:
76154188 mov edi,edi
7615418a push ebp
7615418b mov ebp,esp
7615418d push ecx
7615418e push ebx
7615418f push esi
76154190 mov esi,dword ptr [ebp+8]
76154193 mov ebx,dword ptr [esi+0ACh]
This is an example of a sentinel pointer marking the end of a linked list, for example, although NULL pointers having 0 value are usually used. Also -1 value can be used to assign an invalid pointer value.
- Dmitry Vostokov @ DumpAnalysis.org -
When looking at a stack trace of one crashed process we noticed an invalid code pointer. It is not a NULL code pointer but has the same stack trace pattern:
0:000> kL
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0013dfb4 00401791 0×5e388583
0013fdf4 0040189f Application!RequestData+0×3e1
0013fee4 00401d0a Application!main+0×3f
0013ffc0 77e4f23b Application!mainCRTStartup+0×16c
0013fff0 00000000 kernel32!BaseProcessStart+0×23
When we look at raw stack data and examine the backward disassembly of the return address we see that invalid code was called from RequestData function and WinDbg stack trace reconstruction is not suspicious (it is structurally and semantically correct):
0:000> dds esp l10
0013dfb8 00401791 Application!RequestData+0x3e1
0013dfbc 00000140
0013dfc0 0013ee50
0013dfc4 00000fa4
0013dfc8 00000000
0013dfcc 00000000
0013dfd0 00000ece
0013dfd4 0013ffc0
0013dfd8 7ffdc000
0013dfdc 00000140
0013dfe0 0000054c
0013dfe4 50000002
0013dfe8 4b4919ac
0013dfec 00000000
0013dff0 00000000
0013dff4 003f003c
0:000> .asm no_code_bytes
Assembly options: no_code_bytes
0:000> ub 00401791
Application!RequestData+0x3c8:
00401778 Application!RequestData+0x3d0 (00401780)
0040177a lea ebx,[ebx]
00401780 push 0
00401782 push eax
00401783 lea ecx,[esp+esi+0E30h]
0040178a push ecx
0040178b push edi
0040178c call Application!recv (0040e382)
When seeing recv call we might suspect that the crash happened just inside that function because the raw stack data upwards (lower addresses) doesn’t have any execution residue left from nested function calls:
0:000> dds esp-100 esp
0013deb8 00000000
0013debc 00000000
0013dec0 00000000
0013dec4 00000000
0013dec8 00000000
0013decc 00000000
0013ded0 00000000
0013ded4 00000000
0013ded8 00000000
0013dedc 00000000
0013dee0 00000000
0013dee4 00000000
0013dee8 00000000
0013deec 00000000
0013def0 00000000
0013def4 00000000
0013def8 00000000
0013defc 00000000
0013df00 00000000
0013df04 00000000
0013df08 00000000
0013df0c 00000000
0013df10 00000000
0013df14 00000000
0013df18 00000000
0013df1c 00000000
0013df20 00000000
0013df24 00000000
0013df28 00000000
0013df2c 00000000
0013df30 00000000
0013df34 00000000
0013df38 00000000
0013df3c 00000000
0013df40 00000000
0013df44 00000000
0013df48 00000000
0013df4c 00000000
0013df50 00000000
0013df54 00000000
0013df58 00000000
0013df5c 00000000
0013df60 00000000
0013df64 00000000
0013df68 00000000
0013df6c 00000000
0013df70 00000000
0013df74 00000000
0013df78 00000000
0013df7c 00000000
0013df80 00000000
0013df84 00000000
0013df88 00000000
0013df8c 00000000
0013df90 00000000
0013df94 00000000
0013df98 00000000
0013df9c 00000000
0013dfa0 00000000
0013dfa4 00000000
0013dfa8 00000000
0013dfac 00000000
0013dfb0 00000000
0013dfb4 00000000
0013dfb8 00401791 Application!RequestData+0x3e1
So we follow recv call forward disassembly (notice that the first jump is indirect):
0:000> u 0040e382
Application!recv:
0040e382 jmp dword ptr [Application!_imp__recv (00410180)]
Application!closesocket:
0040e388 jmp dword ptr [Application!_imp__closesocket (00410170)]
Application!WSAGetLastError:
0040e38e jmp dword ptr [Application!_imp__WSAGetLastError (00410174)]
Application!send:
0040e394 jmp dword ptr [Application!_imp__send (00410178)]
Application!connect:
0040e39a jmp dword ptr [Application!_imp__connect (0041017c)]
Application!htons:
0040e3a0 jmp dword ptr [Application!_imp__htons (00410198)]
Application!setsockopt:
0040e3a6 jmp dword ptr [Application!_imp__setsockopt (00410184)]
Application!socket:
0040e3ac jmp dword ptr [Application!_imp__socket (00410188)]
0:000> dps 00410180 l10
00410180 71ad2f7f ws2_32!recv
00410184 71ad2d47 ws2_32!setsockopt
00410188 71ad410c ws2_32!socket
0041018c 71ad7ca1 ws2_32!gethostbyname
00410190 71ad4f3b ws2_32!WSAStartup
00410194 71ad7b5b ws2_32!gethostname
00410198 71ad28bc ws2_32!htons
0041019c 71ad3da8 ws2_32!WSACleanup
004101a0 00000000
004101a4 00000000
004101a8 00000000
004101ac 00000000
004101b0 00000000
004101b4 45cd184e
004101b8 00000000
004101bc 00000002
0:000> u 71ad2f7f
ws2_32!recv:
71ad2f7f jmp 7fd60000
71ad2f84 sub esp,10h
71ad2f87 push ebx
71ad2f88 xor ebx,ebx
71ad2f8a cmp dword ptr [ws2_32!PrologPointer (71ae4044)],offset ws2_32!Prolog_v2 (71ad6067)
71ad2f94 push esi
71ad2f95 je ws2_32!recv+0×18 (71ad6207)
71ad2f9b lea eax,[ebp-8]
0:000> u 7fd60000
7fd60000 jmp DllA!recv_patch (612101b6)
7fd60005 mov edi,edi
7fd60007 push ebp
7fd60008 mov ebp,esp
7fd6000a jmp ws2_32!recv+0×5 (71ad2f84)
7fd6000f add byte ptr [eax],al
7fd60011 add byte ptr [eax],al
7fd60013 add byte ptr [eax],al
Finally we see that sockets library functions were patched by a 3rd-party module DllA and we need to contact its vendor.
- Dmitry Vostokov @ DumpAnalysis.org -
Previously announced memory dump analysis certification will have x86/x64-based tracks for Windows and Unix (including Linux / FreeBSD / Mac OS X). Each track consists of 3 exams, each having its own set of requirements and scope:
More information will be available later. The initiative is supported by OpenTask.
- Dmitry Vostokov @ DumpAnalysis.org -
This is an example from the sample question set of the forthcoming memory dump analysis certification planned by Memory Analysis and Debugging Institute. You can reply to this post with your answers.
Q. Interpret the fragment from WinDbg output below.
0:002> !teb
[...]
TEB at 7efaf000
[...]
0:002> dds poi(7efaf000+8) poi(7efaf000+4)
01192000 00000000
01192004 00000000
01192008 00000000
[...]
0119e448 00010020
0119e44c 00030002
0119e450 00050004
0119e454 00070006
0119e458 00090008
0119e45c 000b000a
0119e460 000d000c
0119e464 000f000e
0119e468 00110010
0119e46c 00130012
0119e470 00150014
0119e474 00170016
0119e478 00190018
0119e47c 001b001a
0119e480 001d001c
0119e484 001f001e
0119e488 00210020
0119e48c 00230022
0119e490 00250024
0119e494 00270026
0119e498 00290028
0119e49c 002b002a
0119e4a0 002d002c
0119e4a4 002f002e
0119e4a8 00310030
0119e4ac 00330032
0119e4b0 00350034
0119e4b4 00370036
0119e4b8 00390038
0119e4bc 003b003a
0119e4c0 003d003c
0119e4c4 003f003e
0119e4c8 00410040 MyService!__InternalCxxFrameHandler+0x5c
0119e4cc 00430042
0119e4d0 00450044
0119e4d4 00470046
0119e4d8 00490048
0119e4dc 004b004a
0119e4e0 004d004c
0119e4e4 004f004e
0119e4e8 00510050 advapi32!`string'+0x164
0119e4ec 00530052 advapi32!GetPerflibKeyValue+0x184
0119e4f0 00550054 advapi32!`string'+0x20c
0119e4f4 00570056 advapi32!_NULL_IMPORT_DESCRIPTOR+0x2714
0119e4f8 00590058 advapi32!szPerflibSectionName <PERF> (advapi32+0x90058)
0119e4fc 005b005a shlwapi!_CRT_INIT+0xaf
0119e500 005d005c shlwapi!_OpenProgidKey+0xee
0119e504 005f005e shlwapi!_pRawDllMain <PERF> (shlwapi+0x5005e)
0119e508 00410060 MyService!__InternalCxxFrameHandler+0x7c
0119e50c 00430042
0119e510 00450044
0119e514 00470046
0119e518 00490048
0119e51c 004b004a
0119e520 004d004c
0119e524 004f004e
0119e528 00510050 advapi32!`string'+0x164
0119e52c 00530052 advapi32!GetPerflibKeyValue+0x184
0119e530 00550054 advapi32!`string'+0x20c
0119e534 00570056 advapi32!_NULL_IMPORT_DESCRIPTOR+0x2714
0119e538 00590058 advapi32!szPerflibSectionName <PERF> (advapi32+0x90058)
0119e53c 007b005a
0119e540 007d007c
0119e544 007f007e
[...]
Choose all answers that are valid:
a. ASCII string fragment
b. Raw stack data
c. Exception handling
d. UNICODE string fragment
e. Partial stack traces
f. Performance monitoring
g. Execution residue
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes the script featured in part 1 doesn’t work because of the lack of symbols or something else:
***
***
*** Your debugger is not using the correct symbols
***
*** In order for this command to work properly, your symbol path
*** must point to .pdb files that have full type information.
***
*** Certain .pdb files (such as the public OS symbols) do not
*** contain the required information. Contact the group that
*** provided you with these symbols if you need this command to
*** work.
***
*** Type referenced: ntdll!_NT_TIB
***
Couldn’t resolve error at ‘ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′
^ Extra character error in ‘~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? $t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; !teb; dps @$t1 @$t2′
This is the case where !teb WinDbg command doesn’t work and we can cope with this as shown in the following post:
Coping with missing symbolic information
Therefore we can adjust our user-mode script to use hard-coded offsets and delineate raw stack outputs by the output of kv WinDbg command:
~*e r? $t0 = @$teb; r? $t1 = @$t0+8; r? $t2 = @$t0+4; kv 100; dps poi(@$t1) poi(@$t2)
- Dmitry Vostokov @ DumpAnalysis.org -
A server was hanging and a complete memory dump was generated. Usual checks for resource contention or CPU spikes like !locks and !running WinDbg commands didn’t reveal anything. Therefore we decided to list all threads in the system using !stacks and !process 0 ff commands. The former command gives a birds eye overview of threads and it was found suspicious that many threads in different processes had 0 ticks waiting and had this top level call:
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[88d9cd88 explorer.exe]
Proc.Thread .Thread Ticks ThreadState Blocker
1ecc.001eb4 88d99db0 0000000 Blocked 3rdPartyAVDriver+0×1394
[…]
[862bad88 userinit.exe]
4394.005570 878f0790 0000000 Blocked 3rdPartyAVDriver+0×1394
Inspection of the latter command detailed output of thread stacks showed that all these suspicious threads were in fact blocked when loading a DLL:
0: kd> .process /r /p 862bad88
Implicit process is now 862bad88
Loading User Symbols
0: kd> .thread 878f0790
Implicit thread is now 878f0790
0: kd> kL 100
ChildEBP RetAddr
9aeb529c 80833e95 nt!KiSwapContext+0x26
9aeb52c8 80829027 nt!KiSwapThread+0x2e5
9aeb5310 b8dff394 nt!KeDelayExecutionThread+0x2ab
WARNING: Stack unwind information not available. Following frames may be wrong.
9aeb5338 b8dffe35 3rdPartyAVDriver+0×1394
9aeb5354 b8e00208 3rdPartyAVDriver+0×1e35
9aeb5398 b8e0e45a 3rdPartyAVDriver+0×2208
9aeb53d0 8081e087 3rdPartyAVDriver+0×1045a
9aeb53e4 b7d6e41b nt!IofCallDriver+0×45
9aeb53f4 b7d6a3ee DriverA!Dispatch+0xfe
9aeb5404 8081e087 DriverA!KUI_dispatch+0×6d
9aeb5418 b8bdb4e1 nt!IofCallDriver+0×45
9aeb543c b8bdb755 DriverB!PassThrough+0xd3
9aeb545c 8081e087 DriverB!Dispatch+0×153
9aeb5470 b6a7b81d nt!IofCallDriver+0×45
9aeb5498 b6a6c5bb DriverC!LowerPassThrough+0×48
9aeb54bc b6a6cc32 DriverC+0×75bb
9aeb5554 b6a7aa57 DriverC+0×7c32
9aeb5564 b6a7aaa7 DriverC+0×15a57
9aeb558c 8081e087 DriverC!DispatchPassThrough+0×48
9aeb55a0 808fb13b nt!IofCallDriver+0×45
9aeb5688 80939c6a nt!IopParseDevice+0xa35
9aeb5708 80935d9e nt!ObpLookupObjectName+0×5b0
9aeb575c 808ece57 nt!ObOpenObjectByName+0xea
9aeb57d8 808ee0f1 nt!IopCreateFile+0×447
9aeb5834 808f1e31 nt!IoCreateFile+0xa3
9aeb5874 8088b19c nt!NtOpenFile+0×27
9aeb5874 7c94860c nt!KiFastCallEntry+0xfc
0007e1fc 7c947339 ntdll!KiFastSystemCallRet
0007e200 7c952653 ntdll!NtOpenFile+0xc
0007e278 7c9525db ntdll!LdrpCreateDllSection+0×4e
0007e364 7c95425c ntdll!LdrpMapDll+0×297
0007e610 7c95408d ntdll!LdrpLoadDll+0×1e4
0007e88c 7c801bf7 ntdll!LdrLoadDll+0×198
0007e8f4 7c81c70b kernel32!LoadLibraryExW+0×1b2
0007e908 7c93a352 kernel32!LoadLibraryW+0×11
0007ee38 7c95348d ntdll!LdrpCallInitRoutine+0×14
0007ef40 7c954339 ntdll!LdrpRunInitializeRoutines+0×367
0007f1d4 7c95408d ntdll!LdrpLoadDll+0×3cd
0007f450 7c801bf7 ntdll!LdrLoadDll+0×198
0007f4b8 7c81c70b kernel32!LoadLibraryExW+0×1b2
0007f4cc 77e556fb kernel32!LoadLibraryW+0×11
0007f558 77e3dc66 USER32!LoadAppDlls+0×179
0007f694 77e3df10 USER32!ClientThreadSetup+0×1c4
0007f698 7c948556 USER32!__ClientThreadSetup+0×5
0007f698 80831348 ntdll!KiUserCallbackDispatcher+0×2e
9aeb5b4c 8091f9e1 nt!KiCallUserMode+0×4
9aeb5ba4 bf8bc7c3 nt!KeUserModeCallback+0×8f
9aeb5bcc bf8bbc93 win32k!xxxClientThreadSetup+0×21
9aeb5ccc bf8b6c46 win32k!xxxCreateThreadInfo+0×804
9aeb5cdc bf8b7ae4 win32k!UserThreadCallout+0×71
9aeb5cf8 80947a1b win32k!W32pThreadCallout+0×3a
9aeb5d54 8088ae9e nt!PsConvertToGuiThread+0×153
9aeb5d58 00000000 nt!KiBBTUnexpectedRange+0xc
The functions shown in blue are known from past issues to be pass through forwarding IRP to the lower drivers in a device driver stack.
Actually the thread stack above belongs to the main thread blocked while loading application DLLs. The conclusion was to check 3rdPartyAVDriver.sys vendor.
- Dmitry Vostokov @ DumpAnalysis.org -
