Crash Dump Analysis

I was inspecting the list of timers in a complete memory dump and found that in x64 W2K8 R2 (it may be also in other versions) there is a timer set to fire for the end of our century 2001 - 2100:

1: kd> !timer
Dump system timers
[...]
    fffff80001c31b80   f690c4d5 0064364d [12/31/2099 23:00:00.510]  nt!ExpCenturyDpcRoutine (DPC @ fffff80001c31bc0)
[…]

Its disassembly shows at that time the following work item will be executed:

1: kd> uf nt!ExpCenturyDpcRoutine
[...] 
fffff800`01ae81ab 488d0dae991400  lea     rcx,[nt!ExpCenturyWorkItem (fffff800`01c31b60)]
[...]

I don’t want to dig myself far into the details here May be someone from MS provides some comments what we should expect at the turn of the century? What should I tell my children to expect if they still run the Windows OS?

PS. Long live the Windows OS! (my the most sincere comment; no pun intended).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump is a double buzzword.

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A few days ago I was in a hotel bar invited to celebrate an event. Later that night we were trying to sing songs and I came up with a few stanzas. Today I finished the composition:

Solution Number One.
Bang, Bang, Bang…

Solution Number Two.
Poo, Poo, Poo…

Solution Number Three.
Wee, Wee, Wee…

Solution Number Four.
Oh, Oh, Oh…

Solution Number Five.
Ay, Ay, Ay…

Solution Number Six.
Fix, Fix, Fix!

I’ll try to add some music later on…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Started testing marketing materials to increase Dublin people awareness of memory dump analysis. Here are the first exemplars of what I’m going to wear by the end of this week while commuting and in local pubs

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I accidentally use du WinDbg command (to interpret memory as Unicode) instead of da (to interpret memory as ASCII):

0:000> db 07329f28 l20
07329f28  68 69 64 70 6e 73 74 3d-26 74 78 74 74 6f 3d 26  hidpnst=&txtto=&
07329f38  74 78 74 63 63 3d 26 74-78 74 62 63 63 3d 26 74  txtcc=&txtbcc=&t

0:000> da 07329f28 l20
07329f28  "hidpnst=&txtto=&txtcc=&txtbcc=&t"

0:000> du 07329f28 l10
07329f28  "楨灤獮㵴琦瑸潴.硴捴㵣琦瑸换㵣琦"

This time I tried to get extra hidden meaning from a process dump taken after the process suffered a CPU spike by using Google translator and got this text (I put more lengthy Unicode sequence and removed some offensive words):

"Luan Xian Zhen Qi-bin 㵴 cisternae. Huasong 㵣 Qi, Qi-bin-bin for 㵣 pull 㵪 䕒 .. 䉉 Ya Hui material. Hong SHIKA King. Huajiayuyan nuts .. 䐰 〥 䅁 evil force. Rafter Hui Qi 䤫 Mi cat deterrent Junying hydrogen walk. cisternae Huzhao Man cat Wuzhou Wen Zhen Zhao Zhen Pan scene file Shan. prison Shang Tang. Jue Shi Pan. sewage knock Xi. generous Zhen. 䤫. ice. conflict. cisternae Zhao askance nuts. rafter .. On unfeigned domain knock. Kagesue Mankuo. 㜲 Ruo Yi enemy luster of gems. cisternae Yu Wei Shan scene. Tan knock Shan. tally Xia Pan Ying. rafter. Xia. luster of gems tumultuous. Jing Feng-Tou Airuo enemy luster of gems Yixian … additionally . Tu. civet eliminating the lot Shan Ying RB Thieme, Jr.-Voltage trapping Feng-潷 Man. Tan knock Ruo Yi Xian cat enemy luster of gems. rafter Shi Feng-Tou. Mu. Minli Bang domain sewage Huitangyuzhao Su-hai.-Voltage Jiumi. rafter. Qing Wei Jun. 歳 Mi hai 䤫 Panyu. Zhucuoqufang .. 䐰 〥. 䐰 〥 䥁 hydrogen walk. rafter. Mount Zao Man. .. Run-Voltage Rendering. Tang Ying Yi. Shisuqingshi Fangmaosheji Yu Zhao 䤫 Su-. tide. tatami knock Feng-generous. rafter. Min luster of gems. Que Tu Mei Shi Tang Pan Ying. Jijue-Voltage. rafter. Wei Hui Mongoose Feng-. hunting. rafter. revolves Recent-Voltage sewage 䤫. stay Jiao RB Thieme, Jr soup.潷 Han.’m setback Xun. Han Tun petty. Liaohe. 䥔 end of Tu Feng-generous. rafter Xiang Shan Li Tu. trapping the end of sleep ZHEJIANG NORMAL Feng-Tou Yu Xun Jing Wen Fang 䤫 .. 䠫 pine and methods of disease. tatami knock Feng-generous. apply Feng-evil force fell Junying Su-Ao Po .. knock .. Tan Li Shan Jie look askance alone. ㅆ Guang Tang rafter. pool just cultural and"

From the translation I see previously hidden notions of gems, disease and evil forces :-)

ASCII->Unicode->translation->ASCII

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I do Google search for my site and find various cached dumpanalysis.org site evaluations ranging from 4,000 USD to 200,000 USD. It’s interesting to see that people evaluate how much this site worth. Today I stumbled across another unbelievable evaluation from valuemysite dot com (> 2,000,000 USD) so I checked myself:

“apple.com” is worth $340,512,336 USD
“amazon.com” is worth $340,512,336 USD
“microsoft.com” is worth $340,512,336 USD
“ibm.com” is worth $49,876,253 USD
“dumpanalysis.org” is worth $1,989,157 USD
“google.com” is worth $1,738,123 USD

Looks like some sites have an upper evaluation limit or reached an equilibrium

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Yes, it can. Here’s the Dump2Picture image of a kernel memory dump (3 GB) from a 128 GB system:

Now it’s time to listen to Klaus Schulze album In Blue again.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I knew it was my destiny!

kd> !analyze -vostokov

[...]

MANUALLY_INITIATED_CRASH (e2)
The user manually initiated this crash dump.
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Debugging Details:
------------------

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Being a software engineer, the author penetrated a software technical support department of a major software company rising to a management position. There he started collecting various management bits and tips promising everyone to write a management book. After moving back to engineering he became a director of several software research, education, publishing and software behavior analysis consultancy institutions including a museum. This book is an anthology of selected and edited blog posts from his Management Bits and Tips blog.

What this book has to do with the crash dump analysis then? Considering metaphorically an organization as a software machine, teams as processes and individuals as threads the author had applied his unique knowledge of software crashes and hangs to organizational project failures.

• Title: Management Bits: An Anthology from Reductionist Manager
• Author: Dmitry Vostokov
• Publisher: OpenTask (September 2010)
• Language: English
• Product Dimensions: 19.8 x 12.9
• Paperback: 100 pages
• ISBN-13: 978-1906717131

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The title of this blog post is a bugtated Sherlock Holmes phrase “… the curious incident of the dog in the night-time.” from Silver Blaze short story. To see why please watch a video at the end of this post.

Last week I was in St. Petersburg where I visited a bookshop Singer House 

and bought “A Grammar of the Coptic Language” book (in Russian) to practice with ancient memory dumps:

Before that I was circling on an 18th-century coach (seems to be a model if we look at its door handle):

After the riding I was looking around and spotted the Tsar (click on a picture to watch the movie):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally found music appropriate for ETW / CDF trace analysis. It’s Andre Gagnon’s album Escape:

Here’s my version of track titles (some of them are also appropriate for crash dump analysis) with my comments in italics:

1. Non Fatal Error
2. Trace Dance (Samba)
3. En Hive
4. Char, The
5. L”Debug”
6. “Memoria”L
7. Process Hearts (cores)
8. Holidays (, but always looking back)
9. WOW (64)
10. DA+TA Master
11. Concert for 4 Threads (“Concertino” doesn’t sound good here)
12. Toc-Cat-ta of Strings
13. Bugville Promenade (along bug clusters?)
14. MOVS
15. The Sea Named Trace (after Solaris movie)
16. Catching The Bottle (it is often difficult to find a relevant problem message in a billion-line trace)
17. Debug Me Tender (DebugLove?)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Don’t give your modules and build folders funny names. When your application or system crashes people will laugh. Recently had seen a driver build path (PDB paths, !dh command) involving words “dust”, “devil” and “missile”. A missile driver may sound like a winner against competitors but looks funny in a crash dump WinDbg output. Another case was a module having words “screw” and “driver” in lmv command output.

Another piece of advice is not to name your modules “fault tolerant”. This looks funny on crash stacks:

STACK_TEXT: 
0016f0ac 776d1faf ntdll!RtlpLowFragHeapFree+0x31
0016f0c4 655b9ed9 ntdll!RtlFreeHeap+0x105
0016f0dc 7650f1cc ModuleA!FaultTolerantHeap::FreeHeap+0x61
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

An exercise in de-analysis

This anti-pattern is called Wild Explanation. This is a free floating explanation based on loose associations. Its extreme version uses Gödel incompleteness theorems (undecidable crashes and hangs), quantum mechanics (in small time delta any bug can appear and disappear without being caught) or hydrodynamics (code fluidity, turbulence around processor cores) for abnormal software behaviour explanation (similar to Fashionable Nonsense in philosophy, humanities and social sciences). Its milder version is slightly modified original analysis monologue I found when searching Google for stack traces (I provide my comments below):

“Two hints make me assume this can be a physical memory problem:”

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: e37cc20e, memory referenced.
Arg2: 00000001, value 0 = read operation, 1 = write operation.
Arg3: 8083fe2c, If non-zero, the instruction address which referenced the bad memory
     address.
Arg4: 00000000, (reserved)

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx” referenced memory at “0x%08lx”. The memory could not be “%s”.

I wouldn’t be so quick. Check Hardware Error pattern post and comments there. So let’s de-analyze the analysis.

“c0000005 is Access Denied where C is virtual memory, meaning usually disk cache”

c is just STATUS_SEVERITY_ERROR (0×3 or 0y11) 11000000000000000000000000000101 and doesn’t have any connection to cache or crash beyond a mere coincidence. See NTSTATUS bit descriptions: http://msdn.microsoft.com/en-us/library/cc231200(PROT.10).aspx

“It could be RAM and/or it could be savrt.sys if you’re using something like an antivirus.
Check to see if you need updates for them, Norton, Symantec, McCaffee.”

Of course, it is either RAM or antivirus, what else? What about a virus? All household names.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx” referenced memory at “0x%08lx”.  ”The memory could not be “%s”.” means there is something wrong with the debugger itself; those should be addresses, not percent signs and alphanumeric garbage.”

Now even a debugger is corrupt. RAM again? It is not a garbage, they’re format codes (like for printf C function) to print actual addresses.

“8083fe2c” this address called a pointer, that referenced “e37cc20e” again, that “e” has specific meaning, some virtual device or another, probably CD ROM, trying to transfer data to RAM.  This implies either SCSI or PCI buss or DMA buss timing problems.”

RAM again… Evidence mounts stronger and stronger. Its all about timing. A bus buzzes.

“Which is usually caused by a bad driver, not RAM, but sometimes replacing the RAM “masks” out the failure.”

You never know… A bit of doubt?

“This normally happens with 64-bit machines that take advantage of the 64-bit superfast address access and indexing, known as Vector Indirect Addressing, which is, apparently, too fast for even recent Operating System code.”

I always suspected these machines run x64 times faster… I can only imagine horrors when Vector Indirect Addressing hits an old Operating System code. However, the crash under discussion is x86…

“A write to the “e” device; so, is it some kind of CDR or DVDR or other storage device?”

I guess that was a rhetorical question… or let me finish this thread of thought to connect dots, e-devices, virtual memory, disk cache and CD ROM. What if a page file was configured on a DVD-R?

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Pictures taken today during weekend nature walks (debugging walkthroughs) in Dublin suburbs:

Before A Debugging Session

Looking For Bugs

Found A Bug

Looking For More Bugs

With fix-privet,
Dr. DebugLove 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Anthology lacks full color inserts. This is rather a current limitation of POD technology that OpenTask publisher uses at the present time. The solution previously announced a year ago is to print a separate full color title with selected articles and illustrations. Finally, it is about to be published next month. The book front and back covers are collages from covers of individual memory dump analysis volumes:

TOC will be available soon. Here is the book data:

• Title: Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3
• Author: Dmitry Vostokov
• Publisher: OpenTask (May 2010)
• Language: English
• Product Dimensions: 21.6 x 14.0
• Paperback: 110 pages
• ISBN-13: 978-1906717698

OpenTask also plans a separate color supplement for volumes 4 and 5 once they are published. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Twitter page for DumpAnalysis now has the background picture of a memory dump generated by Dump2Picture:

Hope you enjoy the new design

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

• Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24
• Paperback: 1600 pages
• Publisher: Opentask (31 May 2010)
• ISBN-13: 978-1-906717-99-5

Information about individual volumes:

• Volume 1
• Volume 2
• Volume 3

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After checking that Vista still has old MS-DOS real mode 16-bit debug.exe with commands similar to WinDbg ones I tried to debug notepad.exe:

C:\Users\user>debug
-?
assemble     A [address]
compare      C range address
dump         D [range]
enter        E address [list]
fill         F range list
go           G [=address] [addresses]
hex          H value1 value2
input        I port
load         L [address] [drive] [firstsector] [number]
move         M range address
name         N [pathname] [arglist]
output       O port byte
proceed      P [=address] [number]
quit         Q
register     R [register]
search       S range list
trace        T [=address] [value]
unassemble   U [range]
write        W [address] [drive] [firstsector] [number]
allocate expanded memory        XA [#pages]
deallocate expanded memory      XD [handle]
map expanded memory pages       XM [Lpage] [Ppage] [handle]
display expanded memory status  XS

As expected it debugs the MSDOS (MZ) portion of notepad.exe:

C:\Users\user>debug c:\windows\system32\notepad.exe
-u
17DB:0000 0E            PUSH    CS
17DB:0001 1F            POP     DS
17DB:0002 BA0E00        MOV     DX,000E
17DB:0005 B409          MOV     AH,09
17DB:0007 CD21          INT     21
17DB:0009 B8014C        MOV     AX,4C01
17DB:000C CD21          INT     21
17DB:000E 54            PUSH    SP
17DB:000F 68            DB      68
17DB:0010 69            DB      69
17DB:0011 7320          JNB     0033
17DB:0013 7072          JO      0087
17DB:0015 6F            DB      6F
17DB:0016 67            DB      67
17DB:0017 7261          JB      007A
17DB:0019 6D            DB      6D
17DB:001A 206361        AND     [BP+DI+61],AH
17DB:001D 6E            DB      6E
17DB:001E 6E            DB      6E
17DB:001F 6F            DB      6F
-t

AX=0000  BX=FFFF  CX=FE00  DX=0000  SP=00B6  BP=0000  SI=0000  DI=0000
DS=17CB  ES=17CB  SS=17DB  CS=17DB  IP=0001   NV UP EI PL NZ NA PO NC
17DB:0001 1F            POP     DS
-t

AX=0000  BX=FFFF  CX=FE00  DX=0000  SP=00B8  BP=0000  SI=0000  DI=0000
DS=17DB  ES=17CB  SS=17DB  CS=17DB  IP=0002   NV UP EI PL NZ NA PO NC
17DB:0002 BA0E00        MOV     DX,000E
-t

AX=0000  BX=FFFF  CX=FE00  DX=000E  SP=00B8  BP=0000  SI=0000  DI=0000
DS=17DB  ES=17CB  SS=17DB  CS=17DB  IP=0005   NV UP EI PL NZ NA PO NC
17DB:0005 B409          MOV     AH,09
-g
This program cannot be run in DOS mode.

Program terminated normally
-

So I was looking for a real MSDOS program to debug and thought that debug.exe would be a natural choice. Unfortunately there was an illegal instruction during double debugging:

C:\Users\user>debug c:\windows\system32\debug.exe
-g
-g

So it looks like WinDbg double debugging is much more robust despite the bigger file size (debug.exe is only 21KB).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -