Software Diagnostics Library

I’ve just found that although I covered Stack Overflow in kernel mode I didn’t do this for user mode. In fact this is one of the simplest patterns to see in crash dumps. It has its own characteristic exception code and stack trace:

FAULTING_IP:
StackOverflow!SoFunction+27
00401317 6a00            push    0

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00401300 (StackOverflow!SoFunction+0x00000010)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00082ffc

0:000> kL
ChildEBP RetAddr 
00083000 00401317 StackOverflow!SoFunction+0x10
00083010 00401317 StackOverflow!SoFunction+0×27
00083020 00401317 StackOverflow!SoFunction+0×27
00083030 00401317 StackOverflow!SoFunction+0×27
00083040 00401317 StackOverflow!SoFunction+0×27
00083050 00401317 StackOverflow!SoFunction+0×27
00083060 00401317 StackOverflow!SoFunction+0×27
00083070 00401317 StackOverflow!SoFunction+0×27
00083080 00401317 StackOverflow!SoFunction+0×27
00083090 00401317 StackOverflow!SoFunction+0×27
000830a0 00401317 StackOverflow!SoFunction+0×27
000830b0 00401317 StackOverflow!SoFunction+0×27
000830c0 00401317 StackOverflow!SoFunction+0×27
000830d0 00401317 StackOverflow!SoFunction+0×27
000830e0 00401317 StackOverflow!SoFunction+0×27
000830f0 00401317 StackOverflow!SoFunction+0×27
00083100 00401317 StackOverflow!SoFunction+0×27
00083110 00401317 StackOverflow!SoFunction+0×27
00083120 00401317 StackOverflow!SoFunction+0×27
00083130 00401317 StackOverflow!SoFunction+0×27

There could be thousands of stack frames:

0:000> kL 2000
[...]
000a2fa0 00401317 StackOverflow!SoFunction+0x27
000a2fb0 00401317 StackOverflow!SoFunction+0x27
000a2fc0 00401317 StackOverflow!SoFunction+0x27
000a2fd0 00401317 StackOverflow!SoFunction+0x27
000a2fe0 00401317 StackOverflow!SoFunction+0x27
000a2ff0 00401317 StackOverflow!SoFunction+0x27

To reach the bottom and avoid over scrolling we can dump the raw stack data, search for the end of the repeating pattern of StackOverflow!SoFunction+0×27 and try to manually reconstruct the bottom of the stack trace:

0:000> !teb
TEB at 7efdd000
    ExceptionList:        0017fdf0
    StackBase:            00180000
    StackLimit:           00081000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efdd000
    EnvironmentPointer:   00000000
    ClientId:             00001dc4 . 00001b74
    RpcHandle:            00000000
    Tls Storage:          7efdd02c
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00081000 00180000
[...]
0017fc74  00401317 StackOverflow!SoFunction+0×27
0017fc78  00000000
0017fc7c  a3a8ea65
0017fc80  0017fc90
0017fc84  00401317 StackOverflow!SoFunction+0×27
0017fc88  10001843
0017fc8c  a3a8ea95
0017fc90  0017fca0
0017fc94  00401317 StackOverflow!SoFunction+0×27
0017fc98  0017fcb8
0017fc9c  a3a8ea85
0017fca0  0017fcb0
0017fca4  00401317 StackOverflow!SoFunction+0×27
0017fca8  00000003
0017fcac  a3a8eab5
0017fcb0  0017fcc0
0017fcb4  00401317 StackOverflow!SoFunction+0×27
0017fcb8  76c68738 user32!_EndUserApiHook+0×11
0017fcbc  a3a8eaa5
0017fcc0  0017fcd0
0017fcc4  00401317 StackOverflow!SoFunction+0×27
0017fcc8  76c6a6cc user32!DefWindowProcW+0×94
0017fccc  a3a8ead5
0017fcd0  0017fce0
0017fcd4  00401317 StackOverflow!SoFunction+0×27
0017fcd8  0037311e
0017fcdc  a3a8eac5
0017fce0  0017fcf0
0017fce4  00401317 StackOverflow!SoFunction+0×27
0017fce8  0017fcd0
0017fcec  a3a8eaf5
0017fcf0  0017fd00
0017fcf4  00401317 StackOverflow!SoFunction+0×27
0017fcf8  76c6ad0f user32!NtUserBeginPaint+0×15
0017fcfc  a3a8eae5
0017fd00  0017fd5c
0017fd04  00401272 StackOverflow!WndProc+0xe2
0017fd08  00401190 StackOverflow!WndProc
0017fd0c  00000003
0017fd10  cf017ada
[…]

We use the extended version of k WinDbg command and supply EBP, ESP and EIP to see in what function it started:

0:000> r
eax=a3b739e5 ebx=00000000 ecx=ac430000 edx=ffefd944 esi=0037311e edi=00000000
eip=00401300 esp=00082ff8 ebp=00083000 iopl=0         nv up ei ng nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
StackOverflow!SoFunction+0×10:
00401300 89442404        mov     dword ptr [esp+4],eax ss:002b:00082ffc=00000000

0:000> k L=0017fcf0 00082ff8 00401300
ChildEBP RetAddr 
0017fcb0 00401317 StackOverflow!SoFunction+0×10
0017fd00 00401272 StackOverflow!SoFunction+0×27
0017fd5c 76c687af StackOverflow!WndProc+0xe2
0017fd88 76c68936 user32!InternalCallWinProc+0×23
0017fe00 76c6a571 user32!UserCallWinProcCheckWow+0×109
0017fe5c 76c6a5dd user32!DispatchClientMessage+0xe0
0017fe98 77ccee2e user32!__fnDWORD+0×2b
0017fedc 0040107d ntdll!KiUserCallbackDispatcher+0×2e
0017ff08 0040151e StackOverflow!wWinMain+0×7d
00402ba0 20245c8b StackOverflow!__tmainCRTStartup+0×176

- Dmitry Vostokov @ DumpAnalysis.org -

My friend was typing a long message in IE to one of his old schoolmates that he had just found on Internet. He spent about an hour writing and rewriting and when finally hit the Send button he got a page saying that connection was probably lost. Going back in URL history brought the empty edit box and all data was lost. Or was it? He called me and I immediately advised him to save a crash dump of iexplore.exe using Task Manager (Vista). I also asked him for a word he used to start his message. It was “Hello” in Russian. I got his dump file and opened it in WinDbg. Because the language of his message was Russian I assumed that it was still there in local buffers or heap entries in UNICODE format so I typed “ello” in Notepad and saved this in a Unicode text file. Loading it in a binary editor (I used Visual C++) showed the following sequence of bytes:

40 04 38 04 32 04 35 04 42 04

Then I did a search in WinDbg for this sequence from the first loaded module address till the end of user space:

0:000> lm
start    end        module name
003c0000 0045b000   iexplore
[...]

0:000> s 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04
[...]
048971e4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04  @.8.2.5.B.,. ...
[...]
08530fe4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...]
201ea65c 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...] 

The number of found entries was big and I decided to output every entry into a file using the following script: 

.foreach ( address { s-[1]b 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04 }) {.writemem c:\dmitry\ieout${address}.txt ${address}-10 ${address}+1000}

I got numerous files:

C:\dmitry>dir ieout*.txt
[...]
09/06/2008  08:53               4112 ieout0x048971e4.txt
09/06/2008  08:53               4112 ieout0x0489784c.txt
09/06/2008  08:53               4112 ieout0x0489b854.txt
09/06/2008  08:53               4112 ieout0x0489bc5c.txt
[...]

I combined all of them into one big file and sent it to my friend:

C:\dmitry>type ieout0x*.txt >ieoutall.txt

The file contained not only the final message but all intermediate typing histories too. He was very happy.

- Dmitry Vostokov @ DumpAnalysis.org -

Looking at memory dumps every day and writing about them has an unfortunate implication: every state of the world looks like a gigantic memory dump to me. Everything is memory and every state is memory dump. The current state of the world is an infinite (or an immense) number of memuons^*. Infinite can be any cardinal number greater or equal to that of natural numbers. In any case we can say it is N bits where this number is either finite or ∞. Therefore we have 2^N possible memory states (S). The set of possible transitions between them (S -> S) has the number of 2^N ^ 2^N elements. Which is the memory itself and we have transitions between its states too. Ad infinitum we have a limiting process from which arises the perceived flow of events. Anyway there is much to elaborate here. I’ll come back to this later.

(*) Memuon is an indivisible entity similar to a bit of information.

This is my current philosophy I’m dead serious.

- Dmitry Vostokov @ DumpAnalysis.org -

If you remember I promised to review this CD:

 The first Memory Analysis CD album

Finally it arrived by post and here it is:

I’ve just finished listening to it. It is wonderful! Clearly belongs to my Music for Debugging collection. Here is my commentary (italics) on track titles:

1. Chameleon 
   It crashes then hangs then spikes then leaks. You never know what happens next…
2. Silence Before The Storm 
   Waiting for the problem to start dumping memory
3. Appearances 
   Multiple occurrences of the issue ease crash dump collection
4. Memory Analysis 
   Memory dump analysis activity
5. Voices From Heaven 
   It’s my blog full of crash dump analysis patterns
6. Figure In The Fog 
   It’s that DLL! I see it clearly!!!
7. Diamond Tear 
   Customer is happy
8. The Toy Soldier 
   WinDbg
9. Tears In Dragon’s Eyes 
   Operating system vendor is happy too
10. Forgotten Song 
   No pasaran (bugs)
11. Skies
   We are debugging gods!

Enjoy :-) 

- Dmitry Vostokov @ DumpAnalysis.org -

Another pattern that we need to pay attention is called Thread Starvation. This happens when some threads or processes have higher priority and favored by OS thread dispatcher effectively starving other threads. If prioritized threads are CPU-bound we also see Spiking Thread pattern. However, if their thread priorities were normal they would have been preempted by other threads and latter threads would not be starved. Here is one example where 2 threads from 2 different applications but from one user session are spiking on 2 processors (threads from other processors have above normal and normal priority):

System Uptime: 0 days 6:40:41.143

0: kd> !running

System Processors f (affinity mask)
  Idle Processors None

     Prcb      Current   Next    
  0  f773a120  89864c86 ................
  1  f773d120  89f243d2 ................
  2  f7737120  89f61398 ................
  3  f773f120  897228a0 ................

0: kd> .thread /r /p 89f61398
Implicit thread is now 89f61398
Implicit process is now 88bcc2e0
Loading User Symbols

0: kd> !thread 89f61398 1f
THREAD 89f61398  Cid 16f8.2058  Teb: 7ffdf000 Win32Thread: bc41aea8 RUNNING on processor 2
Not impersonating
DeviceMap                 e48a6508
Owning Process            88bcc2e0       Image:         application.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      7201654                 LargeStack
UserTime                  01:24:06.687
KernelTime                00:14:53.828
Win32 Start Address application (0×0040a52c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba336000 Current ba335d00 Base ba336000 Limit ba330000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012e09c 762c3b7d USER32!IsWindowVisible
0012e0c4 762d61bb MSVBVM50!RbyCountVisibleDesks+0×3c
0012e0d0 004831f6 MSVBVM50!rtcDoEvents+0×7
0012e348 0046d1ae application+0×831f6
0012e3a0 762ce5a9 application+0×6d1ae
0012e3dc 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012e3f8 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012e434 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012e514 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012e55c 762b1aa3 MSVBVM50!EvtErrFire+0×18
0012e5ac 7739bffa MSVBVM50!CThreadPool::GetThreadData+0xf
0012e58c 762cd13b USER32!CallHookWithSEH+0×21
0012e5ac 7739bffa MSVBVM50!VBDefControlProc_2787+0xad
0012e618 762d3348 USER32!CallHookWithSEH+0×21
0012e640 762cda44 MSVBVM50!PushCtlProc+0×2e
0012e668 762cd564 MSVBVM50!CommonGizWndProc+0×4e
0012e6b8 7739b6e3 MSVBVM50!StdCtlWndProc+0×171
0012e6e4 7739b874 USER32!InternalCallWinProc+0×28
0012e75c 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012e7c4 773a16e5 USER32!DispatchMessageWorker+0×327
0012e7d4 762d616e USER32!DispatchMessageA+0xf
0012e828 762d6054 MSVBVM50!ThunderMsgLoop+0×97
0012e874 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012e8b4 004831f6 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012e8c0 00d3b3c8 application+0×831f6
00184110 00000000 0xd3b3c8

0: kd> .thread /r /p 897228a0
Implicit thread is now 897228a0
Implicit process is now 897348a8
Loading User Symbols

0: kd> !thread 897228a0 1f 100
THREAD 897228a0  Cid 2984.2988  Teb: 7ffdf000 Win32Thread: bc381488 RUNNING on processor 3
IRP List:
    89794bb8: (0006,0220) Flags: 00000000  Mdl: 8a145878
Not impersonating
DeviceMap                 e3ec0360
Owning Process            897348a8       Image:         application2.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      10239625                 LargeStack
UserTime                  02:38:18.890
KernelTime                00:29:36.187
Win32 Start Address application2 (0×00442e4c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init f1d90000 Current f1d8fd00 Base f1d90000 Limit f1d88000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012f66c 762d61bb USER32!_SEH_prolog+0×5
0012f678 00fdb0b9 MSVBVM50!rtcDoEvents+0×7
0012f92c 00fca760 application2+0xbdb0b9
0012fa20 762ce5a9 application2+0xbca760
0012fa40 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012fa5c 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012fa98 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012fb78 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012fb90 76330b2b MSVBVM50!EvtErrFire+0×18
0012fbf0 762cd13b MSVBVM50!ErrDefMouse_100+0×16d
0012fca4 762cda44 MSVBVM50!VBDefControlProc_2787+0xad
0012fccc 7631c826 MSVBVM50!CommonGizWndProc+0×4e
0012fd08 762cd523 MSVBVM50!StdCtlPreFilter_50+0×9e
0012fd5c 7739b6e3 MSVBVM50!StdCtlWndProc+0×130
0012fd88 7739b874 USER32!InternalCallWinProc+0×28
0012fe00 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012fe68 773a16e5 USER32!DispatchMessageWorker+0×327
0012fe78 762d616e USER32!DispatchMessageA+0xf
0012fea8 762bb78f MSVBVM50!ThunderMsgLoop+0×97
0012feb8 762d60cb MSVBVM50!MsoFInitPx+0×39
0012fecc 762d6054 MSVBVM50!CMsoCMHandler::FPushMessageLoop+0×1a
0012ff18 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012fef8 762d5f8e nt!KiDeliverApc+0×11f
0012ff18 762d5f55 MSVBVM50!SCM_MsoCompMgr::FPushMessageLoop+0×2f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ff18 762d5f55 nt!KiDeliverApc+0×11f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ffd4 0012ffc8 nt!KiDeliverApc+0×11f
00000000 00000000 0×12ffc8

What we see here is unusually high Priority and BasePriority values (24 and 24). This means that the base priority for these processes was most likely artificially increased to Realtime. Most processes have base priority 8 (Normal):

0: kd> !thread 88780db0 1f
THREAD 88780db0  Cid 44a8.1b8c  Teb: 7ffaf000 Win32Thread: bc315d20 WAIT: (Unknown) UserMode Non-Alertable
    887b8650  Semaphore Limit 0x7fffffff
    88780e28  NotificationTimer
Not impersonating
DeviceMap                 e1085298
Owning Process            889263a0       Image:         explorer.exe
Wait Start TickCount      1565543        Ticks: 4194 (0:00:01:05.531)
Context Switch Count      7                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6754000 Current b6753c0c Base b6754000 Limit b6750000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
b6753c24 80833e8d nt!KiSwapContext+0×26
b6753c50 80829b74 nt!KiSwapThread+0×2e5
b6753c98 809249cd nt!KeWaitForSingleObject+0×346
b6753d48 8088ac4c nt!NtReplyWaitReceivePortEx+0×521
b6753d48 7c8285ec nt!KiFastCallEntry+0xfc
01a2fe18 7c82783b ntdll!KiFastSystemCallRet
01a2fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
01a2ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×198
01a2ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
01a2ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
01a2ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
01a2ffec 00000000 kernel32!BaseThreadStart+0×34

Some important system processes like csrss.exe have base priority 13 (High) but their threads wait most of the time and this doesn’t create any problems:

0: kd> !thread 887eb3d0 1f
THREAD 887eb3d0  Cid 4cf4.2bd4  Teb: 7ffaf000 Win32Thread: bc141cc0 WAIT: (Unknown) UserMode Non-Alertable
    888769b0  SynchronizationEvent
Not impersonating
DeviceMap                 e1000930
Owning Process            8883f7c0       Image:         csrss.exe
Wait Start TickCount      1540456        Ticks: 29281 (0:00:07:37.515)
Context Switch Count      40                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address winsrv!ConsoleInputThread (0×75a81b18)
Stack Init b5c5a000 Current b5c59bac Base b5c5a000 Limit b5c55000 Call 0
Priority 15 BasePriority 13 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b5c59bc4 80833e8d nt!KiSwapContext+0×26
b5c59bf0 80829b74 nt!KiSwapThread+0×2e5
b5c59c38 bf89b1c3 nt!KeWaitForSingleObject+0×346
b5c59c94 bf89b986 win32k!xxxSleepThread+0×1be
b5c59cec bf89da22 win32k!xxxRealInternalGetMessage+0×46a
b5c59d4c 8088ac4c win32k!NtUserGetMessage+0×3f
b5c59d4c 7c8285ec nt!KiFastCallEntry+0xfc
00ffff64 7739c811 ntdll!KiFastSystemCallRet
00ffff84 75a81c47 USER32!NtUserGetMessage+0xc
00fffff4 00000000 winsrv!ConsoleInputThread+0×16c

However it is very unusual for a process to have Realtime base priority. I can speculate what had really happened before the system crash was forced. The system administrator noticed two applications consuming CPU over the long period of time and decided to intervene. Unfortunately his hand slipped when he was browsing Task Manager Set Priority menu and Realtime was selected instead of Low. Human error…

See also the similar starvation pattern resulted from threads with normal priority.

- Dmitry Vostokov @ DumpAnalysis.org -

Due to the request from blog readers I made my old Practical Foundations of Debugging lectures available in PDF format:

1. Memory, registers and simple arithmetic
2. Number representations and pointers
3. Bytes, words, double words and pointers to memory
4. Instruction pointer and disassembling a program with pointers
5. Memory and stacks
6. Frame pointer and local variables (Part 1)
7. Frame pointer and local variables (Part 2)
8. Function parameters
9. Function pointer parameters (Part 1)
10. Function pointer parameters (Part 2)
11. Virtual Memory, Processes and Threads (Part 1)
12. Virtual Memory, Processes and Threads (Part 2)
13. Arrays and structures in memory (Part 1)
14. Arrays and structures in memory (Part 2)

x64 version:

1. Memory, registers and simple arithmetic
2. Number representations and pointers

I keep both versions (HTMP and PDF) on the following pages where updates or corrections will be posted in the future:

Practical Foundations of Debugging (x86)
Practical Foundations of Debugging (x64)

- Dmitry Vostokov @ DumpAnalysis.org -

Just discovered that China has its own 1000 page Windows debugging book:

ISBN: 9787121064074 (June, 2008)

Author is Zhang Yinkui (Raymond Zhang).

Google translation is quite impressive:

Automatically translated

Automatically translated Table of Contents

Original

- Dmitry Vostokov @ DumpAnalysis.org -

A. Both have online versions of processor manuals. But Intel also ships them in paper format for free (the paper is worse in the latest revision than 2 years ago but books are a bit lighter now):

Information on how to order them can be found here:

http://www.intel.com/products/processor/manuals/index.htm

AMD docs can be downloaded from here:

http://developer.amd.com/documentation/guides/Pages/default.aspx

Comparing both online manuals I see sometimes that certain concepts are explained better in AMD docs and vice versa, so it is recommended to check both. It is also evident that AMD and Intel had to rephrase instruction descriptions differently when they talk about the same things, for example, BOUND instruction:

AMD:  Check Array Bound
Intel:  Check Array Index Against Bounds

- Dmitry Vostokov @ DumpAnalysis.org -

Crash dump analysis is a support activity. Therefore understanding software support business is important. I recently started abbreviating book titles for my personal book reading accounting purposes and here accidentally emerges the ASS abbreviation that reminds me every time that support guys cover “asses” of software engineers designing and developing software and expose their own “asses” when talking to customers :-)

I’m sorry if I offended someone here…

The Art of Software Support

The book was on my shelf for 4 years and only today I got the message :-) 

- Dmitry Vostokov @ DumpAnalysis.org -

Motivated by The Science of Sherlock Holmes I plan to write a book about the history of debugging with the following preliminary product details and tentative release date (which may come earlier if I have enough time):

• Title: The Science of Dr. Watson: An Illustrated History of Debugging
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2010)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-07-0
• Paperback: 256 pages

- Dmitry Vostokov @ DumpAnalysis.org -

More books to come in 2009. One of them is full-color book illustrated with beautiful visual images emerging from inherent modularized structure of modern operating systems and applications. Preliminary product details:

• Title: Computer Memory Visualization
• Authors: Jamie Fenton, Dmitry Vostokov
• Publisher: Opentask (01 February 2009)
• Language: English
• Product Dimensions: 28 x 21.6
• ISBN-13: 978-1-906717-06-3
• Paperback: 64 pages

- Dmitry Vostokov @ DumpAnalysis.org -

I was visiting Microsoft in March and here are some pictures I took outside the building where I worked (click to enlarge):

I also visited the company store where people were buying “Microsoft Brain” toy and saw one of the thickest software engineering books (1488 pages) there. Initially I hesitated to buy it because of its price but subsequently bought it nevertheless upon my return to Ireland:

Software Engineering Foundations: A Software Science Perspective

- Dmitry Vostokov @ DumpAnalysis.org -

Full-color special edition is available now. PART 6: Fun with Crash Dumps that features memory dump visualization pictures is the most impressive there. All screenshots and diagrams are color too. The book is thicker, heavier and much more expensive. Print on demand color books are very pricey. It is only available on Lulu because my Ingram distributor, Lightning Source, doesn’t print color books with more than 480 pages:

Memory Dump Analysis Anthology Collector’s Edition, Volume 1 

- Dmitry Vostokov @ DumpAnalysis.org -

Just discovered this album from Polish band Gutter Sirens on Amazon:

Memory Analysis

I haven’t listened to it yet. Will post a review as soon as it arrives

- Dmitry Vostokov @ DumpAnalysis.org -

ManagementBits Blog:

Management Bit and Tip 0×400

Managing Reading via Cooperative Multireading

LiterateScientist Blog:

The Science of Sherlock Holmes

Is it Safe to Eat?

- Dmitry Vostokov @ DumpAnalysis.org -

Just discovered that priority goes to Mikel Lasa

Memory Dump: 1960-1990 (Poesía vasca, hoy)

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes certain files are opened but not closed when not needed and this prevents other applications and users from using, deleting or replacing them. Sometimes on behalf of certain API calls another process opens them. One of the questions I was asked recently is how to find that process. The answer that I found is very simple: just list all handles from all processes and search for that file name there. This works in kernel and complete memory dumps and also in live kernel debugging session including its local kernel debugging variant. The following WinDbg command lists all objects (we need to open a log because the output is usually of several megabytes):

3: kd> !for_each_process "!handle 0 3 @#Process"

!handle 0 3 @#Process
processor number 3, process 8a392818
PROCESS 8a392818  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: bfc51000  ObjectTable: e1001e00  HandleCount: 2650.
    Image: System

Handle table at e16a3000 with 2650 Entries in use
0004: Object: 8a392818  GrantedAccess: 001f0fff Entry: e1004008
Object: 8a392818  Type: (8a392e70) Process
    ObjectHeader: 8a392800 (old version)
        HandleCount: 3  PointerCount: 235

0008: Object: 8a391db0  GrantedAccess: 00000000 Entry: e1004010
Object: 8a391db0  Type: (8a392ca0) Thread
    ObjectHeader: 8a391d98 (old version)
        HandleCount: 1  PointerCount: 1

000c: Object: e101bca0  GrantedAccess: 00000000 Entry: e1004018
Object: e101bca0  Type: (8a37db00) Key
    ObjectHeader: e101bc88 (old version)
        HandleCount: 1  PointerCount: 3
        Directory Object: 00000000  Name: \REGISTRY

[...]

1fac: Object: 88ec72b0  GrantedAccess: 00000003 (Protected) Entry: e1ed7f58
Object: 88ec72b0  Type: (8a36f900) File
    ObjectHeader: 88ec7298 (old version)
        HandleCount: 1  PointerCount: 2
        Directory Object: 00000000  Name: \Documents and Settings\MyUserName\NTUSER.DAT {HarddiskVolume1}

[...]

07fc: Object: e1000768  GrantedAccess: 00000003 Entry: e2fefff8
Object: e1000768  Type: (8a387e70) KeyedEvent
    ObjectHeader: e1000750 (old version)
        HandleCount: 273  PointerCount: 274
        Directory Object: e1001a48  Name: CritSecOutOfMemoryEvent

processor number 3, process 8873f3b8
PROCESS 8873f3b8  SessionId: 6  Cid: 4c1c    Peb: 7ffdf000  ParentCid: 42bc
    DirBase: 49dbb000  ObjectTable: e325f788  HandleCount:  90.
    Image: PROFLWIZ.EXE

Handle table at e36c3000 with 90 Entries in use
0004: Object: e1000768  GrantedAccess: 00000003 Entry: e36c3008
Object: e1000768  Type: (8a387e70) KeyedEvent
    ObjectHeader: e1000750 (old version)
        HandleCount: 273  PointerCount: 274
        Directory Object: e1001a48  Name: CritSecOutOfMemoryEvent

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

Following the introductory division of memory analysis into two broad categories I decided to plan yet another book with the following title and preliminary product details:

• Title: Memory Analysis Forensics and Intelligence: An Integral Approach
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2009)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-05-6
• Paperback: 256 pages

As you might have noticed, I prefer to put 128, 256 or 512 pages in my book announcements. What that would say about the author’s background? I would like to set 1024 pages for my memory dump analysis anthology series but 740 or 800 pages is the limit for POD technology I use.

- Dmitry Vostokov @ DumpAnalysis.org -

Software ageing can be the cause of the problem. Sometimes a look at the following WinDbg output can give irresistible temptation to suggest periodic reboots:

Debug session time: Wed April 28 15:36:52.330 2008 (GMT+0)
System Uptime: 124 days 6:27:16.658

The suggested pattern name is Overaged System.

- Dmitry Vostokov @ DumpAnalysis.org -

While working on WDN book I noticed that it is possible to analyze managed code in complete memory dumps. We just need to switch to the process in question and load SOS DLL (if memory dumps are from 64-bit Windows we need to run 64-bit WinDbg because it needs to load 64-bit SOS from Microsoft.NET \ Framework64 \ vX.X.XXXXX folder).

Here is some command output from a complete memory dump generated from SystemDump when running TestDefaultDebugger.NET application where we try to find and disassemble IL code for this function:

namespace WindowsApplication1
{
  public partial class Form1 : Form
  {
[...]
  private void button1_Click(object sender, EventArgs e)
  {
  System.Collections.Stack stack = new System.Collections.Stack();stack.Pop();
  }
  }
}

Loading Dump File [C:\W2K3\MEMORY_NET.DMP]
Kernel Complete Dump File: Full address space is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x64
Product: Server, suite: Enterprise TerminalServer
Built by: 3790.srv03_sp2_gdr.070321-2337
Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`01198b00

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
[...]
PROCESS fffffadfe7287610
    SessionId: 0  Cid: 0ad8    Peb: 7fffffdf000  ParentCid: 0acc
    DirBase: 12cd9000  ObjectTable: fffffa800163c260  HandleCount: 114.
    Image: TestDefaultDebugger.NET.exe

PROCESS fffffadfe67905a0
    SessionId: 0  Cid: 085c    Peb: 7fffffd4000  ParentCid: 0acc
    DirBase: 232e2000  ObjectTable: fffffa8000917e10  HandleCount:  55.
    Image: SystemDump.exe

kd> .process /r /p fffffadfe7287610
Implicit process is now fffffadf`e7287610
Loading User Symbols

kd> .loadby sos mscorwks

kd> !threads

ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                              PreEmptive                                                Lock
       ID OSID        ThreadOBJ     State   GC     GC Alloc Context                  Domain           Count APT Exception
       1  a94 0000000000161150      6020 Enabled  0000000000000000:0000000000000000 000000000014ccb0     0 STA
       2  604 00000000001688b0      b220 Enabled  0000000000000000:0000000000000000 000000000014ccb0     0 MTA (Finalizer)

kd> !process fffffadfe7287610 4
PROCESS fffffadfe7287610
    SessionId: 0  Cid: 0ad8    Peb: 7fffffdf000  ParentCid: 0acc
    DirBase: 12cd9000  ObjectTable: fffffa800163c260  HandleCount: 114.
    Image: TestDefaultDebugger.NET.exe

        THREAD fffffadfe668cbf0  Cid 0ad8.0a94  Teb: 000007fffffdd000 Win32Thread: fffff97ff4df2830 WAIT
        THREAD fffffadfe727e6d0  Cid 0ad8.0f54  Teb: 000007fffffdb000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe72d5bf0  Cid 0ad8.0604  Teb: 000007fffffd9000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe679cbf0  Cid 0ad8.06b0  Teb: 000007fffffd7000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe67d23d0  Cid 0ad8.0b74  Teb: 000007fffffd5000 Win32Thread: fffff97ff4b99010 WAIT

kd> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000000002a41030
generation 1 starts at 0x0000000002a41018
generation 2 starts at 0x0000000002a41000
ephemeral segment allocation context: (0x0000000002a8d528, 0x0000000002a8dfe8)
         segment            begin         allocated             size
00000000001a1260 0000064274e28f60  0000064274e5f610 0x00000000000366b0(222896)
00000000001a1070 000006427692ffe8  000006427695af20 0x000000000002af38(175928)
0000000000164f60 00000642787c7380  0000064278809150 0x0000000000041dd0(269776)
0000000002a40000 0000000002a41000  0000000002a8dfe8 0x000000000004cfe8(315368)
Large object heap starts at 0x0000000012a41000
         segment            begin         allocated             size
0000000012a40000 0000000012a41000  0000000012a4e738 0x000000000000d738(55096)
Total Size           0xfdad8(1039064)
------------------------------
GC Heap Size           0xfdad8(1039064)

kd> !gchandles
Bad MethodTable for Obj at 0000000002a7a7b8
Bad MethodTable for Obj at 0000000002a7a750
Bad MethodTable for Obj at 0000000002a445b0
GC Handle Statistics:
Strong Handles: 25
Pinned Handles: 7
Async Pinned Handles: 0
Ref Count Handles: 1
Weak Long Handles: 30
Weak Short Handles: 63
Other Handles: 0
Statistics:
              MT    Count    TotalSize Class Name
[...]
0000064280016580        1          464 WindowsApplication1.Form1
[…]

kd> !dumpmt -md 0000064280016580
EEClass: 0000064280143578
Module: 0000064280012e00
Name: WindowsApplication1.Form1
mdToken: 02000002  (C:\TestDefaultDebugger.NET.exe)
BaseSize: 0×1d0
ComponentSize: 0×0
Number of IFaces in IFaceMap: 15
Slots in VTable: 375
————————————–
MethodDesc Table
           Entry       MethodDesc      JIT Name
[…]
0000064280150208 00000642800164d0      JIT WindowsApplication1.Form1.InitializeComponent()
0000064280150210 00000642800164e0      JIT WindowsApplication1.Form1..ctor()
0000064280150218 00000642800164f0     NONE WindowsApplication1.Form1.button1_Click(System.Object, System.EventArgs)

kd> !dumpil 00000642800164f0
ilAddr = 00000000004021bc
IL_0000: newobj System.Collections.Stack::.ctor
IL_0005: stloc.0
IL_0006: ldloc.0
IL_0007: callvirt System.Collections.Stack::Pop
IL_000c: pop
IL_000d: ret

- Dmitry Vostokov @ DumpAnalysis.org -