Software Diagnostics Library

Instead of providing the list of recommended books on debugging topics here is the picture that shows most of them that I own:

Hope you recognize books that you own and/or read too. Some of them were reviewed by me already:

Advanced Windows Debugging

Debugging by Thinking: A Multidisciplinary Approach

Microsoft Windows Internals: Microsoft Windows Server 2003, Windows XP, and Windows 2000

Why Programs Fail: A Guide to Systematic Debugging

and more reviews will come soon.

- Dmitry Vostokov @ DumpAnalysis.org -

In June I reported passing 15,000 visits / month mark and today from Google Analytics I see the passing of 20,000 visits / month mark:

At the time of this writing Google Analytics reports the higher number of visits than StatPress Wordpress plugin (14,653 visits / month) but this is probably due to the fact that Analytics counts visits for Russian version, portal and forum too. At the same time StatPress shows more page views for English blog alone than Analytics and the overall projected page views number is close to 1,000,000 / year (not including feeds):

One thing is certain is that the number of visitors has doubled since the beginning of this year. 

- Dmitry Vostokov @ DumpAnalysis.org -

CDA Patterns translated by Heejune Kim also appear on the new MSDN blog called !analyze -v (Korean version).

- Dmitry Vostokov @ DumpAnalysis.org -

“There are many rare” crashes “in the World, which Fortune never brings to Light” again.

Thomas Fuller, Gnomologia: Adagies and Proverbs

- Dmitry Vostokov @ DumpAnalysis.org -

Previously I introduced Hooked Functions pattern where I used !chkimg WinDbg command and today after accidentally discovering yet another patched DLL module in one process I created this simple command to check all modules:

!for_each_module !chkimg -lo 50 -d !${@#ModuleName} -v

0:000:x86> !for_each_module !chkimg -lo 50 -d !${@#ModuleName} -v
[...]
Scanning section:    .text
Size: 74627
Range to scan: 71c01000-71c13383
71c02430-71c02434  5 bytes - WS2_32!WSASend
[ 8b ff 55 8b ec:e9 cb db 1c 0d ]
71c0279b-71c0279f  5 bytes - WS2_32!select (+0x36b)
[ 6a 14 68 58 28:e9 60 d8 15 0d ]
71c0290e-71c02912  5 bytes - WS2_32!WSASendTo (+0x173)
[ 8b ff 55 8b ec:e9 ed d6 1b 0d ]
71c02cb2-71c02cb6  5 bytes - WS2_32!closesocket (+0x3a4)
[ 8b ff 55 8b ec:e9 49 d3 19 0d ]
71c02e12-71c02e16  5 bytes - WS2_32!WSAIoctl (+0x160)
[ 8b ff 55 8b ec:e9 e9 d1 1e 0d ]
71c02ec2-71c02ec6  5 bytes - WS2_32!send (+0xb0)
[ 8b ff 55 8b ec:e9 39 d1 14 0d ]
71c02f7f-71c02f83  5 bytes - WS2_32!recv (+0xbd)
[ 8b ff 55 8b ec:e9 7c d0 17 0d ]
71c03c04-71c03c08  5 bytes - WS2_32!WSAGetOverlappedResult (+0xc85)
[ 8b ff 55 8b ec:e9 f7 c3 1f 0d ]
71c03c75-71c03c79  5 bytes - WS2_32!recvfrom (+0x71)
[ 8b ff 55 8b ec:e9 86 c3 16 0d ]
71c03d14-71c03d18  5 bytes - WS2_32!sendto (+0x9f)
[ 8b ff 55 8b ec:e9 e7 c2 13 0d ]
71c03da8-71c03dac  5 bytes - WS2_32!WSACleanup (+0x94)
[ 8b ff 55 8b ec:e9 53 c2 25 0d ]
71c03f38-71c03f3c  5 bytes - WS2_32!WSASocketW (+0x190)
[ 6a 20 68 08 40:e9 c3 c0 11 0d ]
71c0446a-71c0446e  5 bytes - WS2_32!connect (+0x532)
[ 8b ff 55 8b ec:e9 91 bb 18 0d ]
71c04f3b-71c04f3f  5 bytes - WS2_32!WSAStartup (+0xad1)
[ 6a 14 68 60 50:e9 c0 b0 29 0d ]
71c06162-71c06166  5 bytes - WS2_32!shutdown (+0x1227)
[ 8b ff 55 8b ec:e9 99 9e 12 0d ]
71c069e9-71c069ed  5 bytes - WS2_32!WSALookupServiceBeginW (+0x887)
[ 8b ff 55 8b ec:e9 12 96 0f 0d ]
71c06c91-71c06c95  5 bytes - WS2_32!WSALookupServiceNextW (+0x2a8)
[ 8b ff 55 8b ec:e9 6a 93 10 0d ]
71c06ecd-71c06ed1  5 bytes - WS2_32!WSALookupServiceEnd (+0x23c)
[ 8b ff 55 8b ec:e9 2e 91 0e 0d ]
71c090be-71c090c2  5 bytes - WS2_32!WSAEventSelect (+0x21f1)
[ 8b ff 55 8b ec:e9 3d 6f 20 0d ]
71c09129-71c0912d  5 bytes - WS2_32!WSACreateEvent (+0x6b)
[ 33 c0 50 50 6a:e9 d2 6e 22 0d ]
71c0938e-71c09392  5 bytes - WS2_32!WSACloseEvent (+0x265)
[ 6a 0c 68 c8 93:e9 6d 6c 24 0d ]
71c093d9-71c093dd  5 bytes - WS2_32!WSAWaitForMultipleEvents (+0x4b)
[ 8b ff 55 8b ec:e9 22 6c 1a 0d ]
71c093ea-71c093ee  5 bytes - WS2_32!WSAEnumNetworkEvents (+0x11)
[ 8b ff 55 8b ec:e9 11 6c 21 0d ]
71c09480-71c09484  5 bytes - WS2_32!WSARecv (+0x96)
[ 8b ff 55 8b ec:e9 7b 6b 1d 0d ]
71c0eecb-71c0eecf  5 bytes - WS2_32!WSACancelAsyncRequest (+0x5a4b)
[ 8b ff 55 8b ec:e9 30 11 26 0d ]
71c10d39-71c10d3d  5 bytes - WS2_32!WSAAsyncSelect (+0x1e6e)
[ 8b ff 55 8b ec:e9 c2 f2 26 0d ]
71c10ee3-71c10ee7  5 bytes - WS2_32!WSAConnect (+0x1aa)
[ 8b ff 55 8b ec:e9 18 f1 22 0d ]
71c10f9f-71c10fa3  5 bytes - WS2_32!WSAAccept (+0xbc)
[ 8b ff 55 8b ec:e9 5c f0 27 0d ]
Total bytes compared: 74627(100%)
Number of errors: 140
140 errors : !WS2_32 (71c02430-71c10fa3)
[...]

CMDTREE.TXT was also updated with this command.

- Dmitry Vostokov @ DumpAnalysis.org -

“Some” processes “are very busy, and yet do nothing.”

Thomas Fuller, Gnomologia: Adagies and Proverbs

- Dmitry Vostokov @ DumpAnalysis.org -

Thanks to Roberto Farah who blogged about .cmdtree command I was able to create the first version of cmdtree.txt for Crash Dump Analysis Checklist to include common commands that I use. It can be found here:

CMDTREE.TXT for .cmdtree WinDbg command

The link to the file was also added to:

http://www.windbg.org/

Here is the screenshot of it:

It is useful for complex commands including command sequences and scripts. You can also include special guidelines and reminders as well. In the latter cases you can specify empty command or output something you prefer with .echo, for example command templates to fill manually.

This seems to be similar to EasyDbg. 

- Dmitry Vostokov @ DumpAnalysis.org -

“But perhaps the” OS “is suspended on the” finger “of some” developer.

Anton Chekhov, Notebook

- Dmitry Vostokov @ DumpAnalysis.org -

“Think before you” debug!

Pythagoras, Teachings

This bugtation is also illustrated graphically here:

Bugtations: a semiotic approach

- Dmitry Vostokov @ DumpAnalysis.org -

Now a bit of theory behind bugtations. If you know about semiotics you will instantly recognize bugtations as paradigmatic substitutions in quotations (syntagmatic axis). This is better illustrated on the following picture adopted from Daniel Chandler’s book Semiotics, The Basics (2nd ed., figure 3.1, page 84) using this simple bugtation created from Pythagoras famous quotation:

All bugtations are featured here:

http://www.dumpanalysis.org/Bugtations 

- Dmitry Vostokov @ DumpAnalysis.org -

“The only way to read” code “without being bored is to” browse “it at random and, having found something that interests you, close the” browser “and meditate.”

Charles-Joseph, Prince de Ligne, Mes écarts

- Dmitry Vostokov @ DumpAnalysis.org -

“Crash dumps are fossils” 

Here is the draft TOC for the previously announced book Crash Dump Analysis for System Administrators and Support Engineers (Windows Edition) which I’m co-authoring with Thomas Monahan, a Lead Escalation Engineer at Citrix Systems:

Table of Contents

 - Dmitry Vostokov @ DumpAnalysis.org -

Another pattern that logically flows from Tool Chain is called Tool Box. Their principal difference is that the former launches subordinated tools in a sequence to reach the common goal and the latter is the tool that launches other tools in any sequence independently:

Often tool boxes are implemented as toolbars. Another boundary example is the so called resource kit tools where an HTML page or taskbar menu serves the role of tool box.

- Dmitry Vostokov @ DumpAnalysis.org -

Opentask plans to publish full-color children’s book series called “Baby Turing”:

• Title: Baby Turing
• Authors: Alexandra Vostokova, Dmitry Vostokov
• Publisher: Opentask (01 November 2008)
• Language: English
• Product Dimensions: 21.6 x 21.6
• ISBN-13: 978-1-906717-26-1
• Paperback: 16 pages

- Dmitry Vostokov @ DumpAnalysis.org -

Comments “are not” code.

Gertrude Stein, The Autobiography of Alice B. Toklas

- Dmitry Vostokov @ DumpAnalysis.org -

“You can take better care of your” code “than another can.”

Ralph Waldo Emerson, Journals

- Dmitry Vostokov @ DumpAnalysis.org -

“One can best feel in dealing with” systems “how primitive” debugging “still is.”

Albert Einstein, Communication to Leó Szilárd

- Dmitry Vostokov @ DumpAnalysis.org -

Dealing with a problem today where NTSD failed to save a dump file either because of improper configuration for a default postmortem debugger or for other reasons I advised to use ADPlus from Debugging Tools for Windows in crash mode but found that I haven’t yet written about it. In the past I didn’t recommend ADPlus because no one wanted to install something extra on production environments. 

I had to show how ADPlus works for crashes because the usual Microsoft article was too big for an immediate grasp:

How to use ADPlus to troubleshoot “hangs” and “crashes”

Here is the quick polished tutorial: 

1. Download and install Debugging Tools for Windows appropriate for your application or service platform. For example, if your service is 32-bit but runs on x64 you need to download 32-bit package. Refer to http://www.windbg.org for quick download links.

2. Get ready for the test and download TesDefaultDebugger package.

3. Open a command prompt elevated as Administrator and cd to Debugging Tools for Windows installation folder.

4. Run ADPlus command:

5. Skip any warnings related to script interpreter if any:

6. Another command line window appears with ADPlus switches:

7. Close it, go back to the first command line window we used to launch-test ADPlus and type this command:

ADPlus -crash -pn TestDefaultDebugger64.exe

8. Skip warnings from step 5 if any and the symbol path warning if it appears too:

9. The second command line window is opened with the following expected output because we didn’t run TestDefaultDebugger64.exe:

10. Close it and launch TestDefaultDebugger64.exe from the package downloaded in step 2:

11. Go back to the first command line window and repeat the command from step 7. You can also specify PID by -p <PID> instead of -pn <Name> option. Skip warnings from steps 5 and 8 if any and you would see the following message showing where ADPlus will store logs and memory dumps if any:

There are also 2 additional command line windows appear. One is showing which PID the debugger was attached to:

and the other showing the output of attached console debugger, CDB by default:

12. We can dismiss the message from the previous step and wait for the crash to occur when we push the big button on TestDefaultDebugger window from step 10. We see the crash instantaneously in debugger console window if it is still running:

The following message box might appear and that depends on AEDebug registry key and WER settings which are beyond the scope of this post:

In case it appears you can simply choose to close the program.

13. That’s it. All files appear in this folder:

C:\Program Files\Debugging Tools for Windows 64-bit\Crash_Mode__Date_09-12-2008__Time_16-55-5151:

- Dmitry Vostokov @ DumpAnalysis.org -

Usually after writing and using a tool we think about an extension of it or we find another tool that is suitable as that extension. In such cases it is better to reuse existing components and adapt the former tool to use the latter. Tool Chain is a good pattern name here. A programming analogy could be a compiler and linker:

Typical example could be a logging tool that now acquires log viewer functionality by implementing a button that launches a separate log viewer or vice versa, a log viewer that can now do logging. Another simple example is a command file that launches different tools in a sequence. The tools might not be related by the data they produce and operate upon: this not the same as well-known architectural Filters and Pipes pattern. What is important is the fact that tools are related buy overall tool chain goal, for example, to debug problems by recording and analyzing log files or monitor some process activity and periodically create memory dumps:

- Dmitry Vostokov @ DumpAnalysis.org -

Last week I noticed this book on Amazon and I would have passed over it if I didn’t discover 2600 magazine when browsing Microsoft Encyclopedia of Security a few months ago This morning it arrived in post and I’m looking forward reading it during my lunch time. Anthology books are well suited for such breaks because articles are not long and usually self sufficient to learn about something discrete:

The Best of 2600: A Hacker Odyssey

- Dmitry Vostokov @ DumpAnalysis.org -