Software Diagnostics Library

Recently I was asked how to measure complexity of technical support cases especially ones that require memory dump analysis. My first response was that it is a subjective qualitative measure based mostly on experience and feeling. However after careful consideration I understood that nothing has changed for the last 5 years: the nature and causes of system or application hangs and crashes still the same regardless of OS types and versions. Therefore the complexity measure shifts from a case description and its artifacts to an analyst, a memory dump reader. Here the number of queries, questions asked or commands executed to gather information for analysis, can be a good approximation to the measure of complexity. For example, 5 years ago I started with a few commands like !analyze -v, kv and dd and progressed to an elaborate checklist. Here the natural logarithm can be used to approximate the measure:

C = ln (N_dc), where N_dc is the number of debugging commands used.

Initially the complexity was ln(3) ≈ 1.1 and now if someone uses 10 commands on average or asks 10 questions the complexity is ln(10) ≈ 2.3. The analysis is more than 2 times complex than it was.

- Dmitry Vostokov @ DumpAnalysis.org -

I noticed that Amazon sells used copies of my Curriculum Vitae:

- Dmitry Vostokov @ DumpAnalysis.org -

OpenTask publishes the following sequel to Resume and CV book:

Title: Salary Figures: A Codebook of Expectations
Author: Dmitry Vostokov
ISBN-13: 978-1906717469
Paperback: 100 pages (1 January 2009)
Product Dimensions: 20.3 x 12.7
Book Annotation: After publishing Curriculum Vitae as a book the author started receiving e-mails asking for his salary expectation. Being shy to discuss such matters openly he devised a codebook of salaries. Now everyone could answer a question about his or her salary expectations by referring to specific page. Also colleagues and friends could say to each other: “I’m still on page X” or “Last month I turned the page Y over”. The last 10 pages of the codebook specify various perks. The response template to the question of salary expectations is: “My salary expectation is on page Z plus benefits listed on pages N and M”. (*) X,Y,Z,N,M are page numbers from 3 to 100. Good luck with job hunting

Typeset in Chilling font the book is suitable for wide salary ranges, from 1,000 to 1,000,000.

- Dmitry Vostokov @ DumpAnalysis.org -

I’ve written 10 pattern-driven crash dump analysis case studies so far and now put them on one page:

Pattern Cooperation

More case studies will be added in the future.

- Dmitry Vostokov @ DumpAnalysis.org -

Once upon a time I was analyzing a memory dump and in the process become upset. I realized that the dump was vacuous and afterwards learnt that the customer forced the dump of a normal system with steady-state computational activities inside. At the same time computational precognition, the ability to see in advance what would happen with the system, always fascinated me from the very beginning of my memory dump analysis work. In computational precognition phrase, the word “computational” refers to computers and “precognition” refers to humans but this should not be the problem as the boundary between these two categories is not sharp and can be shifted in either direction. This is an opening of the new post series about investigation of occult, paranormal and supernatural in the realm of computable. In short, about psi-computation.

- Dmitry Vostokov @ DumpAnalysis.org -

Finally previously announced full-color coffee table book is on sale:

DLL List Landscape: The Art from Computer Memory Space

Memory Dump Analysis Anthology, Volume 1 and Volume 2 have numerous articles related to computer memory visualization techniques using Dump2Picture and Microsoft debugger WinDbg.

I’m also working on Computer Memory Visualization book together with Jamie Fenton that was originally conceived as a coffee table book for general readers but now becomes more software engineering and science oriented the more I work on it. Coffee table status now comes to DLL List Landscape book.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash “must be distinguished from” hang “with which it is often confounded.”

Sydney Smith, A Memoir of the Reverend Sydney Smith by his daughter, Lady Holland, with a Selection from his Letters

- Dmitry Vostokov @ DumpAnalysis.org -

Here is the simplified example of executive resource wait chain:

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...

Resource @ 0x88094118    Exclusively owned
    Contention Count = 1461106
    NumberOfExclusiveWaiters = 172
     Threads: 87571600-01<*>
     Threads Waiting On Exclusive Access:
              87a0cd70       86e478b0       86d73270       87463908      
              86ed5020       872d3a08       87a0b228       87985020      
              870e4430       870adb00       88197500       86e06db0      
              87030db0       86d86db0       88a22288       86db07a0      
              86815c50       87524628       899d2020       86da03f0      
              86fc8db0       86e43b40       86d86ac8       87320690      
              86da2020       879c0108       86d8f7a8       86876370      
              87565150       87142020       879ddd30       86ff8990      
              86e5c770       867a7200       87a97c50       86e21020      
              86dac6e8       876d6db0       876fadb0       86e36408      
              86e621c8       8770adb0       86fd7c50       86db6ba8      
              86b87020       867ea2f8       870b60e8       889dc6d8      
              877ebae0       86e267a8       88a8a9f0       8737e5e8      
              86fc0780       87993c98       88aead28       872bedb0      
              899e5628       87523770       870aaaf0       8717b3b0      
              86e19db0       86e11db0       86e5a7a0       87038448      
              8743adb0       8816b9a0       880955f8       867f3db0      
              875c3430       8714a4f8       879b6020       87642598      
              86ec2b40       884a7c50       87200020       86880db0      
              86e2f988       866fb020       86ddfdb0       867c1bd8      
              86645020       868c0db0       87613db0       872b0020      
              88a56898       8770d9e0       8680b418       87014db0      
              865e0720       868c7af0       8733aaf0       86929508      
              8798f928       879cd378       8822ec50       8721adb0      
              876b25a0       87b5b598       8684baf0       86e48db0      
              86eb5b90       86d969a8       87039db0       87486020      
              86d8f3c0       8680edb0       86fddb88       885c2cb0      
              870ba890       86e2e4f8       8695b948       86e6fa28      
              88a42b88       86e58af0       86ddd2e0       8695b540      
              86817520       86975800       86817020       88b40b50      
              87271620       8695b2d0       867b44c8       880b6af8      
              870e1898       87c711e0       87a77210       8676bdb0      
              86734630       86878db0       86fd0c50       872a81f8      
              86e09020       880cf4f8       87178970       868a1508      
              870a9db0       8692c020       867a4020       868c9c50      
              890c74e0       8687c9a8       8692c4f8       880cf238      
              8708cac0       86ef5db0       86fa9db0       87158330      
              87979868       87a4f510       879a3510       87a1cdb0      
              87094020       87095db0       8705d2a8       87b0d5b0      
              870c0020       879eb660       8737e2e0       86ea7918      
              86e46a28       87a49198       87d61db0       87067db0      
              8730e598       86f97db0       8668d020       89d671b8      
              8732a5c8       89a00bb8       867fa020       86e2a020

KD: Scanning for held locks..

Resource @ 0x88aaabe8    Exclusively owned
    Contention Count = 97373
     Threads: 87178598-01<*>
KD: Scanning for held locks.

Resource @ 0x87712650    Exclusively owned
    Contention Count = 41716
    NumberOfExclusiveWaiters = 2
     Threads: 87178598-01<*>
     Threads Waiting On Exclusive Access:
              87571600       879f5648

KD: Scanning for held locks...

Resource @ 0x87736048    Exclusively owned
    Contention Count = 29109
    NumberOfExclusiveWaiters = 1
     Threads: 87ab30d0-01<*>
     Threads Waiting On Exclusive Access:
              87178598

KD: Scanning for held locks...

21056 total locks, 4 locks currently held

This is straight forward single wait chain (172 threads -> 87571600 -> 87178598 -> 87ab30d0) culminating in thread 87ab30d0 which loops in kernel mode (Spiking Thread):

0: kd> !thread 87ab30d0
THREAD 87ab30d0  Cid 3814.322c  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 3
Not impersonating
DeviceMap                 e1006e10
Owning Process            889d6d88       Image:         Application.exe
Wait Start TickCount      2518917        Ticks: 0
Context Switch Count      4057707            
UserTime                  00:00:00.000
KernelTime                01:26:13.906
*** WARNING: Unable to verify timestamp for driverA.sys
*** ERROR: Module load completed but symbols could not be loaded for driverA.dll
Start Address driverA (0xbfa1c930)
Stack Init ae8ec000 Current ae8eae7c Base ae8ec000 Limit ae8e9000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
f773d3b0 ae8eaf40 00000010 00000000 00000000 driverA+0×25880

- Dmitry Vostokov @ DumpAnalysis.org -

“It is better to offer no” fix “than a” wrong “one.”

George Washington, Letter to Harriet Washington

- Dmitry Vostokov @ DumpAnalysis.org -

Here we can look at one process dump with many patterns seen inside. Default WinDbg analysis command !analyze -v points to invalid handle exception perhaps at DLL initialization time during thread attach to DllA module:

STACK_TEXT: 
0296fa68 7c90eb93 ntdll!KiRaiseUserExceptionDispatcher+0x37
0296fa7c 10001252 ntdll!KiFastSystemCallRet+0x4
WARNING: Stack unwind information not available. Following frames may be wrong.
0296faa8 771215f8 DllA!DllMain+0×202
0296fbec 100014b0 OLEAUT32!DllMain+0×2c
0296fc0c 7c9011a7 DllA!DllMain+0×460
0296fc2c 7c918f65 ntdll!LdrpCallInitRoutine+0×14
0296fca0 7c918dde ntdll!LdrpInitializeThread+0xc0
0296fd18 7c90eac7 ntdll!_LdrpInitialize+0×219
00000000 00000000 ntdll!KiUserApcDispatcher+0×7

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c90eb74 (ntdll!KiRaiseUserExceptionDispatcher+0x00000037)
   ExceptionCode: c0000008 (Invalid handle)
  ExceptionFlags: 00000000
NumberParameters: 0
Thread tried to close a handle that was invalid or illegal to close

We may stop here after applying lmv command to DllA and recommending to upgrade / remove that component. But let’s look a bit deeper inside that crash dump. If we list all thread stacks (stack trace collection) we would see another thread with unhandled exception processing stack:

0:000> ~*kL

.  0  Id: a1c.e78 Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr 
0012da34 7c90e9ab ntdll!KiFastSystemCallRet
0012da38 7c86372c ntdll!ZwWaitForMultipleObjects+0xc
0012e1a8 77c32f0f kernel32!UnhandledExceptionFilter+0×8e4
0012e1c4 0041808b msvcrt!_XcptFilter+0×161
0012ffc0 7c816fd7 Application!WinMainCRTStartup+0×14f
0012fff0 00000000 kernel32!BaseProcessStart+0×23

   1  Id: a1c.2ec Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr 
02faff84 7c90e9ab ntdll!KiFastSystemCallRet
02faff88 5b890f8c ntdll!ZwWaitForMultipleObjects+0xc
02faffb4 7c80b683 NETAPI32!NetbiosWaiter+0x73
02faffec 00000000 kernel32!BaseThreadStart+0x37

   2  Id: a1c.c14 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr 
036afe1c 7c90e9ab ntdll!KiFastSystemCallRet
036afe20 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc
036afebc 7e4195f9 kernel32!WaitForMultipleObjectsEx+0x12c
036aff18 7e4196a8 USER32!RealMsgWaitForMultipleObjectsEx+0x13e
036aff34 00450d91 USER32!MsgWaitForMultipleObjects+0x1f
036aff80 77c3a3b0 Application!ThreadProc+0x61
036affb4 7c80b683 msvcrt!_endthreadex+0xa9
036affec 00000000 kernel32!BaseThreadStart+0x37

   3  Id: a1c.15c Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr 
0417ff78 7c90e31b ntdll!KiFastSystemCallRet
0417ff7c 71a5d320 ntdll!ZwRemoveIoCompletion+0xc
0417ffb4 7c80b683 mswsock!SockAsyncThread+0x5a
0417ffec 00000000 kernel32!BaseThreadStart+0x37

#  4  Id: a1c.96c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr 
0296fa68 7c90eb93 ntdll!KiRaiseUserExceptionDispatcher+0x37
0296fa7c 10001252 ntdll!KiFastSystemCallRet+0x4
WARNING: Stack unwind information not available. Following frames may be wrong.
0296faa8 771215f8 DllA!DllMain+0x202
0296fbec 100014b0 OLEAUT32!DllMain+0x2c
0296fc0c 7c9011a7 DllA!DllMain+0x460
0296fc2c 7c918f65 ntdll!LdrpCallInitRoutine+0x14
0296fca0 7c918dde ntdll!LdrpInitializeThread+0xc0
0296fd18 7c90eac7 ntdll!_LdrpInitialize+0x219
00000000 00000000 ntdll!KiUserApcDispatcher+0x7

Seems we have multiple exceptions here. Let’s extract thread 0 exception:

0:000> kv
ChildEBP RetAddr  Args to Child             
0012da34 7c90e9ab 7c86372c 00000002 0012dbac ntdll!KiFastSystemCallRet
0012da38 7c86372c 00000002 0012dbac 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0012e1a8 77c32f0f 0012e1f0 00000000 00000000 kernel32!UnhandledExceptionFilter+0×8e4
0012e1c4 0041808b 00000000 0012e1f0 77c35cf5 msvcrt!_XcptFilter+0×161
0012ffc0 7c816fd7 00160000 001ae3c6 7ffdd000 Application!WinMainCRTStartup+0×14f
0012fff0 00000000 00417f3c 00000000 78746341 kernel32!BaseProcessStart+0×23

0:000> .exptr 0012e1f0

----- Exception record at 0012e2e4:
ExceptionAddress: 77c47fd4 (msvcrt!wcslen+0x00000008)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 04649000
Attempt to read from address 04649000

----- Context record at 0012e300:
eax=04649000 ebx=00000000 ecx=0464006c edx=04648fb4 esi=04648fd0 edi=00000000
eip=77c47fd4 esp=0012e5cc ebp=0012e5cc iopl=0  nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00010206
msvcrt!wcslen+0x8:
77c47fd4 668b08          mov     cx,word ptr [eax]        ds:0023:04649000=????

0:000> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
0012e5cc 7301561a 04648fd0 00000030 00000018 msvcrt!wcslen+0×8
0012e5f0 73016c32 04648fd0 04afefe8 00000000 DllB!UnicodeToAnsiString+0×105
[…]

We see invalid pointer access violation while calculating string length. If we look at invalid address we see that UNICODE string crosses page boundary into a reserved page:

0:000> dd 04648fd0
04648fd0  0060004d 00620066 00680072 0020006f
04648fe0  00200034 00630022 007100ea 00710060
04648ff0  00200073 0060006e 0076006f 006d0066
04649000  ???????? ???????? ???????? ????????
04649010  ???????? ???????? ???????? ????????
04649020  ???????? ???????? ???????? ????????
04649030  ???????? ???????? ???????? ????????
04649040  ???????? ???????? ???????? ????????

0:000> !address 04648fd0
    04648000 : 04648000 - 00001000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageIsVAD

0:000> !address 04649000
    045e0000 : 04649000 - 00001000
                    Type     00040000 MEM_MAPPED
                    State    00002000 MEM_RESERVE
                    Usage    RegionUsageIsVAD

And we also notice full page heap enabled to catch possible heap corruption (dynamic memory corruption):

0:000> !gflag
Current NtGlobalFlag contents: 0x02000000
    hpa - Place heap allocations at ends of pages

This explains why we see invalid handle exception which is normally ignored by runtime unless we enable Application Verifier. Looking at DllB version data we see that it is the old component that needs to be upgraded.

- Dmitry Vostokov @ DumpAnalysis.org -

A short note: sometimes there are difficulties to explain the nature of software faults to not so technically oriented customers or casual inquirers. Here the language of software astrology can greatly help in assembling the right phrases and provide easily understandable analogies.

- Dmitry Vostokov @ DumpAnalysis.org -

“To achieve great” fixes “we must” debug “as though we were never going to” stop.

Luc de Clapiers, marquis de Vauvenargues, Reflexions and Maximes

- Dmitry Vostokov @ DumpAnalysis.org -

Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.

The Day of Larger Than Computation

Modules, products or systems built on December 2 have tremendous execution power. No matter how small their code they will exert an influence on their surrounding execution environment. Less evolved components built on that day can do great amount of damage to themselves and other modules. Computation is their God. When provoked by testing or debugging they are confrontational but not very aggressive. Often December 2 modules, products or systems see computation as a struggle where they must emerge as a victor. They are fighting not for their resources but for the certain basic computational values they were programmed for. Integrity is very important for them. The great challenge for December 2 components is to reconcile their computational individualism and their built-in computational paths. Often they stray from the latter. They constantly learn throughout their complex computational life what is true and what is false. Although December 2 modules, products or systems health is built-in they need regular yearly checkups with a software maintenance engineer otherwise small problems go too long without being found and fixed. Idle periods of activity are very important to their computational health. If they have a sibling component built on the same date they behave like subordinated to it.

DLL, SYS and EXE born on this date:

MSVCR80.dll Sat Dec 02 17:50:32 2006
rdbss.sys   Thu Dec 02 20:37:11 2004
Mup.sys     Thu Dec 02 20:37:23 2004
ftdisk.sys  Thu Dec 02 22:29:58 2004 
hal.dll     Thu Dec 02 22:29:15 2004

Weaknesses: Manipulative computation.

Strengths: Dynamic computation, lucid code, human orientation.

Advice: Watch your debugging temper. Regardless of what customers say, fixing bugs is not everything. Be self-assure, less judgemental and condemning to software. Acknowledge your debugging weaknesses and past mistakes.

- Dmitry Vostokov @ DumpAnalysis.org -

An interesting observation on how people perceive visualized computer memory where every byte, word or double word is interpreted as a pixel. The printing company initially rejected the interior of my DLL Art book containing pictures from process memory dumps because they thought that the art images were corrupt in PDF file I submitted. They accepted the book after I told them that images were normal and not corrupt. So I hope in one or two weeks the book will be in print.

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg.org has been updated to include a sorted command check list, a link to MSDN help and a link to yet another book related to WinDbg. All changes are highlighted in red on the following page screenshot:

- Dmitry Vostokov @ DumpAnalysis.org -

Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.

The Day of Debugging License

Modules, products or systems built on December 1 win customers over despite denying the rules of protocol. They can provide impression of simplicity but this is not the case. Their internals can be very complex and their perceived simplicity is the direct consequence of their user interface. Modules are not fully aware of what they are doing and seen as being driven by external components. Modules, products or systems built on this day are very busy with computation and have little time to care about users despite their built-in human-computer interaction. However they strive to calculate the impossible in all domains. They love to interact with other components with opposite behaviour. December 1 components are free modules and exert the full computation capabilities on the right data arrived at the right time. Working too many hours can seriously damage their internals and they may loose touch with their built-in goals. Sometimes December 1 modules, products or systems outrageous behaviour need to be amended to become more tolerable and not to hang. They need to be idle from time to time to avoid burn-out.

DLL, SYS and EXE born on this date:

VERSION.dll    Wed Dec 01 01:37:27 1999
nvcoaft51.sys  Wed Dec 01 11:55:40 2004
dump_m5289.sys Wed Dec 01 02:49:17 2004
CFGMGR32.DLL   Wed Dec 01 15:37:31 1999
MPRAPI.DLL     Wed Dec 01 15:37:29 1999
ICMP.DLL       Wed Dec 01 15:37:29 1999
RTUTILS.DLL    Wed Dec 01 15:37:27 1999

Weaknesses: Misdirected computation, unawareness of environment.

Strengths: Energetic computation, UI extroverts.

Advice: Keep a handle on your desire to debug. Beware of damaging other processes and alienating users with a overly direct debugging approach.

- Dmitry Vostokov @ DumpAnalysis.org -

This December starts with easy and obvious patterns that I forgot to write about. Integer division by zero is one of the most frequent exceptions. It is easily recognizable in process crash dumps by the processor instruction that caused this exception type (DIV or IDIV):

FAULTING_IP:
DLL!FindHighestID+278
1b2713c4 f775e4 div dword ptr [ebp-0×1c]

EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 1b2713c4 (DLL!FindHighestID+0x00000278)
ExceptionCode: c0000094 (Integer divide-by-zero)
ExceptionFlags: 00000000
NumberParameters: 0

or

FAULTING_IP:
Application+263d8
004263d8 f7fe idiv eax,esi

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 004263d8 (Application+0x000263d8)
ExceptionCode: c0000094 (Integer divide-by-zero)
ExceptionFlags: 00000000
NumberParameters: 0

ERROR_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION} Integer division by zero.

- Dmitry Vostokov @ DumpAnalysis.org -

A bit of digression on the first winter day in Ireland. What does it mean to be meanigful as astrology? It is meaningful as Tr O(Log y):

AsTrOLogy ↔ As Tr O(Log y) ↔ As O(Log y) ↔ bound (or dominated in case of small-o notation) by Log y after some y >= y₀ or bound (or dominated in case of small-o notation) by -logy. So Astrology is bounded and dominated by Y (Why?) lowered in magnitude by Log. Bounded and dominated by the this ultimate question…

Tr is trace operation in linear algebra. Tr (a number) = a number by definition of a 1×1 matrix
O is Big O notation

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Literate Scientist blog:

The Third Reich: A New History

Bird Flu: A Virus of Our Own Hatching

Discrete Mathematics 

This month was very busy for me as a publisher, an editor and a writer and I didn’t have much time to read. Hope in December I write more book reviews. 

- Dmitry Vostokov @ DumpAnalysis.org -

Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.

The Day of Measured Testing

Modules built on November 30 have a built-in capacity for  overcoming challenges of hostile environments. They are capable of bringing surprises to security attacks, for example. One can learn a lot about them by studying their traces or doing reverse engineering. November 30 components do their work to the utmost degree of quality with a little waste of CPU and memory. Message boxes they pop up have a subtle sense of thought-provoking humour but it can also be a full blown thigh-slapping. November 30 systems are very defensive when attacked. They are stubbornly resistant to reverse engineering but at the same time very open to honest debugging. 

DLL, SYS and EXE born on this date: 

tifsfilt.sys Tue Nov 30 07:16:27 2004
alrsvc.dll   Tue Nov 30 17:31:14 1999
ntkrpamp.exe Fri Nov 30 14:54:49 2007
Tppwrif.sys  Tue Nov 30 02:38:22 2004

Weaknesses: Over-reactive to code and data injection, funny behaviour.

Strengths: Thorough developed, dynamic responsiveness.

Advice: Improvise during troubleshooting and debugging. Admire control vs. spontaneity balance. Laugh at your failures.

- Dmitry Vostokov @ DumpAnalysis.org -