Archive for December 16th, 2008

Crash Dump Analysis AntiPatterns (Part 12)

Tuesday, December 16th, 2008

Sometimes a memory dump is not useful because of missing memory space, hence the name of this anti-pattern: Missing Space. This usually happens for non-bugcheck type issues like CPU spikes, application or session hangs where user space analysis is needed. Here is an example of Spiking Thread pattern in a kernel memory dump where analysis cannot be done because threads were spiking mostly in user space not included in kernel memory dumps:

Loading Dump File [MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

1: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

     Prcb      Current   Next   
  0  ffdff120  88d14b40            ................
  1  f7737120  891dddb0            ................

1: kd> !thread 88d14b40
THREAD 88d14b40  Cid 02b0.5298  Teb: 7ff7b000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    87e3ad50: (0006,0094) Flags: 00000900  Mdl: 00000000
    8a8485f8: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e1001878
Owning Process            8a9831a8       Image:         ApplicationA.exe
Wait Start TickCount      1282379        Ticks: 0
Context Switch Count      2950            
UserTime                  00:00:15.109
KernelTime                00:00:00.125
Win32 Start Address 0×77c7b0f5
Start Address 0×77e617ec
Stack Init ae302000 Current ae301c38 Base ae302000 Limit ae2ff000 Call 0
Priority 11 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
0269fd6c 00000000 00000000 00000000 00000000 0×77e61684

1: kd> !thread 891dddb0
THREAD 891dddb0  Cid 2de4.3f58  Teb: 7ffd8000 Win32Thread: bc487c18 RUNNING on processor 1
Not impersonating
DeviceMap                 e6aba8b0
Owning Process            88b50d88       Image:         ApplicationB.exe
Wait Start TickCount      1282379        Ticks: 0
Context Switch Count      46241                 LargeStack
UserTime                  00:00:35.296
KernelTime                00:00:06.656
Win32 Start Address 0×0b80e53c
Start Address 0×77e617ec
Stack Init b234c000 Current b234bd00 Base b234c000 Limit b2346000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
b234bcc8 f65c0532 000000e2 00000000 00000000 nt!KeBugCheckEx+0×1b
b234bcf8 f65bfd2c 00866d40 bbf000c6 00000000 i8042prt!I8xProcessCrashDump+0×256
b234bd40 808395a5 8a107c28 8a866c88 0101000a i8042prt!I8042KeyboardInterruptService+0×225
b234bd40 432de484 8a107c28 8a866c88 0101000a nt!KiInterruptDispatch+0×49
WARNING: Frame IP not in any known module. Following frames may be wrong.
01bbf150 00000000 00000000 00000000 00000000 0×432de484

The solution here is to request either a complete memory dump or user dumps of identified processes. The former was requested already but the crash dump type in Control Panel was not checked before forcing the dump of the system via standard keyboard method. Therefore this is also an instance of Wrong Dump anti-pattern.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in AntiPatterns, Crash Dump Analysis, Crash Dump Patterns, Debugging | 1 Comment »

The Measure of Debugging and Memory Dump Analysis Complexity

Tuesday, December 16th, 2008

Recently I was asked how to measure complexity of technical support cases especially ones that require memory dump analysis. My first response was that it is a subjective qualitative measure based mostly on experience and feeling. However after careful consideration I understood that nothing has changed for the last 5 years: the nature and causes of system or application hangs and crashes still the same regardless of OS types and versions. Therefore the complexity measure shifts from a case description and its artifacts to an analyst, a memory dump reader. Here the number of queries, questions asked or commands executed to gather information for analysis, can be a good approximation to the measure of complexity. For example, 5 years ago I started with a few commands like !analyze -v, kv and dd and progressed to an elaborate checklist. Here the natural logarithm can be used to approximate the measure:

C = ln (Ndc), where Ndc is the number of debugging commands used.

Initially the complexity was ln(3) ≈ 1.1 and now if someone uses 10 commands on average or asks 10 questions the complexity is ln(10) ≈ 2.3. The analysis is more than 2 times complex than it was.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Crash Dump Analysis, Debugging, Science of Memory Dump Analysis, Software Technical Support | No Comments »

Fun with CV

Tuesday, December 16th, 2008

I noticed that Amazon sells used copies of my Curriculum Vitae:

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Books, History | 1 Comment »