Software Diagnostics Library

Sometimes simple comparison of crash signatures is not enough to find similar support incidents. We then traverse stack trace frames to find Least Common Frame matching similar stack traces in a database. For example, consider this signature:

0:026> r
eax=011349ec ebx=01136738 ecx=79f943e1 edx=00000000 esi=011349ec edi=0888f3b8
eip=00dfbef8 esp=0888f348 ebp=0888f3c8 iopl=0  nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00000246
00dfbef8 3902  cmp     dword ptr [edx],eax  ds:0023:00000000=????????

0:026> k 100
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0888f3c8 792a842f 0xdfbef8
0888f3e4 792a839b mscorlib_ni+0x1e842f
0888f3fc 79e71b4c mscorlib_ni+0x1e839b
0888f40c 79e821b9 mscorwks!CallDescrWorker+0×33
0888f48c 79e8281f mscorwks!CallDescrWorkerWithHandler+0xa3
0888f4ac 79e82860 mscorwks!DispatchCallBody+0×1e
0888f510 79e828d1 mscorwks!DispatchCallDebuggerWrapper+0×3d
0888f544 79ec50f5 mscorwks!DispatchCallNoEH+0×51
0888f5a0 79e9848f mscorwks!AddTimerCallback_Worker+0×66
0888f5b4 79e9842b mscorwks!Thread::DoADCallBack+0×32a
0888f648 79e98351 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0888f684 79e984dd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0888f6ac 79ec4a84 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
0888f6c4 79ec5075 mscorwks!ManagedThreadBase::ThreadPool+0×13
0888f70c 79ec50a4 mscorwks!AddTimerCallbackEx+0×83
0888f720 79ec514a mscorwks!AddTimerCallback+0×10
0888f75c 79ec4e0c mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
0888f7a8 79ec471e mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
0888f7bc 79ec4892 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0888f814 79f75715 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
0888ffb4 7c80b729 mscorwks!Thread::intermediateThreadProc+0×49
0888ffec 00000000 kernel32!BaseThreadStart+0×37

Most likely we won’t find any similar stack trace when searching for 0xdfbef8. The search for mscorlib_ni+0×1e842f brings several results but they are not crashes but hangs with the frame being in the middle of call stack. The same is for mscorlib_ni+0×1e839b. So we finally try searching for CallDescrWorker+0×33 but with a limit to results having the same application module name. And indeed we find the similar software incident with the same stack trace after our least common frame:

0:004> r
eax=00000024 ebx=03e6f738 ecx=738129d8 edx=00495ef0 esi=01a87c4c edi=019c5f1c
eip=00a92037 esp=03e6f6cc ebp=03e6f6e8 iopl=0 nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00010206
00a92037 ??              ???

0:004> k 100
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
03e6f6c8 737d6bb5 0xa92037
03e6f6e8 737a509f mscorlib_ni+0x216bb5
03e6f6f8 737a834c mscorlib_ni+0x1e509f
03e6f70c 74171b6c mscorlib_ni+0x1e834c
03e6f71c 74182209 mscorwks!CallDescrWorker+0×33
03e6f79c 7418286f mscorwks!CallDescrWorkerWithHandler+0xa3
03e6f7bc 741828b0 mscorwks!DispatchCallBody+0×1e
03e6f820 74182921 mscorwks!DispatchCallDebuggerWrapper+0×3d
03e6f854 742ced79 mscorwks!DispatchCallNoEH+0×51
03e6f8b0 7419846f mscorwks!AddTimerCallback_Worker+0×66
03e6f8c4 7419840b mscorwks!Thread::DoADCallBack+0×32a
03e6f958 74198331 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
03e6f994 741984bd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
03e6f9bc 742ce708 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
03e6f9d4 742cecf9 mscorwks!ManagedThreadBase::ThreadPool+0×13
03e6fa1c 742ced28 mscorwks!AddTimerCallbackEx+0×83
03e6fa30 742cedce mscorwks!AddTimerCallback+0×10
03e6fa6c 742cea90 mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
03e6fab8 742ce3a2 mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
03e6facc 742ce516 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
03e6fb64 74441ec9 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
03e6fc84 76813677 mscorwks!Thread::intermediateThreadProc+0×49
03e6fc90 77219d72 kernel32!BaseThreadInitThunk+0xe
03e6fcd0 77219d45 ntdll!__RtlUserThreadStart+0×70
03e6fce8 00000000 ntdll!_RtlUserThreadStart+0×1b

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

HKLM\S\MountedDevices and basic disk volume partition offset (pp. 667 - 668)

General reparse points; symbolic links and mount points as their applications (p. 669)

Device object -> VPB, !vpb WinDbg command (p. 670) - here’s on my x64 W2K8 system:

0: kd> dt _DEVICE_OBJECT
ntdll!_DEVICE_OBJECT
+0x000 Type             : Int2B
+0x002 Size             : Uint2B
+0x004 ReferenceCount   : Int4B
+0x008 DriverObject     : Ptr64 _DRIVER_OBJECT
+0x010 NextDevice       : Ptr64 _DEVICE_OBJECT
+0x018 AttachedDevice   : Ptr64 _DEVICE_OBJECT
+0x020 CurrentIrp       : Ptr64 _IRP
+0x028 Timer            : Ptr64 _IO_TIMER
+0x030 Flags            : Uint4B
+0x034 Characteristics  : Uint4B
   +0×038 Vpb              : Ptr64 _VPB
+0×040 DeviceExtension  : Ptr64 Void
+0×048 DeviceType       : Uint4B
+0×04c StackSize        : Char
+0×050 Queue            : <unnamed-tag>
+0×098 AlignmentRequirement : Uint4B
+0×0a0 DeviceQueue      : _KDEVICE_QUEUE
+0×0c8 Dpc              : _KDPC
+0×108 ActiveThreadCount : Uint4B
+0×110 SecurityDescriptor : Ptr64 Void
+0×118 DeviceLock       : _KEVENT
+0×130 SectorSize       : Uint2B
+0×132 Spare1           : Uint2B
+0×138 DeviceObjectExtension : Ptr64 _DEVOBJ_EXTENSION
+0×140 Reserved         : Ptr64 Void

0: kd> dt _VPB
ntdll!_VPB
+0x000 Type             : Int2B
+0x002 Size             : Int2B
+0x004 Flags            : Uint2B
+0x006 VolumeLabelLength : Uint2B
+0x008 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x010 RealDevice       : Ptr64 _DEVICE_OBJECT
+0x018 SerialNumber     : Uint4B
+0x01c ReferenceCount   : Uint4B
+0x020 VolumeLabel      : [32] Wchar

FS -> Volume I/O (pp. 674 - 675) - we can also see driver stack from IRP I/O stack locations:

2: kd> !irp fffffa8017492b80
[...]
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
>[  4,34]  1c e0 fffffa800dfe2060 00000000 fffff88001186f30-00000000 Success Error Cancel
\Driver\Disk  partmgr!PmReadWriteCompletion
Args: 00001000 00000000 b99a9000 00000000
[  4, 0]  1c e0 fffffa800dfe2b90 00000000 fffff88001197180-fffffa800da89e20 Success Error Cancel
\Driver\partmgr     volmgr!VmpReadWriteCompletionRoutine
Args: 148ce8c5bed 00000000 b99a9000 00000000
[  4, 0]   c e0 fffffa800da89cd0 00000000 fffff88001968150-fffffa800dfe7190 Success Error Cancel
\Driver\volmgr      volsnap!VspRefCountCompletionRoutine
Args: 00001000 00000000 148ce8c5be9 00000000
[  4, 0]   c e1 fffffa800dfe7040 00000000 fffff88001a464f4-fffff88002777a10 Success Error Cancel pending
\Driver\volsnap     Ntfs!NtfsMasterIrpSyncCompletionRoutine
Args: 00001000 00000000 b996a000 00000000
[  4, 0]   0  0 fffffa800dfed030 fffffa800da958e0 00000000-00000000
\FileSystem\Ntfs
Args: 00001000 00000000 01afc000 00000000
[…]

BitLocker architecture diagram (p.678) - parts can be seen from IRP I/O stack locations:

 kd> !irp 85e7ee00
[...]
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
>[  3,34]  10 e0 857b9030 00000000 8353724e-00000000 Success Error Cancel
\Driver\Disk     partmgr!PmReadWriteCompletion
Args: 00001000 00000000 400d6000 00000000
[  3, 0]  10  0 857b9d18 00000000 00000000-00000000
\Driver\partmgr
Args: 6bad71d7 00000000 400d6000 00000000
[  3, 0]  10 e0 8478b5f0 00000000 835487a4-857bc2f0 Success Error Cancel
\Driver\DriverA   volmgr!VmpReadWriteCompletionRoutine
Args: 00001000 00000000 400d6000 00000000
[  3, 0]   0 e0 857bc238 00000000 872c83e2-857bfb70 Success Error Cancel
\Driver\volmgr   fvevol!FvePassThroughCompletion
Args: 00001000 00000000 6bad70ba 00000000
[  3, 0]   0 e0 857bfab8 00000000 8709807a-859a2118 Success Error Cancel
\Driver\fvevol   Ntfs!NtfsMasterIrpAsyncCompletionRoutine
Args: 00001000 00000000 40097000 00000000
[  3, 0]   0  1 857e2020 8584ca40 00000000-00000000    pending
\FileSystem\Ntfs
Args: 00001000 00000000 0329e000 00000000
[…]

VMK -> FVEK: possibility for rekeying (p. 679)

Maximum protection: TPM+USB+PIN (p. 679)

Diffuser to protect from manipulations with AES-encrypted ciphertext (p. 681)

Because the number of software trace patterns is growing I’m starting another checklist in addition to memory dump analysis checklist. The goal is to help experienced engineers not to miss any important information. The checklist doesn’t prescribe any specific steps, just lists all possible points to double check when looking at a software trace. Of course, it is not complete at the moment and any suggestions are welcome. This post will be modified on the ongoing basis.

General:

• Check overall trace time delta
• Check no trace metafile message density
• Check whether a trace is a multi-part or a circular
• Check for basic facts and the story (software narrative)
• Check for any exceptions, non-false positive errors and periodic errors
• Check for significant events
• Check for discontinuities in the time domain
• Check for message current and acceleration in the frequency domain

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The last common mistake in this set is when an engineer doesn’t compare stack traces or other debugger output to normal reference stack traces. For example, a 3rd party service was hanging on a W2K8 server and a user memory dump was saved for offline analysis. The following thread was identified as blocked waiting for a response:

STACK_TEXT: 
0041f1bc 75a30816 ntdll_77e50000!ZwWaitForSingleObject+0x15
0041f228 76671184 KERNELBASE!WaitForSingleObjectEx+0x98
0041f240 76671138 kernel32!WaitForSingleObjectExImplementation+0x75
0041f254 75b17be6 kernel32!WaitForSingleObject+0x12
0041f2f8 75b18040 sechost!ScSendResponseReceiveControls+0xea
0041f3ac 75b18662 sechost!ScDispatcherLoop+0xc2
0041f3ec 01271bb4 sechost!StartServiceCtrlDispatcherW+0xb0
0041fa7c 01271dd6 ServiceA!WinMain+0×254
0041fb0c 76673677 ServiceA!__tmainCRTStartup+0×160
0041fb18 77e89d42 kernel32!BaseThreadInitThunk+0xe
0041fb58 77e89d15 ntdll_77e50000!__RtlUserThreadStart+0×70
0041fb70 00000000 ntdll_77e50000!_RtlUserThreadStart+0×1b

Unfortunately, this thread wasn’t recognized as a normal main service thread. Typical Internet search for ScSendResponseReceiveControls function points to a sample analysis log where we can find such thread stacks in the variety of other standard services:

THREAD fffffa8005362060  Cid 0a1c.0b68  Teb: 000007fffffde000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8004dc0a60  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a000008c10
Owning Process            fffffa800540e060       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      37051          Ticks: 3239 (0:00:00:50.528)
Context Switch Count      13            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address svchost!wmainCRTStartup (0x00000000ffa9246c)
Stack Init fffff88005abbdb0 Current fffff88005abb900
Base fffff88005abc000 Limit fffff88005ab6000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`05abb940 fffff800`01a93992 nt!KiSwapContext+0x7a
fffff880`05abba80 fffff800`01a95cff nt!KiCommitThreadWait+0x1d2
fffff880`05abbb10 fffff800`01d871d2 nt!KeWaitForSingleObject+0x19f
fffff880`05abbbb0 fffff800`01a8b993 nt!NtWaitForSingleObject+0xb2
fffff880`05abbc20 00000000`7781fefa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05abbc20)
00000000`001ffab8 000007fe`fda910ac ntdll!NtWaitForSingleObject+0xa
00000000`001ffac0 000007fe`ff4eaffb KERNELBASE!WaitForSingleObjectEx+0x79
00000000`001ffb60 000007fe`ff4e9d61 sechost!ScSendResponseReceiveControls+0×13b
00000000`001ffc50 000007fe`ff4e9c16 sechost!ScDispatcherLoop+0×121
00000000`001ffd60 00000000`ffa91d3a sechost!StartServiceCtrlDispatcherW+0×14e
00000000`001ffdb0 00000000`ffa9257a svchost!wmain+0×110
00000000`001ffde0 00000000`776cf56d svchost!ScCreateWellKnownSids+0×2fd
00000000`001ffe20 00000000`77803281 kernel32!BaseThreadInitThunk+0xd
00000000`001ffe50 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Also, studying OS architecture and deliberate practice in memory dump analysis helps in recognition of problem and normal structural and behavioral patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memiosphere is the sphere of operating memory processes where memory signs (studied by memiotics) are created, manipulated and destroyed. This new word was inspired by the similar notion of semiosphere. We can also metaphorically analyze and remember it as mem(ory) + I/O + sphere.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Church of Memory Analysis acquires its first symbol as a cross with 4 pointers and a background natural memory visualization image extracted from one of computer memory dumps:

Dmitry Vostokov
Memoriarch

- Dmitry Vostokov @ Memory Religion Portal -

Similar to Buddhism view of a person as a transient assemblage of physical and psychical elements Memorianity considers a person as an assemblage of memories always in flux. To be saved correctly also means working with memories to overcome their original defects and reorder chains of memory perspectives. This is similar to karmic laws and past actions that determine the nature of rebirth. See also on Morality and Memorianity.

Why it is so difficult to restore everything that was saved? Here André-Marie Ampère’s notion of dynamic resistance when we try to penetrate matter is similar to memuonic resistance when we try to penetrate Memory.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first Webinar to start an in-depth discussion of pattern-driven software troubleshooting, debugging and maintenance:

Date: 25th of March 2011
Time: 18:30 (GMT) 14:30 (EST) 11:30 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/448268158

Topics include:

• A Short History of DumpAnalysis.org
• Memory Dump Analysis Patterns
• Troubleshooting and Debugging Tools (Debugware) Patterns
• Software Trace Analysis Patterns
• From Software Defects to Software Behavior
• Workaround Patterns
• Structural Memory Patterns
• Memory Analysis Domain Pattern Hierarchy
• New Directions

Prerequisites: experience in software troubleshooting and/or debugging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In certain software behavior scenarios such as a memory leak when we see top modules calling OS API functions we might suspect them having defects. However, this might not be the case and these modules were used from Directing Module  keeping references or handles preventing top modules from freeing memory or releasing resources.

For example, a memory dump from a process had 2 growing heap segments and one of them had this recurrent stack trace saved in a user mode stack trace database:

38D2CE78: 02ba8 . 02ba8 [07] - busy (2b90), tail fill
Stack trace (38101) at 83e390:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F
337d0572: ModuleA!XHeapAlloc+0×00000115
[…]
338809e2: ModuleA!Execute+0×000002CD
488b3fc1: ModuleB!Execute+0×000000D3
679b8c64: ModuleC!ExecuteByHandle+0×00000074
[…]
67d241cb: ModuleD!Query+0×0000016B
67ba2ed4: ModuleE!Browse+0×000000E4
[…]
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406
[…]

Initially we suspected ModuleA but found a different recurrent stack trace corresponding to another growing segment:

40C81688: 000c8 . 00058 [07] - busy (40), tail fill
Stack trace (38136) at 83f6a4:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F
7c3416b3: msvcr71!_heap_alloc+0×000000E0
7c3416db: msvcr71!_nh_malloc+0×00000010
67745875: ModuleX!BufAllocate+0×00000015
6775085e: ModuleY!QueryAttribute+0×0000008E
[…]
677502b5: ModuleY!Query+0×00000015
67ba2f19: ModuleE!Browser+0×00000129
[…]
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406
[…]

From the common stack trace fragment (highlighted in blue) we transferred our investigation to ModuleE and indeed the similar software incident (as the latter trace) was found in our troubleshooting database.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another name for Memorianity (Memory Religion) that incorporates M->analysis techniques. I’m working now on the full statement of creed to be published soon. May you be memorized.

Dmitry Vostokov
Memoriarch

- Dmitry Vostokov @ Memory Religion Portal -

Memory->analysis or M->analysis in short is a new pattern-driven psychotherapeutic method based on Unified Memory Theory (another name for Memory Worldview). It has a logo as well:

The name was inspired by M-theory and unified field theories. Please also see my interpretation of M-theory. Another name variant is M-analysis which can be used as a shortcut to M->analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of stack overflow is necessary, so I created this post:

• Stack Overflow (kernel mode)
• Stack Overflow (user mode)
• Stack Overflow (software implementation)
• Stack Overflow (insufficient memory)
• Stack Overflow (managed space)

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Stack Overflow pattern variants in user and kernel mode are ISA (Instruction Set Architecture) and processor architecture oriented. Another pattern variant is software stack implementations where push and pop operations check  stack ADT preconditions and throw a software exception (overflow or underflow) or call an assertion mechanism to display an error message. For the latter example, we look at a bugcheck for the specific stack implementation on Windows: IRP stack locations array. For a graphical reminder on how driver-to-driver communication is implemented by an IRP stack corresponding to a driver stack please refer to UML diagram no. 3 in the old post about using UML for describing device driver design. The following WinDbg command output is from a kernel memory dump:

0: kd> !analyze -v
[...]
NO_MORE_IRP_STACK_LOCATIONS (35)
A higher level driver has attempted to call a lower level driver through the IoCallDriver() interface, but there are no more stack locations in the packet, hence, the lower level driver would not be able to access its parameters, as there are no parameters for it. This is a disasterous situation, since the higher level driver "thinks" it has filled in the parameters for the lower level driver (something it MUST do before it calls it), but since there is no stack location for the latter driver, the former has written off of the end of the packet.  This means that some other memory has probably been trashed at this point.
Arguments:
Arg1: fffffa800500c9e0, Address of the IRP
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
[…]

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffff880`01fe2338 fffff800`016d7732 nt!KeBugCheckEx
fffff880`01fe2340 fffff800`01754f27 nt!KiBugCheck3+0x12
fffff880`01fe2370 fffff880`0177e271 nt! ?? ::FNODOBFM::`string’+0×3f31b
fffff880`01fe23a0 fffff880`0177c138 DriverA!CallProvider+0×161
[…]
fffff880`01fe2cb0 fffff800`0197a7c6 nt!ExpWorkerThread+0×111
fffff880`01fe2d40 fffff800`016b5c26 nt!PspSystemThreadStartup+0×5a
fffff880`01fe2d80 00000000`00000000 nt!KxStartSystemThread+0×16

0: kd> !irp fffffa800500c9e0
Irp is active with 1 stacks 0 is current (= 0xfffffa8006c2e960)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  4, 0]   0 e0 fffffa8004045c50 fffffa8006c2e960 fffff88005a04460-fffffa8005b9c370 Success Error Cancel
        \DriverA DriverB!CompleteRoutine
   Args: 00000008 00000000 00000000 00000000

0: kd> ub fffff880`0177e271
DriverA!CallProvider+0×13e:
fffff880`0177e24e mov     qword ptr [r11-10h],rax
fffff880`0177e252 mov     qword ptr [r11-8],r12
fffff880`0177e256 mov     byte ptr [r11-45h],0E0h
fffff880`0177e25b mov     rcx,qword ptr [rdi+40h]
fffff880`0177e25f call    qword ptr [DriverA!_imp_IoGetAttachedDevice (fffff880`017790b0)]
fffff880`0177e265 mov     rdx,rbp
fffff880`0177e268 mov     rcx,rax
fffff880`0177e26b call    qword ptr [DriverA!_imp_IofCallDriver (fffff880`01779068)]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Learning from Philip Marlowe, a detective:

“I like you,” she said suddenly. “You believe in miracles.”

Raymond Chandler, The Big Sleep

Do you believe in miracles from a driver modifying an arbitrary user space? Or in a miracle of suddenly disappearing software incidents?

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Hardware Error pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

The distinction between class, port and miniport components in storage stack (pp. 646 - 647)

Example: disk.sys as a class driver, ataport.sys and atapi.sys as port and miniport drivers (pp. 647 - 448)

MPIO (multi path I/O), DSM (device-specific modules) and storage stack (pp. 649 - 650)

Old and new naming convention (DRX) for disk device objects (p. 650)

Win32 API disk drive naming (p. 651)

Partition device objects (p. 652)

Volume manager as a bus driver (p. 655)

System vs. boot volume (p. 660)

Volmgr.sys vs. Volmgrx.sys (p. 661)

The advantages of storing volume metadata in a file (p. 662)

Spanned, striped (RAID-0), mirrored (RAID-1), RAID-5 (striped with rotated parity) (pp. 662 - 667)

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Differences between driver and service loading (p. 623)

Tag value precedence redefinition (p. 624)

Verbose !devnode command options (pp. 627 - 628)

DID=VID.PID and DIID=DID.IID (p. 630)

Hybrid sleep (pp. 637-638)

Power dispatch routine (p. 639) - Here’s a dispatch routine for a PCI driver from my x64 W2K8R2 system:

0: kd> !devnode 0 3
Dumping IopRootDeviceNode (= 0xfffffa8003c1ed90)
DevNode 0xfffffa8003c1ed90 for PDO 0xfffffa8003c1db10
InstancePath is "HTREE\ROOT\0"
State = DeviceNodeStarted (0x308)
Previous State = DeviceNodeEnumerateCompletion (0x30d)

[...]

        DevNode 0xfffffa8003e91b10 for PDO 0xfffffa8003e40a20
InstancePath is “PCI\VEN_8086&DEV_2810&SUBSYS_00000000&REV_02\3&172e68dd&0&F8″
ServiceName is “msisadrv”
State = DeviceNodeStarted (0×308)
Previous State = DeviceNodeEnumerateCompletion (0×30d)

[...]

0: kd> !devobj 0xfffffa8003e40a20
Device object (fffffa8003e40a20) is for:
NTPNP_PCI0013 \Driver\pci DriverObject fffffa8003cfe270
Current Irp 00000000 RefCount 0 Type 00000022 Flags 00001040
Dacl fffff9a10008b231 DevExt fffffa8003e40b70 DevObjExt fffffa8003e40f90 DevNode fffffa8003e91b10
ExtensionFlags (0×00000800)
Unknown flags 0×00000800
AttachedDevice (Upper) fffffa8003e3f800
\Driver\ACPI
Device queue is not busy.

0: kd> !drvobj fffffa8003cfe270 f
Driver object (fffffa8003cfe270) is for:
\Driver\pci
Driver Extension List: (id , addr)

Device Object list:
fffffa8003e9da20  fffffa8003e9a060  fffffa8003e99a20  fffffa8003e939f0
fffffa8003e93040  fffffa8003e92660  fffffa8003e92cb0  fffffa8003e42060
fffffa8003e41a20  fffffa8003e41060  fffffa8003e40a20  fffffa8003e40060
fffffa8003e3fa20  fffffa8003e3f060  fffffa8003e3ea20  fffffa8003e3e060
fffffa8003e3da20  fffffa8003e3d060  fffffa8003e3ca20  fffffa8003e3c060
fffffa8003e3ba20  fffffa8003e3b060  fffffa8003e3aa20  fffffa8003e3a060
fffffa8003e37530

DriverEntry:   fffff880013ae1a0 pci!GsDriverEntry
DriverStartIo: 00000000�
DriverUnload:  fffff880013a2fec pci!PciDriverUnload
AddDevice:     fffff8800139ae54 pci!PciAddDevice

Dispatch routines:
[00] IRP_MJ_CREATE                      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[01] IRP_MJ_CREATE_NAMED_PIPE           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[02] IRP_MJ_CLOSE                       fffff80001ab5cfc nt!IopInvalidDeviceRequest
[03] IRP_MJ_READ                        fffff80001ab5cfc nt!IopInvalidDeviceRequest
[04] IRP_MJ_WRITE                       fffff80001ab5cfc nt!IopInvalidDeviceRequest
[05] IRP_MJ_QUERY_INFORMATION           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[06] IRP_MJ_SET_INFORMATION             fffff80001ab5cfc nt!IopInvalidDeviceRequest
[07] IRP_MJ_QUERY_EA                    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[08] IRP_MJ_SET_EA                      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[09] IRP_MJ_FLUSH_BUFFERS               fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0a] IRP_MJ_QUERY_VOLUME_INFORMATION    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0b] IRP_MJ_SET_VOLUME_INFORMATION      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0c] IRP_MJ_DIRECTORY_CONTROL           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0d] IRP_MJ_FILE_SYSTEM_CONTROL         fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0e] IRP_MJ_DEVICE_CONTROL              fffff8800139e6d0 pci!PciDispatchDeviceControl
[0f] IRP_MJ_INTERNAL_DEVICE_CONTROL     fffff80001ab5cfc nt!IopInvalidDeviceRequest
[10] IRP_MJ_SHUTDOWN                    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[11] IRP_MJ_LOCK_CONTROL                fffff80001ab5cfc nt!IopInvalidDeviceRequest
[12] IRP_MJ_CLEANUP                     fffff80001ab5cfc nt!IopInvalidDeviceRequest
[13] IRP_MJ_CREATE_MAILSLOT             fffff80001ab5cfc nt!IopInvalidDeviceRequest
[14] IRP_MJ_QUERY_SECURITY              fffff80001ab5cfc nt!IopInvalidDeviceRequest
[15] IRP_MJ_SET_SECURITY                fffff80001ab5cfc nt!IopInvalidDeviceRequest
[16] IRP_MJ_POWER                       fffff880013848fc pci!PciDispatchPnpPower
[17] IRP_MJ_SYSTEM_CONTROL              fffff8800139e66c pci!PciDispatchSystemControl
[18] IRP_MJ_DEVICE_CHANGE               fffff80001ab5cfc nt!IopInvalidDeviceRequest
[19] IRP_MJ_QUERY_QUOTA                 fffff80001ab5cfc nt!IopInvalidDeviceRequest
[1a] IRP_MJ_SET_QUOTA                   fffff80001ab5cfc nt!IopInvalidDeviceRequest
[1b] IRP_MJ_PNP                         fffff880013848fc pci!PciDispatchPnpPower

!pocaps and !popolicy WinDbg commands (pp. 641 - 643)

Unlike other PnP operations like normal eject power cannot be vetoed by drivers and apps (pp. 643 - 644)

Today we introduce an icon for Wild Code pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

With Windows moving to ARM platforms I’m learning ARM assembly language and system programming. Being mostly exposed to PDP-11, VAX-11 and Intel architectures my interest in ARM systems began with porting my old project to Windows Mobile and writing a tool called WindowHistory Mobile to assist me in debugging. I remember how I was surprised when WinDbg was able to open a crash dump and show ARM code! Then I put it aside. Now with Windows Mobile 7 and Windows 8 approaching with its support of ARM I decided to resume my ARM studies and highly recommend this book:

ARM System Developer’s Guide: Designing and Optimizing System Software

My renewed interest in assembly language also coincides with joining Linkedin Assembly language coders group.

I also plan to create a separate version of my popular Windows Debugging: Practical Foundations book with ARM examples and an ARM section on Asmpedia. If time and resources permit, of course.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, we look at a trace and say it’s Impossible Trace. For example, this fragment shows that the function foo had been called:

#     Module  PID  TID  Message
[...]
1001  ModuleA 202  404  foo: start
1002  ModuleA 202  404  foo: end
[...]

However, if we look at the corresponding source code (PLOT) we would see that something is missing: the function bar must have been called with its own set of trace messages we don’t see in the trace:

void foo()
{
    TRACE("foo: start");
    bar();  
    TRACE("foo: end");
}

void bar()
{
    TRACE("bar: start");
    // some code ... 
    TRACE("bar: end");
}

We suspect the runtime code being modified, perhaps by patching. In other cases of missing messages we can also suspect thrown exceptions or local buffer overflows that led to wrong return address skipping the code with expected tracing statements. The mismatch between the trace and the source code we are looking at is also possible if the old source code didn’t have bar function called.

Note: I’m grateful for this pattern idea to Gary Barton.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -