Software Diagnostics Library

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Scatter/gather (p. 566) - you can find examples of scatter/gather I/O residues left on a thread raw stack in Hardware Activity pattern and corresponding case study:

http://www.dumpanalysis.org/blog/index.php/2010/05/08/crash-dump-analysis-patterns-part-98/ 

and

http://www.dumpanalysis.org/blog/index.php/2010/06/07/irp-distribution-anomaly-inconsistent-dump-execution-residue-hardware-activity-coincidental-symbolic-information-not-my-version-virtualized-system-pattern-cooperation/

IRP (pp. 566 - 567) - here is an expanded IRP structure from x64 W2K8:

0: kd> dt -r1 _IRP
ntdll!_IRP
+0x000 Type             : Int2B
+0x002 Size             : Uint2B
+0x008 MdlAddress       : Ptr64 _MDL
+0x000 Next             : Ptr64 _MDL
+0x008 Size             : Int2B
+0x00a MdlFlags         : Int2B
+0x010 Process          : Ptr64 _EPROCESS
+0x018 MappedSystemVa   : Ptr64 Void
+0x020 StartVa          : Ptr64 Void
+0x028 ByteCount        : Uint4B
+0x02c ByteOffset       : Uint4B
+0x010 Flags            : Uint4B
+0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
+0x000 SystemBuffer     : Ptr64 Void
+0x020 ThreadListEntry  : _LIST_ENTRY
+0x000 Flink            : Ptr64 _LIST_ENTRY
+0x008 Blink            : Ptr64 _LIST_ENTRY
+0x030 IoStatus         : _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B
+0x040 RequestorMode    : Char
+0x041 PendingReturned  : UChar
+0x042 StackCount       : Char
+0x043 CurrentLocation  : Char
+0x044 Cancel           : UChar
+0x045 CancelIrql       : UChar
+0x046 ApcEnvironment   : Char
+0x047 AllocationFlags  : UChar
+0x048 UserIosb         : Ptr64 _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B
+0x050 UserEvent        : Ptr64 _KEVENT
+0x000 Header           : _DISPATCHER_HEADER
+0x058 Overlay          : <unnamed-tag>
+0x000 AsynchronousParameters : <unnamed-tag>
+0x000 AllocationSize   : _LARGE_INTEGER
+0x068 CancelRoutine    : Ptr64     void
+0x070 UserBuffer       : Ptr64 Void
+0x078 Tail             : <unnamed-tag>
+0x000 Overlay          : <unnamed-tag>
+0x000 Apc              : _KAPC
+0x000 CompletionKey    : Ptr64 Void

IRP stack locations (pp. 568 - 569) - here is a corresponding structure from x64 W2K8:

0: kd> dt _IO_STACK_LOCATION
ntdll!_IO_STACK_LOCATION
+0x000 MajorFunction    : UChar
+0x001 MinorFunction    : UChar
+0x002 Flags            : UChar
+0x003 Control          : UChar
+0x008 Parameters       : <unnamed-tag>
+0x028 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x030 FileObject       : Ptr64 _FILE_OBJECT
+0x038 CompletionRoutine : Ptr64     long
+0x040 Context          : Ptr64 Void 

Buffered I/O (p. 570) - this part of IRP references a buffer (user input data is copied there and device output is copied there):

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
      +0×000 SystemBuffer     : Ptr64 Void

These parts of I/O stack location structure handle buffer lengths:

      +0x000 DeviceIoControl  : <unnamed-tag>
         +0×000 OutputBufferLength : Uint4B
+0×008 InputBufferLength : Uint4B
+0×010 IoControlCode    : Uint4B
+0×018 Type3InputBuffer : Ptr64 Void

      +0x000 Read             : <unnamed-tag>
         +0×000 Length           : Uint4B
+0×008 Key              : Uint4B
+0×010 ByteOffset       : _LARGE_INTEGER

      +0x000 Write            : <unnamed-tag>
         +0×000 Length           : Uint4B
+0×008 Key              : Uint4B
+0×010 ByteOffset       : _LARGE_INTEGER

Direct I/O (p. 570) - these parts of IRP handle IOCTL input data (SystemBuffer, via buffering) and IOCTL output/Read/Write data (MdlAddress):

   +0x008 MdlAddress       : Ptr64 _MDL
+0x000 Next             : Ptr64 _MDL
+0x008 Size             : Int2B
+0x00a MdlFlags         : Int2B
+0x010 Process          : Ptr64 _EPROCESS
+0x018 MappedSystemVa   : Ptr64 Void
+0x020 StartVa          : Ptr64 Void
+0x028 ByteCount        : Uint4B
+0x02c ByteOffset       : Uint4B

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
+0x000 SystemBuffer     : Ptr64 Void

Neither I/O (p. 571) - these parts handle input data (IO_STACK_LOCATION.Parameters.DeviceIoControl.Type3InputBuffer) and output data (IRP.UserBuffer):

      +0x000 DeviceIoControl  : <unnamed-tag>
+0x000 OutputBufferLength : Uint4B
+0x008 InputBufferLength : Uint4B
+0x010 IoControlCode    : Uint4B
         +0×018 Type3InputBuffer : Ptr64 Void

   +0×070 UserBuffer       : Ptr64 Void

I/O status block and kernel APC (pp. 575 - 577) - this is a part of IRP  structure:

   +0x030 IoStatus         : _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B

KeSynchronizeExecution (p. 578) - here is a stack trace fragment showing it in action:

[...]
b9ada518 8088d661 SCSIPORT!SpStartIoSynchronized+0x14f
b9ada550 80a60147 nt!KeSynchronizeExecution+0×21
b9ada57c f72523a6 hal!HalBuildScatterGatherList+0×1c7
b9ada5c8 8081cfa2 SCSIPORT!ScsiPortStartIo+0×36a
b9ada5ec f725262f nt!IoStartPacket+0×82
b9ada620 f7252146 SCSIPORT!ScsiPortFdoDispatch+0×270
b9ada63c f7251dc3 SCSIPORT!SpDispatchRequest+0×68
b9ada658 f7251299 SCSIPORT!ScsiPortPdoScsi+0×129
b9ada66c 8081df85 SCSIPORT!ScsiPortGlobalDispatch+0×1d
b9ada680 f723e607 nt!IofCallDriver+0×45
b9ada690 f723e2b2 CLASSPNP!SubmitTransferPacket+0xbb
b9ada6c4 f723e533 CLASSPNP!ServiceTransferRequest+0×1e4
b9ada6e8 8081df85 CLASSPNP!ClassReadWrite+0×159
b9ada6fc f74c80cf nt!IofCallDriver+0×45
b9ada70c 8081df85 PartMgr!PmReadWrite+0×95
b9ada720 f7317053 nt!IofCallDriver+0×45
b9ada73c 8081df85 ftdisk!FtDiskReadWrite+0×1a9
b9ada750 f72bf8bc nt!IofCallDriver+0×45
b9ada768 8081df85 volsnap!VolSnapRead+0×52
b9ada77c f7163a62 nt!IofCallDriver+0×45
b9ada788 f71638d9 Ntfs!NtfsSingleAsync+0×91
b9ada960 f7164156 Ntfs!NtfsNonCachedIo+0×2db
b9adaa4c f7164079 Ntfs!NtfsCommonRead+0xaf5
b9adabf8 8081df85 Ntfs!NtfsFsdRead+0×113
b9adac0c f721cc45 nt!IofCallDriver+0×45
b9adac34 8081df85 fltmgr!FltpDispatch+0×6f
b9adac48 bafd5373 nt!IofCallDriver+0×45
[…]

IRP and layered drivers (pp. 578 - 586) - here’s a UML-style diagram (#3) for IRP flow:

http://www.dumpanalysis.org/blog/index.php/2006/10/08/uml-and-device-drivers/ 

Associated IRP (pp. 585 - 586) - this is a part of IRP structure:

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP

File object vs. thread IRP association (p. 587)

Thread Termination and pending IRP (pp. 589 - 590) - this pattern uses I/O cancellation as an example:

http://www.dumpanalysis.org/blog/index.php/2007/12/14/crash-dump-analysis-patterns-part-42a/ 

Crash Dump is a double buzzword.

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes a software trace is truncated when the trace session was stopped prematurely, often when a problem didn’t manifest itself visually. We can diagnose such traces by their short time duration, missing anchor messages or components necessary for analysis. My favourite example is user session initialization in a Citrix terminal services environment when problem effects are visible only after the session is fully initialized and an application is launched but a truncated CDF trace only shows the launch of winlogon.exe despite the presence of a process creation trace provider or other components that record the process launch sequence and the trace itself lasts only a few seconds after that.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

assert (*p + *p == 2 * (*p) + 1); // for sufficiently fast processors

The reinterpretation of 1 + 1 = 3, Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Coupled Processes (semantics) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A few days ago I was in a hotel bar invited to celebrate an event. Later that night we were trying to sing songs and I came up with a few stanzas. Today I finished the composition:

Solution Number One.
Bang, Bang, Bang…

Solution Number Two.
Poo, Poo, Poo…

Solution Number Three.
Wee, Wee, Wee…

Solution Number Four.
Oh, Oh, Oh…

Solution Number Five.
Ay, Ay, Ay…

Solution Number Six.
Fix, Fix, Fix!

I’ll try to add some music later on…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The one obvious pattern that is shown in many case studies on this blog is Exception Stack Trace (or Exception Thread). This is a stack trace that has exception processing functions, for example:

   9  Id: 1df4.a08 Suspend: -1 Teb: 7fff4000 Unfrozen
ChildEBP RetAddr 
1022f5a8 7c90df4a ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 kernel32!UnhandledExceptionFilter+0×8b9
1022f908 7c839b39 kernel32!BaseThreadStart+0×4d
1022f930 7c9032a8 kernel32!_except_handler3+0×61
1022f954 7c90327a ntdll!ExecuteHandler2+0×26
1022fa04 7c90e48a ntdll!ExecuteHandler+0×24
1022fa04 7c812afb ntdll!KiUserExceptionDispatcher+0xe
1022fd5c 0b82e680 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0×21e640
1022fde8 7753004f DllA+0×21d4f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0×3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0×70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0×36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0×9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0×25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0×1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×1b
1022ffec 00000000 kernel32!BaseThreadStart+0×37

Such exceptions can be detected by the default analysis command (for example, !analyze -v WinDbg command) or by inspecting a stack trace collection. However if we didn’t see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack data. 

In our case we can get the exception information by looking at parameters to in unhandled exception filter:

0:009> kv 3
ChildEBP RetAddr Args to Child
1022f5a8 7c90df4a 7c8648a2 00000002 1022f730 ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 00000002 1022f730 00000001 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 1022f928 7c839b39 1022f930 kernel32!UnhandledExceptionFilter+0×8b9

0:009> .exptr 1022f928

----- Exception record at 1022fa1c:
ExceptionAddress: 7c812afb (kernel32!RaiseException+0x00000053)
  ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
  Parameter[0]: 19930520
  Parameter[1]: 1022fda4
  Parameter[2]: 0b985074
  pExceptionObject: 1022fda4
  _s_ThrowInfo : 0b985074

----- Context record at 1022fa3c:
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e pop esi

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we get memory dumps that are difficult to analyze in full because some if not most of information was omitted during saving. These are usually small memory dumps (contrasted with kernel and complete) and user process minidumps. We can easily recognize that when we open a dump:

User Mini Dump File: Only registers, stack and portions of memory are available

or

Mini Kernel Dump File: Only registers and stack trace are available

The same also applies to user dumps where thread times information is omitted so it is not possible to use !runaway WinDbg command or to a dump saved with various options of .dump command (including privacy-aware) instead of /ma or deprecated /f option. On the contrary, manually erased data in crash dumps looks more like an example of another pattern called Lateral Damage.

The similar cases of abridged dumps are discussed in Wrong Dump and Missing Space antipatterns.

Anyway, we shouldn’t dismiss such dumps and should try to analyze them. For example, some approaches (including using image binaries) are listed in kernel minidump analysis series. We can even see portions of raw stack data in search of execution residue:

0: kd> !thread
GetPointerFromAddress: unable to read from 81d315b0
THREAD 82f49020  Cid 0004.0034  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    Unable to read nt!_IRP @ 8391e008
Not impersonating
GetUlongFromAddress: unable to read from 81d0ad90
Owning Process            82f00ab0       Image:         System
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      4000214     
Context Switch Count      21886            
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0x81c78ea3)
Stack Init 85be0000 Current 85bdf7c0 Base 85be0000 Limit 85bdd000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps 85bdd000 85be0000
85bdd000  ????????
85bdd004  ????????
85bdd008  ????????
85bdd00c  ????????
85bdd010  ????????
85bdd014  ????????
85bdd018  ????????
85bdd01c  ????????
85bdd020  ????????
85bdd024  ????????
85bdd028  ????????
[...]
85bdf8c4  ????????
85bdf8c8  ????????
85bdf8cc  ????????
85bdf8d0  0000000a
85bdf8d4  a112883e
85bdf8d8  0000001b
85bdf8dc  00000000
85bdf8e0  81c28750 nt!KeSetEvent+0x4d
85bdf8e4  85bdf8e8
85bdf8e8  85bdf970
85bdf8ec  81c28750 nt!KeSetEvent+0x4d
85bdf8f0  badb0d00
85bdf8f4  00000000
85bdf8f8  00000000
85bdf8fc  81cf4820 nt!KiInitialPCR+0x120
85bdf900  00000000
85bdf904  85bdf938
85bdf908  81cf4820 nt!KiInitialPCR+0x120
85bdf90c  00000000
85bdf910  81d32300 nt!IopTimerLock
85bdf914  00000000
85bdf918  81fa0000 nt!_NULL_IMPORT_DESCRIPTOR <PERF> (nt+0x3a0000)
85bdf91c  85bd0023
85bdf920  00000023
85bdf924  00000000
85bdf928  81d323c0 nt!KiDispatcherLock
85bdf92c  a1128828
85bdf930  85bdf9b4
85bdf934  85bdfdb0
85bdf938  00000030
85bdf93c  84ca6f40
85bdf940  84ca6f38
85bdf944  00000001
85bdf948  85bdf970
85bdf94c  00000000
85bdf950  81c28750 nt!KeSetEvent+0x4d
85bdf954  00000008
85bdf958  00010246
85bdf95c  00000000
85bdf960  84ca68a0
[...]
85bdfd2c  82f49020
85bdfd30  835ca4d0
85bdfd34  a6684538
85bdfd38  81cfde7c nt!ExWorkerQueue+0x3c
85bdfd3c  00000001
85bdfd40  00000000
85bdfd44  85bdfd7c
85bdfd48  81c78fa0 nt!ExpWorkerThread+0xfd
85bdfd4c  835ca4d0
85bdfd50  00000000
85bdfd54  82f49020
85bdfd58  00000000
85bdfd5c  00000000
85bdfd60  0069000b
85bdfd64  00000000
85bdfd68  00000001
85bdfd6c  00000000
85bdfd70  835ca4d0
85bdfd74  81da9542 nt!PnpDeviceEventWorker
85bdfd78  00000000
85bdfd7c  85bdfdc0
85bdfd80  81e254e0 nt!PspSystemThreadStartup+0x9d
85bdfd84  835ca4d0
85bdfd88  85bd4680
85bdfd8c  00000000
85bdfd90  00000000
85bdfd94  00000000
85bdfd98  00000002
85bdfd9c  00000000
85bdfda0  00000000
85bdfda4  00000001
85bdfda8  85bdfd88
85bdfdac  85bdfdbc
85bdfdb0  ffffffff
85bdfdb4  81c8aad5 nt!_except_handler4
85bdfdb8  81c9ddb8 nt!`string'+0x4
85bdfdbc  00000000
85bdfdc0  00000000
85bdfdc4  81c9159e nt!KiThreadStartup+0x16
85bdfdc8  81c78ea3 nt!ExpWorkerThread
85bdfdcc  00000001
85bdfdd0  00000000
85bdfdd4  00000000
85bdfdd8  002e0069
85bdfddc  006c0064
85bdfde0  004c006c
85bdfde4  00000000
85bdfde8  000007f0
85bdfdec  00010000
85bdfdf0  0000027f
85bdfdf4  00000000
85bdfdf8  00000000
85bdfdfc  00000000
85bdfe00  00000000
85bdfe04  00000000
85bdfe08  00001f80
85bdfe0c  0000ffff
85bdfe10  00000000
85bdfe14  00000000
85bdfe18  00000000
[...]
85bdffe4  00000000
85bdffe8  00000000
85bdffec  00000000
85bdfff0  00000000
85bdfff4  00000000
85bdfff8  00000000
85bdfffc  00000000
85be0000  ????????

User minidumps are similar here:

0:001> k
ChildEBP RetAddr 
099bfe147c90daaa ntdll!KiFastSystemCallRet
099bfe18 77e765e3 ntdll!NtReplyWaitReceivePortEx+0xc
099bff80 77e76caf rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×12a
099bff88 77e76ad1 rpcrt4!RecvLotsaCallsWrapper+0xd
099bffa8 77e76c97 rpcrt4!BaseCachedThreadRoutine+0×79
099bffb4 7c80b729 rpcrt4!ThreadStartRoutine+0×1a
099bffec 00000000 kernel32!BaseThreadStart+0×37

0:001> dd 099bfe14
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006
099bfe74  00000006 e1f79698 e39b8b60 00000000
099bfe84  fbe33c40 00000001 e5ce12f8 b279db9c

0:001> dd 099bfe14-20
099bfdf4  ???????? ???????? ???????? ????????
099bfe04  ???????? ???????? ???????? ????????
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006

As a warning here it is possible to conclude that minidumps can also reveal private information especially when ASCII or Unicode buffers are seen on raw stack data.

I was thinking how to name this pattern and Oxford Thesaurus of English suggested the following name: Abridged Dump by analogy with an abridged book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to strong and weak process coupling patterns we also have another variant that I call semantic coupling. Some processes (not necessarily from the same vendor) cooperate to provide certain functionality. The cooperation might not involve trackable and visible inter-process communication such as (A)LPC/RPC or pipes but involve events, shared memory and other possible mechanisms not explicitly visible when we look at memory dumps. In many cases, after finding problems in one or several processes from a semantic group we also look at the remaining processes from that group to see if there are some anomalies there as well. The one example I encounter often can be generalized as follows: we have an ALPC wait chain ProcessA -> ProcessB <-> ProcessC (not necessarily a deadlock) but the crucial piece of functionality is also implemented in ProcessD. Sometimes ProcessD is healthy and the problem resides in ProcessC or ProcessB, and sometimes, when we look at ProcessD we find evidence of an earlier problem pattern there so the focus of recommendations shifts to one of ProcessD modules. The case study is coming soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -