Software Diagnostics Library

There are many different approaches to illustrate virtual to physical memory mapping on systems with paging like Windows. Here is another approach that uses natural memory visualization. An image of a user process was generated and juxtaposed to an image of kernel memory dump generated afterwards to produce the combined picture of the full virtual space. Of course, uncommited regions were not included in it as they were not present in user and kernel dumps. Then, after reboot, the same application was launched again and an image of a complete memory dump was generated. Finally, both images were juxtaposed to produce this approximate picture:

In the virtual memory space to the left we see much more granularity. On the contrary, the physical memory space to the right is more uniform and has a different coloring.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Scatter/gather (p. 566) - you can find examples of scatter/gather I/O residues left on a thread raw stack in Hardware Activity pattern and corresponding case study:

http://www.dumpanalysis.org/blog/index.php/2010/05/08/crash-dump-analysis-patterns-part-98/ 

and

http://www.dumpanalysis.org/blog/index.php/2010/06/07/irp-distribution-anomaly-inconsistent-dump-execution-residue-hardware-activity-coincidental-symbolic-information-not-my-version-virtualized-system-pattern-cooperation/

IRP (pp. 566 - 567) - here is an expanded IRP structure from x64 W2K8:

0: kd> dt -r1 _IRP
ntdll!_IRP
+0x000 Type             : Int2B
+0x002 Size             : Uint2B
+0x008 MdlAddress       : Ptr64 _MDL
+0x000 Next             : Ptr64 _MDL
+0x008 Size             : Int2B
+0x00a MdlFlags         : Int2B
+0x010 Process          : Ptr64 _EPROCESS
+0x018 MappedSystemVa   : Ptr64 Void
+0x020 StartVa          : Ptr64 Void
+0x028 ByteCount        : Uint4B
+0x02c ByteOffset       : Uint4B
+0x010 Flags            : Uint4B
+0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
+0x000 SystemBuffer     : Ptr64 Void
+0x020 ThreadListEntry  : _LIST_ENTRY
+0x000 Flink            : Ptr64 _LIST_ENTRY
+0x008 Blink            : Ptr64 _LIST_ENTRY
+0x030 IoStatus         : _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B
+0x040 RequestorMode    : Char
+0x041 PendingReturned  : UChar
+0x042 StackCount       : Char
+0x043 CurrentLocation  : Char
+0x044 Cancel           : UChar
+0x045 CancelIrql       : UChar
+0x046 ApcEnvironment   : Char
+0x047 AllocationFlags  : UChar
+0x048 UserIosb         : Ptr64 _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B
+0x050 UserEvent        : Ptr64 _KEVENT
+0x000 Header           : _DISPATCHER_HEADER
+0x058 Overlay          : <unnamed-tag>
+0x000 AsynchronousParameters : <unnamed-tag>
+0x000 AllocationSize   : _LARGE_INTEGER
+0x068 CancelRoutine    : Ptr64     void
+0x070 UserBuffer       : Ptr64 Void
+0x078 Tail             : <unnamed-tag>
+0x000 Overlay          : <unnamed-tag>
+0x000 Apc              : _KAPC
+0x000 CompletionKey    : Ptr64 Void

IRP stack locations (pp. 568 - 569) - here is a corresponding structure from x64 W2K8:

0: kd> dt _IO_STACK_LOCATION
ntdll!_IO_STACK_LOCATION
+0x000 MajorFunction    : UChar
+0x001 MinorFunction    : UChar
+0x002 Flags            : UChar
+0x003 Control          : UChar
+0x008 Parameters       : <unnamed-tag>
+0x028 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x030 FileObject       : Ptr64 _FILE_OBJECT
+0x038 CompletionRoutine : Ptr64     long
+0x040 Context          : Ptr64 Void 

Buffered I/O (p. 570) - this part of IRP references a buffer (user input data is copied there and device output is copied there):

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
      +0×000 SystemBuffer     : Ptr64 Void

These parts of I/O stack location structure handle buffer lengths:

      +0x000 DeviceIoControl  : <unnamed-tag>
         +0×000 OutputBufferLength : Uint4B
+0×008 InputBufferLength : Uint4B
+0×010 IoControlCode    : Uint4B
+0×018 Type3InputBuffer : Ptr64 Void

      +0x000 Read             : <unnamed-tag>
         +0×000 Length           : Uint4B
+0×008 Key              : Uint4B
+0×010 ByteOffset       : _LARGE_INTEGER

      +0x000 Write            : <unnamed-tag>
         +0×000 Length           : Uint4B
+0×008 Key              : Uint4B
+0×010 ByteOffset       : _LARGE_INTEGER

Direct I/O (p. 570) - these parts of IRP handle IOCTL input data (SystemBuffer, via buffering) and IOCTL output/Read/Write data (MdlAddress):

   +0x008 MdlAddress       : Ptr64 _MDL
+0x000 Next             : Ptr64 _MDL
+0x008 Size             : Int2B
+0x00a MdlFlags         : Int2B
+0x010 Process          : Ptr64 _EPROCESS
+0x018 MappedSystemVa   : Ptr64 Void
+0x020 StartVa          : Ptr64 Void
+0x028 ByteCount        : Uint4B
+0x02c ByteOffset       : Uint4B

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP
+0x000 IrpCount         : Int4B
+0x000 SystemBuffer     : Ptr64 Void

Neither I/O (p. 571) - these parts handle input data (IO_STACK_LOCATION.Parameters.DeviceIoControl.Type3InputBuffer) and output data (IRP.UserBuffer):

      +0x000 DeviceIoControl  : <unnamed-tag>
+0x000 OutputBufferLength : Uint4B
+0x008 InputBufferLength : Uint4B
+0x010 IoControlCode    : Uint4B
         +0×018 Type3InputBuffer : Ptr64 Void

   +0×070 UserBuffer       : Ptr64 Void

I/O status block and kernel APC (pp. 575 - 577) - this is a part of IRP  structure:

   +0x030 IoStatus         : _IO_STATUS_BLOCK
+0x000 Status           : Int4B
+0x000 Pointer          : Ptr64 Void
+0x008 Information      : Uint8B

KeSynchronizeExecution (p. 578) - here is a stack trace fragment showing it in action:

[...]
b9ada518 8088d661 SCSIPORT!SpStartIoSynchronized+0x14f
b9ada550 80a60147 nt!KeSynchronizeExecution+0×21
b9ada57c f72523a6 hal!HalBuildScatterGatherList+0×1c7
b9ada5c8 8081cfa2 SCSIPORT!ScsiPortStartIo+0×36a
b9ada5ec f725262f nt!IoStartPacket+0×82
b9ada620 f7252146 SCSIPORT!ScsiPortFdoDispatch+0×270
b9ada63c f7251dc3 SCSIPORT!SpDispatchRequest+0×68
b9ada658 f7251299 SCSIPORT!ScsiPortPdoScsi+0×129
b9ada66c 8081df85 SCSIPORT!ScsiPortGlobalDispatch+0×1d
b9ada680 f723e607 nt!IofCallDriver+0×45
b9ada690 f723e2b2 CLASSPNP!SubmitTransferPacket+0xbb
b9ada6c4 f723e533 CLASSPNP!ServiceTransferRequest+0×1e4
b9ada6e8 8081df85 CLASSPNP!ClassReadWrite+0×159
b9ada6fc f74c80cf nt!IofCallDriver+0×45
b9ada70c 8081df85 PartMgr!PmReadWrite+0×95
b9ada720 f7317053 nt!IofCallDriver+0×45
b9ada73c 8081df85 ftdisk!FtDiskReadWrite+0×1a9
b9ada750 f72bf8bc nt!IofCallDriver+0×45
b9ada768 8081df85 volsnap!VolSnapRead+0×52
b9ada77c f7163a62 nt!IofCallDriver+0×45
b9ada788 f71638d9 Ntfs!NtfsSingleAsync+0×91
b9ada960 f7164156 Ntfs!NtfsNonCachedIo+0×2db
b9adaa4c f7164079 Ntfs!NtfsCommonRead+0xaf5
b9adabf8 8081df85 Ntfs!NtfsFsdRead+0×113
b9adac0c f721cc45 nt!IofCallDriver+0×45
b9adac34 8081df85 fltmgr!FltpDispatch+0×6f
b9adac48 bafd5373 nt!IofCallDriver+0×45
[…]

IRP and layered drivers (pp. 578 - 586) - here’s a UML-style diagram (#3) for IRP flow:

http://www.dumpanalysis.org/blog/index.php/2006/10/08/uml-and-device-drivers/ 

Associated IRP (pp. 585 - 586) - this is a part of IRP structure:

   +0x018 AssociatedIrp    : <unnamed-tag>
+0x000 MasterIrp        : Ptr64 _IRP

File object vs. thread IRP association (p. 587)

Thread Termination and pending IRP (pp. 589 - 590) - this pattern uses I/O cancellation as an example:

http://www.dumpanalysis.org/blog/index.php/2007/12/14/crash-dump-analysis-patterns-part-42a/ 

Crash Dump is a double buzzword.

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes a software trace is truncated when the trace session was stopped prematurely, often when a problem didn’t manifest itself visually. We can diagnose such traces by their short time duration, missing anchor messages or components necessary for analysis. My favourite example is user session initialization in a Citrix terminal services environment when problem effects are visible only after the session is fully initialized and an application is launched but a truncated CDF trace only shows the launch of winlogon.exe despite the presence of a process creation trace provider or other components that record the process launch sequence and the trace itself lasts only a few seconds after that.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

assert (*p + *p == 2 * (*p) + 1); // for sufficiently fast processors

The reinterpretation of 1 + 1 = 3, Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Coupled Processes (semantics) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A few days ago I was in a hotel bar invited to celebrate an event. Later that night we were trying to sing songs and I came up with a few stanzas. Today I finished the composition:

Solution Number One.
Bang, Bang, Bang…

Solution Number Two.
Poo, Poo, Poo…

Solution Number Three.
Wee, Wee, Wee…

Solution Number Four.
Oh, Oh, Oh…

Solution Number Five.
Ay, Ay, Ay…

Solution Number Six.
Fix, Fix, Fix!

I’ll try to add some music later on…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The one obvious pattern that is shown in many case studies on this blog is Exception Stack Trace (or Exception Thread). This is a stack trace that has exception processing functions, for example:

   9  Id: 1df4.a08 Suspend: -1 Teb: 7fff4000 Unfrozen
ChildEBP RetAddr 
1022f5a8 7c90df4a ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 kernel32!UnhandledExceptionFilter+0×8b9
1022f908 7c839b39 kernel32!BaseThreadStart+0×4d
1022f930 7c9032a8 kernel32!_except_handler3+0×61
1022f954 7c90327a ntdll!ExecuteHandler2+0×26
1022fa04 7c90e48a ntdll!ExecuteHandler+0×24
1022fa04 7c812afb ntdll!KiUserExceptionDispatcher+0xe
1022fd5c 0b82e680 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0×21e640
1022fde8 7753004f DllA+0×21d4f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0×3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0×70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0×36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0×9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0×25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0×1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×1b
1022ffec 00000000 kernel32!BaseThreadStart+0×37

Such exceptions can be detected by the default analysis command (for example, !analyze -v WinDbg command) or by inspecting a stack trace collection. However if we didn’t see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack data. 

In our case we can get the exception information by looking at parameters to in unhandled exception filter:

0:009> kv 3
ChildEBP RetAddr Args to Child
1022f5a8 7c90df4a 7c8648a2 00000002 1022f730 ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 00000002 1022f730 00000001 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 1022f928 7c839b39 1022f930 kernel32!UnhandledExceptionFilter+0×8b9

0:009> .exptr 1022f928

----- Exception record at 1022fa1c:
ExceptionAddress: 7c812afb (kernel32!RaiseException+0x00000053)
  ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
  Parameter[0]: 19930520
  Parameter[1]: 1022fda4
  Parameter[2]: 0b985074
  pExceptionObject: 1022fda4
  _s_ThrowInfo : 0b985074

----- Context record at 1022fa3c:
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e pop esi

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we get memory dumps that are difficult to analyze in full because some if not most of information was omitted during saving. These are usually small memory dumps (contrasted with kernel and complete) and user process minidumps. We can easily recognize that when we open a dump:

User Mini Dump File: Only registers, stack and portions of memory are available

or

Mini Kernel Dump File: Only registers and stack trace are available

The same also applies to user dumps where thread times information is omitted so it is not possible to use !runaway WinDbg command or to a dump saved with various options of .dump command (including privacy-aware) instead of /ma or deprecated /f option. On the contrary, manually erased data in crash dumps looks more like an example of another pattern called Lateral Damage.

The similar cases of abridged dumps are discussed in Wrong Dump and Missing Space antipatterns.

Anyway, we shouldn’t dismiss such dumps and should try to analyze them. For example, some approaches (including using image binaries) are listed in kernel minidump analysis series. We can even see portions of raw stack data in search of execution residue:

0: kd> !thread
GetPointerFromAddress: unable to read from 81d315b0
THREAD 82f49020  Cid 0004.0034  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    Unable to read nt!_IRP @ 8391e008
Not impersonating
GetUlongFromAddress: unable to read from 81d0ad90
Owning Process            82f00ab0       Image:         System
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      4000214     
Context Switch Count      21886            
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0x81c78ea3)
Stack Init 85be0000 Current 85bdf7c0 Base 85be0000 Limit 85bdd000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps 85bdd000 85be0000
85bdd000  ????????
85bdd004  ????????
85bdd008  ????????
85bdd00c  ????????
85bdd010  ????????
85bdd014  ????????
85bdd018  ????????
85bdd01c  ????????
85bdd020  ????????
85bdd024  ????????
85bdd028  ????????
[...]
85bdf8c4  ????????
85bdf8c8  ????????
85bdf8cc  ????????
85bdf8d0  0000000a
85bdf8d4  a112883e
85bdf8d8  0000001b
85bdf8dc  00000000
85bdf8e0  81c28750 nt!KeSetEvent+0x4d
85bdf8e4  85bdf8e8
85bdf8e8  85bdf970
85bdf8ec  81c28750 nt!KeSetEvent+0x4d
85bdf8f0  badb0d00
85bdf8f4  00000000
85bdf8f8  00000000
85bdf8fc  81cf4820 nt!KiInitialPCR+0x120
85bdf900  00000000
85bdf904  85bdf938
85bdf908  81cf4820 nt!KiInitialPCR+0x120
85bdf90c  00000000
85bdf910  81d32300 nt!IopTimerLock
85bdf914  00000000
85bdf918  81fa0000 nt!_NULL_IMPORT_DESCRIPTOR <PERF> (nt+0x3a0000)
85bdf91c  85bd0023
85bdf920  00000023
85bdf924  00000000
85bdf928  81d323c0 nt!KiDispatcherLock
85bdf92c  a1128828
85bdf930  85bdf9b4
85bdf934  85bdfdb0
85bdf938  00000030
85bdf93c  84ca6f40
85bdf940  84ca6f38
85bdf944  00000001
85bdf948  85bdf970
85bdf94c  00000000
85bdf950  81c28750 nt!KeSetEvent+0x4d
85bdf954  00000008
85bdf958  00010246
85bdf95c  00000000
85bdf960  84ca68a0
[...]
85bdfd2c  82f49020
85bdfd30  835ca4d0
85bdfd34  a6684538
85bdfd38  81cfde7c nt!ExWorkerQueue+0x3c
85bdfd3c  00000001
85bdfd40  00000000
85bdfd44  85bdfd7c
85bdfd48  81c78fa0 nt!ExpWorkerThread+0xfd
85bdfd4c  835ca4d0
85bdfd50  00000000
85bdfd54  82f49020
85bdfd58  00000000
85bdfd5c  00000000
85bdfd60  0069000b
85bdfd64  00000000
85bdfd68  00000001
85bdfd6c  00000000
85bdfd70  835ca4d0
85bdfd74  81da9542 nt!PnpDeviceEventWorker
85bdfd78  00000000
85bdfd7c  85bdfdc0
85bdfd80  81e254e0 nt!PspSystemThreadStartup+0x9d
85bdfd84  835ca4d0
85bdfd88  85bd4680
85bdfd8c  00000000
85bdfd90  00000000
85bdfd94  00000000
85bdfd98  00000002
85bdfd9c  00000000
85bdfda0  00000000
85bdfda4  00000001
85bdfda8  85bdfd88
85bdfdac  85bdfdbc
85bdfdb0  ffffffff
85bdfdb4  81c8aad5 nt!_except_handler4
85bdfdb8  81c9ddb8 nt!`string'+0x4
85bdfdbc  00000000
85bdfdc0  00000000
85bdfdc4  81c9159e nt!KiThreadStartup+0x16
85bdfdc8  81c78ea3 nt!ExpWorkerThread
85bdfdcc  00000001
85bdfdd0  00000000
85bdfdd4  00000000
85bdfdd8  002e0069
85bdfddc  006c0064
85bdfde0  004c006c
85bdfde4  00000000
85bdfde8  000007f0
85bdfdec  00010000
85bdfdf0  0000027f
85bdfdf4  00000000
85bdfdf8  00000000
85bdfdfc  00000000
85bdfe00  00000000
85bdfe04  00000000
85bdfe08  00001f80
85bdfe0c  0000ffff
85bdfe10  00000000
85bdfe14  00000000
85bdfe18  00000000
[...]
85bdffe4  00000000
85bdffe8  00000000
85bdffec  00000000
85bdfff0  00000000
85bdfff4  00000000
85bdfff8  00000000
85bdfffc  00000000
85be0000  ????????

User minidumps are similar here:

0:001> k
ChildEBP RetAddr 
099bfe147c90daaa ntdll!KiFastSystemCallRet
099bfe18 77e765e3 ntdll!NtReplyWaitReceivePortEx+0xc
099bff80 77e76caf rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×12a
099bff88 77e76ad1 rpcrt4!RecvLotsaCallsWrapper+0xd
099bffa8 77e76c97 rpcrt4!BaseCachedThreadRoutine+0×79
099bffb4 7c80b729 rpcrt4!ThreadStartRoutine+0×1a
099bffec 00000000 kernel32!BaseThreadStart+0×37

0:001> dd 099bfe14
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006
099bfe74  00000006 e1f79698 e39b8b60 00000000
099bfe84  fbe33c40 00000001 e5ce12f8 b279db9c

0:001> dd 099bfe14-20
099bfdf4  ???????? ???????? ???????? ????????
099bfe04  ???????? ???????? ???????? ????????
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006

As a warning here it is possible to conclude that minidumps can also reveal private information especially when ASCII or Unicode buffers are seen on raw stack data.

I was thinking how to name this pattern and Oxford Thesaurus of English suggested the following name: Abridged Dump by analogy with an abridged book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to strong and weak process coupling patterns we also have another variant that I call semantic coupling. Some processes (not necessarily from the same vendor) cooperate to provide certain functionality. The cooperation might not involve trackable and visible inter-process communication such as (A)LPC/RPC or pipes but involve events, shared memory and other possible mechanisms not explicitly visible when we look at memory dumps. In many cases, after finding problems in one or several processes from a semantic group we also look at the remaining processes from that group to see if there are some anomalies there as well. The one example I encounter often can be generalized as follows: we have an ALPC wait chain ProcessA -> ProcessB <-> ProcessC (not necessarily a deadlock) but the crucial piece of functionality is also implemented in ProcessD. Sometimes ProcessD is healthy and the problem resides in ProcessC or ProcessB, and sometimes, when we look at ProcessD we find evidence of an earlier problem pattern there so the focus of recommendations shifts to one of ProcessD modules. The case study is coming soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Started testing marketing materials to increase Dublin people awareness of memory dump analysis. Here are the first exemplars of what I’m going to wear by the end of this week while commuting and in local pubs

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Blog - Binary log.

Examples: This program blogged. I’m reading its blog now.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we show the possible signs of the classical Thread Starvation.
Suppose we have two running threads with a priority 8:

0: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    ffdff120  89a92020            O...............
  1    f7737120  89275020            W...............

0: kd> !thread 89a92020
THREAD 89a92020  Cid 11d8.27d8  Teb: 7ffd9000 Win32Thread: bc1e6860 RUNNING on processor 0
[...]
Priority 8 BasePriority 8 PriorityDecrement 0

0: kd> !thread 89275020
THREAD 89275020  Cid 1cd0.2510  Teb: 7ffa9000 Win32Thread: bc343180 RUNNING on processor 1
[...]
Priority 8 BasePriority 8 PriorityDecrement 0

If we have other threads ready with the same priority contending for the same processors other threads with less priority might starve (shown in red):

0: kd> !ready
Processor 0: Ready Threads at priority 8
    THREAD 894a1db0  Cid 1a98.25c0  Teb: 7ffde000 Win32Thread: bc19cea8 READY
    THREAD 897c4818  Cid 11d8.1c5c  Teb: 7ffa2000 Win32Thread: bc2c5ba8 READY
    THREAD 8911fd18  Cid 2730.03f4  Teb: 7ffd9000 Win32Thread: bc305830 READY
Processor 0: Ready Threads at priority 7
    THREAD 8a9e5ab0  Cid 0250.0470  Teb: 7ff9f000 Win32Thread: 00000000 READY
    THREAD 8a086838  Cid 0250.0654  Teb: 7ff93000 Win32Thread: 00000000 READY
    THREAD 8984b8b8  Cid 0250.1dc4  Teb: 7ff99000 Win32Thread: 00000000 READY
    THREAD 8912a4c0  Cid 0f4c.2410  Teb: 7ff81000 Win32Thread: 00000000 READY
    THREAD 89e5c570  Cid 0f4c.01c8  Teb: 00000000 Win32Thread: 00000000 READY
Processor 0: Ready Threads at priority 6
    THREAD 8a9353b0  Cid 1584.1598  Teb: 7ff8b000 Win32Thread: bc057698 READY
    THREAD 8aba2020  Cid 1584.15f0  Teb: 7ff9f000 Win32Thread: bc2a0ea8 READY
    THREAD 8aab17a0  Cid 1584.01a8  Teb: 7ff92000 Win32Thread: bc316ea8 READY
    THREAD 8a457020  Cid 1584.0634  Teb: 7ff8d000 Win32Thread: bc30fea8 READY
    THREAD 8a3d4020  Cid 1584.1510  Teb: 7ff8f000 Win32Thread: bc15b8a0 READY
    THREAD 8a5f5db0  Cid 1584.165c  Teb: 7ff9d000 Win32Thread: bc171be8 READY
    THREAD 8a297020  Cid 0f4c.0f54  Teb: 7ffde000 Win32Thread: bc20fda0 READY
    THREAD 8a126020  Cid 1584.175c  Teb: 7ffa9000 Win32Thread: 00000000 READY
    THREAD 8a548478  Cid 0250.07b0  Teb: 7ff9a000 Win32Thread: 00000000 READY
    THREAD 8a478020  Cid 0944.0988  Teb: 7ffd9000 Win32Thread: 00000000 READY
    THREAD 8986ad08  Cid 1d2c.1cf0  Teb: 7ffa8000 Win32Thread: bc285800 READY
    THREAD 897f4db0  Cid 1d2c.2554  Teb: 7ffdb000 Win32Thread: bc238e80 READY
    THREAD 89a2e618  Cid 1d2c.1de4  Teb: 7ffdd000 Win32Thread: bc203908 READY
Processor 0: Ready Threads at priority 0
    THREAD 8b184db0  Cid 0004.0008  Teb: 00000000 Win32Thread: 00000000 READY
Processor 1: Ready Threads at priority 8
    THREAD 89d89db0  Cid 1b10.20ac  Teb: 7ffd7000 Win32Thread: bc16e680 READY
    THREAD 891f24a8  Cid 1e2c.20d0  Teb: 7ffda000 Win32Thread: bc1b9ea8 READY
    THREAD 89214db0  Cid 1e2c.24d4  Teb: 7ffd7000 Win32Thread: bc24ed48 READY
    THREAD 89a28020  Cid 1b10.21b4  Teb: 7ffa7000 Win32Thread: bc25b3b8 READY
    THREAD 891e03b0  Cid 1a98.05c4  Teb: 7ffdb000 Win32Thread: bc228bb0 READY
    THREAD 891b0020  Cid 1cd0.0144  Teb: 7ffde000 Win32Thread: bc205ea8 READY
Processor 1: Ready Threads at priority 7
    THREAD 898367a0  Cid 0f4c.1cd4  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 8a1ac020  Cid 0f4c.1450  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 8aa1ab90  Cid 0f4c.11b0  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 89cc92e0  Cid 0f4c.1b34  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 89579020  Cid 0f4c.2220  Teb: 00000000 Win32Thread: 00000000 READY
Processor 1: Ready Threads at priority 6
    THREAD 8a487db0  Cid 1584.14bc  Teb: 7ffa2000 Win32Thread: bc304ea8 READY
    THREAD 8a3ce020  Cid 1584.0630  Teb: 7ff8e000 Win32Thread: bc293c20 READY
    THREAD 8a1b6db0  Cid 1584.1590  Teb: 7ff8c000 Win32Thread: bc310ea8 READY
    THREAD 8a1fe6e0  Cid 1584.15ec  Teb: 7ffa1000 Win32Thread: bc15bea8 READY
    THREAD 8ac0adb0  Cid 1584.156c  Teb: 7ff8a000 Win32Thread: bc153be8 READY
    THREAD 8b1e35a0  Cid 1584.15f4  Teb: 7ff9e000 Win32Thread: bc0567e8 READY
    THREAD 8a3288e8  Cid 1584.14b8  Teb: 7ff9a000 Win32Thread: bc2fbea8 READY
    THREAD 8a5056a0  Cid 1584.1518  Teb: 7ff91000 Win32Thread: bc337ea8 READY
    THREAD 891afdb0  Cid 1d2c.27e8  Teb: 7ffaf000 Win32Thread: bc217c18 READY
    THREAD 8a07d308  Cid 1d2c.2548  Teb: 7ffae000 Win32Thread: bc235750 READY
    THREAD 8a055d18  Cid 1584.17d0  Teb: 7ffd5000 Win32Thread: 00000000 READY
    THREAD 8ac0b770  Cid 0250.0268  Teb: 7ffde000 Win32Thread: bc2349d8 READY
    THREAD 8a0eeb40  Cid 1584.1560  Teb: 7ffdc000 Win32Thread: 00000000 READY

Here we should also analyze stack traces for running and ready threads with priority 8 and check kernel and user times. If we find anything common between them we should also check ready threads with lower priority to see if that commonality is unique to threads with priority 8. See also the similar pattern: Busy System and the similar starvation pattern resulted from realtime priority threads.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I accidentally use du WinDbg command (to interpret memory as Unicode) instead of da (to interpret memory as ASCII):

0:000> db 07329f28 l20
07329f28  68 69 64 70 6e 73 74 3d-26 74 78 74 74 6f 3d 26  hidpnst=&txtto=&
07329f38  74 78 74 63 63 3d 26 74-78 74 62 63 63 3d 26 74  txtcc=&txtbcc=&t

0:000> da 07329f28 l20
07329f28  "hidpnst=&txtto=&txtcc=&txtbcc=&t"

0:000> du 07329f28 l10
07329f28  "楨灤獮㵴琦瑸潴.硴捴㵣琦瑸换㵣琦"

This time I tried to get extra hidden meaning from a process dump taken after the process suffered a CPU spike by using Google translator and got this text (I put more lengthy Unicode sequence and removed some offensive words):

&quot;Luan Xian Zhen Qi-bin 㵴 cisternae. Huasong 㵣 Qi, Qi-bin-bin for 㵣 pull 㵪 䕒 .. 䉉 Ya Hui material. Hong SHIKA King. Huajiayuyan nuts .. 䐰 〥 䅁 evil force. Rafter Hui Qi 䤫 Mi cat deterrent Junying hydrogen walk. cisternae Huzhao Man cat Wuzhou Wen Zhen Zhao Zhen Pan scene file Shan. prison Shang Tang. Jue Shi Pan. sewage knock Xi. generous Zhen. 䤫. ice. conflict. cisternae Zhao askance nuts. rafter .. On unfeigned domain knock. Kagesue Mankuo. 㜲 Ruo Yi enemy luster of gems. cisternae Yu Wei Shan scene. Tan knock Shan. tally Xia Pan Ying. rafter. Xia. luster of gems tumultuous. Jing Feng-Tou Airuo enemy luster of gems Yixian … additionally . Tu. civet eliminating the lot Shan Ying RB Thieme, Jr.-Voltage trapping Feng-潷 Man. Tan knock Ruo Yi Xian cat enemy luster of gems. rafter Shi Feng-Tou. Mu. Minli Bang domain sewage Huitangyuzhao Su-hai.-Voltage Jiumi. rafter. Qing Wei Jun. 歳 Mi hai 䤫 Panyu. Zhucuoqufang .. 䐰 〥. 䐰 〥 䥁 hydrogen walk. rafter. Mount Zao Man. .. Run-Voltage Rendering. Tang Ying Yi. Shisuqingshi Fangmaosheji Yu Zhao 䤫 Su-. tide. tatami knock Feng-generous. rafter. Min luster of gems. Que Tu Mei Shi Tang Pan Ying. Jijue-Voltage. rafter. Wei Hui Mongoose Feng-. hunting. rafter. revolves Recent-Voltage sewage 䤫. stay Jiao RB Thieme, Jr soup.潷 Han.’m setback Xun. Han Tun petty. Liaohe. 䥔 end of Tu Feng-generous. rafter Xiang Shan Li Tu. trapping the end of sleep ZHEJIANG NORMAL Feng-Tou Yu Xun Jing Wen Fang 䤫 .. 䠫 pine and methods of disease. tatami knock Feng-generous. apply Feng-evil force fell Junying Su-Ao Po .. knock .. Tan Li Shan Jie look askance alone. ㅆ Guang Tang rafter. pool just cultural and&quot;

From the translation I see previously hidden notions of gems, disease and evil forces :-)

ASCII->Unicode->translation->ASCII

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

8

A separating digit of user space and kernel space in equi-bipartitional 32-bit virtual memory, for example, on x86 Windows systems:

RetAddr
80833491 nt!KiSwapContext+0×26
80828f2b nt!KiSwapThread+0×2e5
8099490a nt!KeDelayExecutionThread+0×2ab
808897cc nt!NtDelayExecution+0×84
7c82860c nt!KiFastCallEntry+0xfc
7c826f69 ntdll!KiFastSystemCallRet
7c83d454 ntdll!NtDelayExecution+0xc
77e6482f ntdll!RtlpTimerThread+0×47
00000000 kernel32!BaseThreadStart+0×34

lkd> lm
start end module name
[…]
77bd0000 77c19000 GDI32
77c20000 77cbf000 RPCRT4
77cd0000 77d5b000 OLEAUT32
77e20000 77eb2000 USER32
77ec0000 77f12000 SHLWAPI
77f30000 77fdc000 ADVAPI32
7c800000 7c915000 kernel32
7c920000 7c9e8000 ntdll
7c9f0000 7d1f9000 SHELL32
7e020000 7e02f000 SAMLIB
80800000 80a5d000 nt
80a5d000 80a89000 hal
bf800000 bf9d0000 win32k
[…]

This number doesn’t continue to be used in the same way on x64 Windows systems, for example:

RetAddr Call Site
fffff800`0167e082 nt!KiSwapContext+0×7a
fffff800`0167a34b nt!KiCommitThreadWait+0×1d2
fffff800`01973def nt!KeWaitForMultipleObjects+0×271
fffff800`019746f6 nt!ObpWaitForMultipleObjects+0×294
fffff800`01675ed3 nt!NtWaitForMultipleObjects+0xe5
00000000`77ca030a nt!KiSystemServiceCopyEnd+0×13

lkd> lm
start end module name
00000000`774a0000 00000000`7756d000 USER32
00000000`77570000 00000000`7769d000 kernel32
00000000`776a0000 00000000`77826000 ntdll
00000000`ffe60000 00000000`ffec0000 services
000007fe`fca90000 000007fe`fca97000 wshtcpip
000007fe`fcae0000 000007fe`fcb0c000 NTMARTA
000007fe`fcbe0000 000007fe`fcc34000 schannel
000007fe`fce10000 000007fe`fce5f000 mswsock
000007fe`fcf60000 000007fe`fcf69000 credssp
000007fe`fcf90000 000007fe`fcf97000 wship6
000007fe`fd0d0000 000007fe`fd207000 CRYPT32
000007fe`fd260000 000007fe`fd279000 MSASN1
000007fe`fd5a0000 000007fe`fd5b6000 NCObjAPI
000007fe`fd5c0000 000007fe`fd625000 SCESRV
000007fe`fd630000 000007fe`fd665000 apphelp
000007fe`fd680000 000007fe`fd6a6000 AUTHZ
000007fe`fd950000 000007fe`fd96c000 Secur32
000007fe`fd970000 000007fe`fd996000 USERENV
000007fe`fd9a0000 000007fe`fd9bc000 SAMLIB
000007fe`fdc80000 000007fe`fdce4000 GDI32
000007fe`fdd10000 000007fe`fdd1d000 LPK
000007fe`fdd20000 000007fe`fde28000 ADVAPI32
000007fe`fde30000 000007fe`fdecc000 msvcrt
000007fe`fded0000 000007fe`fdefd000 IMM32
000007fe`fdf00000 000007fe`fe0d8000 ole32
000007fe`fe0e0000 000007fe`fe0e7000 NSI
000007fe`fe190000 000007fe`fe22a000 USP10
000007fe`fe230000 000007fe`fe373000 RPCRT4
000007fe`ff340000 000007fe`ff395000 WLDAP32
000007fe`ff540000 000007fe`ff584000 WS2_32
000007fe`ff7d0000 000007fe`ff8d2000 MSCTF
fffff800`01802000 fffff800`01848000 hal
fffff800`01848000 fffff800`01d60000 nt
fffff960`000f0000 fffff960`003a1000 win32k
[…]

Synonyms:

Antonyms:

Also: virtual memory, user space, kernel space, x86, x64, 7

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Similar to different C/C++ styles like where to put the right brace we have User/Kernel Space/Mode architecture diagramming styles. Some prefer to put User part on top (Windows Internals team) and some prefer to put Kernel on top (like me). Marc Sherman in the comment here explains the former style as “calling down into the kernel”. Originally I thought about a psychological explanation where you put on top what you value the most or use the most. However, the reason I put Kernel on top is because I value Space over Mode in depicting memory and dependencies. In stack traces from complete memory dumps you have kernel portions on top as well. Also Google and Bing favour “stack grows down” slightly over “stack grows up” and I prefer “down” as well. Additionally, if you value User on top you would be forced to put Managed on top as well. If you put Kernel on top than Managed is at the bottom like on this flag that became a logo of Memory Dump Analysis Services. So here are two diagrams where I prefer the first (Kernel on top) with any stack growing down (in address decrement sense) and any stack trace from WinDbg having Kernel on top too:

The second diagram has any stack growing up:

I also suggest this variant (but people who write and read from right to left may prefer its reflection):

Another diagram style that is consistent with the traditional depiction of Privilege Mode rings (here Kernel is also on top but can be put in any direction):

What style do you favour? Please let me know.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Hooked Functions (kernel space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I do Google search for my site and find various cached dumpanalysis.org site evaluations ranging from 4,000 USD to 200,000 USD. It’s interesting to see that people evaluate how much this site worth. Today I stumbled across another unbelievable evaluation from valuemysite dot com (> 2,000,000 USD) so I checked myself:

“apple.com” is worth $340,512,336 USD
“amazon.com” is worth $340,512,336 USD
“microsoft.com” is worth $340,512,336 USD
“ibm.com” is worth $49,876,253 USD
“dumpanalysis.org” is worth $1,989,157 USD
“google.com” is worth $1,738,123 USD

Looks like some sites have an upper evaluation limit or reached an equilibrium

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Hooked Functions (user space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Viewing the loaded driver list (pp. 546 - 547) - if we don’t see company information in lmv command output we can examine raw driver data like in this pattern: http://www.dumpanalysis.org/blog/index.php/2007/08/16/crash-dump-analysis-patterns-part-22/

DriverEntry (p. 548) - consider this as similar to main (console) or WinMain (Win32). For example, if you are writing a Windows service you have to register certain functions with SCM.

Dispatch routines (p. 548) - if you know C++ consider them as class functions for a device object where DeviceObject is a this parameter (C++ class function implementation in C where an implicit this becomes the first function argument):

NTSTATUS (*PDRIVER_DISPATCH) (IN PDEVICE_OBJECT DeviceObject, IN PIRP Irp);

and a driver object can be seen as a container for a virtual function table (vtable) for a device object (purely from implementation perspective): devObj->DriverObject->MajorFunction[IRP_MJ_XXX]

Relationship between device and driver objects (pp. 553 - 554) - long time ago when I was preparing a presentation about Windows drivers for escalation engineers I created some UML diagrams you can see in the following blog post: http://www.dumpanalysis.org/blog/index.php/2006/10/08/uml-and-device-drivers/ 

AttachedDevice vs. AttachedTo (p.554)

File object structure and extension (pp. 556 - 557) - Here are driver, device and file object structures from x64 W2K8:

0: kd> dt _DRIVER_OBJECT
ntdll!_DRIVER_OBJECT
+0x000 Type             : Int2B
+0x002 Size             : Int2B
+0x008 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x010 Flags            : Uint4B
+0x018 DriverStart      : Ptr64 Void
+0x020 DriverSize       : Uint4B
+0x028 DriverSection    : Ptr64 Void
+0x030 DriverExtension  : Ptr64 _DRIVER_EXTENSION
+0x038 DriverName       : _UNICODE_STRING
+0x048 HardwareDatabase : Ptr64 _UNICODE_STRING
+0x050 FastIoDispatch   : Ptr64 _FAST_IO_DISPATCH
+0x058 DriverInit       : Ptr64     long
+0x060 DriverStartIo    : Ptr64     void
+0x068 DriverUnload     : Ptr64     void
+0x070 MajorFunction    : [28] Ptr64     long

0: kd> dt _DEVICE_OBJECT
ntdll!_DEVICE_OBJECT
+0x000 Type             : Int2B
+0x002 Size             : Uint2B
+0x004 ReferenceCount   : Int4B
+0x008 DriverObject     : Ptr64 _DRIVER_OBJECT
+0x010 NextDevice       : Ptr64 _DEVICE_OBJECT
+0x018 AttachedDevice   : Ptr64 _DEVICE_OBJECT
+0x020 CurrentIrp       : Ptr64 _IRP
+0x028 Timer            : Ptr64 _IO_TIMER
+0x030 Flags            : Uint4B
+0x034 Characteristics  : Uint4B
+0x038 Vpb              : Ptr64 _VPB
+0x040 DeviceExtension  : Ptr64 Void
+0x048 DeviceType       : Uint4B
+0x04c StackSize        : Char
+0x050 Queue            : <unnamed-tag>
+0x098 AlignmentRequirement : Uint4B
+0x0a0 DeviceQueue      : _KDEVICE_QUEUE
+0x0c8 Dpc              : _KDPC
+0x108 ActiveThreadCount : Uint4B
+0x110 SecurityDescriptor : Ptr64 Void
+0x118 DeviceLock       : _KEVENT
+0x130 SectorSize       : Uint2B
+0x132 Spare1           : Uint2B
+0x138 DeviceObjectExtension : Ptr64 _DEVOBJ_EXTENSION
+0x140 Reserved         : Ptr64 Void

0: kd> dt _FILE_OBJECT
ntdll!_FILE_OBJECT
+0x000 Type             : Int2B
+0x002 Size             : Int2B
+0x008 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x010 Vpb              : Ptr64 _VPB
+0x018 FsContext        : Ptr64 Void
+0x020 FsContext2       : Ptr64 Void
+0x028 SectionObjectPointer : Ptr64 _SECTION_OBJECT_POINTERS
+0x030 PrivateCacheMap  : Ptr64 Void
+0x038 FinalStatus      : Int4B
+0x040 RelatedFileObject : Ptr64 _FILE_OBJECT
+0x048 LockOperation    : UChar
+0x049 DeletePending    : UChar
+0x04a ReadAccess       : UChar
+0x04b WriteAccess      : UChar
+0x04c DeleteAccess     : UChar
+0x04d SharedRead       : UChar
+0x04e SharedWrite      : UChar
+0x04f SharedDelete     : UChar
+0x050 Flags            : Uint4B
+0x058 FileName         : _UNICODE_STRING
+0x068 CurrentByteOffset : _LARGE_INTEGER
+0x070 Waiters          : Uint4B
+0x074 Busy             : Uint4B
+0x078 LastLock         : Ptr64 Void
+0x080 Lock             : _KEVENT
+0x098 Event            : _KEVENT
+0x0b0 CompletionContext : Ptr64 _IO_COMPLETION_CONTEXT
+0x0b8 IrpListLock      : Uint8B
+0x0c0 IrpList          : _LIST_ENTRY
+0x0d0 FileObjectExtension : Ptr64 Void