Software Diagnostics Library

This pattern (we call it String Hint) covers traces of ASCII and UNICODE strings that look suspicious such as website, password and HTTP forms or strange names that intuitively shouldn’t be present according to the purpose of a module or its container process (example is taken from Victimware presentation case study):

0:005> s-sa 00040000 L1d000
0004004d  "!This program cannot be run in D"
0004006d  "OS mode."
00040081  "3y@"
000400b8  "Rich"
000401d0  ".text"
000401f7  "`.rdata"
0004021f  "@.data"
00040248  ".reloc"
[...]
00054018  "GET /stat?uptime=%d&downlink=%d&"
00054038  "uplink=%d&id=%s&statpass=%s&comm"
00054058  "ent=%s HTTP/1.0"
000540ac  "%s%s%s"
000540d8  "ftp://%s:%s@%s:%d"
000540fc  "Accept-Encoding:"
00054118  "Accept-Encoding:"
00054130  "0123456789ABCDEF"
00054144  "://"
00054160  "POST %s HTTP/1.0"
00054172  "Host: %s"
0005417c  "User-Agent: %s"
0005418c  "Accept: text/html"
0005419f  "Connection: Close"
000541b2  "Content-Type: application/x-www-"
000541d2  "form-urlencoded"
000541e3  "Content-Length: %d"
000541fc  "id="
00054208  "POST %s HTTP/1.1"
0005421a  "Host: %s"
00054224  "User-Agent: %s"
00054234  "Accept: text/html"
00054247  "Connection: Close"
0005425a  "Content-Type: application/x-www-"
0005427a  "form-urlencoded"
0005428b  "Content-Length: %d"
000542a4  "id=%s&base="
000542b8  "id=%s&brw=%d&type=%d&data="
000542d8  "POST %s HTTP/1.1"
000542ea  "Host: %s"
000542f4  "User-Agent: %s"
00054304  "Accept: text/html"
00054317  "Connection: Close"
0005432a  "Content-Type: application/x-www-"
0005434a  "form-urlencoded"
0005435b  "Content-Length: %d"
00054378  "id=%s&os=%s&plist="
00054390  "POST %s HTTP/1.1"
000543a2  "Host: %s"
000543ac  "User-Agent: %s"
000543bc  "Accept: text/html"
000543cf  "Connection: Close"
000543e2  "Content-Type: application/x-www-"
00054402  "form-urlencoded"
00054413  "Content-Length: %d"
00054430  "id=%s&data=%s"
00054440  "POST %s HTTP/1.1"
00054452  "Host: %s"
0005445c  "User-Agent: %s"
0005446c  "Accept: text/html"
0005447f  "Connection: Close"
00054492  "Content-Type: application/x-www-"
000544b2  "form-urlencoded"
000544c3  "Content-Length: %d"
000544e0  "GET %s HTTP/1.0"
000544f1  "Host: %s"
000544fb  "User-Agent: %s"
0005450b  "Connection: close"
00054528  "POST /get/scr.html HTTP/1.0"
00054545  "Host: %s"
0005454f  "User-Agent: %s"
0005455f  "Connection: close"
00054572  "Content-Length: %d"
00054586  "Content-Type: multipart/form-dat"
000545a6  "a; boundary=--------------------"
000545c6  "-------%d"
000545d4  "-----------------------------%d"
000545f8  "%sContent-Disposition: form-data"
00054618  "; name="id""
00054630  "%sContent-Disposition: form-data"
00054650  "; name="screen"; filename="%d""
00054670  "Content-Type: application/octet-"
00054690  "stream"
000546a0  "%s(%d) : %s"
000546ac  "%s failed with error %d: %s"
000546c8  "%02X"
000546d8  "BlackwoodPRO"
000546e8  "FinamDirect"
000546f4  "GrayBox"
000546fc  "MbtPRO"
00054704  "Laser"
0005470c  "LightSpeed"
00054718  "LTGroup"
00054720  "Mbt"
00054724  "ScotTrader"
00054730  "SaxoTrader"
00054740  "Program:   %s"
0005474f  "Username:  %s"
0005475e  "Password:  %s"
0005476d  "AccountNO: %s"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Analysis Patterns (MAPs) including memory dump, malware, software trace (TAPs), and other patterns and pattern catalogs from Software Diagnostics Institute form the very rich semantic network. Now it is possible (by using a metaphorical bijection) to create a catalog of General Patterns of Abnormal Structure and Behaviour including software, hardware, biological behavior including animal (ethology) and human behavior, sociological and historical behavior including economics, business and finance, ethics and law, and even behavior of chemical and physical systems. Such “GAPs of Structure and Behavior” may include wait chains, spikes, deadlocks, etc. We provide more specific examples in the forthcoming parts. So we are a few steps closer to realization of my old dangerous idea of a parameterized science of universal memory dumps by the so called science files or might event a general diagnostics discipline.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

What is a role of perceived memory deficient matter in Memorianity where Memory consists of memories (Memorianic Prophecy 0m3)? Matter are boundaries of memories as illustrated on this schematic diagram:

We can also reverse monistic aspect pluralism of Memoidealism and consider substances as boundaries of memories too.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I don’t know how I missed it. It was the first real life slang I heard almost 10 years ago during hot political and critical crash dump analysis session.

DD That - Analyze that simply.

Although more correctly would be to say DP That at that time when I heard it 64-bit computing wasn’t a mainstream yet. Based on dd WinDbg command to dump raw binary data starting from a given 32-bit memory address.

Examples: I dd-ed that and found an ASCII.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

STATUS - Something important to check for just now.

Examples: If only programmers checked statuses of their functions more often than statuses on Facebook…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Injected code address may not be in the address ranges of loaded modules. In such cases, in the execution call history we would see plain EIP and RIP return addresses on stack traces. We call this pattern RIP Stack Trace partly because we have seen these addresses after something had gone wrong and a process crashed:

0:005> k
ChildEBP RetAddr
02aec974 77655620 ntdll!KiFastSystemCallRet
02aec978 77683c62 ntdll!NtWaitForSingleObject+0xc
02aec9fc 77683d4b ntdll!RtlReportExceptionEx+0x14b
02aeca3c 7769fa87 ntdll!RtlReportException+0x3c
02aeca50 7769fb0d ntdll!RtlpTerminateFailureFilter+0x14
02aeca5c 775f9bdc ntdll!RtlReportCriticalFailure+0x6b
02aeca70 775f4067 ntdll!_EH4_CallFilterFunc+0x12
02aeca98 77655f79 ntdll!_except_handler4+0x8e
02aecabc 77655f4b ntdll!ExecuteHandler2+0x26
02aecb6c 77655dd7 ntdll!ExecuteHandler+0x24
02aecb6c 7769faf8 ntdll!KiUserExceptionDispatcher+0xf
02aecee0 776a0704 ntdll!RtlReportCriticalFailure+0x5b
02aecef0 776a07f2 ntdll!RtlpReportHeapFailure+0x21
02aecf24 7766b1a5 ntdll!RtlpLogHeapFailure+0xa1
02aecf6c 7765730a ntdll!RtlpCoalesceFreeBlocks+0x4b9
02aed064 77657545 ntdll!RtlpFreeHeap+0x1e2
02aed080 75e47e4b ntdll!RtlFreeHeap+0x14e
02aed0c8 77037277 kernel32!GlobalFree+0x47
02aed0dc 774b4a1f ole32!ReleaseStgMedium+0x124
02aed0f0 77517feb urlmon!ReleaseBindInfo+0x4c
02aed100 774d9a87 urlmon!CINet::ReleaseCNetObjects+0x3d
02aed118 774d93f0 urlmon!CINetHttp::OnWininetRequestHandleClosing+0x60
02aed12c 76432078 urlmon!CINet::CINetCallback+0x2de
02aed274 76438f5d wininet!InternetIndicateStatus+0xfc
02aed2a4 7643937a wininet!HANDLE_OBJECT::~HANDLE_OBJECT+0xc9
02aed2c0 7643916b wininet!INTERNET_CONNECT_HANDLE_OBJECT::~INTERNET_CONNECT_HANDLE_OBJECT+0x209
02aed2cc 76438d5e wininet!HTTP_REQUEST_HANDLE_OBJECT::`vector deleting destructor'+0xd
02aed2dc 76434e72 wininet!HANDLE_OBJECT::Dereference+0x22
02aed2e8 76439419 wininet!DereferenceObject+0x21
02aed310 76439114 wininet!_InternetCloseHandle+0x9d
02aed330 0004aaaf wininet!InternetCloseHandle+0x11e
WARNING: Frame IP not in any known module. Following frames may be wrong.
02aed33c 774c5d25 0×4aaaf
02aed358 774c5d95 urlmon!CINet::TerminateRequest+0×82
02aed364 774c5d7c urlmon!CINet::MyUnlockRequest+0×10
02aed370 774c5d63 urlmon!CINetProtImpl::UnlockRequest+0×10
02aed37c 774c5d49 urlmon!CINetEmbdFilter::UnlockRequest+0×11
02aed388 774b743d urlmon!CINet::UnlockRequest+0×13
02aed394 774b73e1 urlmon!COInetProt::UnlockRequest+0×11
02aed3a8 774b7530 urlmon!CTransaction::UnlockRequest+0×36
02aed3b4 774b74e0 urlmon!CTransData::~CTransData+0×3a
02aed3c0 774b74c9 urlmon!CTransData::`scalar deleting destructor’+0xd
02aed3d8 774e221f urlmon!CTransData::Release+0×25
02aed3e0 774b6d0a urlmon!CReadOnlyStreamDirect::~CReadOnlyStreamDirect+0×1a
02aed3ec 774b7319 urlmon!CReadOnlyStreamDirect::`vector deleting destructor’+0xd
02aed404 774b72be urlmon!CReadOnlyStreamDirect::Release+0×25
02aed410 774b71f4 urlmon!CBinding::~CBinding+0xb9
02aed41c 774b71dd urlmon!CBinding::`scalar deleting destructor’+0xd
02aed434 6b20b0e8 urlmon!CBinding::Release+0×25
02aed448 6b20b0ba mshtml!ATL::AtlComPtrAssign+0×2b
02aed458 6b20b8de mshtml!ATL::CComPtr<IBindCallbackInternal>::operator=+0×15
02aed464 6b20b8aa mshtml!CBindingXSSFilter::TearDown+0×2b
02aed46c 6b20b887 mshtml!BindingXSSFilter_TearDown+0×19
02aed478 6b0da61a mshtml!CStreamProxy::Passivate+0×12
02aed484 6b0ddf3a mshtml!CBaseFT::Release+0×1d
02aed4ac 6b0e0b70 mshtml!CDwnBindData::TerminateBind+0×11d
02aed4b8 6b11a2a9 mshtml!CDwnBindData::TerminateOnApt+0×14
02aed4ec 6b105066 mshtml!GlobalWndOnMethodCall+0xfb
02aed50c 7742fd72 mshtml!GlobalWndProc+0×183
02aed538 7742fe4a user32!InternalCallWinProc+0×23
02aed5b0 7743018d user32!UserCallWinProcCheckWow+0×14b
02aed614 7743022b user32!DispatchMessageWorker+0×322
02aed624 6ecac1d5 user32!DispatchMessageW+0xf
02aef72c 6ec5337e ieframe!CTabWindow::_TabWindowThreadProc+0×54c
02aef7e4 760f426d ieframe!LCIETab_ThreadProc+0×2c1
02aef7f4 75e4d0e9 iertutil!CIsoScope::RegisterThread+0xab
02aef800 776319bb kernel32!BaseThreadInitThunk+0xe
02aef840 7763198e ntdll!__RtlUserThreadStart+0×23
02aef858 00000000 ntdll!_RtlUserThreadStart+0×1b

However, such addresses need to be checked whether they belong to .NET CLR JIT code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We skip parts 5 (Module Collection), 6 (No Component Symbols, for looking at import tables), 7 (Stack Trace Collection, for listing active processes, threads and their stack traces), 8 (Hidden Module), and 9 (Hidden Process). The new pattern here is called Driver Device Collection and can be used to compare the current list of device and driver objects with some saved reference list to find out any changes. This listing can be done by using !object command:

0: kd> !object \Driver
[...]

0: kd> !object \FileSystem
[...]

0: kd> !object \Device
[...]

Note that the collection is called Driver Device and not Device Driver.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The next pattern is closely linked to packed and/or obfuscated code. We call it Pre-Obfuscation Residue. Depending on a level of obfuscation and/or packing some initial code and data structures and patterns including fragments of strings may leak in post-obfuscation data giving a clue to intended software behavior:

0:000> s-sa 00000000`00fd4000 L6000
[...]
00000000`00fd943d  "o__"
00000000`00fd9449  "91!We"
00000000`00fd945d  "H5!"
00000000`00fd94d2  "zQ@"
00000000`00fd94dd  "ommandS"
00000000`00fd94f4  “IsDeb”
00000000`00fd94fd  “uggerP”
00000000`00fd9507  “Enc”
00000000`00fd950c  “v)3Po4t”
00000000`00fd9515  “DeXU”
00000000`00fd9520  “xFe”
00000000`00fd952a  “5Eb”
00000000`00fd9533  “SI=l8kev”
00000000`00fd953e  “Z_1m”
00000000`00fd9547  “@IF”
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Packed Code is frequent ingredient of armored malware. Here we demonstrate a few WinDbg commands to detect UPX packed modules with little or no expected strings:

0:000> !dh 00000000`00fd40b0

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
3 number of sections
time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL

OPTIONAL HEADER VALUES
10B magic #
11.00 linker version
6000 size of code
1000 size of initialized data
F000 size of uninitialized data
15600 address of entry point
10000 base of code
----- new -----
0000000010000000 image base
1000 section alignment
200 file alignment
2 subsystem (Windows GUI)
6.00 operating system version
0.00 image version
6.00 subsystem version
17000 size of image
1000 size of headers
0 checksum
0000000000100000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
140  DLL characteristics
Dynamic base
NX compatible
16274 [      AC] address [size] of Export Directory
161DC [      98] address [size] of Import Directory
16000 [     1DC] address [size] of Resource Directory
0 [       0] address [size] of Exception Directory
0 [       0] address [size] of Security Directory
16320 [      10] address [size] of Base Relocation Directory
0 [       0] address [size] of Debug Directory
0 [       0] address [size] of Description Directory
0 [       0] address [size] of Special Directory
0 [       0] address [size] of Thread Storage Directory
157CC [      48] address [size] of Load Configuration Directory
0 [       0] address [size] of Bound Import Directory
0 [       0] address [size] of Import Address Table Directory
0 [       0] address [size] of Delay Import Directory
0 [       0] address [size] of COR20 Header Directory
0 [       0] address [size] of Reserved Directory
SECTION HEADER #1
UPX0 name
F000 virtual size
1000 virtual address
0 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E0000080 flags
Uninitialized Data
(no align specified)
Execute Read Write

SECTION HEADER #2
UPX1 name
6000 virtual size
10000 virtual address
5A00 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E0000040 flags
Initialized Data
(no align specified)
Execute Read Write

SECTION HEADER #3
.rsrc name
1000 virtual size
16000 virtual address
400 size of raw data
5E00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
(no align specified)
Read Write

0:000> s-sa 00000000`00fd40b0 L6600
00000000`00fd40fd  "!This program cannot be run in D"
00000000`00fd411d  "OS mode."
00000000`00fd4188  "Rich"
00000000`00fd4290  “UPX0″
00000000`00fd42b8  “UPX1″
00000000`00fd42e0  “.rsrc”
00000000`00fd448b  “3.08″
00000000`00fd4490  “UPX!”
00000000`00fd449b  “YhHM4″
00000000`00fd44d1  “vqx”
[…]

Such in-memory modules (not yet initialized by a loader) can be saved to disk using .writemem command and unpacked. Once loaded and relocated into some address they still have UPX sections but also they now have more strings:

0:000> s-sa 00000000`691c0000 L300
00000000`691c004d  "!This program cannot be run in D"
00000000`691c006d  "OS mode."
00000000`691c00d8  "Rich"
00000000`691c01e0  “UPX0″
00000000`691c0207  “`UPX1″
00000000`691c022f  “`.rsrc”
[…]
00000000`691d620b  “uGC”
00000000`691d621c  “KERNEL32.DLL”
00000000`691d622a  “LoadLibraryA”
00000000`691d6238  “GetProcAddress”
00000000`691d6248  “VirtualProtect”
00000000`691d6258  “VirtualAlloc”
00000000`691d6266  “VirtualFree”
[…]

0:000> s-su 00000000`691c0000 L(00000000`691d7000-00000000`691c0000)
[...]
00000000`691c8178  “http://www.dumpanalysis.com”
00000000`691c8260  “mscoree.dll”
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Whereas some false positives can be considered soft debugger bugs false negatives can have more severe impact on software behavior analysis especially in malware analysis. We name this pattern Debugger Omission. Typical example here is current .imgscan command which according to documentation should by default scan virtual process space for MZ/PE signatures. Unfortunately it doesn’t detect such signatures in resource pages (we haven’t checked stack regions yet):

0000000000fd0000 image base

SECTION HEADER #4
.rsrc name
6430 virtual size
4000 virtual address
6600 size of raw data
1600 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

0:000> .imgscan /r 00000000`00fd4000 L200

0:000> s -[l2]sa 00000000`00fd4000 l200
00000000`00fd40b0  "MZ"
00000000`00fd40fd  "!This program cannot be run in D"
00000000`00fd411d  "OS mode."
00000000`00fd4188  "Rich"
00000000`00fd4198  "PE"

0:000> !dh 00000000`00fd40b0

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
3 number of sections
time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL
[...]

Another other analysis scenarios found will be added to this pattern. Milder version of it includes !analyze -v that shows us a breakpoint instead of an exception violation from a parallel thread.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The new SF novella to be published this year: Holes of Infinity (ISBN 978-1-908043436). Book description:

The year 1000001. Intergalactic flight is in deep past. Humans switch between holes on manifold computers to navigate through parallel universes. Some universes are fine-tuned for usage as a library storage and others as universal computers. A researcher finds an ancient desktop machine description in one of the universal libraries and builds a working copy.  Fascinated by the ability of ancients to switch between windows to navigate through different views of the same data the researcher discovers a long sought fine-tuning constant that will transform one of universes into a world suitable for colonization. A new era begins…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

These are not really testing patterns but patterns of a user or program behaviour resulting in abnormalities such as colored screens (blue and grey), crash dumps and other support artefacts including performance alerts and UI problem patterns. The first such pattern is called Space Precondition. It is often a case that a process checks for free disk space before proceeding, for example, with updates. This precondition may be violated by a user filling disk after the check but before or during installation. Such was the case with our recent Mac OS X update where we did some copying in the background while the update was downloaded that resulted in insufficient space after reboot, then a grey screen after that and subsequent disk corruption and finally reinstalling OS and loss of several hours better spent with other pattern categories.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

An operating system for inter-universe flight in year 1000001. Simple inter-galactic space flight was already achieved in deep past. Its operating system name was lost.

Disclaimer: Due to so distant future no connection with existing current operating systems is assumed.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Static program analysis is used to eliminate certain coding errors that may lead to abnormal software behaviour. So it is naturally a part of software diagnostics but at source code level. Our goal here is to identify certain patterns directly linkable to patterns we see in memory dumps and software logs and collect them into a catalog. One such pattern candidate is called Loop Construct. It covers conditional and unconditional loops, for example, in one of modern languages:

extern bool soonToBeTrue; 

int mediumValue = ...;

while (true)

{

  TRACE("Waiting");

  sleep(mediumValue);

  if (soonToBeTrue)

  {

    break;

  }

  doHeavyWork();

}

while (--pControl->aFewPasses)

{

  TRACE("Waiting");

  sleep(mediumValue);

  doHeavyWork();

}

Such loops may potentially lead to Spiking Thread memory dump analysis and High Message Current and Density trace analysis patterns. Of course, we shouldn’t suspect every loop but only some that have potential to be altered by Local Buffer Overflow (for mediumValue) or Shared Buffer Overwrite (for Control.aFewPasses) or by a race condition (soonToBeTrue).

We expect things to get more interesting when we start associating source code that uses certain API with patterns of abnormal behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you need to get various real life software traces with millions of lines from complex software environments to learn pattern-driven software log analysis you can us Process Monitor as a modeling tool. Here you can abstract from their “monitoring” and API interception context and consider trace messages as emitted from various processes and threads (like Citrix CDF traces). This approach was used in Accelerated Windows Software Trace Analysis training and Debugging TV Frames episode 0×19.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have a Broken Link for some reason, either from memory corruption, Lateral Damage or Truncated Dump. For example, an active process list enumeration stops after showing some processes (!for_each_thread and !vm also don’t work):

0: kd> !process 0 ff

[...]

TYPE mismatch for process object at fffffa80041da5c0

0: kd> !validatelist nt!PsActiveProcessHead
Blink at address fffffa80041da748 does not point back to previous at fffffa8005bc8cb8

Here we can either try to repair or navigate links manually or use other means such as dumping pool allocations for process structures with Proc pool tag:

0: kd> !poolfind Proc

Searching NonPaged pool (fffffa80032fc000 : ffffffe000000000) for Tag: Proc

*fffffa80033879a0 size:  510 previous size:   a0  (Allocated) Proc (Protected)
*fffffa80033ffad0 size:  530 previous size:  280  (Allocated) Proc (Protected)
*fffffa80041a2af0 size:  510 previous size:   90  (Allocated) Proc (Protected)
*fffffa800439c5c0 size:  530 previous size:   80  (Allocated) Proc (Protected)
[...]
*fffffa8007475ad0 size:  530 previous size:   30  (Allocated) Proc (Protected)
*fffffa80074e8490 size:  530 previous size:  100  (Allocated) Proc (Protected)
*fffffa80075ee0b0 size:  530 previous size:   b0  (Free)      Pro.
*fffffa800761d000 size:  530 previous size:    0  (Free)      Pro.
*fffffa8007645ad0 size:  530 previous size:   b0  (Allocated) Proc (Protected)

0: kd> dc fffffa8007645ad0
fffffa80`07645ad0  0253000b e36f7250 07644030 fffffa80  ..S.Pro.0.d.....
fffffa80`07645ae0  00001000 00000528 00000068 fffff800  ....(...h.......
fffffa80`07645af0  01a1a940 fffff800 00080090 00490024  @...........$.I.
fffffa80`07645b00  000000c4 00000000 00000008 00000000  ................
fffffa80`07645b10  00000000 00000000 00080007 00300033  ............3.0.
fffffa80`07645b20  01a1a940 fffff800 013cfeae fffff8a0  @.........<.....
fffffa80`07645b30  00580003 00000000 05ba19a0 fffffa80  ..X………….
fffffa80`07645b40  05ba19a0 fffffa80 07645b48 fffffa80  ……..H[d…..

0: kd> !process fffffa80`07645b30 ff
PROCESS fffffa8007645b30
SessionId: 0  Cid: 14c4    Peb: 7fffffd4000  ParentCid: 02c4
DirBase: 7233e000  ObjectTable: fffff8a0014d4220  HandleCount: 399.
Image: AppA.exe
VadRoot fffffa80072bc5b0 Vads 239 Clone 0 Private 24675. Modified 23838. Locked 0.
DeviceMap fffff8a0000088f0
Token                             fffff8a000f28060
ElapsedTime                       00:00:53.066
UserTime                          00:00:00.000
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         0
QuotaPoolUsage[NonPagedPool]      0
Working Set Sizes (now,min,max)  (11960, 50, 345) (47840KB, 200KB, 1380KB)
PeakWorkingSetSize                74346
VirtualSize                       331 Mb
PeakVirtualSize                   478 Mb
PageFaultCount                    92214
MemoryPriority                    BACKGROUND
BasePriority                      8
CommitCharge                      25905

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we provide examples of threads waiting for pushlocks as they are not normally seen in crash dumps:

THREAD fffffa80033b5b50  Cid 0004.0030  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880021d9750  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          fffffa800439c620       Image:         AppA.exe
Wait Start TickCount      30819          Ticks: 14746574 (2:15:54:08.028)
Context Switch Count      2800
UserTime                  00:00:00.000
KernelTime                00:00:00.374
Win32 Start Address nt!ExpWorkerThread (0xfffff8000189e530)
Stack Init fffff880021d9db0 Current fffff880021d9470
Base fffff880021da000 Limit fffff880021d4000 Call 0
Priority 12 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`021d94b0 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`021d95f0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`021d9680 fffff800`018c4bf8 nt!KeWaitForSingleObject+0×19f
fffff880`021d9720 fffff800`01c2915d nt!ExfAcquirePushLockShared+0×138
fffff880`021d97a0 fffff800`01c6da31 nt!MmEnumerateAndReferenceImages+0×6d
[…]
fffff880`021d9cb0 fffff800`01b2be5a nt!ExpWorkerThread+0×111
fffff880`021d9d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`021d9d80 00000000`00000000 nt!KxStartSystemThread+0×16

THREAD fffffa8003c9d600  Cid 0004.00ac  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880023d1b30  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      177686         Ticks: 14599707 (2:15:15:56.888)
Context Switch Count      1590
UserTime                  00:00:00.000
KernelTime                00:00:00.124
Win32 Start Address 0xfffff80001bac754
Stack Init fffff880023d1db0 Current fffff880023d1850
Base fffff880023d2000 Limit fffff880023cc000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`023d1890 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`023d19d0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`023d1a60 fffff800`01886183 nt!KeWaitForSingleObject+0×19f
fffff880`023d1b00 fffff800`01cd9982 nt!ExfAcquirePushLockExclusive+0×188
[…]
fffff880`023d1d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`023d1d80 00000000`00000000 nt!KxStartSystemThread+0×16

Instead of explaining what a pushlock is we provide a link to ntdebugging blog article.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of process related patterns is necessary, so I created this post:

• Virtualized Process (WOW64)
• Coupled Processes (strong)
• Coupled Processes (weak)
• Coupled Processes (semantics)
• Manual Dump (process)
• Wait Chain (process objects)
• No Process Dumps
• Special Process
• Zombie Processes
• Process Factory
• Missing Process
• Fat Process Dump
• Value Adding Process
• Frozen Process
• Hidden Process
• One-Thread Process
• Crashed Process
• Past Process

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of thread related patterns is necessary, so I created this post:

• Spiking Thread
• Waiting Thread Time (kernel dumps)
• Waiting Thread Time (user dumps)
• Missing Thread (user space)
• Missing Thread (kernel space)
• Passive Thread (user space)
• Passive System Thread (kernel space)
• Main Thread
• Wait Chain (thread objects)
• Suspended Thread
• Blocked Thread (software)
• Blocked Thread (hardware)
• Blocked Thread (timeout)
• Thread Starvation (realtime priority)
• Thread Starvation (normal priority)
• Affine Thread
• CLR Thread
• Thread Age
• Special Thread (.NET CLR)
• No Current Thread
• Active Thread (Mac OS X)
• One-Thread Process
• Thread Cluster
• Ghost Thread
• Thread Poset
• Active Thread (Windows)
• Not My Thread
• Data Correlation (CPU times)
• Shared Thread

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Although we briefly mentioned session pool in Insufficient Memory (kernel pool) pattern we decided to factor it into a separate (sub)pattern and provide WinDbg commands to analyze possible leaks. The following output shows the sequence of commands that gives you an idea although the example itself was taken from a healthy dump so no red coloring (from my memory leaks in session pool happened mostly in 32-bit past):

1: kd> !vm 4

Terminal Server Memory Usage By Session:

Session ID 0 @ fffff8800324d000:
Paged Pool Usage:        4128K
Commit Usage:            7488K

Session ID 1 @ fffff88002f65000:
Paged Pool Usage:       32852K
Commit Usage:           36488K

1: kd> !session
Sessions on machine: 2
Valid Sessions: 0 1
Error in reading current session

1: kd> !session -s 1
Sessions on machine: 2
Implicit process is now fffffa80`07d79730
Using session 1

1: kd> !poolused 8
Sorting by Session Tag

Pool Used:
NonPaged            Paged
Tag    Allocs     Used    Allocs     Used
TOTAL           4     4208      9500 33475120
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -