Software Diagnostics Library

This is a new methodology I’m working upon. The idea came from reading “About the Author” page in a book I got yesterday in my post:

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting

I post a review here and on Amazon when finished reading. Just a few words now. This is the first career book I’m reading where I find pages in roman numerals useful. The page xiii itself looks like a good template (or an example) for a business-oriented CV summary. Thinking now about updating my CV book (2nd edition?):

Resume and CV: As a Book

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for False Positive Dump pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While reading two balanced books about Trotsky I started to admire the Russian signature “С коммунистическим приветом, <имя>” (”S kommunisticheskim privetom, <name>”) that can be translated as “With communist greetings, <name>”. Did they laugh in their red sleeves? When I was at a primary school I loved History (that was long before I saw a computer at Moscow University and I loved Chemistry in secondary and high schools). In fact, to realize my childhood dream, OpenTask, an iterative and incremental publisher, plans to publish a centennial balanced 2 volume bilingual history of Russian revolutions (the work has began already): http://www.opentask.com/history-titles

While commuting today I devised a similar but rectangular 2×2 greeting:

With fix-privet,
Dr. Debuglove

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When the performance is a performative and the performative is a performance we think about performability:

A virtualization is virtualization.

Peter Brook, The Empty Space

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Clock cycle counter for measuring CPU activity  (p. 382)

Process Explorer usage to inspect hung threads (p. 383) - useful for coupled processes (http://www.dumpanalysis.org/blog/index.php/2007/09/26/crash-dump-analysis-patterns-part-28/) and could be great with simultaneous WinDbg session to inspect wait chains (http://www.dumpanalysis.org/blog/index.php/2009/02/17/wait-chain-patterns/)

Process Explorer shows both thread and WOW64 thread stacks on x64 (p. 384)

Thread stack and context query limitations for protected processes (pp. 384 - 386)

Thread pool mechanism was moved into kernel space in Vista (p. 387)

TpWorkerFactory and I/O completion ports and KQUEUE (pp. 387 - 388) - see also a “brief guide” to I/O completion ports: http://www.dumpanalysis.org/blog/index.php/2007/11/27/understanding-io-completion-ports/ 

The mystery of ntdll!TppWorkerThread in stack traces (pp. 389 - 390)

Today we introduce an icon for Dynamic Memory Corruption (kernel pool) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

W32THREAD (p. 371) - One candidate in _ETHREAD that points to it is Tcb.Win32Thread. One interesting code I found on how to extract window message queues from it: http://www.cc.gatech.edu/~brendan/volatility/dl/threadqueues.py. _W32THREAD structure on x64 W2K8 (we also see that is points to _ETHREAD):

0: kd> dt _W32THREAD
win32k!_W32THREAD
+0x000 pEThread         : Ptr64 _ETHREAD
+0x008 RefCount         : Uint4B
+0x010 ptlW32           : Ptr64 _TL
+0x018 pgdiDcattr       : Ptr64 Void
+0x020 pgdiBrushAttr    : Ptr64 Void
+0x028 pUMPDObjs        : Ptr64 Void
+0x030 pUMPDHeap        : Ptr64 Void
+0x038 pUMPDObj         : Ptr64 Void
+0x040 pProxyPort       : Ptr64 Void
+0x048 pClientID        : Ptr64 Void
+0x050 GdiTmpTgoList    : _LIST_ENTRY

!thread output fields (p. 376) - Stack Base and Limit fields can be useful to dump raw stack data via dps command to see execution residue or when reconstructing stack trace, see, for example, this pattern: http://www.dumpanalysis.org/blog/index.php/2009/10/23/crash-dump-analysis-patterns-part-88/

tlist utility (p. 377)

Thread creation calls (pp. 380 - 381) - a condensed view of top level function calls on x64 W2K8:

0: kd> uf /c CreateThread
kernel32!CreateThread (00000000`7731c1c0)
kernel32!CreateThread+0x28 (00000000`7731c1e8):
call to kernel32!CreateRemoteThread (00000000`7731c200)

0: kd> uf /c CreateRemoteThread
Flow analysis was incomplete, some code may be missing
kernel32!CreateRemoteThread (00000000`7731c200)
kernel32!CreateRemoteThread+0x134 (00000000`7731c334):
    call to ntdll!NtCreateThreadEx (00000000`77477790)
  kernel32!CreateRemoteThread+0×166 (00000000`7731c366):
call to ntdll!RtlAllocateActivationContextStack (00000000`77456900)
kernel32!CreateRemoteThread+0×1b4 (00000000`7731c3b4):
call to ntdll!RtlQueryInformationActivationContext (00000000`77456b20)
kernel32!CreateRemoteThread+0×241 (00000000`7731c441):
    call to ntdll!CsrClientCallServer (00000000`7747a460)
  kernel32!CreateRemoteThread+0×281 (00000000`7731c47d):
    call to ntdll!ZwResumeThread (00000000`77477230)
  kernel32!CreateRemoteThread+0×38b (00000000`7731c4ae):
call to kernel32!_security_check_cookie (00000000`7732c200)

0: kd> uf /c NtCreateThreadEx
ntdll!NtCreateThreadEx (00000000`77477790)
no calls found

0: kd> uf NtCreateThreadEx
ntdll!NtCreateThreadEx:
00000000`77477790 4c8bd1          mov     r10,rcx
00000000`77477793 b8a5000000      mov     eax,0A5h
00000000`77477798 0f05            syscall
00000000`7747779a c3              ret

0: kd> uf /c nt!NtCreateThreadEx
nt!NtCreateThreadEx (fffff800`01af60fc)
nt!NtCreateThreadEx+0x3d (fffff800`01af6139):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0x5b (fffff800`01af6157):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0x99 (fffff800`01af6195):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0xc8 (fffff800`01af61c4):
call to nt!PspBuildCreateProcessContext (fffff800`01af5204)
nt!NtCreateThreadEx+0x1e1 (fffff800`01af62dd):
    call to nt!PspCreateThread (fffff800`01af5d40)
  nt!NtCreateThreadEx+0×1f0 (fffff800`01af62ec):
call to nt!PspDeleteCreateProcessContext (fffff800`01af68f0)

0: kd> uf /c nt!PspCreateThread
nt!PspCreateThread (fffff800`01af5d40)
nt!PspCreateThread+0x102 (fffff800`01af5e42):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!PspCreateThread+0x15b (fffff800`01af5e9b):
call to nt!ObfReferenceObject (fffff800`01883250)
nt!PspCreateThread+0x22f (fffff800`01af5f6f):
call to nt!PspAllocateThread (fffff800`01af6338)
nt!PspCreateThread+0x243 (fffff800`01af5f83):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateThread+0x2a6 (fffff800`01af5fe6):
call to nt!PspInsertThread (fffff800`01af4c10)
nt!PspCreateThread+0x318 (fffff800`01af6058):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateThread+0x32a (fffff800`01af606a):
call to nt!_security_check_cookie (fffff800`01895e50)
nt!PspCreateThread+0x36a (fffff800`01af60aa):
call to nt!ObfReferenceObject (fffff800`01883250)
nt!PspCreateThread+0x3a2 (fffff800`01af60e2):
call to nt!ExfAcquireRundownProtection (fffff800`0184f66c)
nt! ?? ::NNGAKEGL::`string'+0x2816e (fffff800`01b3628e):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x281ad (fffff800`01b362ca):
call to nt!ExfReleaseRundownProtection (fffff800`0184f690)
nt! ?? ::NNGAKEGL::`string'+0x281ce (fffff800`01b362eb):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x281d8 (fffff800`01b362f5):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt! ?? ::NNGAKEGL::`string'+0x281e7 (fffff800`01b36304):
call to nt!ExfReleaseRundownProtection (fffff800`0184f690)
nt! ?? ::NNGAKEGL::`string'+0x281ff (fffff800`01b3631c):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x2821a (fffff800`01b36337):
call to nt!PspTerminateThreadByPointer (fffff800`01ad30dc)

Today we introduce an icon for Dynamic Memory Corruption (process heap) pattern:

B/W

Color

Another alternative I considered to use is a chain metaphor but decided that it is more appropriate for linked lists.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just launched my game Blackjack, Jackpot and Slot applets written in pure Java 1.0 in 1997 and they still work in IE (I do launch them occasionally every few years):

http://vostokov.opentask.com/Applets.htm

Can’t believe it!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Multiple Exceptions (kernel mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just noticed that Memory Dump Analysis Anthology, Volume 3 is on a metaphysics bestseller list on Amazon DE today (the volume indeed has a few articles related to Memoidealism and memory dumps + memory traces worldview):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I borrowed a pattern icon idea from the book I’m reading now: Algorithms in a Nutshell

From now on, every memory dump analysis pattern (and later trace analysis patterns) will have platform-independent pictorial representation. Today we introduce an icon for Multiple Exceptions (user mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have a sequence of Activity Regions with increasing values of Statement Current, like depicted here:

The boundaries of regions may be blurry and arbitrarily drawn. Nevertheless, the current is visibly increasing or decreasing, hence the name of this pattern: Trace Acceleration, by analogy with physical acceleration, second-order derivative. We can also metaphorically use here the notion of a partial derivative for trace statement current and acceleration for Threads of Activity and Adjoint Threads of Activity but whether it is useful remains to be seen.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is an extension of Thread of Activity pattern based on the concept of multibraiding and it is called Adjoint Thread of Activity correspondingly. I’m going to illustrate it soon when I publish a synthetic case study involving several software trace analysis patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While reading Software Studies: A Lexicon (Leonardo Books) I came upon an article about Concurrent Versions System written by Simon Yuill. It features a patched quotation from Proudhon:

“The highest perfection of society is found in the union of order and anarchy.” ->

-> “The highest perfection of software is found in the union of order and anarchy.”

Therefore, a bugtation as paradigmatic substitution can be seen as specialised topical patching. It is also augmented with topical coloring:

“The highest perfection of” debugging “is found in the union of order and anarchy.”

Pierre-Joseph Proudhon, What is Property?

Double patching now?

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

NtCreateProcess (pp. 349 - 351) -  a condensed view of top level function calls on x64 W2K8:

0: kd> uf /c nt!NtCreateProcess
nt!NtCreateProcess (fffff800`01c51770)
nt!NtCreateProcess+0x64 (fffff800`01c517d4):
call to nt!NtCreateProcessEx (fffff800`01c516e0)

0: kd> uf /c nt!NtCreateProcessEx
nt!NtCreateProcessEx (fffff800`01c516e0)
nt!NtCreateProcessEx+0x7d (fffff800`01c5175d):
call to nt!PspCreateProcess (fffff800`01c51410)

0: kd> uf /c nt!PspCreateProcess
nt!PspCreateProcess (fffff800`01c51410)
nt!PspCreateProcess+0xd0 (fffff800`01c514e0):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!PspCreateProcess+0xff (fffff800`01c5150f):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateProcess+0x146 (fffff800`01c51556):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!PspCreateProcess+0x1a6 (fffff800`01c515b6):
call to nt!PspAllocateProcess (fffff800`01aac690)
nt!PspCreateProcess+0x202 (fffff800`01c51612):
call to nt!PspInsertProcess (fffff800`01aa6520)
nt!PspCreateProcess+0x21b (fffff800`01c5162b):
call to nt!PspDoHandleSweepSingle (fffff800`01b92770)
nt!PspCreateProcess+0x26f (fffff800`01c5167f):
call to nt!SeDeleteAccessState (fffff800`01b02f8c)
nt!PspCreateProcess+0x27a (fffff800`01c5168a):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateProcess+0x287 (fffff800`01c51697):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateProcess+0x294 (fffff800`01c516a4):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateProcess+0x2a7 (fffff800`01c516b7):
call to nt!_security_check_cookie (fffff800`01895e50)

NtCreateUserProcess (pp. 351 - 360) - a condensed view of top level function calls on x64 W2K8:

0: kd> uf /c nt!NtCreateUserProcess
nt!NtCreateUserProcess (fffff800`01ab2238)
nt!NtCreateUserProcess+0x97 (fffff800`01ab22cf):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateUserProcess+0xb4 (fffff800`01ab22ec):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateUserProcess+0x184 (fffff800`01ab23bc):
call to nt!ExRaiseDatatypeMisalignment (fffff800`01bddd20)
nt!NtCreateUserProcess+0x1c2 (fffff800`01ab23fb):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateUserProcess+0x1dd (fffff800`01ab2416):
call to nt!PspBuildCreateProcessContext (fffff800`01af5204)
nt!NtCreateUserProcess+0x207 (fffff800`01ab2440):
call to nt!PspCaptureCreateInfo (fffff800`01aad390)
nt!NtCreateUserProcess+0x2d1 (fffff800`01ab250a):
call to nt!ZwOpenFile (fffff800`01873480)
nt!NtCreateUserProcess+0x311 (fffff800`01ab254a):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!NtCreateUserProcess+0x378 (fffff800`01ab25b1):
call to nt!ZwCreateSection (fffff800`01873760)
nt!NtCreateUserProcess+0x3af (fffff800`01ab25e8):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!NtCreateUserProcess+0x412 (fffff800`01ab264b):
call to nt!PspCaptureProcessParameters (fffff800`01aae128)
nt!NtCreateUserProcess+0x483 (fffff800`01ab26bc):
call to nt!PspAllocateProcess (fffff800`01aac690)
nt!NtCreateUserProcess+0x546 (fffff800`01ab277f):
call to nt!ObfReferenceObject (fffff800`01883250)
nt!NtCreateUserProcess+0x630 (fffff800`01ab2869):
call to nt!PspAllocateThread (fffff800`01af6338)
nt!NtCreateUserProcess+0x69f (fffff800`01ab28d8):
call to nt!PspInsertProcess (fffff800`01aa6520)
nt!NtCreateUserProcess+0x70e (fffff800`01ab2947):
call to nt!PspInsertThread (fffff800`01af4c10)
nt!NtCreateUserProcess+0x74f (fffff800`01ab2988):
call to nt!PspCreateObjectHandle (fffff800`01b01e10)
nt!NtCreateUserProcess+0x775 (fffff800`01ab29ae):
call to nt!memmove (fffff800`0186fce0)
nt!NtCreateUserProcess+0x7ca (fffff800`01ab2a03):
call to nt!PspUpdateCreateInfo (fffff800`01aadc9c)
nt!NtCreateUserProcess+0x7d9 (fffff800`01ab2a12):
call to nt!SeDeleteAccessState (fffff800`01b02f8c)
nt!NtCreateUserProcess+0x7e9 (fffff800`01ab2a22):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!NtCreateUserProcess+0x7f1 (fffff800`01ab2a2a):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!NtCreateUserProcess+0x7fe (fffff800`01ab2a37):
call to nt!PspDeleteCreateProcessContext (fffff800`01af68f0)
nt!NtCreateUserProcess+0x810 (fffff800`01ab2a49):
call to nt!_security_check_cookie (fffff800`01895e50)
nt!NtCreateUserProcess+0x862 (fffff800`01ab2a9b):
call to nt!ZwOpenFile (fffff800`01873480)
nt!NtCreateUserProcess+0x884 (fffff800`01ab2abd):
call to nt!PspUpdateCreateInfo (fffff800`01aadc9c)
nt! ?? ::NNGAKEGL::`string'+0x4f944 (fffff800`01b55164):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt! ?? ::NNGAKEGL::`string'+0x4f9a5 (fffff800`01b551c5):
call to nt!PspUpdateCreateInfo (fffff800`01aadc9c)
nt! ?? ::NNGAKEGL::`string'+0x4fa80 (fffff800`01b55298):
call to nt!PspGetContextThreadInternal (fffff800`01b02660)
nt! ?? ::NNGAKEGL::`string'+0x4faf3 (fffff800`01b55303):
call to nt!ExfTryToWakePushLock (fffff800`0186b924)
nt! ?? ::NNGAKEGL::`string'+0x4fb21 (fffff800`01b55325):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x4fb3e (fffff800`01b55342):
call to nt!PspDoHandleSweepSingle (fffff800`01b92770)
nt! ?? ::NNGAKEGL::`string'+0x4fb92 (fffff800`01b55392):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x4fba0 (fffff800`01b553a0):
call to nt!PspDoHandleSweepSingle (fffff800`01b92770)
nt! ?? ::NNGAKEGL::`string'+0x4fbb2 (fffff800`01b553b2):
call to nt!PsTerminateProcess (fffff800`01b94140)

The check for import of disallowed API during post-process initialization (p. 361)

CsrCreateProcess (pp. 361 - 362) - Here’s a condensed view of top level function calls on x64 W2K8:

0: kd> uf /c CsrCreateProcess
CSRSRV!CsrCreateProcess (000007fe`fd8c76c8)
CSRSRV!CsrCreateProcess+0x18 (000007fe`fd8c76e0):
call to CSRSRV!CsrpCreateProcess (000007fe`fd8c7280)

0: kd> uf /c CsrpCreateProcess
CSRSRV!CsrpCreateProcess (000007fe`fd8c7280)
CSRSRV!CsrpCreateProcess+0x2e (000007fe`fd8c72ae):
call to ntdll!RtlEnterCriticalSection (00000000`77478920)
CSRSRV!CsrpCreateProcess+0x66 (000007fe`fd8c72e6):
call to CSRSRV!CsrCreateThread (000007fe`fd8c77fc)
CSRSRV!CsrpCreateProcess+0x78 (000007fe`fd8c72f8):
call to ntdll!ZwClose (00000000`77476e00)
CSRSRV!CsrpCreateProcess+0x83 (000007fe`fd8c7303):
call to CSRSRV!CsrAllocateProcess (000007fe`fd8c715c)
CSRSRV!CsrpCreateProcess+0xa4 (000007fe`fd8c7324):
call to CSRSRV!CsrGetProcessLuid (000007fe`fd8c8790)
CSRSRV!CsrpCreateProcess+0x114 (000007fe`fd8c7394):
call to CSRSRV!memcpy (000007fe`fd8cadec)
CSRSRV!CsrpCreateProcess+0x1ab (000007fe`fd8c742b):
call to ntdll!NtSetInformationProcess (00000000`77476ed0)
CSRSRV!CsrpCreateProcess+0x1d2 (000007fe`fd8c7452):
call to ntdll!NtSetInformationProcess (00000000`77476ed0)
CSRSRV!CsrpCreateProcess+0x257 (000007fe`fd8c74d7):
call to ntdll!NtSetInformationProcess (00000000`77476ed0)
CSRSRV!CsrpCreateProcess+0x277 (000007fe`fd8c74f7):
call to ntdll!RtlFreeHeap (00000000`77478c80)
CSRSRV!CsrpCreateProcess+0x2d8 (000007fe`fd8c7558):
call to ntdll!NtQueryInformationThread (00000000`77476f60)
CSRSRV!CsrpCreateProcess+0x2f0 (000007fe`fd8c7570):
call to ntdll!RtlFreeHeap (00000000`77478c80)
CSRSRV!CsrpCreateProcess+0x2fd (000007fe`fd8c757d):
call to CSRSRV!CsrAllocateThread (000007fe`fd8c7b94)
CSRSRV!CsrpCreateProcess+0x32d (000007fe`fd8c75ad):
call to CSRSRV!CsrInsertThread (000007fe`fd8c7bfc)
CSRSRV!CsrpCreateProcess+0x344 (000007fe`fd8c75c4):
call to ntdll!RtlFreeHeap (00000000`77478c80)
CSRSRV!CsrpCreateProcess+0x356 (000007fe`fd8c75d6):
call to ntdll!RtlFreeHeap (00000000`77478c80)
CSRSRV!CsrpCreateProcess+0x365 (000007fe`fd8c75e5):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)
CSRSRV!CsrpCreateProcess+0x393 (000007fe`fd8c7613):
call to CSRSRV!CsrSetBackgroundPriority (000007fe`fd8c712c)
CSRSRV!CsrpCreateProcess+0x3b6 (000007fe`fd8c7636):
call to CSRSRV!CsrInsertProcess (000007fe`fd8c71f0)
CSRSRV!CsrpCreateProcess+0x3d8 (000007fe`fd8c7658):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)

No elevation, virtualization and compatibility checks for protected processes (p. 362)

KiThreadStartup (p. 363) - it looks like on x64 W2K8 it is KxStartUserThread that has this high-level call structure:

0: kd> uf /c nt!KxStartUserThread
nt!KxStartUserThread (fffff800`018b56e0)
nt!KiStartUserThread+0x12 (fffff800`018b5756):
unresolvable call: call    qword ptr [rsp+10h]
nt!KiStartUserThread+0x9f (fffff800`018b57e3):
call to nt!KiInitiateUserApc (fffff800`0189d710)
nt!KiStartUserThread+0xbc (fffff800`018b5800):
call to nt!KiRestoreDebugRegisterState (fffff800`01878860)

PspUserThreadStartup (p. 363) - high-level call structure on x64 W2K8

0: kd> uf /c PspUserThreadStartup
nt!PspUserThreadStartup (fffff800`01b01ae4)
nt!PspUserThreadStartup+0xa1 (fffff800`01b01b85):
call to nt!MmGetSessionLocaleId (fffff800`01b028a4)
nt!PspUserThreadStartup+0xdc (fffff800`01b01bc0):
call to nt!DbgkCreateThread (fffff800`01b02cc0)
nt!PspUserThreadStartup+0x100 (fffff800`01b01be4):
call to nt!PfProcessCreateNotification (fffff800`01ab46cc)
nt!PspUserThreadStartup+0x121 (fffff800`01b01c05):
call to nt!PspInitializeThunkContext (fffff800`01b028e4)
nt! ?? ::NNGAKEGL::`string'+0x42263 (fffff800`01b48d5a):
call to nt!ExfAcquirePushLockExclusive (fffff800`0186aa60)
nt! ?? ::NNGAKEGL::`string'+0x4226b (fffff800`01b48d62):
call to nt!ExfReleasePushLockExclusive (fffff800`018c4b98)
nt! ?? ::NNGAKEGL::`string'+0x42283 (fffff800`01b48d7a):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x42299 (fffff800`01b48d90):
call to nt!PspTerminateThreadByPointer (fffff800`01ad30dc)

System-wide cookie in SharedUserData for pointer encoding/decoding API (p. 363)

LdrInitializeThunk (p. 364) - high-level call structure on x64 W2K8

0: kd> uf /c LdrInitializeThunk
ntdll!LdrInitializeThunk (00000000`774568d0)
ntdll!LdrInitializeThunk+0x9 (00000000`774568d9):
    call to ntdll!LdrpInitialize (00000000`77456990)
  ntdll!LdrInitializeThunk+0×13 (00000000`774568e3):
    call to ntdll!ZwContinue (00000000`77477140)
  ntdll!LdrInitializeThunk+0×1a (00000000`774568ea):
call to ntdll!RtlRaiseStatus (00000000`774e8fa0)
ntdll!RtlAllocateActivationContextStack+0×29 (00000000`7745692d):
call to ntdll!RtlAllocateHeap (00000000`774789b0)

0: kd> uf /c LdrpInitialize
Matched: 00000000`774567f0 ntdll!LdrpInitialize = <no type information>
Matched: 00000000`77456990 ntdll!LdrpInitialize = <no type information>
Ambiguous symbol error at ‘LdrpInitialize’

0: kd> uf /c 00000000`77456990
Flow analysis was incomplete, some code may be missing
ntdll!LdrpInitialize (00000000`77456990)
ntdll!LdrpInitialize+0xaa (00000000`7745689a):
    call to ntdll!LdrpInitializeThread (00000000`77470770)
  ntdll!LdrpInitialize+0xaf (00000000`7745689f):
call to ntdll!ZwTestAlert (00000000`77478490)
ntdll! ?? ::FNODOBFM::`string’+0×20948 (00000000`7746bb8b):
call to ntdll!RtlInitializeSRWLock (00000000`774687f0)
ntdll! ?? ::FNODOBFM::`string’+0×20954 (00000000`7746bb97):
    call to ntdll!LdrpInitializeProcess (00000000`7746ca20)
  ntdll! ?? ::FNODOBFM::`string’+0×20b40 (00000000`7746d540):
call to ntdll!InitSecurityCookie (00000000`7746d560)
ntdll! ?? ::FNODOBFM::`string’+0×20ae4 (00000000`7746e52f):
call to ntdll!NtDelayExecution (00000000`77477050)

0: kd> uf /c ntdll!LdrpInitializeThread
ntdll!LdrpInitializeThread (00000000`77470770)
ntdll!LdrShutdownThread+0x139 (00000000`77437544):
call to ntdll!LdrpCallTlsInitializers (00000000`77437630)
ntdll!LdrpInitializeThread+0x16d (00000000`774376f8):
call to ntdll!LdrpCallTlsInitializers (00000000`77437630)
ntdll!LdrShutdownThread+0x124 (00000000`77448199):
call to ntdll!RtlActivateActivationContextUnsafeFast (00000000`77478bf0)
ntdll!LdrShutdownThread+0x149 (00000000`774481b5):
unresolvable call: call    rsi
ntdll!LdrShutdownThread+0x151 (00000000`774481bd):
call to ntdll!RtlDeactivateActivationContextUnsafeFast (00000000`77478b00)
ntdll!LdrShutdownThread+0x68 (00000000`77448238):
call to ntdll!RtlEnterCriticalSection (00000000`77478920)
ntdll!LdrShutdownThread+0x1cd (00000000`774483cf):
call to ntdll!LdrpFreeTls (00000000`774482f0)
ntdll!LdrShutdownThread+0x1e1 (00000000`774483e3):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)
ntdll!LdrShutdownThread+0x1e6 (00000000`774483e8):
call to ntdll!LdrpCleanupThreadTlsData (00000000`77448490)
ntdll!LdrShutdownThread+0x213 (00000000`77448415):
call to ntdll!RtlFreeHeap (00000000`77478c80)
ntdll!LdrShutdownThread+0x246 (00000000`77448448):
call to ntdll!RtlFreeActivationContextStack (00000000`774480a0)
ntdll!LdrpInitializeThread+0x264 (00000000`774706bf):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)
ntdll!LdrpInitializeThread+0x43 (00000000`774707b3):
call to ntdll!RtlAllocateActivationContextStack (00000000`77456900)
ntdll!LdrpInitializeThread+0x5f (00000000`774707cf):
call to ntdll!RtlEnterCriticalSection (00000000`77478920)
ntdll!LdrpInitializeThread+0x65 (00000000`774707d5):
call to ntdll!LdrpAllocateTls (00000000`774569d0)
ntdll!LdrpInitializeThread+0x13e (00000000`774708ae):
call to ntdll!RtlActivateActivationContextUnsafeFast (00000000`77478bf0)
ntdll!LdrpInitializeThread+0x161 (00000000`774708d5):
unresolvable call: call    rsi
ntdll!LdrpInitializeThread+0x17c (00000000`774708e1):
call to ntdll!RtlDeactivateActivationContextUnsafeFast (00000000`77478b00)
ntdll!LdrpInitializeThread+0x1a9 (00000000`7749017c):
call to ntdll!RtlRaiseStatus (00000000`774e8fa0)
ntdll!LdrpInitializeThread+0x1b5 (00000000`77490188):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)
ntdll!LdrpInitializeThread+0x1d0 (00000000`774901a3):
call to ntdll!NtDelayExecution (00000000`77477050)
ntdll!LdrpInitializeThread+0x1dc (00000000`774901af):
call to ntdll!RtlEnterCriticalSection (00000000`77478920)
ntdll!LdrpInitializeThread+0x233 (00000000`7749020a):
call to ntdll!RtlActivateActivationContextUnsafeFast (00000000`77478bf0)
ntdll!LdrpInitializeThread+0x245 (00000000`7749021c):
call to ntdll!LdrpCallTlsInitializers (00000000`77437630)
ntdll!LdrpInitializeThread+0x250 (00000000`77490227):
call to ntdll!RtlDeactivateActivationContextUnsafeFast (00000000`77478b00)
ntdll!LdrShutdownThread+0x1ab (00000000`7749027e):
call to ntdll!RtlActivateActivationContextUnsafeFast (00000000`77478bf0)
ntdll!LdrShutdownThread+0x1bd (00000000`77490290):
call to ntdll!LdrpCallTlsInitializers (00000000`77437630)
ntdll!LdrShutdownThread+0x1c8 (00000000`7749029b):
call to ntdll!RtlDeactivateActivationContextUnsafeFast (00000000`77478b00)
ntdll! ?? ::FNODOBFM::`string'+0x15c61 (00000000`774bd160):
call to ntdll!NtDelayExecution (00000000`77477050)
ntdll! ?? ::FNODOBFM::`string'+0x15c6e (00000000`774bd16d):
call to ntdll!RtlRaiseStatus (00000000`774e8fa0)
ntdll! ?? ::FNODOBFM::`string'+0x15cb0 (00000000`774bd1a6):
call to ntdll!RtlEnterCriticalSection (00000000`77478920)
ntdll! ?? ::FNODOBFM::`string'+0x15cbc (00000000`774bd1b2):
call to ntdll!RtlLeaveCriticalSection (00000000`77478960)
ntdll! ?? ::FNODOBFM::`string'+0x15cd7 (00000000`774bd1cd):
call to ntdll!RtlFreeHeap (00000000`77478c80)
ntdll! ?? ::FNODOBFM::`string'+0x15cfd (00000000`774bd1f3):
call to ntdll!RtlFreeHeap (00000000`77478c80)

0: kd> uf /c ntdll!LdrpInitializeProcess
Flow analysis was incomplete, some code may be missing
ntdll!LdrpInitializeProcess (00000000`7746ca20)
ntdll!LdrpInitializeProcess+0xf88 (00000000`7746bc0d):
call to ntdll!LdrpUpdateOrderLinks (00000000`774644c0)
ntdll!LdrpInitializeProcess+0xf9c (00000000`7746bc21):
call to ntdll!RtlInsertInvertedFunctionTable (00000000`77464e50)
ntdll!LdrpInitializeProcess+0xfa4 (00000000`7746bc29):
call to ntdll!LdrpAllocateDataTableEntry (00000000`77464380)
ntdll!LdrpInitializeProcess+0x1098 (00000000`7746bc76):
call to ntdll!RtlImageNtHeaderEx (00000000`7747dc00)
ntdll!LdrpInitializeProcess+0x10f1 (00000000`7746bccd):
call to ntdll!RtlAppendUnicodeStringToString (00000000`774574b0)
ntdll!LdrpInitializeProcess+0x110f (00000000`7746bceb):
call to ntdll!LdrpUpdateOrderLinks (00000000`774644c0)
ntdll!LdrpInitializeProcess+0x1123 (00000000`7746bcff):
call to ntdll!RtlInsertInvertedFunctionTable (00000000`77464e50)
ntdll!LdrpInitializeProcess+0x1128 (00000000`7746bd04):
call to ntdll!RtlInitializeHistoryTable (00000000`7746da90)
ntdll!LdrpInitializeProcess+0x11c9 (00000000`7746bd4f):
call to ntdll!RtlpInitCurrentDir (00000000`7746db70)
ntdll!LdrpInitializeProcess+0x1648 (00000000`7746bdca):
call to ntdll!LdrLoadDll (00000000`77463e30)
ntdll!LdrpInitializeProcess+0x16ba (00000000`7746bdf9):
call to ntdll!LdrGetProcedureAddressEx (00000000`7747dd10)
ntdll!LdrpInitializeProcess+0x171f (00000000`7746be16):
call to ntdll!LdrpWalkImportDescriptor (00000000`77466390)
ntdll!LdrpInitializeProcess+0x18cd (00000000`7746be5b):
call to ntdll!LdrpInitializeTls (00000000`7746e380)
ntdll!LdrpInitializeProcess+0x1940 (00000000`7746be88):
call to ntdll!LdrpRunInitializeRoutines (00000000`77464650)
ntdll!LdrpInitializeProcess+0x138e (00000000`7746bedf):
call to ntdll!LdrLoadDll (00000000`77463e30)
ntdll!LdrpInitializeProcess+0x13ff (00000000`7746bf0d):
call to ntdll!LdrGetProcedureAddressEx (00000000`7747dd10)
ntdll!LdrpInitializeProcess+0x1475 (00000000`7746bf3b):
call to ntdll!LdrGetProcedureAddressEx (00000000`7747dd10)
ntdll!LdrpInitializeProcess+0x14eb (00000000`7746bf69):
call to ntdll!LdrGetProcedureAddressEx (00000000`7747dd10)
ntdll!LdrpInitializeProcess+0x19f5 (00000000`7746bfc5):
call to ntdll!_security_check_cookie (00000000`7747acb0)
ntdll!LdrpInitializeProcess+0x32 (00000000`7746ca52):
call to ntdll!RtlSetUnhandledExceptionFilter (00000000`7746c2d0)
ntdll!LdrpInitializeProcess+0xe9 (00000000`7746ca9a):
call to ntdll!RtlInitNlsTables (00000000`7746c920)
ntdll!LdrpInitializeProcess+0xf6 (00000000`7746caa7):
call to ntdll!RtlResetRtlTranslations (00000000`7746c410)
ntdll!LdrpInitializeProcess+0xfe (00000000`7746caaf):
call to ntdll!RtlpInitSRWLock (00000000`7746c530)
ntdll!LdrpInitializeProcess+0x103 (00000000`7746cab4):
call to ntdll!RtlpInitConditionVariable (00000000`7746c550)
ntdll!LdrpInitializeProcess+0x213 (00000000`7746cb7d):
call to ntdll!RtlImageNtHeader (00000000`774567b0)
ntdll!LdrpInitializeProcess+0x273 (00000000`7746cbd7):
call to ntdll!LdrpInitializeExecutionOptions (00000000`7746c6b0)
ntdll!LdrpInitializeProcess+0x353 (00000000`7746cc2f):
call to ntdll!RtlImageDirectoryEntryToData (00000000`7746a940)
ntdll!LdrpInitializeProcess+0x3cd (00000000`7746cc95):
call to ntdll!RtlNormalizeProcessParams (00000000`7746c2f0)
ntdll!LdrpInitializeProcess+0x423 (00000000`7746cce3):
call to ntdll!RtlImageDirectoryEntryToData (00000000`7746a940)
ntdll!LdrpInitializeProcess+0x448 (00000000`7746cd02):
call to ntdll!memset (00000000`77478830)
ntdll!LdrpInitializeProcess+0x58c (00000000`7746cd53):
call to ntdll!RtlpInitDeferredCriticalSection (00000000`7746c640)
ntdll!LdrpInitializeProcess+0x7d5 (00000000`7746ceb5):
call to ntdll!RtlInitializeCriticalSection (00000000`77455d20)
ntdll!LdrpInitializeProcess+0x7fb (00000000`7746cedb):
call to ntdll!RtlInitializeHeapManager (00000000`7746c7a0)
ntdll!LdrpInitializeProcess+0x84b (00000000`7746cf2a):
call to ntdll!RtlCreateHeap (00000000`77466ed0)
ntdll!LdrpInitializeProcess+0x8e2 (00000000`7746cf51):
call to ntdll!RtlAllocateActivationContextStack (00000000`77456900)
ntdll!LdrpInitializeProcess+0x8f6 (00000000`7746cf65):
call to ntdll!EtwpInitializeDll (00000000`7746c250)
ntdll!LdrpInitializeProcess+0x916 (00000000`7746cf85):
call to ntdll!RtlCreateTagHeap (00000000`7746d320)
ntdll!LdrpInitializeProcess+0x942 (00000000`7746cfb1):
call to ntdll!RtlCreateTagHeap (00000000`7746d320)
ntdll!LdrpInitializeProcess+0x962 (00000000`7746cfd1):
call to ntdll!RtlpInitEnvironmentBlock (00000000`7746d380)
ntdll!LdrpInitializeProcess+0x96f (00000000`7746cfde):
call to ntdll!RtlpInitParameterBlock (00000000`7746d7f0)
ntdll!LdrpInitializeProcess+0xa5e (00000000`7746d068):
call to ntdll!RtlInitUnicodeString (00000000`7747ad10)
ntdll!LdrpInitializeProcess+0xa73 (00000000`7746d07d):
call to ntdll!RtlAppendUnicodeStringToString (00000000`774574b0)
ntdll!LdrpInitializeProcess+0xa87 (00000000`7746d091):
call to ntdll!RtlAppendUnicodeStringToString (00000000`774574b0)
ntdll!LdrpInitializeProcess+0xaf0 (00000000`7746d0fe):
call to ntdll!ZwOpenDirectoryObject (00000000`77477290)
ntdll!LdrpInitializeProcess+0xc2a (00000000`7746d171):
call to ntdll!ZwOpenSymbolicLinkObject (00000000`77477cb0)
ntdll!LdrpInitializeProcess+0xc6b (00000000`7746d1b2):
call to ntdll!ZwQuerySymbolicLinkObject (00000000`77477f60)
ntdll!LdrpInitializeProcess+0xc7a (00000000`7746d1c1):
call to ntdll!ZwClose (00000000`77476e00)
ntdll!LdrpInitializeProcess+0xe50 (00000000`7746d24d):
call to ntdll!LdrpAllocateDataTableEntry (00000000`77464380)
ntdll!LdrpInitializeProcess+0xee4 (00000000`7746d289):
call to ntdll!RtlImageNtHeaderEx (00000000`7747dc00)
ntdll!LdrpInitializeProcess+0x30d (00000000`77473eb0):
call to ntdll!NtQueryInformationProcess (00000000`77476ea0)
ntdll!LdrpInitializeProcess+0x635 (00000000`77473ef0):
call to ntdll!RtlSetBits (00000000`77466c00)
ntdll!LdrpInitializeProcess+0x873 (00000000`77473f19):
call to ntdll!RtlCreateHeap (00000000`77466ed0)
ntdll!LdrpInitializeProcess+0xbb8 (00000000`774744f9):
call to ntdll!ZwOpenDirectoryObject (00000000`77477290)
ntdll!LdrpInitializeProcess+0xe10 (00000000`77474554):
call to ntdll!RtlAppendUnicodeStringToString (00000000`774574b0)
ntdll!LdrpInitializeProcess+0x77 (00000000`77490a96):
call to ntdll!NtQueryVirtualMemory (00000000`77476f40)
ntdll!LdrpInitializeProcess+0xb3 (00000000`77490ad2):
call to ntdll!NtQueryVirtualMemory (00000000`77476f40)
ntdll!LdrpInitializeProcess+0x2d2 (00000000`77490b92):
call to ntdll!LdrpLogDbgPrint (00000000`774ec5a0)
ntdll!LdrpInitializeProcess+0x2db (00000000`77490b9d):
call to ntdll!DbgBreakPoint (00000000`77476060)
ntdll!LdrpInitializeProcess+0x720 (00000000`77490d34):
call to ntdll!LdrQueryImageFileExecutionOptions (00000000`77473260)
ntdll!LdrpInitializeProcess+0x790 (00000000`77490da4):
call to ntdll!LdrpLogDbgPrint (00000000`774ec5a0)
ntdll!LdrpInitializeProcess+0x79a (00000000`77490dae):
call to ntdll!DbgBreakPoint (00000000`77476060)
ntdll!LdrpInitializeProcess+0x7c6 (00000000`77490dda):
call to ntdll!RtlControlStackTraceDataBase (00000000`774e3cd0)
ntdll!LdrpInitializeProcess+0x8ac (00000000`77490e27):
call to ntdll!LdrpLogDbgPrint (00000000`774ec5a0)
ntdll!LdrpInitializeProcess+0x8bf (00000000`77490e3a):
call to ntdll!DbgBreakPoint (00000000`77476060)
ntdll!LdrpInitializeProcess+0x9ff (00000000`77490e8c):
call to ntdll!RtlQueryImageFileKeyOption (00000000`77473320)
ntdll!LdrpInitializeProcess+0xb0f (00000000`77490ee2):
call to ntdll!RtlInitUnicodeString (00000000`7747ad10)
ntdll!LdrpInitializeProcess+0xcea (00000000`77490f57):
call to ntdll!LdrpLogDbgPrint (00000000`774ec5a0)
ntdll!LdrpInitializeProcess+0xd01 (00000000`77490f6e):
call to ntdll!LdrpInitializationFailure (00000000`774ed120)
ntdll!LdrpInitializeProcess+0xd3f (00000000`77490f82):
call to ntdll!RtlAllocateHeap (00000000`774789b0)
ntdll!LdrpInitializeProcess+0xd7d (00000000`77490fc0):
call to ntdll!LdrpLogDbgPrint (00000000`774ec5a0)
ntdll!LdrpInitializeProcess+0xd90 (00000000`77490fd3):
call to ntdll!DbgBreakPoint (00000000`77476060)

Private vs. shared assemblies (p. 365)

Memoriarch

The highest title in Memorianic religious hierarchy. Equivalent to similar titles of Pope, Patriarch or Archbishop. Derived from Memoria (Latin, memory) and arch (Latin root, chief, first, rule).

- Dmitry Vostokov @ Memory Religion Portal - 

This post was motivated during my work on a memory dump differing tool called DumpLogic that can do logical and arithmetic operations between memory snapshots, for example, take a difference between them for further visualization. This tool is forthcoming next week and it resulted in another simple tool called DumpFilter. The latter allows to filter certain unsigned integer (DWORD) values from a memory dump (or any binary file) by replacing them with 0xFFFFFFFF and all other values with 0×00000000. The resultant binary file can be visualized by any data visualization package or transformed to a bitmap file using Dump2Picture to see distribution of filtered values.

As a filtering example I used TestDefaultDebugger64 to generate a process user memory dump. It was converted to a BMP file by Dump2Picture: 

Then I filtered only AV exception code 0xc0000005:

C:\>DumpFilter tdd64.dmp tdd64.bin <dwords.txt

dwords.txt just contained one line 

c0000005

It is possible to filter many values. Just put more lines to dwords.txt file. tdd64.bin was converted to tdd64.bmp by Dump2Picture:

C:\>Dump2Picture tdd64.bin tdd64.bmp

Because the image had only black and while RGBA colors I saved it as a B/W bitmap (click to enlarge, it is a 3236×3236 1.3Mb bitmap):

Every EV exception code is a white dot there but it is difficult to see them unless magnified. So I enlarged them manually on the following map:

I put them on the original image too. We can see that exception processing spans many areas:

The tool and the sample dwords.txt file (for c0000005 and 80000003) can be downloaded from this location:

Download DumpFilter

Another example: Night Sky memory space art image is just a fragment after filtering all 1 values from another process memory dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here is a synthetic case study to show various wait chain patterns. The complete memory dump from a frozen system is inconsistent, saved by LiveKd. Stack trace collection shows many threads waiting for LPC replies:

THREAD 87432520  Cid 0b10.0dd8  Teb: 7ff83000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    8743270c  Semaphore Limit 0x1
Waiting for reply to LPC MessageId 007ade85:
Current LPC port d676e560
Not impersonating
DeviceMap                 d6f05be8
Owning Process            87581c78       Image:         ProcessA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4093415        Ticks: 659565 (0:02:51:45.703)
Context Switch Count      111255            
UserTime                  00:00:27.781
KernelTime                00:00:02.015
Win32 Start Address MSVCR71!_threadstartex (0×7c3494f6)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6439000 Current b6438c08 Base b6439000 Limit b6436000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b6438c10 00000000 0×0

Checking MessageId by using !lpc command gives us the following LPC server thread that is waiting for a mutant owned by thread 866d63e8 (this is equivalent that thread 85b209d0 is waiting for thread 866d63e8):

THREAD 85b209d0  Cid 1524.2770  Teb: 7ff78000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    85e9ba00  Mutant - owning thread 866d63e8
Not impersonating
DeviceMap                 d64008d8
Owning Process            87200880       Image:         ProcessB.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4093415        Ticks: 659565 (0:02:51:45.703)
Context Switch Count      28            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×007ade85
LPC Server thread working on message Id 7ade85
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b42e5000 Current b42e4c60 Base b42e5000 Limit b42e2000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b42e4c68 00000000 0xa000a02

Thread 866d63e8 is waiting for a process 86b33b30:

THREAD 866d63e8  Cid 1524.3984  Teb: 7ff6e000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    86b33b30  ProcessObject
    866d6460  NotificationTimer
Not impersonating
DeviceMap                 d64008d8
Owning Process            87200880       Image:         ProcessB.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4755080     
Context Switch Count      4767            
UserTime                  00:00:00.015
KernelTime                00:00:00.000
Win32 Start Address 0×007a5051
LPC Server thread working on message Id 7a5051
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init ab459000 Current ab458c60 Base ab459000 Limit ab456000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
ab458c78 8083d5b1 nt!KiSwapContext+0×26
ab458ca4 8083df9e nt!KiSwapThread+0×2e5
ab458cec 8092ae67 nt!KeWaitForSingleObject+0×346
ab458d50 80833bef nt!NtWaitForSingleObject+0×9a
ab458d50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ ab458d64)
0499fbec 7c827d29 ntdll!KiFastSystemCallRet
0499fbf0 76548721 ntdll!ZwWaitForSingleObject+0xc
0499fc0c 7654aa54 ProcessB!WaitForProcess+0×4a
[…]
0499ffec 00000000 kernel32!BaseThreadStart+0×34

We find the following thread in the process 86b33b30 (there is only one thread left where we expect many of them in ProcessC):

THREAD 85a6fdb0  Cid 5c0c.26f4  Teb: 7ffdf000 Win32Thread: b9b778a0 WAIT: (Unknown) KernelMode Non-Alertable
    86686030  SynchronizationEvent
    85a6fe28  NotificationTimer
Not impersonating
DeviceMap                 d6767248
Owning Process            86b33b30       Image:         ProcessC.EXE
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4755681     
Context Switch Count      77668                 LargeStack
UserTime                  00:00:00.437
KernelTime                00:00:04.421
*** ERROR: Module load completed but symbols could not be loaded for ProcessC.EXE
Win32 Start Address 0×300010cc
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init 9ad92000 Current 9ad91a40 Base 9ad92000 Limit 9ad8d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
9ad91a58 8083d5b1 nt!KiSwapContext+0×26
9ad91a84 8083df9e nt!KiSwapThread+0×2e5
9ad91acc 8081e05b nt!KeWaitForSingleObject+0×346
9ad91b08 8082e012 nt!ExpWaitForResource+0xd5
9ad91b28 b6a9c1ee nt!ExAcquireResourceExclusiveLite+0×8d
WARNING: Stack unwind information not available. Following frames may be wrong.
9ad91b38 b6ab09d3 DriverA+0×21ee
[…]
9ad91c3c 80840153 DriverA+0×1ed6
9ad91c50 8092b51f nt!IofCallDriver+0×45
9ad91c64 8092b454 nt!IopSynchronousServiceTail+0×10b
9ad91d00 8092b574 nt!IopXxxControlFile+0×60f
9ad91d34 80833bef nt!NtDeviceIoControlFile+0×2a
9ad91d34 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ 9ad91d64)
0012d4e0 00000000 ntdll!KiFastSystemCallRet

It is blocked by DriverA waiting for an executive resource. Fortunately !locks command works for this inconsistent dump and we finally reach thread 86ba5638:

0: kd> !locks

Resource @ 0x85610d30    Exclusively owned
    Contention Count = 4
    NumberOfExclusiveWaiters = 1
     Threads: 86ba5638-01<*>
     Threads Waiting On Exclusive Access:
              85a6fdb0

This thread belongs to another instance of ProcessC.exe (different process 8690dd88):

0: kd> !thread 86ba5638 1f
THREAD 86ba5638  Cid 186c.1574  Teb: 7ffdf000 Win32Thread: b9a28a70 WAIT: (Unknown) KernelMode Non-Alertable
    8944e3d4  SynchronizationEvent
Not impersonating
DeviceMap                 d6767248
Owning Process            8690dd88       Image:         ProcessC.EXE
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4074148        Ticks: 678832 (0:02:56:46.750)
Context Switch Count      95896                 LargeStack
UserTime                  00:00:00.593
KernelTime                00:00:05.343
*** ERROR: Module load completed but symbols could not be loaded for ProcessC.EXE
Win32 Start Address 0×300010cc
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init 99ef2000 Current 99ef19c0 Base 99ef2000 Limit 99eec000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
99ef19d8 8083d5b1 nt!KiSwapContext+0×26
99ef1a04 8083df9e nt!KiSwapThread+0×2e5
99ef1a4c 80853f3f nt!KeWaitForSingleObject+0×346
99ef1a64 8081d45f nt!KiAcquireFastMutex+0×13
99ef1a70 b6a9c70d nt!ExAcquireFastMutex+0×20
WARNING: Stack unwind information not available. Following frames may be wrong.
99ef1a7c b6aaf22a DriverA+0×270d
[…]
99ef1c3c 80840153 DriverA+0×1ed6
99ef1c50 8092b51f nt!IofCallDriver+0×45
99ef1c64 8092b454 nt!IopSynchronousServiceTail+0×10b
99ef1d00 8092b574 nt!IopXxxControlFile+0×60f
99ef1d34 80833bef nt!NtDeviceIoControlFile+0×2a
99ef1d34 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ 99ef1d64)
0012db08 00000000 ntdll!KiFastSystemCallRet

We see this thread is also blocked by DriverA. We also check thread waiting times. All threads involved in wait chains have their Ticks value less than 86ba5638. This means that thread 86ba5638 was blocked earlier. We contact DriverA vendor for any possible updates.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -