Software Diagnostics Library

When reading and analyzing software traces we always compare them to Master Trace. Another name for this pattern borrowed from narrative theory is Archetype. When looking at the software trace from a system we either know the correct sequence of Activity Regions, expect certain Background and Foreground Components, Event Sequence Order or mentally construct a model based on our experience and Implementation Discourse. For the latter example software engineers internalize software master narratives when they construct code and write tracing code for supportability. For the former example it is important to have a repository of traces corresponding to master traces. This helps in finding deviations after Bifurcation Point. Consider such comparisons similar to regression testing when we check the computation output against the expected prerecorded sequence.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Our sponsor, Memory Dump Analysis Services has published common questions and answers regarding their operation and competitive advantage over traditional software support organizations:

http://www.dumpanalysis.com/common-qa 

I like their bold statement that they don’t have any competitors because they are one step further by providing an audit. This reminds me the statement I learnt from school: Marxism is all powerful because it is true.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is the 3rd revision of memory map visualization tools list where I include IDA Pro and a visualizer from Debug Analyzer.NET (every category is presented in chronological order of my encounter with links):

1. Synthetic

• Numerous debuggers / disassemblers that show memory maps for easy navigation, for example, IDA Pro
• VMMap
• KernelMAP v0.0.1
• MemSpyy
• Virtual memory usage and GC heap usage
• Plugin and / or script-driven visualizers: Debug Analyzer.NET

2. Natural

a. Static

• Dump2Picture (Windows)
• dump2pic (Linux)
• 2D and 3D visualization using general-purpose tools like ParaView.

b. Semi-dynamic

• WinDbg scripts

c. Dynamic

• Haywire

Please let me know any other approaches or links you know.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Blocked Thread (software) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Window of Opportunity - WYSIWYG. Requires scrolling or search to get most of it.

Examples: He opened a log file in notepad and was staring at it with disbelief. There was no error. After some time he closed the window of opportunity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Dump - The last hope.

Examples: He was so desperate on site to ask whether a complete dump will do.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Pre-analysis - Avoiding crash dump analysis anti-patterns like wrong dump or zippocricy. Sometimes it goes with wild explanation no one asked for. Checking whether a software trace is empty before sending it. 

Examples: He is always doing !locks pre-analysis before sending any dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Debugging is defaulting.

Dmitry Vostokov, “It’s time to stop faulting!” programme from the independent Irish political candidate for the next general election who was writing programs in the past

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services announces the launch of CollaborativeA3System for its new and existing customers:

“… is a way to report and discuss memory dump analysis using using well-known and time-proven iterative and incremental processes. Several corporate users can view, comment on and learn various aspects of an analysis report gradually improving initial preliminary analysis to its final deliverable form.”

Source: http://www.dumpanalysis.com/collaborative-analysis-audit-access-system

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the great surprises in 2010 was the popularity of WinDbg.org page. I had never thought it was so popular! Originally forwarded to a subdomain of Crash Dump Analysis Portal when introduced in February, 2008 as a way to quickly download Debugging Tools for Windows and set up symbols it was made a full domain in July, 2010 with corresponding Google Analytics setup. Since then it enjoys 7,000 - 9,000 visits per month with almost 93% return rate with Microsoft, Intel, HP, Road Runner, Dell, Citrix, EMC, CA, Samsung, Cisco, Symantec, AMD, IBM, Oracle, and McAfee among its biggest users. I myself find it very useful in the field when I need to quickly setup debugging components. To the best of my knowledge the page remains the fastest way to that. There will be more useful information linked to it in 2011.

Another trend is the rise of the online version of Debugged! MZ/PE magazine that was completely redesigned in August, 2010 and now approaches 2,000 visitors per month. In 2011 there are some new initiatives coming for this online magazine.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Stack Trace is a general pattern and there can always be found fine-grained patterns in stack traces as well. Here we discuss the general category of such stack trace patterns called Technology-Specific Subtrace (TSST) and give examples related to COM technology.

Consider this trace:

1: kd> k250
ChildEBP RetAddr            
8d5d2808 82a7eb15 nt!KiSwapContext+0x26
8d5d2840 82a7d403 nt!KiSwapThread+0x266
8d5d2868 82a772cf nt!KiCommitThreadWait+0x1df
8d5d28e0 82550d75 nt!KeWaitForSingleObject+0x393
8d5d293c 82550e10 win32k!xxxRealSleepThread+0x1d7
8d5d2958 824ff4b0 win32k!xxxSleepThread+0x2d
8d5d29cc 825547e8 win32k!xxxInterSendMsgEx+0xb1c
8d5d2a1c 825546a4 win32k!xxxSendMessageTimeout+0x13b
8d5d2a44 82533843 win32k!xxxSendMessage+0×28
8d5d2b08 824fd865 win32k!xxxCalcValidRects+0xf7
8d5d2b64 82502c98 win32k!xxxEndDeferWindowPosEx+0×100
8d5d2b84 825170c9 win32k!xxxSetWindowPos+0xf6
8d5d2c08 82517701 win32k!xxxActivateThisWindow+0×2b1
8d5d2c38 82517537 win32k!xxxActivateWindow+0×144
8d5d2c4c 824fd9dd win32k!xxxSwpActivate+0×44
8d5d2ca4 82502c98 win32k!xxxEndDeferWindowPosEx+0×278
8d5d2cc4 824fff82 win32k!xxxSetWindowPos+0xf6
8d5d2d10 82a5342a win32k!NtUserSetWindowPos+0×140
8d5d2d10 76ee64f4 nt!KiFastCallEntry+0×12a (TrapFrame @ 8d5d2d34)
01e2cea0 7621358d ntdll!KiFastSystemCallRet
01e2cea4 6a8fa0eb USER32!NtUserSetWindowPos+0xc
01e2cf14 6a894b13 IEFRAME!SHToggleDialogExpando+0×15a
01e2cf28 6a894d5d IEFRAME!EleDlg::ToggleExpando+0×20
01e2d74c 6a895254 IEFRAME!EleDlg::OnInitDlg+0×229
01e2d7b8 762186ef IEFRAME!EleDlg::DlgProcEx+0×189
01e2d7e4 76209eb2 USER32!InternalCallWinProc+0×23
01e2d860 7620b98b USER32!UserCallDlgProcCheckWow+0xd6
01e2d8a8 7620bb7b USER32!DefDlgProcWorker+0xa8
01e2d8c4 762186ef USER32!DefDlgProcW+0×22
01e2d8f0 76218876 USER32!InternalCallWinProc+0×23
01e2d968 76217631 USER32!UserCallWinProcCheckWow+0×14b
01e2d9a8 76209b1d USER32!SendMessageWorker+0×4d0
01e2da64 76235500 USER32!InternalCreateDialog+0xb0d
01e2da94 76235553 USER32!InternalDialogBox+0xa7
01e2dab4 76235689 USER32!DialogBoxIndirectParamAorW+0×37
01e2dad8 6a5d4952 USER32!DialogBoxParamW+0×3f
01e2db00 6a5d5024 IEFRAME!Detour_DialogBoxParamW+0×47
01e2db24 6a8956df IEFRAME!SHFusionDialogBoxParam+0×32
01e2db58 6a8957bb IEFRAME!EleDlg::ShowDialog+0×398
01e2e638 6a8959d3 IEFRAME!ShowDialogBox+0xb6
01e2eb9c 6a9013ed IEFRAME!ShowElevationPrompt+0×1dd
01e2f010 7669fc8f IEFRAME!CIEUserBrokerObject::BrokerCoCreateInstance+0×202
01e2f040 76704c53 RPCRT4!Invoke+0×2a
01e2f448 76d9d936 RPCRT4!NdrStubCall2+0×2d6
01e2f490 76d9d9c6 ole32!CStdStubBuffer_Invoke+0xb6
01e2f4d8 76d9df1f ole32!SyncStubInvoke+0×3c
01e2f524 76cb213c ole32!StubInvoke+0xb9
01e2f600 76cb2031 ole32!CCtxComChnl::ContextInvoke+0xfa
01e2f61c 76d9a754 ole32!MTAInvoke+0×1a
01e2f64c 76d9dcbb ole32!AppInvoke+0xab
01e2f72c 76d9a773 ole32!ComInvokeWithLockAndIPID+0×372
01e2f778 7669f34a ole32!ThreadInvoke+0×302
01e2f7b4 7669f4da RPCRT4!DispatchToStubInCNoAvrf+0×4a
01e2f80c 7669f3c6 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×16c
01e2f834 766a0cef RPCRT4!RPC_INTERFACE::DispatchToStub+0×8b
01e2f86c 7669f882 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xb2
01e2f8b8 7669f7a4 RPCRT4!LRPC_SCALL::DispatchRequest+0×23b
01e2f8d8 7669f763 RPCRT4!LRPC_SCALL::QueueOrDispatchCall+0xbd
01e2f8f4 7669f5ff RPCRT4!LRPC_SCALL::HandleRequest+0×34f
01e2f928 7669f573 RPCRT4!LRPC_SASSOCIATION::HandleRequest+0×144
01e2f960 7669ee4f RPCRT4!LRPC_ADDRESS::HandleRequest+0xbd
01e2f9dc 7669ece7 RPCRT4!LRPC_ADDRESS::ProcessIO+0×50a
01e2f9e8 766a1357 RPCRT4!LrpcServerIoHandler+0×16
01e2f9f8 76ecd3a3 RPCRT4!LrpcIoComplete+0×16
01e2fa20 76ed0748 ntdll!TppAlpcpExecuteCallback+0×1c5
01e2fb88 76e11174 ntdll!TppWorkerThread+0×5a4
01e2fb94 76efb3f5 kernel32!BaseThreadInitThunk+0xe
01e2fbd4 76efb3c8 ntdll!__RtlUserThreadStart+0×70
01e2fbec 00000000 ntdll!_RtlUserThreadStart+0×1b

In the middle of the stack trace we see COM interface invocation in IEFRAME module. The similar stack trace fragment can be found in the following stack trace where COM IRemUnknown interface implementation resides in .NET CLR mscorwks module:

0:000> kL
ChildEBP RetAddr 
0018a924 68b5f8f0 mscorwks!SafeReleaseHelper+0x77
0018a958 68b04a99 mscorwks!SafeRelease+0x2f
0018a98c 68b04860 mscorwks!IUnkEntry::Free+0x68
0018a9a0 68b049b5 mscorwks!RCW::ReleaseAllInterfaces+0x18
0018a9d0 68b049e1 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
0018aa00 68c0a108 mscorwks!RCW::Cleanup+0x22
0018aa0c 68c0a570 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
0018aa3c 68bd4b3d mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
0018aa4c 75dd8c2e mscorwks!CtxEntry::EnterContextCallback+0×89
0018aa68 763c586c ole32!CRemoteUnknown::DoCallback+0×7a
0018aa84 764405f1 rpcrt4!Invoke+0×2a
0018ae88 75efd936 rpcrt4!NdrStubCall2+0×2ea
0018aed0 75efd9c6 ole32!CStdStubBuffer_Invoke+0xb6
0018af18 75efdf1f ole32!SyncStubInvoke+0×3c
0018af64 75e1223c ole32!StubInvoke+0xb9
0018b040 75e12131 ole32!CCtxComChnl::ContextInvoke+0xfa
0018b05c 75e130fa ole32!MTAInvoke+0×1a
0018b088 75efde47 ole32!STAInvoke+0×46
0018b0bc 75efdcbb ole32!AppInvoke+0xab
0018b19c 75efe34c ole32!ComInvokeWithLockAndIPID+0×372
0018b1c4 75e12ed2 ole32!ComInvoke+0xc5
0018b1d8 75e12e91 ole32!ThreadDispatch+0×23
0018b21c 75a06238 ole32!ThreadWndProc+0×161
0018b248 75a068ea user32!InternalCallWinProc+0×23
0018b2c0 75a07d31 user32!UserCallWinProcCheckWow+0×109
0018b320 75a07dfa user32!DispatchMessageWorker+0×3bc
0018b330 75ddd6be user32!DispatchMessageW+0xf
0018b360 75ddd66d ole32!CCliModalLoop::PeekRPCAndDDEMessage+0×4c
0018b390 75ddd57e ole32!CCliModalLoop::FindMessage+0×30
0018b3f0 75ddd633 ole32!CCliModalLoop::HandleWakeForMsg+0×41
0018b408 75dd1117 ole32!CCliModalLoop::BlockFn+0xc3
0018b488 68a6c905 ole32!CoWaitForMultipleHandles+0xcd
0018b4a8 68a6c866 mscorwks!NT5WaitRoutine+0×51
0018b514 68a6c7ca mscorwks!MsgWaitHelper+0xa5
0018b534 68b5fbe4 mscorwks!Thread::DoAppropriateAptStateWait+0×28
0018b5b8 68b5fc79 mscorwks!Thread::DoAppropriateWaitWorker+0×13c
0018b608 68b5fdf9 mscorwks!Thread::DoAppropriateWait+0×40
0018b664 68a1c5b6 mscorwks!CLREvent::WaitEx+0xf7
0018b678 68b1adb4 mscorwks!CLREvent::Wait+0×17
0018b6c8 68b1ab2a mscorwks!WKS::GCHeap::FinalizerThreadWait+0xfb
0018b764 08fa12c1 mscorwks!GCInterface::RunFinalizers+0×99
[…]

A TSST usually spans several modules. In any stack trace we can also find several TSST that may be overlapping. For example, in the first stack trace above we can discern fragments of COM, RPC, LPC, GUI Dialog, Window Management, and Window Messaging subtraces. In the second trace we can also see GC, Modal Loop, COM Wrapper, and Interface Management stack frames.

The closest software trace analysis pattern here is Implementation Discourse.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Year of Dump Analysis, 0×7DA, was another remarkable year for DumpAnalysis.org. Here is the list of major achievements to report:

- Proposed software narratological framework

- Extended the concept of multithreading with adjoint threading

- After recognizing victimware as a counterpart to crimeware established Victimware.org (currently points to Memory Dump, Software Trace, Debugging and Malware Analysis Portal)

- Established the Dublin School of Security

- Proposed using category theory for memory dump analysis as an alternative to set-theoretical frameworks

- Secured the first Memory Dump Analysis Audit Service and its subsidiary Tool Objects: Unified Troubleshooting and Debugging to become portal sponsors

- Completely redesigned Debugging Experts Magazine Online

- Contributed to two issues of Debugged! MZ/PE magazine: Multithreading and Tools for Software Problem Solving

- Published the 4th volume of Memory Dump Analysis Anthology with color supplement

- Prepared material for the 5th volume of Memory Dump Analysis Anthology (to be published in January, 2011)

- Organized the first Debugging Jokes Competition and the first Tell Your Windows Debugging Story Competition

- Established the Metaphysical Society of Ireland to promote memory dump worldview

- Helped to deliver Fundamentals of Complete Crash and Hang Memory Dump Analysis webinar

- Established the Museum of Debugging and Memory Dumps (the 1st exhibition opens in 2011)

- Introduced Unified Debugging patterns, Workaround patterns and Structural Memory Analysis patterns

- Proposed and organized the first Memory Analysts and Debuggers Day (to be celebrated annually)

- Continued working on CARE research project and published its reference architecture

- Unified memory dump and software trace analysis patterns as software behavior patterns

- Memory Dump Analysis Anthology got international recognition when Korean translation of 1st volume was published

… and much more including many new memory dump and software trace analysis patterns, pattern icons, pattern models, and pattern-driven analysis case studies.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

According to Google Analytics the number of visits / year increased by 0.7% since 2009 with almost 156,000 unique visitors (1% increase) from 184 countries (183 in 2009) and more than 33% of them are coming back. 1,322 visits were via 12 mobile operating systems (512% increase). Here are the top 100 network locations out of 28,612 (1% increase):

Almost 71,000 Google search keywords pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to more than 1,100 web sites that mention the portal and this blog with the first top 100:

Top 25 visiting countries:

More than 9,000 portal and blog pages were viewed a total of almost 387,000 times with top 100 content pages:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally I got a few good crash dumps illustrating Livelock pattern when 2 threads are looping while acquiring and releasing a resource but not progressing. We have these signs in selected WinDbg output below:

- high contention patterns or context switch counts

- increased CPU time in user and / or kernel mode

- at least one livelocked thread is scheduled for execution

- one of livelocked threads has unusual priority boost

- the same thread stack trace for both livelocked threads having similar stats like spent time and context switch counts

- zero waiting ticks

1: kd> !locks

Resource @ 0xfffffa8008464528    Exclusively owned
    Contention Count = 43743004
    NumberOfExclusiveWaiters = 1
     Threads: fffffa8008315b60-01<*>
     Threads Waiting On Exclusive Access:
              fffffa8005769660

41080 total locks, 1 locks currently held

1: kd> !running

Prcbs  Current           Next            
  1    fffff88001e68180  fffff88001e72fc0  fffffa8008315b60  …………….

We have these stack traces from stack trace collection:

THREAD fffffa8008315b60  Cid 0724.2a28  Teb: 000007fffff9c000 Win32Thread: 0000000000000000 ????
IRP List:
    fffffa80082e5990: (0006,0118) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a009f434e0
Owning Process            fffffa8005726360       Image:         ProcessA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      522197         Ticks: 0
Context Switch Count      21665144           
UserTime                  00:00:40.373
KernelTime                00:02:42.791
Win32 Start Address 0×000007fef6939430
Stack Init fffff88007321db0 Current fffff88007321520
Base fffff88007322000 Limit fffff8800731c000 Call 0
Priority 8 BasePriority 6 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`07321560 fffff800`0168a992 nt!KiSwapContext+0×7a
fffff880`073216a0 fffff800`0168ccff nt!KiCommitThreadWait+0×1d2
fffff880`07321730 fffff800`0164c242 nt!KeWaitForSingleObject+0×19f
fffff880`073217d0 fffff800`0168b5ac nt!ExpWaitForResource+0xae
fffff880`07321840 fffff880`04608d91 nt!ExAcquireResourceExclusiveLite+0×14f
fffff880`073218b0 fffff880`04609e4e DriverA!foo+0×42
[…]
fffff880`07321a10 fffff800`0199ef66 nt!IopXxxControlFile+0×607
fffff880`07321b40 fffff800`01682993 nt!NtDeviceIoControlFile+0×56
fffff880`07321bb0 00000000`76ffff2a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`07321c20)
00000000`03a1f488 000007fe`fd06b399 ntdll!NtDeviceIoControlFile+0xa
00000000`03a1f490 00000000`76ea610f KERNELBASE!DeviceIoControl+0×75
00000000`03a1f500 000007fe`f74f3d7c kernel32!DeviceIoControlImplementation+0×7f
[…]

THREAD fffffa8005769660  Cid 0724.10b0  Teb: 000007fffffa6000 Win32Thread: 0000000000000000 WAIT: (WrResource) KernelMode Non-Alertable
    fffffa8006661f20  SynchronizationEvent
IRP List:
    fffffa8006b1ac10: (0006,0118) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a009f434e0
Owning Process            fffffa8005726360       Image:         ProcessA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      522197         Ticks: 0
Context Switch Count      21601988           
UserTime                  00:00:30.147
KernelTime                00:02:30.972
Win32 Start Address 0×000007fef6939430
Stack Init fffff880071bbdb0 Current fffff880071bb520
Base fffff880071bc000 Limit fffff880071b6000 Call 0
Priority 7 BasePriority 6 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`071bb560 fffff800`0168a992 nt!KiSwapContext+0×7a
fffff880`071bb6a0 fffff800`0168ccff nt!KiCommitThreadWait+0×1d2
fffff880`071bb730 fffff800`0164c242 nt!KeWaitForSingleObject+0×19f
fffff880`071bb7d0 fffff800`0168b5ac nt!ExpWaitForResource+0xae
fffff880`071bb840 fffff880`04608d91 nt!ExAcquireResourceExclusiveLite+0×14f
fffff880`071bb8b0 fffff880`04609e4e DriverA!foo+0×42
[…]
fffff880`071bba10 fffff800`0199ef66 nt!IopXxxControlFile+0×607
fffff880`071bbb40 fffff800`01682993 nt!NtDeviceIoControlFile+0×56
fffff880`071bbbb0 00000000`76ffff2a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`071bbc20)
00000000`033bf708 000007fe`fd06b399 ntdll!NtDeviceIoControlFile+0xa
00000000`033bf710 00000000`76ea610f KERNELBASE!DeviceIoControl+0×75
00000000`033bf780 000007fe`f74f3d7c kernel32!DeviceIoControlImplementation+0×7f
[…]

In both traces we have DriverA as Blocking Module.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Wait Chain (named pipes) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Beer time - Time to socialize and discuss the politics of debugging. For some engineers it is actually the time of debugging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finger exercise - Repeated application of certain keyboard debugging commands like t (trace).

Examples: Doing 15 minute finger exercise to locate a pointer change.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

GI Index (Google Insights for Search) shows that crash dump analysis and debugging slowly decline in popularity:

GI for memory dump analysis searches coincides with the birth of DumpAnalysis.org in later 2006 (the only mystery for me is mid 2005 peak):

The birth of memory dump analysis as a discipline probably reinforced the emergence of memory forensics in the early 2006:

We also see that malware analysis is on the rise and it is not a surprise that DumpAnalysis.org is moving to MARS:

Perhaps mid 2005 peak for memory dump analysis searches coincides with the interest in malware analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is the variant of the general Wait Chain pattern where threads are waiting for named pipes. This is visible when we examining the pending IRP of a blocked thread:

THREAD 88ec9020  Cid 17a0.2034  Teb: 7ffad000 Win32Thread: bc28c6e8 WAIT: (Unknown) UserMode Non-Alertable
    89095f48  Semaphore Limit 0x10000
IRP List:
    89a5a370: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 d6c30c48
Owning Process            88fffd88       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      5632994        Ticks: 2980 (0:00:00:46.562)
Context Switch Count      2269                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×00a262d0
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b204c000 Current b204bc60 Base b204c000 Limit b2048000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
b204bc78 80833ec5 nt!KiSwapContext+0×26
b204bca4 80829c14 nt!KiSwapThread+0×2e5
b204bcec 8093b174 nt!KeWaitForSingleObject+0×346
b204bd50 8088b41c nt!NtWaitForSingleObject+0×9a
b204bd50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b204bd64)
058fcabc 7c827d29 ntdll!KiFastSystemCallRet
058fcac0 77e61d1e ntdll!ZwWaitForSingleObject+0xc
058fcb30 77e61c8d kernel32!WaitForSingleObjectEx+0xac
058fcb44 00f98b4a kernel32!WaitForSingleObject+0×12
[…]
058fffec 00000000 kernel32!BaseThreadStart+0×34

0: kd> !irp 89a5a370
Irp is active with 1 stacks 1 is current (= 0×89a5a3e0)
No Mdl: No System Buffer: Thread 88ec9020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 89ebee90 891d4f90 00000000-00000000    pending
                 \FileSystem\Npfs
                                   Args: 00000100 00000000 00000000 00000000

0: kd> !fileobj 891d4f90

\ServiceB\SVC

Device Object: 0x89ebee90   \FileSystem\Npfs
Vpb is NULL

Flags:  0x40080
                Named Pipe
                Handle Created

FsContext: 0xdaeca230 FsContext2: 0x8949bdb0
Private Cache Map: 0x00000001
CurrentByteOffset: 0

The pipe chain can also extend from thread to thread and even cross machine boundary. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Resolution rush - The rush of software technical support and maintenance engineers to provide the resolution to a suddenly escalated incident.

Examples: After it crashed 3 times in a row at the customer site our VP was called and we all got the resolution rush.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -