Crash Dump Analysis Patterns (Part 28)

Sometimes we have a problem that some functionality is not available or it is unresponsive when we request it. We can suppose that the process implementing that functionality has crashed or hangs. If we know the relationship between processes we can request several user dumps at once or a complete memory dump to analyze the dependency between processes by looking at their stack traces. This is an example of the system level crash dump analysis pattern that I call Coupled Processes.

Process relationship can be implemented via different interprocess communication mechanisms (IPC), for example, Remote Procedure Call (RPC) via LPC (Local Procedure Call) which can be easily identified in stack traces.

My favorite example here is when some application tries to print and hangs. Printing API is exported from WINSPOOL.DLL and it forwards via RPC most requests to Windows Print Spooler service. Therefore it is logical to take two dumps, one from that application and one from spoolsv.exe. Similar example is from Citrix Presentation Server environments related to printer autocreation when there are dependencies between Citrix Printing Service CpSvc.exe and spoolsv.exe. Therefore if new user connections hang and restarting both printing services resolves the issue then we might need to analyze dumps from both services together to confirm this Procedure Call Chain and find the problem 3rd-party printing component or driver.

Back to my favorite example. In the hang application we have the following thread:

  18  Id: 2130.6320 Suspend: 1 Teb: 7ffa8000 Unfrozen
ChildEBP RetAddr
01eae170 7c821c94 ntdll!KiFastSystemCallRet
01eae174 77c72700 ntdll!NtRequestWaitReplyPort+0xc
01eae1c8 77c713ba rpcrt4!LRPC_CCALL::SendReceive+0x230
01eae1d4 77c72c7f rpcrt4!I_RpcSendReceive+0x24
01eae1e8 77ce219b rpcrt4!NdrSendReceive+0x2b
01eae5d0 7307c9ef rpcrt4!NdrClientCall2+0x22e
01eae5e8 73082d8d winspool!RpcAddPrinter+0x1c
01eaea70 0040d81a winspool!AddPrinterW+0x102
01eaef58 0040ee7c App!AddNewPrinter+0x816
...
...
...

Notice winspool and rpcrt4 modules. The application is calling spooler service using RPC to add a new printer and waiting for a reply back. Looking at spooler service dump shows several threads displaying message boxes and waiting for user input: 

  20  Id: 790.5950 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr  Args to Child
03deea70 7739d02f 77392bf3 00000000 00000000 ntdll!KiFastSystemCallRet
03deeaa8 7738f122 03dd0058 00000000 00000001 user32!NtUserWaitMessage+0xc
03deead0 773a1722 77380000 00123690 00000000 user32!InternalDialogBox+0xd0
03deed90 773a1004 03deeeec 03dae378 03dae160 user32!SoftModalMessageBox+0x94b
03deeee0 773b1a28 03deeeec 00000028 00000000 user32!MessageBoxWorker+0x2ba
03deef38 773b19c4 00000000 03defb9c 03def39c user32!MessageBoxTimeoutW+0x7a
03deef58 773b19a0 00000000 03defb9c 03def39c user32!MessageBoxExW+0x1b
03deef74 021f265b 00000000 03defb9c 03def39c user32!MessageBoxW+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
03deef88 00000000 03dae160 03deffec 03dae16a PrinterDriver!UninstallerInstall+0×2cb

Dumping the 3rd parameter of MessageBoxW using WinDbg du command shows the message:

“Installation of the software for your printer is now complete. Restart your computer to make the new settings active.”

Another example when one process starts another and is waiting for it to finish:

0 Id: 2a34.24d0 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0007ec8c 7c822124 ntdll!KiFastSystemCallRet
0007ec90 77e6bad8 ntdll!NtWaitForSingleObject+0xc
0007ed00 77e6ba42 kernel32!WaitForSingleObjectEx+0xac
0007ed14 01002f4c kernel32!WaitForSingleObject+0x12
0007f79c 01003137 userinit!ExecApplication+0x2d3
0007f7dc 0100366b userinit!ExecProcesses+0x1bb
0007fe68 010041fd userinit!StartTheShell+0x132
0007ff1c 010056f1 userinit!WinMain+0x263
0007ffc0 77e523e5 userinit!WinMainCRTStartup+0x186
0007fff0 00000000 kernel32!BaseProcessStart+0x23

- Dmitry Vostokov @ DumpAnalysis.org -

15 Responses to “Crash Dump Analysis Patterns (Part 28)”

  1. Crash Dump Analysis » Blog Archive » Exception and deadlock: pattern cooperation Says:

    […] in this post. The process AppA was hanging and causing another process AppB to hang too (see Coupled Processes pattern). If we look at AppA locked critical sections we would see a loader […]

  2. Crash Dump Analysis » Blog Archive » Coupled processes, wait chains, message box, waiting thread time, paged out data, incorrect stack trace, hidden exception, unknown component and execution residue: pattern cooperation Says:

    […] identify this blocked thread that was processing an LPC request and waiting for an LPC reply from a coupled process, a spooler […]

  3. Crash Dump Analysis » Blog Archive » Blocked LPC thread, coupled processes, stack trace collection and blocked GUI thread: pattern cooperation Says:

    […] after identifying the blocked LPC thread in ServiceA process. We know that ServiceA always asks (coupled with) ServiceB and, indeed, when looking at the latter stack trace collection we see a GUI […]

  4. Software Generalist » Blog Archive » Computational Collectives Says:

    […] rethink boundaries between a collective and its environment. As a simple example, consider coupled processes, or wait chains. Note that this is not the same and it is not used in the same sense as a […]

  5. Crash Dump Analysis » Blog Archive » Statement current, coupled processes, wait chain, spiking thread, hidden exception, and not my version: memory dump and trace analysis pattern cooperation Says:

    […] usually restored to normal operation when one service (ServiceA) is restarted. That service was coupled with ServiceB and their memory dumps were saved and delivered for analysis. Unfortunately, […]

  6. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 92) Says:

    […] failure scenarios we should check these processes for their presence (and also the presence of any coupled processes), hence the name of this pattern: Missing Process. However, if the vital process is present we […]

  7. Software Generalist » Blog Archive » Reading Notebook: 11-March-10 Says:

    […] Explorer usage to inspect hung threads (p. 383) - useful for coupled processes (http://www.dumpanalysis.org/blog/index.php/2007/09/26/crash-dump-analysis-patterns-part-28/) and could be great with simultaneous WinDbg session to inspect wait chains […]

  8. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 28b) Says:

    […] introduced Coupled Processes pattern involves an active request (or an action) and an active wait for a response (or the […]

  9. Crash Dump Analysis » Blog Archive » Strong process coupling, stack trace collection, critical section coruption and wait chains, message box, self-diagnosis and hidden exception and dynamic memory corruption: pattern cooperation Says:

    […] print spooler service process was hanging and blocking print-related requests from other coupled processes. Default analysis of its dump doesn’t show any problem (it shows normal service main […]

  10. Software Generalist » Blog Archive » Reading Notebook: 12-May-10 Says:

    […] So according to memory dump analysis pattern terminology these 2 processes are strongly coupled and this fact can be used for analysis logon problems in terminal services environments: http://www.dumpanalysis.org/blog/index.php/2007/09/26/crash-dump-analysis-patterns-part-28/  […]

  11. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 47) Says:

    […] Today we introduce an icon for Coupled Processes (strong) pattern: […]

  12. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 101) Says:

    […] that wait for a response from another machine (for example, via RPC) and for most of the time Coupled Processes pattern covers that if we assume that processes in that pattern are not restricted to same […]

  13. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 103) Says:

    […] link: Memory Dump Analysis ServicesIn addition to strong and weak process coupling patterns we also have another variant that I call semantic coupling. Some […]

  14. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 111) Says:

    […] additional value or to coexist peacefully in a larger environment. The system thus becomes coupled strongly and / or weekly with other processes it was never intended to work with as opposed to […]

  15. Dmitry Vostokov Says:

    Good diagrams explaining basic printing architecture:
    http://blogs.technet.com/b/askperf/archive/2007/06/19/basic-printing-architecture.aspx
    Also about print driver isolation in W2K8 R2/W7
    http://blogs.technet.com/b/askperf/archive/2009/10/08/windows-7-windows-server-2008-r2-print-driver-isolation.aspx

Leave a Reply

You must be logged in to post a comment.