Comments on: Crash Dump Analysis Patterns (Part 19)

By: Dmitry Vostokov

Dmitry Vostokov — Tue, 22 May 2012 23:02:11 +0000

Example: a cllent LPC thread was waiting but the corresponding server thread value was missing in the output of !lpc extension. Because the server process name was known from the output the server thread was found by the same waiting value:

THREAD 894e13a8 Cid 02d4.0e28 Teb: 7ffdf000 Win32Thread: e6e7cea8 WAIT: (Unknown) KernelMode Non-Alertable 894e1594 Semaphore Limit 0x1 Waiting for reply to LPC MessageId 3786440d: Current LPC port e6edd680 IRP List: 894cb818: (0006,01d8) Flags: 00000884 Mdl: 00000000 Impersonation token: e76aa028 (Level Impersonation) DeviceMap e558a008 Owning Process 895497d0 Image: ApplicationA.exe Attached Process N/A Image: N/A Wait Start TickCount 9426151 Ticks: 162368 (0:00:42:17.000) Context Switch Count 1297 LargeStack UserTime 00:00:00.031 KernelTime 00:00:00.359 Start Address 0x0103e1b0 Stack Init b7ad8000 Current b7ad7174 Base b7ad8000 Limit b7ad3000 Call 0 Priority 15 BasePriority 15 PriorityDecrement 0

0: kd> !lpc message 3786440d Searching message 3786440d in threads ... Client thread 894e13a8 waiting a reply from 3786440d Searching thread 894e13a8 in port rundown queues ...

Server communication port 0xe71e7f08 Handles: 1 References: 1 The LpcDataInfoChainHead queue is empty Connected port: 0xe6edd680 Server connection port: 0xe4f348e0

Client communication port 0xe6edd680 Handles: 0 References: 1 The LpcDataInfoChainHead queue is empty

Server connection port e4f348e0 Name: ServiceBPort Handles: 1 References: 156 Server process : 8a75f438 (ServiceB.exe) Queue semaphore : 8a6c4908 Semaphore state 0 (0x0) The message queue is empty The LpcDataInfoChainHead queue is empty Done.

THREAD 8a6439d8 Cid 0234.02b0 Teb: 7ffa5000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Alertable 8a63b7f0 NotificationEvent IRP List: 8a229a50: (0006,0094) Flags: 00000900 Mdl: 00000000 Not impersonating DeviceMap e1001840 Owning Process 8a75f438 Image: ServiceB.exe Attached Process N/A Image: N/A Wait Start TickCount 9426151 Ticks: 162368 (0:00:42:17.000) Context Switch Count 31433 UserTime 00:00:00.375 KernelTime 00:00:02.625 Win32 Start Address 0x4ab8f3f1 Start Address kernel32!BaseThreadStartThunk (0x77e617ec) Stack Init ba466000 Current ba465c60 Base ba466000 Limit ba463000 Call 0 Priority 13 BasePriority 9 PriorityDecrement 0 ChildEBP RetAddr ba465c78 80833491 nt!KiSwapContext+0x26 ba465ca4 80829a82 nt!KiSwapThread+0x2e5 ba465cec 80938e0a nt!KeWaitForSingleObject+0x346 ba465d50 808897ec nt!NtWaitForSingleObject+0x9a ba465d50 7c82845c nt!KiFastCallEntry+0xfc (TrapFrame @ ba465d64) 00ebf250 7c827b79 ntdll!KiFastSystemCallRet 00ebf254 77e61d1e ntdll!ZwWaitForSingleObject+0xc 00ebf2c4 77c6a927 kernel32!WaitForSingleObjectEx+0xac 00ebf2e0 77c69cbf RPCRT4!UTIL_WaitForSyncIO+0x20 00ebf304 77c6cef0 RPCRT4!UTIL_GetOverlappedResultEx+0x1d 00ebf32c 77c6b02a RPCRT4!CO_SyncRecv+0x71 00ebf354 77c6972f RPCRT4!OSF_CCONNECTION::TransSendReceive+0xb8 00ebf3dc 77c6969c RPCRT4!OSF_CCONNECTION::SendFragment+0x2ae 00ebf434 77c69842 RPCRT4!OSF_CCALL::SendNextFragment+0x1e2 00ebf47c 77c69aba RPCRT4!OSF_CCALL::FastSendReceive+0x148 00ebf498 77c69a3d RPCRT4!OSF_CCALL::SendReceiveHelper+0x5b 00ebf4c8 77c7feb0 RPCRT4!OSF_CCALL::SendReceive+0x41 00ebf4d4 77c80845 RPCRT4!I_RpcSendReceive+0x24 00ebf4e8 77ce415a RPCRT4!NdrSendReceive+0x2b 00ebf8d8 7d1fac12 RPCRT4!NdrClientCall2+0x22e [...]

By: Crash Dump Analysis » Blog Archive » ALPC wait chain, waiting thread time and semantic process coupling: pattern cooperation

Mon, 16 Aug 2010 18:08:38 +0000

[…] deadlock. We could also see that threads waiting for ServiceA.exe sometimes had the greater waiting time than threads waiting for ServiceC so it could be the case that the problem initially started with […]

By: Crash Dump Analysis » Blog Archive » Stack trace collection, missing threads, waiting time, critical section and LPC wait chains: pattern cooperation

Fri, 13 Aug 2010 18:50:47 +0000

[…] the question arises: who was freezing first, ServiceA or ServiceB? We can compare waiting times to answer. We see that waiting time for ServiceB request thread is 3:05:19 and for ServiceA […]

By: Crash Dump Analysis » Blog Archive » Stack trace collection, special process, LPC and critical section wait chains, blocked thread, coupled machines, thread waiting time and IRP distribution anomaly: pattern cooperation

Sun, 18 Jul 2010 14:08:54 +0000

[…] above is blocked trying to set the current directory residing on another server SERVER_B. Its waiting time is almost 13 min 34 […]

By: Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 35)

Mon, 10 May 2010 15:44:00 +0000

[…] Today we introduce an icon for Waiting Thread Time (kernel dumps) pattern: […]

By: Crash Dump Analysis » Blog Archive » Inconsistent dump, stack trace collection, LPC, thread, process, executive resource wait chains, missing threads and waiting thread time: pattern cooperation

Sat, 27 Feb 2010 02:14:12 +0000

[…] see this thread is also blocked by DriverA. We also check thread waiting times. All threads involved in wait chains have their Ticks value less than 86ba5638. This means that […]

By: Crash Dump Analysis » Blog Archive » Wait chain, blocked thread, waiting thread time, IRP distribution anomaly and stack trace collection: pattern cooperation

Thu, 17 Dec 2009 17:35:31 +0000

[…] The blocking thread 86d14ae8 had been blocked waiting for a notification event for more than 2 hours: […]

By: Crash Dump Analysis » Blog Archive » Truncated dump, stack trace collection, waiting thread time and wait chains: pattern cooperation

Thu, 10 Sep 2009 15:18:36 +0000

[…] but most of kernel and user space stack traces not present in the output. ServiceA has many threads waiting for LPC requests during last 5 minutes (the bugcheck was forced after 3 - 4 minutes the issue […]

By: Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 19b)

Thu, 10 Jul 2008 13:46:59 +0000

[…] I wrote about how to calculate thread waiting time in kernel and complete memory dumps (Waiting Thread Time pattern). Now I show how to do it for user dumps. Unfortunately this information is not available […]

By: Dmitry Vostokov

Dmitry Vostokov — Wed, 07 Nov 2007 01:23:38 +0000

!stacks command shows Ticks data for threads