Software Diagnostics Library

The following memory dump of a hanging process was manually generated:

Loading Dump File [Process.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER'

Applying default analysis command shows the following stack traces:

0:000> .kframes 100
Default stack trace depth is 0n256 frames

0:000> !analyze -v 

[...] 

STACK_TEXT: 
009ef258 7c827d0b ntdll!KiFastSystemCallRet
009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc
009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0×1a3
009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8
009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0×128
009ef374 77e764ea ntdll!LdrGetProcedureAddress+0×18
009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0×46f
009ef5f4 7c34957c msvcr71!_XcptFilter+0×15f
009ef600 7c34246e msvcr71!_endthreadex+0xb7
009ef628 7c828752 msvcr71!_except_handler3+0×61
009ef64c 7c828723 ntdll!ExecuteHandler2+0×26
009ef6f4 7c82855e ntdll!ExecuteHandler+0×24
009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
009efa00 7c82a319 ntdll!RtlpFindEntry+0×68
009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0×606
009efc6c 7c3416db msvcr71!_heap_alloc+0xe0
009efc74 7c360947 msvcr71!_nh_malloc+0×10
009efc80 0285f893 msvcr71!operator new+0xb
[…]
009effb8 77e64829 msvcr71!_endthreadex+0xa0
009effec 00000000 kernel32!BaseThreadStart+0×34

[...]

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT_STATUS_BREAKPOINT

We see the exeption was dispatched because of heap corruption and the unhandled exception filter is blocked waiting for a critical section. We can immediately recommend to enable full page heap. However, let’s explore the dump file further. By listing all threads we find that there were 2 exceptions with the second one having the following stack trace:

0:000> ~*kb

[...]

  98  Id: be4.2ca4 Suspend: 1 Teb: 7ff68000 Unfrozen
ChildEBP RetAddr  Args to Child             
0f83c80c 7c827d0b 7c83d236 00000154 00000000 ntdll!KiFastSystemCallRet
0f83c810 7c83d236 00000154 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0f83c84c 7c83d281 00000154 00000004 0f83c8b0 ntdll!RtlpWaitOnCriticalSection+0×1a3
0f83c86c 7c82dabf 7c8877a0 00000000 00000000 ntdll!RtlEnterCriticalSection+0xa8
0f83c90c 7c82dab1 00000000 77e767cc 00000000 ntdll!LdrpGetProcedureAddress+0×128
0f83c928 77e764ea 00000000 77e767cc 00000000 ntdll!LdrGetProcedureAddress+0×18
0f83cb8c 77e792a3 0f83cbb4 77e61ac1 0f83cbbc kernel32!UnhandledExceptionFilter+0×46f
0f83cb94 77e61ac1 0f83cbbc 00000000 0f83cbbc kernel32!BaseThreadStart+0×4a
0f83cbbc 7c828752 0f83cca0 0f83ffdc 0f83ccbc kernel32!_except_handler3+0×61
0f83cbe0 7c828723 0f83cca0 0f83ffdc 0f83ccbc ntdll!ExecuteHandler2+0×26
0f83cc88 7c82855e 0f83c000 0f83ccbc 0f83cca0 ntdll!ExecuteHandler+0×24
0f83cc88 7c82be3e 0f83c000 0f83ccbc 0f83cca0 ntdll!KiUserExceptionDispatcher+0xe
0f83cf94 7c82a319 00340178 00000050 0b425f54 ntdll!RtlpFindEntry+0×68
0f83d1c0 7c3416b3 00340000 00000000 00000278 ntdll!RtlAllocateHeap+0×606
0f83d200 7c3416db 00000278 7c3416f8 00000278 msvcr71!_heap_alloc+0xe0
0f83d208 7c3416f8 00000278 00000000 003214fd msvcr71!_nh_malloc+0×10
0f83d214 003214fd 00000278 0f83d23c 023e5912 msvcr71!malloc+0xf
[…]
0f83f8b4 77ce33e1 0cf4ca00 0f83fa98 0000000c rpcrt4!Invoke+0×30
0f83fcb4 77ce35c4 00000000 00000000 0bf93d8c rpcrt4!NdrStubCall2+0×299
0f83fcd0 77c7ff7a 0bf93d8c 0beae6b8 0bf93d8c rpcrt4!NdrServerCall2+0×19
0f83fd04 77c8042d 0cf4f53e 0bf93d8c 0f83fdec rpcrt4!DispatchToStubInCNoAvrf+0×38
0f83fd58 77c80353 0000006f 00000000 0cf74358 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
0f83fd7c 77c7e0d4 0bf93d8c 00000000 0cf74358 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
0f83fdbc 77c7e080 0bf93d8c 0bf93d44 00000000 rpcrt4!RPC_INTERFACE::DispatchToStubWithObject+0xc0
0f83fdfc 77c812f0 001c85c0 0bf0f510 00189d08 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×41e
0f83fe20 77c88678 0bf0f548 0f83fe38 001c85c0 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0f83ff84 77c88792 0f83ffac 77c8872d 0bf0f510 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0f83ff8c 77c8872d 0bf0f510 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
0f83ffac 77c7b110 00145520 0f83ffec 77e64829 rpcrt4!BaseCachedThreadRoutine+0×9d
0f83ffb8 77e64829 0be74ce0 00000000 00000000 rpcrt4!ThreadStartRoutine+0×1b
0f83ffec 00000000 77c7b0f5 0be74ce0 00000000 kernel32!BaseThreadStart+0×34

[...] 

 102  Id: be4.2ac0 Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr  Args to Child             
134df91c 7c827d0b 7c83d236 000004e4 00000000 ntdll!KiFastSystemCallRet
134df920 7c83d236 000004e4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
134df95c 7c83d281 000004e4 00000004 00340000 ntdll!RtlpWaitOnCriticalSection+0×1a3
134df97c 7c82a264 00340608 00000000 0000008c ntdll!RtlEnterCriticalSection+0xa8
134dfba4 7c3423aa 00340000 00000008 0000008c ntdll!RtlAllocateHeap+0×313
134dfbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6
134dfbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138
134dfc1c 7c82ed97 7c34229f 7c340000 00000002 ntdll!LdrpCallInitRoutine+0×14
134dfcb8 7c82ec9f 134dfd28 134dfd28 00000000 ntdll!LdrpInitializeThread+0×10d
134dfd14 7c8284c5 134dfd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×25

If we inspect the list of locked critical sections and compare stack trace parameters for threads 2ac0 and 2ca4 we would see them deadlocked:

0:000> !locks

CritSec ntdll!LdrpLoaderLock+0 at 7c8877a0
WaiterWoken        No
LockCount          6
RecursionCount     1
OwningThread       2ac0
EntryCount         0
ContentionCount    36
*** Locked

CritSec +340608 at 00340608
WaiterWoken        No
LockCount          32
RecursionCount     1
OwningThread       2ca4
EntryCount         0
ContentionCount    6f
*** Locked

[...]

Looking at other locks (omitted here) we can find additional wait chains but we employ default hang analysis command to find one chain for us:

0:000> !analyze -v -hang

[...]

LOADERLOCK_BLOCKED_API:  UnhandledExceptionFilter:LdrGetProcedureAddress: LdrpGetProcedureAddress:

LOADERLOCK_OWNER_API:  _LdrpInitialize:LdrpInitializeThread:LdrpCallInitRoutine:

DERIVED_WAIT_CHAIN: 

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   2   be4.c20 Critical Section       -->
x  98  be4.2ca4 Critical Section       -->
x  102 be4.2ac0 Critical Section       --^

WAIT_CHAIN_COMMAND:  ~2s;k;;~98s;k;;~102s;k;;

BLOCKING_THREAD:  00002ca4

DEFAULT_BUCKET_ID:  APPLICATION_HANG_DEADLOCK_HeapCorruption

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_DEADLOCK_HeapCorruption

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

I’m sorry to announce that the book has been delayed and the publication date has been changed to 30th of November, 2009. I promise this delay is the last one and kindly ask you to be patient. As a bonus or compensation for it, the book will also cover Windows 7.

- Dmitry Vostokov @ DumpAnalysis.org -

Reading stack traces like English verse (remeber to read from bottom to top):

0:01> ~8kL
ChildEBP RetAddr 
009ef258 7c827d0b ntdll!KiFastSystemCallRet
009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc
009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0x1a3
009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8
009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0x128
009ef374 77e764ea ntdll!LdrGetProcedureAddress+0x18
009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0x46f
009ef5f4 7c34957c msvcr71!_XcptFilter+0x15f
009ef600 7c34246e msvcr71!_endthreadex+0xb7
009ef628 7c828752 msvcr71!_except_handler3+0x61
009ef64c 7c828723 ntdll!ExecuteHandler2+0x26
009ef6f4 7c82855e ntdll!ExecuteHandler+0x24
009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
009efa00 7c82a319 ntdll!RtlpFindEntry+0x68
009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0x606
009efc6c 7c3416db msvcr71!_heap_alloc+0xe0
009efc74 7c360947 msvcr71!_nh_malloc+0x10
009efc80 0285f893 msvcr71!operator new+0xb
009efca8 02852e38 SQLModule!ODBCDelete+0xf3
009efd54 0269acff Store!ProcessDeletes+0x3d
009eff38 0269badb Store!UpdateStore+0xe
009eff58 00323499 Common!WorkItem+0x15c
009eff84 7c349565 Common!WorkItemThread+0x339
009effb8 77e64829 msvcr71!_endthreadex+0xa0
009effec 00000000 kernel32!BaseThreadStart+0x34

The new thread started
To work through items
It got an item
Handled to the store
To run delete requests
Through Oh-Dee-Bee-See
It tried to alloc
But crashed in malloc
While browsing the heap
Exception was dispatched
And handler called at once
But couldn’t find a filter
And called default one
That filter needed help
And looked for its address
But halted in suspense
While entering crit sec.

- Dmitry Vostokov @ DumpAnalysis.org -

Based on John Moore 8 science criteria we can consider Memory Dump Analysis (MDA) as a science:

1. MDA is based on data (memory dumps) collected in the field or re-pro / test environment.

2. Data (memory dumps) is collected to answer troubleshooting, debugging or forensics and intelligence questions. Observations in memory dumps are made to support or refute these questions.

3. Analysis of data (via memory dump analyzers, debuggers and log analyzers) is done objectively.

4. Troubleshooting, debugging or forensics hypotheses are developed and they are consistent with observations and compatible with general conceptual computer memory framework.

5. Troubleshooting, debugging or forensics hypotheses are tested and several comparable competing ones may be developed at any one time.

6. Generalizations are made that are valid universally within the domain of MDA.

7. The facts are confirmed independently.

8. Previously puzzling facts are explained.

It is also interesting to generalize the domain of MDA to empirical data collection via the so called universal memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org -

Every debugging engineer needs to know how the code is interpreted or compiled. Debugging complex problems or doing memory analysis on general-purpose operating systems often requires understanding the syntax and semantics of several programming languages and their run-time support. The knowledge of optimization techniques is also important for low-level debugging when the source code is not available. The following book provides an overview of all important concepts and discusses almost 50 languages. I read the first edition 6 years ago and I liked it so much that I’m now reading the second edition.

Programming Language Pragmatics, Second Edition

- Dmitry Vostokov @ DumpAnalysis.org -

A new book is planned by OpenTask with the following preliminary details:

State and Event: Categorical Foundations of Being and Time (ISBN: 978-1906717643)

- Dmitry Vostokov @ DumpAnalysis.org -

Just found on Amazon this forthcoming book:

Windows® Sysinternals Administrator’s Reference (Inside Out)

- Dmitry Vostokov @ DumpAnalysis.org -

In December, 2000 I decided to apply for an Irish working visa after receiving an offer from Ericsson. Although offered compensation for a Senior Software Designer position was less than I had in Moscow at that time working for 2 companies simultaneously, I decided to accept the offer because of 3 primary reasons:

1. To learn spoken English

2. To work for one company only instead of many and dedicate free time to learning, reading and socializing

3. Working visa conditions such as the freedom to change an employer and virtually unlimited duration (permission to stay is renewed every 2 years)

If I had a US H-1B visa offer at the same time I would have definitely chosen the Irish one because I consider being tied to an employer in the case of H-1B as a kind of modern slavery. So Irish immigration is more progressive in this regard.  When in March, 2003 I got my redundancy in another Irish company I was calm because I knew that I could find another employer in Ireland and I didn’t have to leave the country like many engineers did leave USA during dot-com crash and what we see now when USA companies lay off H-1B workers. Therefore I had my working visa renewed 3-4 times and after 8 years I got today a stamp that allows me to stay indefinitely without any condition (practically until my passport expires). This is very good and allows me to proceed further with Memory Analysis and Debugging Institute and associated publishing activities. I also applied for Irish citizenship that if granted gives me freedom to visit other EU countries on demand and ease access to USA and Canada. Last year I got an invitation to Canada to participate in the development of Windows Driver Foundation exam but I had to abandon my visit because of simultaneous passport change and the need to renew my stay in Ireland that happened to coincide with the visit dates.

- Dmitry Vostokov @ DumpAnalysis.org -

In one kernel memory dump we can see the signs of a busy system where all processors are executing non-idle threads:

0: kd> !running

System Processors ff (affinity mask)
  Idle Processors 0

     Prcb      Current   Next   
  0  ffdff120  8b223928            ................
  1  f772f120  8a765380            ................
  2  f7737120  89365db0            ................
  3  f773f120  8833adb0            ................
  4  f7747120  889bbdb0            ................
  5  f774f120  8c085db0            ................
  6  f7757120  8aa79698            ................
  7  f775f120  896c0668            ................

When inspecting them we see that some are kernel worker threads without a process context, for example:

0: kd> !thread 8aa79698 1f
THREAD 8aa79698  Cid 0004.6edc  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 6
Not impersonating
DeviceMap                 d66008d0
Owning Process            8d15d648       Image:         System
Wait Start TickCount      2548878        Ticks: 3 (0:00:00:00.046)
Context Switch Count      248713            
UserTime                  00:00:00.000
KernelTime                00:00:00.906
Start Address nt!ExpWorkerThread (0×80881860)
Stack Init acfbc000 Current acfbbcec Base acfbc000 Limit acfb9000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
acfbbaec b19483f3 driver+0×3f65
acfbbb6c 8081e095 driver+0×23f3
acfbbb80 af36044a nt!IofCallDriver+0×45
[…]
acfbbdac 8094bea4 nt!ExpWorkerThread+0xeb
acfbbddc 8088f57e nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

some threads with an associated process context are running in kernel space:

0: kd> !thread 889bbdb0 1f
THREAD 889bbdb0  Cid 6c58.6f98  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 4
Not impersonating
DeviceMap                 d66008d0
Owning Process            89ad8b18       Image:         csrss.exe
Wait Start TickCount      2548880        Ticks: 1 (0:00:00:00.015)
Context Switch Count      129536            
UserTime                  00:00:00.000
KernelTime                00:00:00.312
Start Address displaydriver!Thread (0xbfad4a60)
Stack Init a439d000 Current a439cc70 Base a439d000 Limit a439a000 Call 0
Priority 13 BasePriority 10 PriorityDecrement 3
ChildEBP RetAddr
a439c004 bfad707f displaydriver!CalcRegion+0×30
[…]
a439cddc 8088f57e nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

and some threads with an associated process context are running in user space:

0: kd> !thread 8c085db0 1f
THREAD 8c085db0  Cid 2318.231c  Teb: 7ffdd000 Win32Thread: b4b5ebe8 RUNNING on processor 5
Not impersonating
DeviceMap                 dc1a71f0
Owning Process            8b02e458       Image:         Application.EXE
Wait Start TickCount      2548881        Ticks: 0
Context Switch Count      725122                 LargeStack
UserTime                  00:00:01.625
KernelTime                00:00:03.062
Win32 Start Address 0×30001084
Start Address 0×7c8217f8
Stack Init ad648000 Current ad647c50 Base ad648000 Limit ad642000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr
0013fb7c 00000000 0×7c81b910

Because none of them consumed much CPU the patter of Spiking Thread is ruled out and CPU load can be explained by the number of active user sessions and this appears to be normal:

0: kd> !session
Sessions on machine: 50

However looking at ERESOURCE locks we see many blocked threads and signs of possible wait chains:

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****

Resource @ 0x8cbfaa68    Exclusively owned
    Contention Count = 22969
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 109
     Threads: 8a961db0-01<*> 8bf532b0-01   
     Threads Waiting On Exclusive Access:
              8b4532f0       884fc648       88c58a00       8a751360      
              88ed64f8       89aa6738       89870db0       881dedb0      
              8a6d7b40       8b4a4db0       89818ad0       8afcedb0      
              8a2ca020       88684db0       8b411020       89d595c0      
              8d1573f0       88d06020       8aed8b38       8a8c9020      
              8a5a0a50       8a1f63b0       89b66688       89bf1db0      
              880dab18       882e6730       895d8020       88e6d3f0      
              896e6748       89802100       8a604508       8907c5e8      
              8890a020       885e2300       8a061bd8       88445340      
              88113db0       8a680db0       89b53370       88c3f2a0      
              88a774f8       8834ddb0       89d78888       88386020      
              897ca8d8       8b3532d0       882341d0       8a4a9b80      
              87e7c4f8       895e5db0       8846f4e8       89df3db0      
              889b8b40       89d82db0       89e4b720       8aadadb0      
              8aa63020       88852020       8a249ba8       891b8c20      
              8b3f95f0       8aace760       8b470020       897ad388      
              8c07dba8       8a331628       896c74d0       8997cb40      
              88e133c8       886eddb0       8864e518       89ab5698      
              88d8bdb0       89996db0       8ac54d28       87f42020      
              882b1020       8857fdb0       895f3db0       88b0ab40      
              8a1aadb0       8b819020       8b3bf388       88315660      
              8a45db18       883fbdb0       88f53db0       87f209a0      
              8978ddb0       8840c868       8823c1c8       88277db0      
              89c0a8c8       88322940       8a475db0       8a6ad460      
              8a35a4c8       88e3da40       886b1b40       8886a2a0      
              8897d750       8b30bdb0       8a123020       8b0ad7f8      
              8a256930       885cedb0       88ec8db0       887d7ba8      
              88175b90      

Resource @ 0x8b8f09a8    Shared 1 owning threads
    Contention Count = 123597
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 6
     Threads: 88200840-01    8a92ddb0-01<*>
     Threads Waiting On Exclusive Access:
              8a317db0       8d151840       899acdb0       8a961db0
              891ac940       89ee5db0      

Resource @ 0x8ac79f08    Exclusively owned
    Contention Count = 717691
    NumberOfExclusiveWaiters = 12
     Threads: 8a5193f0-01<*>
     Threads Waiting On Exclusive Access:
              880e7b40       8a60adb0       8a543108       8a4be020      
              8a77c360       8a470730       87f12db0       8a4618d0      
              895c5600       8a942b98       8a453b40       8a3bf020      

Resource @ 0x8a73ed28    Exclusively owned
    Contention Count = 4
    NumberOfExclusiveWaiters = 2
     Threads: 8a45db18-01<*>
     Threads Waiting On Exclusive Access:
              8a412db0       8a542268      

Resource @ 0x8a621bf8    Exclusively owned
    Contention Count = 8532
    NumberOfExclusiveWaiters = 3
     Threads: 8a412db0-01<*>
     Threads Waiting On Exclusive Access:
              8a5193f0       8a60cdb0       8a595c78      

Resource @ 0x8a4c8b20    Exclusively owned
    Contention Count = 1
    NumberOfExclusiveWaiters = 1
     Threads: 8a92ddb0-01<*>
     Threads Waiting On Exclusive Access:
              89524a70      

Resource @ 0x8a43b0e8    Exclusively owned
    Contention Count = 1135854
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 9
     Threads: 8aaa3020-01<*> 88efb400-01   
     Threads Waiting On Exclusive Access:
              89f883b0       8a273a70       89f82c10       89fd9020      
              89ec0db0       89571290       89edcdb0       88930400      
              8845f4c8      

Resource @ 0x89f7dbe8    Exclusively owned
    Contention Count = 2
    NumberOfExclusiveWaiters = 2
     Threads: 891b8c20-01<*>
     Threads Waiting On Exclusive Access:
              89ecedb0       89fc3020      

Resource @ 0x89f82f28    Exclusively owned
    Contention Count = 26674
    NumberOfExclusiveWaiters = 2
     Threads: 89fc3020-01<*>
     Threads Waiting On Exclusive Access:
              8aaa3020       8a02db40      

Resource @ 0x89315320    Exclusively owned
    Contention Count = 509247
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 19
     Threads: 89261428-01<*> 89313a08-01   
     Threads Waiting On Exclusive Access:
              89cc7db0       8ad26528       8970db68       88ef64d0      
              8a629020       89450798       8825c9a8       89206378      
              8a7c7b90       89162890       8ae7c020       883318e0      
              88bd6358       89367db0       8952aaa0       8a817b40      
              881d65b8       8ab74db0       889202c0      

Resource @ 0x893872d8    Exclusively owned
    Contention Count = 5079
    NumberOfExclusiveWaiters = 3
     Threads: 896e6748-01<*>
     Threads Waiting On Exclusive Access:
              89261428       893bc3c8       892a88a8      

Resource @ 0x8924adf8    Exclusively owned
    Contention Count = 1
    NumberOfExclusiveWaiters = 1
     Threads: 88ed64f8-01<*>
     Threads Waiting On Exclusive Access:
              89146660      

Resource @ 0x890281b0    Exclusively owned
    Contention Count = 4
    NumberOfExclusiveWaiters = 4
     Threads: 88d06020-01<*>
     Threads Waiting On Exclusive Access:
              88b5c528       88c5aa98       87ef77b8       88c48b40      

Resource @ 0x88d40440    Exclusively owned
    Contention Count = 13
    NumberOfExclusiveWaiters = 1
     Threads: 899acdb0-01<*>
     Threads Waiting On Exclusive Access:
              895e6db0      

Resource @ 0x88ed0c20    Exclusively owned
    Contention Count = 2
    NumberOfExclusiveWaiters = 2
     Threads: 895e6db0-01<*>
     Threads Waiting On Exclusive Access:
              88ad7540       88b5f620      

Resource @ 0x894e7990    Exclusively owned
    Contention Count = 3852647
    NumberOfExclusiveWaiters = 12
     Threads: 881b14b8-01<*>
     Threads Waiting On Exclusive Access:
              88a13db0       87f12020       8aad7a20       8820a020      
              8824bdb0       88213db0       88eefdb0       88ab7550      
              889fe808       89df17a0       8aa83430       8a8f73c8      

Resource @ 0x88559288    Exclusively owned
    Contention Count = 7422
    NumberOfExclusiveWaiters = 3
     Threads: 880dab18-01<*>
     Threads Waiting On Exclusive Access:
              881b14b8       88311020       882ab660      

Resource @ 0x8aff12b0    Exclusively owned
    Contention Count = 6
    NumberOfExclusiveWaiters = 1
     Threads: 89524a70-01<*>
     Threads Waiting On Exclusive Access:
              8a92ddb0      

62174 total locks, 75 locks currently held

Starting with the thread 8a961db0 that blocks 109 other threads we can unravel the following deadlock:

109 threads -> 8a961db0 -> 8a92ddb0 -> 89524a70 -> 8a92ddb0 -> 89524a70 -> …

Looking at threads involved in the deadlock we see that they belong to the same process and deadlocked in kernel space when running through driverA.sys code:

0: kd> !thread 89524a70 1f
THREAD 89524a70  Cid 1fdc.26cc  Teb: 7ffdd000 Win32Thread: b4d0fea8 WAIT: (Unknown) KernelMode Non-Alertable
    89170648  SynchronizationEvent
    89524ae8  NotificationTimer
IRP List:
    88e7a008: (0006,0268) Flags: 00000000  Mdl: 00000000
    8a7cc228: (0006,0268) Flags: 00000000  Mdl: 00000000
    89e67b90: (0006,0268) Flags: 00000000  Mdl: 00000000 Not impersonating
DeviceMap                 e2e671d0
Owning Process            88c37020       Image:         ApplicationA.exe
Wait Start TickCount      2548760        Ticks: 121 (0:00:00:01.890)
Context Switch Count      4850                 LargeStack
UserTime                  00:00:00.734
KernelTime                00:00:01.718
Win32 Start Address 0×00404054
Start Address 0×7c8217f8
Stack Init 91971000 Current 91970278 Base 91971000 Limit 9196a000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr
91970290 80833ec5 nt!KiSwapContext+0×26
919702bc 80829bc0 nt!KiSwapThread+0×2e5
91970304 8087e0db nt!KeWaitForSingleObject+0×346
91970340 8087e2f5 nt!ExpWaitForResource+0xd5
91970360 b0a1cf6d nt!ExAcquireResourceExclusiveLite+0×8d
91970374 b0a08cef driverA+0×2ef6d
919703cc b0a089cc driverA+0×1acef
919703f4 b0a209d9 driverA+0×1a9cc
9197056c b0a20386 driverA+0×329d9
91970630 b0a1dc32 driverA+0×32386
919706e8 b0a20508 driverA+0×2fc32
919707ac b0a1eec1 driverA+0×32508
919708a0 b0a21e90 driverA+0×30ec1
91970930 b0a171c9 driverA+0×33e90
919709c4 b0a16c9d driverA+0×291c9
91970a38 b0a600b3 driverA+0×28c9d
91970a84 b0a45dda driverA+0×720b3
91970afc b0a4657a driverA+0×57dda
91970b48 8081e095 driverA+0×5857a
91970b5c f7876d28 nt!IofCallDriver+0×45
91970b88 8081e095 fltmgr!FltpDispatch+0×152
91970b9c f74fc6ca nt!IofCallDriver+0×45
91970bb4 f7876d28 driverB+0×56ca
91970be0 8081e095 fltmgr!FltpDispatch+0×152
91970bf4 b195a4e1 nt!IofCallDriver+0×45
91970c18 b195a5d0 driverC!PassThrough+0xd1
91970c28 8081e095 driverC!Dispatch+0×80
91970c3c f7876d28 nt!IofCallDriver+0×45
91970c68 8081e095 fltmgr!FltpDispatch+0×152
91970c7c 808f7601 nt!IofCallDriver+0×45
91970c90 808f5339 nt!IopSynchronousServiceTail+0×10b
91970d38 8088ac9c nt!NtWriteFile+0×663
91970d38 7c9485ec nt!KiFastCallEntry+0xfc

0: kd> !thread 8a92ddb0 1f
THREAD 8a92ddb0  Cid 1fdc.7b98  Teb: 7ffa9000 Win32Thread: b4deeae8 WAIT: (Unknown) KernelMode Non-Alertable
    8b422388  SynchronizationEvent
    8a92de28  NotificationTimer
Not impersonating
DeviceMap                 e2e671d0
Owning Process            88c37020       Image:         ApplicationA.exe
Wait Start TickCount      2548760        Ticks: 121 (0:00:00:01.890)
Context Switch Count      956                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Win32 Start Address 0×01381fa0
Start Address 0×7c8217ec
Stack Init 917c1000 Current 917c034c Base 917c1000 Limit 917bd000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr
917c0364 80833ec5 nt!KiSwapContext+0×26
917c0390 80829bc0 nt!KiSwapThread+0×2e5
917c03d8 8087e0db nt!KeWaitForSingleObject+0×346
917c0414 8087e2f5 nt!ExpWaitForResource+0xd5
917c0434 b0a1cf6d nt!ExAcquireResourceExclusiveLite+0×8d
917c0448 b0a08cef driverA+0×2ef6d
917c04a0 b0a089cc driverA+0×1acef
917c04c8 b0a13787 driverA+0×1a9cc
917c053c b0a0bfaa driverA+0×25787
917c057c b0a0c3b3 driverA+0×1dfaa
917c0858 b0a0ccaf driverA+0×1e3b3
917c0934 b0a6074c driverA+0×1ecaf
917c097c b0a4f9d2 driverA+0×7274c
917c0a18 b0a501f6 driverA+0×619d2
917c0a40 b0a5020c driverA+0×621f6
917c0a4c b0a50442 driverA+0×6220c
917c0a6c b0a50687 driverA+0×62442
917c0ac4 b0a50cb0 driverA+0×62687
917c0b08 b0a50ddd driverA+0×62cb0
917c0b18 8081e095 driverA+0×62ddd
917c0b2c f7876d28 nt!IofCallDriver+0×45
917c0b58 8081e095 fltmgr!FltpDispatch+0×152
917c0b6c f74fc6ca nt!IofCallDriver+0×45
917c0b84 f7876d28 driverB+0×56ca
917c0bb0 8081e095 fltmgr!FltpDispatch+0×152
917c0bc4 b195a4e1 nt!IofCallDriver+0×45
917c0be8 b195a5d0 driverC!PassThrough+0xd1
917c0bf8 8081e095 driverC!Dispatch+0×80
917c0c0c f7876d28 nt!IofCallDriver+0×45
917c0c38 8081e095 fltmgr!FltpDispatch+0×152
917c0c4c 808f7601 nt!IofCallDriver+0×45
917c0c60 808f1b45 nt!IopSynchronousServiceTail+0×10b
917c0c84 afdfebd5 nt!NtQueryDirectoryFile+0×5d
917c0cf8 afdff95d driverD+0×8bd5
917c0d30 8088ac9c driverD+0×995d
917c0d30 7c9485ec nt!KiFastCallEntry+0xfc

Other wait chains seem to be subordinate to the main deadlock chain. 

- Dmitry Vostokov @ DumpAnalysis.org -

Last week I had an opportunity to present a pattern-driven memory dump analysis methodology at a global engineering conference. Now in a series of articles I’m going to clarify certain points and extend it to wider domain of memory analysis including computer memory forensics and intelligence.

Today I post the reworked picture of a waterfall-like analysis process:

Various phases and their relationship will be discussed in subsequent parts together with examples. 

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Management Bits and Tips blog including January posts:

Management Bit and Tip 0×2000

Management Bit and Tip 0×4000

A Thread Was Killed 

- Dmitry Vostokov @ DumpAnalysis.org -

Monthly summary of my Literate Scientist blog including January reviews:

Blog Anniversary

Social Sciences as Sorcery

Literate Scientists and Their Books

Reality Rules

A Brief History of Theology

- Dmitry Vostokov @ DumpAnalysis.org -

A contribution to Software Resistentialism:

Software objects can be classified scientifically into three major categories: those that don’t work, those that crash and those that hang.

Russell Wayne Baker

- Dmitry Vostokov @ DumpAnalysis.org -

“Don’t” guess “it, get a larger” dump.

Anthony’s Law of Force

- Dmitry Vostokov @ DumpAnalysis.org -

Debugger logs (textual output) from commands like !process 0 ff and various scripts can be very long and consist of thousands of pages. I found the following reading technique useful for my daily memory dump analysis activities:

CSA-QSA

Checklists-Skim-Analyze—Questions-Survey-Analyze   

1. First, have a checklist

2. Skim through the log several times

3. Write analysis notes

4. Have a list of questions based on problem description and steps 1-3

5. Survey the log

6. Write analysis notes

Repeat steps 2,3 and 5,6 if necessary.

This technique can also be applied to reading any large logs, for example, voluminous CDF or ETW traces.

- Dmitry Vostokov @ DumpAnalysis.org -

Named after Georg Cantor CAN.TOR.OS^∞ brings computation from the distant future into today. The transfinite worldview and universe of tomorrow into the finite worldview and universe of today. Cantor OS drives transfinite computing and saves transfinite memory dumps. More on this in subsequent parts as I have to come back to finite memory dumps… One cautious note though: transfinite doesn’t mean absolute infinity, or God-like computation, the latter is the realm of Memory Religion. 

^(∞) TOR is a new transfinite operation in addition to finite OR, AND or XOR 

- Dmitry Vostokov @ DumpAnalysis.org -

These dumps are larger than any finite memory dump and contain all of them inside (see the definition of a transfinite number). Think about them as a variant of the Library of Babel where all possible memory snapshots of your Windows or Linux PC are stored including Googol dumps. If you have some code then all possible code defects are there too. An interesting question then arises. If this dump is collected what kind of patterns we can see there? Are these patterns extrapolated infinite versions of finite patterns or there come new ones specific to transfinite computations? More on this in the next parts.

- Dmitry Vostokov @ DumpAnalysis.org -

The first issue of Debugged! MZ/PE magazine is going to be printed on debugged paper (not to confused with common bugs-free paper used in publishing houses and printing factories). Once you open the first issue you would instantly recognize that!

- Dmitry Vostokov @ DumpAnalysis.org -

Looked this evening at Amazon and found that the book achieved #1 status (although it might not be the case at the time when you are reading this post):

- Dmitry Vostokov @ DumpAnalysis.org -

Named after Bernhard Riemann, this programming language gives software defects first-class status as alternative branches of computation, comparable with multivalued functions and Riemann surfaces. Bugs become first-class constructs. It is reflected in the language syntax, semantics and pragmatics. More on this later.

- Dmitry Vostokov @ DumpAnalysis.org -