Software Diagnostics Library

Some fellow debuggers ask me what brand of headphones I use during debugging. It depends on the working environment. In the office I use STAX electrostatic headphones, one of the previous versions of their Basic System that I bought 4 years ago, similar to this one:

http://www.stax.co.jp/Export/SRS2050II.html

I learnt about STAX in 2000, Moscow, when I was obsessed with pure sound and rushed to the nearest dealer to buy the old version of SR-001:

http://www.stax.co.jp/Export/SR001mk2e.html

- Dmitry Vostokov @ DumpAnalysis.org -

If you like me totally immerse in complete memories of computation you probably would like a complete Beethoven collection that I bought last year and now listen to the second time:

Beethoven Edition: Complete Works

- Dmitry Vostokov @ DumpAnalysis.org -

On 15th of April last year I published

The First Windows Memory Dump Analysis Book!

To celebrate its anniversary OpenTask iterative and incremental publisher will publish Volume 3 this year simultaneously with another book featuring selected articles from Volumes 1-3 printed in full color:

Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3 (ISBN: 978-1906717698)

This title will compensate for the lack of color inserts in MDAA volumes.

- Dmitry Vostokov @ DumpAnalysis.org -

I aim to make Debugged! MZ/PE a quarterly publication and the next issue is to be by the end of June. Its main topic is  modeling software bugs:

Debugged! MZ/PE: Modeling Software Defects, June, 2009 (ISBN: 978-1906717681)

If you have an article idea or if you’d like to write an article for this issue please use the following contact form:

http://www.dumpanalysis.org/contact

The deadline is June 15. 

Here is the draft front cover:

The back cover remains secret until release and will feature tips on debugging.

- Dmitry Vostokov @ DumpAnalysis.org - 

NULL Data Pointer is a special version of the more general Invalid Pointer pattern like NULL Code Pointer. The effective address is below 0xFFFF and it is usually a register with 0 value and the small offset, for example:

0: kd> r
Last set context:
eax=8923b008 ebx=00000000 ecx=00000000 edx=8923b008 esi=891312d0 edi=89f0b300
eip=8081c7c4 esp=f1b5d7a4 ebp=f1b5d7a4 iopl=0 nv up ei ng nz ac pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010296
nt!IoIsOperationSynchronous+0xe:
8081c7c4 f6412c02  test    byte ptr [ecx+2Ch],2   ds:0023:0000002c=??

Here, after disassembling the function backwards, we see the succession of dereferences starting from [EBP+8] and this means that a pointer to a structure (an IRP here) was passed to the function and it had a data pointer in it, pointing to another structure and the latter contained an NULL pointer:  

0: kd> ub 8081c7c4
nt!IoIsOperationSynchronous:
8081c7b6 8bff            mov     edi,edi
8081c7b8 55              push    ebp
8081c7b9 8bec            mov     ebp,esp
8081c7bb 8b4508          mov     eax,dword ptr [ebp+8]
8081c7be 8b4860          mov     ecx,dword ptr [eax+60h]
8081c7c1 8b4918          mov     ecx,dword ptr [ecx+18h]

Next post will feature a full case study.

Note: pointers are discussed in great detail in my latest book: Windows Debugging: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org -

When constantly looking at Stack Trace Collections from complete or kernel memory dumps we notice that certain processes are always present and remember them. They are no longer suspicious. The same about thread stacks. Some are always present and some are not suspicious because of their function or status, like Passive Threads or Passive System Threads. Going more fine-grained we can talk about components and their specific functions. For example, certain kernel space components have special filter functions, they get an IRP and pass it down the device stack. It doesn’t take much code to check an IRP and forward it. This is usually reflected in small function offsets, for example:

ChildEBP RetAddr
aeced780 80833ec5 nt!KiSwapContext+0x26
aeced7ac 80829bc0 nt!KiSwapThread+0x2e5
aeced7f4 badffece nt!KeWaitForSingleObject+0x346
WARNING: Stack unwind information not available. Following frames may be wrong.
aeced824 bae00208 AVFilterB+0×1ece
aeced868 bae0e45a AVFilterB+0×2208
aeced8a0 8081e095 AVFilterB+0×1045a
aeced8b4 b946673b nt!IofCallDriver+0×45
aeced8c4 b94626ee driverB!FS_Dispatch+0xfb
aeced8d4 8081e095 driverB!dispatch+0×6e
aeced8e8 b96e04e1 nt!IofCallDriver+0×45
aeced90c b96e0755 driverA!PassThrough+0xd1
aeced92c 8081e095 driverA!Create+0×155
aeced940 b882df08 nt!IofCallDriver+0×45
aeceda5c 8081e095 AVFilterA!DispatchPassThrough+0×48
aeceda70 808fb13b nt!IofCallDriver+0×45
aecedb58 80939c6a nt!IopParseDevice+0xa35
aecedbd8 80935d9e nt!ObpLookupObjectName+0×5b0
aecedc2c 808ece57 nt!ObOpenObjectByName+0xea
aecedca8 808ee0f1 nt!IopCreateFile+0×447
aecedd04 808f1e31 nt!IoCreateFile+0xa3
aecedd44 8088ad3c nt!NtOpenFile+0×27
[…]

Here, if the thread is blocked, AVFilterB is more suspicious than AVFilterA because it is on top of the stack, waiting and AVFilterA just passed an IRP to driverA. DriverA seems also relayed the IRP to driverB and the latter relayed it to AVFilterB.

Another x64 example shows how these filter functions can be identified. They have “Dispatch” or “PassThrough” in their function names:

Child-SP          RetAddr           Call Site
fffffa60`12610880 fffff800`01875f8a nt!KiSwapContext+0x7f
fffffa60`126109c0 fffff800`0187776a nt!KiSwapThread+0x2fa
fffffa60`12610a30 fffff800`01ab16d6 nt!KeWaitForSingleObject+0x2da
[...]
fffffa60`12610fe0 fffffa60`06c5191a rdbss!RxFsdCommonDispatch+0×786
fffffa60`126110d0 fffffa60`07e4f21f rdbss!RxFsdDispatch+0×21a
fffffa60`12611140 fffffa60`011e05f5 mrxsmb!MRxSmbFsdDispatch+0xbf
fffffa60`12611180 fffffa60`011e0130 mup!MupiCallUncProvider+0×159
fffffa60`126111f0 fffffa60`011e17af mup!MupStateMachine+0×120
fffffa60`12611240 fffffa60`00d200b4 mup!MupCreate+0×2c3
fffffa60`126112c0 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
fffffa60`12611370 fffffa60`06d786c7 driverB!FS_Dispatch+0×156
fffffa60`126113a0 fffffa60`06d7894d driverA!PassThrough+0×177
fffffa60`12611400 fffffa60`090b3f30 driverA!Create+0×14d
fffffa60`12611430 fffff800`01aef360 AVDriverA!LowerDevicePassThrough+0×5c
fffffa60`12611700 fffff800`01aefa59 nt!IopParseDevice+0×5e3
fffffa60`126118a0 fffff800`01af3944 nt!ObpLookupObjectName+0×5eb
fffffa60`126119b0 fffff800`01affee0 nt!ObOpenObjectByName+0×2f4
fffffa60`12611a80 fffff800`01b00a0c nt!IopCreateFile+0×290
fffffa60`12611b20 fffff800`0186fdf3 nt!NtCreateFile+0×78
[…]

- Dmitry Vostokov @ DumpAnalysis.org -

Andreas Zeller to publish the second edition of his book this summer:

Why Programs Fail, Second Edition: A Guide to Systematic Debugging

I really enjoyed reading the first edition a few years ago, liked the fact that he uses mathematics to prove propositions of delta debugging, illustrate memory graphs and cause-effect chains. I’m looking forward to reading the second edition.

- Dmitry Vostokov @ DumpAnalysis.org -

Microsoft to add 5th memory dump type to the final version of Windows 7. In addition to kernel, complete, mini and user dump file types new memory dumps will include all open files to allow full data recovery and postmortem process resurrection on another computer. The new coming soon version of WinDbg includes specialized extensions for process instantiation and recursive data recovery near the point of failure:

blogs.technet.com/5thcolumn

- Dmitry Vostokov @ DumpAnalysis.org -

The latest WinDbg 6.11.0001.404 has broken !ntsdexts.locks command (simply !locks in user dumps). I noticed that some time ago and switched to !cs command which works in user and complete memory dumps. I now like it more than !locks because the following command parameters give stack traces for critical section owners:

!cs -l -o -s

The following recent cases studies use !cs command:

Blocked threads, message box and self-diagnosis

Manual dump, wait chain, blocked thread, dynamic memory corruption and historical information: pattern cooperation

Coupled processes, wait chains, message box, waiting thread time, paged out data, incorrect stack trace, hidden exception, unknown component and execution residue: pattern cooperation

Easy to remember mnemonic is cslos: critical sections locked with stacks

This command is also featured on the back cover of March Debugged! magazine issue.

- Dmitry Vostokov @ DumpAnalysis.org -

The paperback version of the book Windows Debugging: Practical Foundations has been temporarily out of print since the last week but this should be resolved very soon in a few days. OpenTask also submitted to print a hardcover version (ISBN: 978-1906717674) and it should be available for sale by the end of this week.

- Dmitry Vostokov @ DumpAnalysis.org - 

Finally it has been published and available for orders from Amazon and other bookstores:

http://www.dumpanalysis.org/Debugged+Magazine

I had to increase the number of pages for the first issue from 16, planned originally, to 28 and this is reflected in the retail price of $10 (originally planned $8) but bookstores should sell it with a discount between 0% and 55%.

More information about the next issue should be ready by the end of the next week.

- Dmitry Vostokov @ DumpAnalysis.org - 

Previously announced Memory Analysis and Debugging Institute was registered in the Republic of Ireland (No. 382026) last week.

- Dmitry Vostokov @ DumpAnalysis.org - 

I’m very proud to announce that my first 2 volumes of Memory Dump Analysis Anthology are to be translated and published in the Republic of Korea this year:

http://www.opentask.com/korean-rights-mdaa-v1-v2

- Dmitry Vostokov @ DumpAnalysis.org -

I was a big fan of Kitaro music for many years during 90s. Today I started re-listening to some of his albums and recommend

Heaven & Earth: Music From The Motion Picture Soundtrack

to listen to when analyzing complete memory dumps from complex issues, looking at user (Earth) and kernel (Heaven) spaces and trying to figure out which module started the conflict.

- Dmitry Vostokov @ DumpAnalysis.org -

Just recently got news about a Microsoft security WinDbg extension released as open source:

http://www.microsoft.com/security/msec/default.mspx

http://www.codeplex.com/msecdbg 

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes a problem like a crash or a hang never happens again, the so called a unique computational event, like the extinction of dinosaurs if we apply biological metaphors. Biology science copes with such events via constructing historical narratives and multiple probabilistic explanations with cross data examinations. The same is true for memory dump analysis where we construct possible explanations based on evidence and collected supporting data. Like Ernst Mayr pointed, we try to answer both questions: “How?” and “Why?”. Usually the answer to the first question is very simple and straightforward, like NULL pointer access (proximate, functional causation) and the answer to the second question is provided by testing various possible historical narratives (ultimate or evolutionary causation) possibly involving an animate agent (a human user of a system).

- Dmitry Vostokov @ DumpAnalysis.org -   

In Windows Vista and Server 2008 LPC is implemented using undocumented Advanced Local Procedure Calls (ALPC) and the new formatting of the !process 0 ff output shows the owner of the port: 

THREAD fffffa801afae6f0  Cid 033c.3c40  Teb: 000007fffffa0000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa801afaea80  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8800f4cc950 : queued at port fffffa8017dbb060 : owned by process fffffa801807b8f0

Instead of !lpc we should use !alpc extension command: 

0: kd> !lpc message fffff8800f4cc950
LPC is now emulated over ALPC. Use !alpc
Arguments:

   /?                   - Print help
   /P [/V] PortAddress  - Dump the ALPC port information
   /M [/V] MsgAddress   - Searches the message and dumps the message information
   /R [/V] ResAddress   - Dumps the information regarding resources
   /G [/V]              - Dumps the global information

0: kd> !alpc /M fffff8800f4cc950

Message @ fffff8800f4cc950
  MessageID             : 0x0BFC (3068)
  CallbackID            : 0x1FEF27 (2092839)
  SequenceNumber        : 0x0004D4D4 (316628)
  Type                  : LPC_REQUEST
  DataLength            : 0x0068 (104)
  TotalLength           : 0x0090 (144)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8018a95e60 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa801afae6f0
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa8017dbb060 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa801807b8f0 (svchost.exe)
  ServerThread          : fffffa801aeec060
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 00000000025599e0
  ServerContext         : 0000000000000000
  PortContext           : 000000000030d640
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

- Dmitry Vostokov @ DumpAnalysis.org -

Looking at one kernel memory dump from x64 Windows Server 2008 I noticed this API call (shown in blue):

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffffa60`138f4720 fffff800`01875f8a nt!KiSwapContext+0x7f
fffffa60`138f4860 fffff800`0187776a nt!KiSwapThread+0x2fa
fffffa60`138f48d0 fffff800`01ab16d6 nt!KeWaitForSingleObject+0x2da
fffffa60`138f4960 fffff800`01ab1667 nt!FsRtlCancellableWaitForMultipleObjects+0x62
fffffa60`138f49c0 fffffa60`06c515e0 nt!FsRtlCancellableWaitForSingleObject+0x27
fffffa60`138f4a00 fffffa60`06c611dc rdbss!RxWaitForStableCondition+0x11c
fffffa60`138f4a40 fffffa60`06c61c07 rdbss!RxFindOrCreateConnections+0x44c
fffffa60`138f4b20 fffffa60`06c56840 rdbss!RxConstructVirtualNetRoot+0xb7
fffffa60`138f4bc0 fffffa60`06c6381a rdbss!RxFindOrConstructVirtualNetRoot+0x594
fffffa60`138f4d30 fffffa60`06c54c42 rdbss!RxCreateTreeConnect+0x13e
fffffa60`138f4dc0 fffffa60`06c2fbf6 rdbss!RxCommonCreate+0x20a
fffffa60`138f4e80 fffffa60`06c5191a rdbss!RxFsdCommonDispatch+0x786
fffffa60`138f4f70 fffffa60`07e4f21f rdbss!RxFsdDispatch+0x21a
fffffa60`138f4fe0 fffffa60`011e05f5 mrxsmb!MRxSmbFsdDispatch+0xbf
fffffa60`138f5020 fffffa60`011e0130 mup!MupiCallUncProvider+0x159
fffffa60`138f5090 fffffa60`011e17af mup!MupStateMachine+0x120
fffffa60`138f50e0 fffffa60`00d200b4 mup!MupCreate+0x2c3
fffffa60`138f5160 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[...]
3rd party filter drivers
[...]
fffffa60`138f55a0 fffff800`01aefa59 nt!IopParseDevice+0x5e3
fffffa60`138f5740 fffff800`01af3944 nt!ObpLookupObjectName+0x5eb
fffffa60`138f5850 fffff800`01affee0 nt!ObOpenObjectByName+0x2f4
fffffa60`138f5920 fffff800`01b00a0c nt!IopCreateFile+0x290
fffffa60`138f59c0 fffff800`0186fdf3 nt!NtCreateFile+0x78
fffffa60`138f5a50 fffff800`01870300 nt!KiSystemServiceCopyEnd+0x13
fffffa60`138f5c58 fffffa60`06c91a5e nt!KiServiceLinkage
fffffa60`138f5c60 fffff800`018913d1 dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc dfsc!DfscGetIpcConnection+0×1f0
fffffa60`138f5e30 fffffa60`06c9bb21 dfsc!DfscRmGetReferral+0×78
fffffa60`138f5ea0 fffffa60`06c91470 dfsc!DfscGetDomainDCReferral+0×31
fffffa60`138f5ef0 fffffa60`06c917ec dfsc!DfscRmValidateDomainIterate+0×5c
fffffa60`138f5f40 fffffa60`06c915f5 dfsc!DfscValidateReferral+0xa0
fffffa60`138f5fb0 fffffa60`06c917ec dfsc!DfscRmValidateRootGetParent+0×75
fffffa60`138f5fe0 fffffa60`06c90825 dfsc!DfscValidateReferral+0xa0
fffffa60`138f6050 fffffa60`06c93905 dfsc!DfscCmValidateState+0×79
fffffa60`138f6090 fffffa60`06c9e759 dfsc!DfscSurrogateCreate+0×7d
fffffa60`138f6100 fffffa60`011e03ab dfsc!DfscSurrogatePreProcess+0xb9
fffffa60`138f6130 fffffa60`011e014f mup!MupCallSurrogatePrePost+0×10b
fffffa60`138f6190 fffffa60`011e17af mup!MupStateMachine+0×13f
fffffa60`138f61e0 fffffa60`00d200b4 mup!MupCreate+0×2c3
fffffa60`138f6260 fffffa60`06d332d6 fltmgr!FltpCreate+0xa4
[…]
3rd party filter drivers
[…]
fffffa60`138f6610 fffff800`01aefa59 nt!IopParseDevice+0×5e3
fffffa60`138f67b0 fffff800`01af3944 nt!ObpLookupObjectName+0×5eb
fffffa60`138f68c0 fffff800`01ac22f1 nt!ObOpenObjectByName+0×2f4
fffffa60`138f6990 fffff800`0186fdf3 nt!NtQueryAttributesFile+0×134
fffffa60`138f6c20 00000000`77285e4a nt!KiSystemServiceCopyEnd+0×13

This API is mentioned in the following presentation and document and can also be found in WDK:

PPT: Windows Memory Management Advances

DOC: Advances in Memory Management 

KeExpandKernelStackAndCallout

Its 3rd parameter is the stack size and we can see it used in disassembly where r8d register is used for 3rd parameter according to x64 calling convention and rcx is used for the first parameter, a function procedure to be executed with a guaranteed kernel stack size:

0: kd> kv 100
Child-SP          RetAddr           : Args to Child                                                           : Call Site
[...]
fffffa60`138f5c60 fffff800`018913d1 : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 00000000`00000000 : dfsc!DfscConnOpenIpcConnectionCallout+0xbe
fffffa60`138f5d20 fffffa60`06c91d08 : fffffa60`06c919a0 fffffa60`138f5df0 fffff880`102128d0 fffffa60`138f5f10 : nt!KeExpandKernelStackAndCalloutEx+0×2e1
fffffa60`138f5db0 fffffa60`06c9bbcc : 00000000`00000000 fffff880`10d6d3f8 00000000`00000000 fffff880`10d6d460 : dfsc!DfscGetIpcConnection+0×1f0
[…]

0: kd> ub fffffa60`06c91d08
dfsc!DfscGetIpcConnection+0×1c6:
fffffa60`06c91cde xor     r9d,r9d
fffffa60`06c91ce1 mov     qword ptr [rsp+50h],rax
fffffa60`06c91ce6 mov     rax,qword ptr [dfsc!DfscGlobalData+0×138 (fffffa60`06c8d758)]
fffffa60`06c91ced mov     r8d,6000h
fffffa60`06c91cf3 mov     qword ptr [rsp+40h],rdi
fffffa60`06c91cf8 mov     byte ptr [rsp+58h],r11b
fffffa60`06c91cfd mov     qword ptr [rsp+20h],rax
fffffa60`06c91d02 call    qword ptr [dfsc!_imp_KeExpandKernelStackAndCalloutEx (fffffa60`06c8b0d0)]

0: kd> ub fffffa60`06c91cde
dfsc!DfscGetIpcConnection+0x199:
fffffa60`06c91cb1 488b88b8000000  mov     rcx,qword ptr [rax+0B8h]
fffffa60`06c91cb8 0fba61100a      bt      dword ptr [rcx+10h],0Ah
fffffa60`06c91cbd 450f42df        cmovb   r11d,r15d
fffffa60`06c91cc1 488b4338        mov     rax,qword ptr [rbx+38h]
fffffa60`06c91cc5 488d542440      lea     rdx,[rsp+40h]
fffffa60`06c91cca 488d0dcffcffff  lea     rcx,[dfsc!DfscConnOpenIpcConnectionCallout (fffffa60`06c919a0)]
fffffa60`06c91cd1 4889442448      mov     qword ptr [rsp+48h],rax
fffffa60`06c91cd6 488d842490000000 lea     rax,[rsp+90h]

It is good sign to see it used in file system stacks because in the past the fixed kernel stacks resulted in stack overflows and double faults:

Stack Overflow Pattern (kernel mode)

- Dmitry Vostokov @ DumpAnalysis.org -

Yesterday I submitted the magazine to print and distribution world-wide. If everything is right it should be available by the end of this month. This first issue features 12 page WinDbg command supplement to pattern-driven memory dump analysis methodology, an overview of Win32dd complete memory dumper and PowerDbg enhancements to debug ASP.NET code. The magazine will only be available in print.

- Dmitry Vostokov @ DumpAnalysis.org -

For one manual user dump of a hanging service neither !analyze -v nor !analyze -v -hang command found the problem:

Loading Dump File [ServiceA.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER'

0:000> !analyze -v -hang

DERIVED_WAIT_CHAIN: 

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   0   678.67c File IO               

WAIT_CHAIN_COMMAND:  ~0s;k;;

BLOCKING_THREAD:  0000067c

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_FileIO

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BlockedOn_FileIO

LAST_CONTROL_TRANSFER:  from 7c82776b to 7c8285ec

STACK_TEXT: 
0012fbc8 7c82776b ntdll!KiFastSystemCallRet
0012fbcc 77e218b2 ntdll!NtReadFile+0xc
0012fc34 77d85edb kernel32!ReadFile+0x16c
0012fc60 77d85f82 advapi32!ScGetPipeInput+0x2a
0012fcd4 77d71ed9 advapi32!ScDispatcherLoop+0x51
0012ff38 004025e0 advapi32!StartServiceCtrlDispatcherW+0xe3
0012ff60 0042c8fd ServiceA!main+0x110
0012ffc0 77e4f23b ServiceA!mainCRTStartup+0x143
0012fff0 00000000 kernel32!BaseProcessStart+0x23

As usual the critical section list was checked for any locked sections and to find wait chains:

0:000> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x7c8877c0
Critical section   = 0×7c8877a0 (ntdll!LdrpLoaderLock+0×0)
LOCKED
LockCount          = 0xA2
WaiterWoken        = No
OwningThread       = 0×00011f64
RecursionCount     = 0×1
LockSemaphore      = 0×188
SpinCount          = 0×00000000
OwningThread DbgId = ~20s
OwningThread Stack =
ChildEBP RetAddr  Args to Child             
0327f91c 7c827d0b 7c83d236 00000754 00000000 ntdll!KiFastSystemCallRet
0327f920 7c83d236 00000754 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0327f95c 7c83d281 00000754 00000004 00370000 ntdll!RtlpWaitOnCriticalSection+0×1a3
0327f97c 7c82a264 00370608 00000000 0000008c ntdll!RtlEnterCriticalSection+0xa8
0327fba4 7c3423aa 00370000 00000008 0000008c ntdll!RtlAllocateHeap+0×313
0327fbe4 7c3422cb 00000001 0000008c 00000000 msvcr71!calloc+0xe6
0327fbfc 7c81a352 7c340000 00000002 00000000 msvcr71!_CRTDLL_INIT+0×138
0327fc1c 7c82ed97 7c34229f 7c340000 00000002 ntdll!LdrpCallInitRoutine+0×14
0327fcb8 7c82ec9f 0327fd28 0327fd28 00000000 ntdll!LdrpInitializeThread+0×10d
0327fd14 7c8284c5 0327fd28 7c800000 00000000 ntdll!_LdrpInitialize+0×16f
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0×25
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
[…]
DebugInfo          = 0×7c887e20
Critical section   = 0×00370608 (+0×370608)
LOCKED
LockCount          = 0×7
WaiterWoken        = No
OwningThread       = 0×000027ac
RecursionCount     = 0×1
LockSemaphore      = 0×754
SpinCount          = 0×00000fa0
OwningThread DbgId = ~17s
OwningThread Stack =
ChildEBP RetAddr  Args to Child             
0170ebe0 7c82775b 77e62610 d0000144 00000004 ntdll!KiFastSystemCallRet
0170ebe4 77e62610 d0000144 00000004 00000000 ntdll!NtRaiseHardError+0xc
0170ee50 77e592a3 0170ee78 77e41ac1 0170ee80 kernel32!UnhandledExceptionFilter+0×51a
0170ee58 77e41ac1 0170ee80 00000000 0170ee80 kernel32!BaseThreadStart+0×4a
0170ee80 7c828752 0170ef64 0170ffdc 0170ef80 kernel32!_except_handler3+0×61
0170eea4 7c828723 0170ef64 0170ffdc 0170ef80 ntdll!ExecuteHandler2+0×26
0170ef4c 7c82855e 0170d000 0170ef80 0170ef64 ntdll!ExecuteHandler+0×24
0170ef4c 7c82a754 0170d000 0170ef80 0170ef64 ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 0170ef80)
0170f258 7c82a82b 0185d6a8 00690043 0170f320 ntdll!RtlpCoalesceFreeBlocks+0×36e
0170f340 7c34218a 00370000 00000000 0185bad8 ntdll!RtlFreeHeap+0×38e
0170f388 004288df 0185bad8 05afd9c4 00000000 msvcr71!free+0xc3
0170f3b4 00428a16 05afd9c4 00001bc4 0185bad8 ServiceA!SetData+0×4df
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
[…]

We see that 162 threads (0xA2) are waiting for 0×7c8877a0 locked critical section whose owner thread #20 is waiting for the critical section 00370608 owned by the thread #17. The latter thread is blocked in an unhandled exception filter raising a hard error message box. Applying the new exception context we confirm heap corruption problem:

0:000> .cxr 0170ef80
eax=0185d6a8 ebx=00370000 ecx=00690043 edx=00000001 esi=0185d6a0 edi=01858030
eip=7c82a754 esp=0170f24c ebp=0170f258 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
ntdll!RtlpCoalesceFreeBlocks+0×36e:
7c82a754 8b09            mov     ecx,dword ptr [ecx]  ds:0023:00690043=????????

0:000> kL
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
0170f258 7c82a82b ntdll!RtlpCoalesceFreeBlocks+0×36e
0170f340 7c34218a ntdll!RtlFreeHeap+0×38e
0170f388 004288df msvcr71!free+0xc3
0170f3b4 00428a16 ServiceA!SetData+0×4df
[…]
0170ffec 00000000 kernel32!BaseThreadStart+0×34

Therefore we suggest to enable full page heap either by using gflags or application verifier. Looking further at other locked critical sections omitted above we notice that one is inaccessible and from an unloaded module:

Cannot read structure field value at 0x5531953c, error 0
Cannot determine if the critical section is locked or not.
-----------------------------------------
Critical section   = 0x55319538 (<Unloaded_DetouredA.DLL>+0×29538)
DebugInfo          = 0×00147108
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

We see this execution residue in the list of unloaded modules too:

0:000> lmt
[...]
Unloaded modules:
[...]
697b0000 697c7000   faultrep.dll
    Timestamp: Sat Feb 17 15:12:57 2007 (45D71B79)
    Checksum:  0001D340
[...]
552f0000 55320000   DetouredA.DLL
    Timestamp: Wed Dec 19 09:38:46 2007 (4768E6A6)
    Checksum:  00030564

Recalling that this hooksware had problems before we also suggest to remove software package that injects DetouredA module to test the stability of the system.

- Dmitry Vostokov @ DumpAnalysis.org -