Software Diagnostics Library

Today found a CD in my old case (the link I give is for MP3 though):

Ancient Journeys: A Vision Of The New World

Here’s my version of track titles:

1. Discovering the Interface
2. Conquering the New Space
3. Frozen System
4. MS DOS  
5. Russian OS
6. The Journeys of RIP
7. The Spiking Threads
8. Not Enough Space
9. Against Bugs

- Dmitry Vostokov @ DumpAnalysis.org -

Some practical engineers asked me how do Debugged! MZ/PE magazine back covers look like from a birds eye view:

One engineer even commented that they look better and better (counterclockwise) :-) 

- Dmitry Vostokov @ DumpAnalysis.org -

If you know any other please let us know. For example, MSDN is great too, there was also a Visual C++-oriented magazine I used to read in the past. May be it is still in print. I plan to make a wider perspective picture later on. Both pictured magazines (Debugged! MZ/PE and NT Insider) are mainly oriented towards unmanaged and native code respectively (the former with a timeless computer science slant) but Debugged! soon to cover native, virtualized, interpreted and managed code environments too and aims to become a “Complete Debugging Magazine” similar to kernel vs. user process vs. complete memory dump file differences.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes a single memory dump analysis session requires an analysis of several memory dump files, for example, comparative analysis of a memory leak issue or several dumps saved to check the consistency of a stack trace and exception address. In the past I used to open every individual memory dump file separately: multiple dumps - multiple WinDbg process instances. Recently I discovered a WinDbg command that allows me to keep several memory dumps open in the same WinDbg instance and share the single command window: .opendump. After opening a memory dump we need to type g to finish its load. Then we can use || selector to switch between memory dumps similar to ~ command we use to switch between threads in user process memory dumps or processors in kernel and complete memory dumps. Here is an example: 

Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\MemoryDumps\1MbNop.exe.2972.dmp]
User Mini Dump File with Full Memory: Only application data is available

Windows Server 2008/Windows Vista Version 6002 (Service Pack 2) MP (2 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Debug session time: Mon Sep 28 21:49:16.000 2009 (GMT+0)
System Uptime: 0 days 5:34:37.445
Process Uptime: 0 days 0:00:03.000

This dump file has a breakpoint exception stored in it.
The stored exception information can be accessed via .ecxr.
1MbNop+0x101011:
00000001`40101011 cc              int     3

0:000> .opendump c:\MemoryDumps\1MbPause.exe.3488.dmp

Loading Dump File [c:\MemoryDumps\1MbPause.exe.3488.dmp]
User Mini Dump File with Full Memory: Only application data is available

Opened 'c:\MemoryDumps\1MbPause.exe.3488.dmp'

||0:0:000> g
Windows Server 2008/Windows Vista Version 6002 (Service Pack 2) MP (2 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Debug session time: Mon Sep 28 21:49:23.000 2009 (GMT+0)
System Uptime: 0 days 5:34:44.410
Process Uptime: 0 days 0:00:01.000

This dump file has a breakpoint exception stored in it.
The stored exception information can be accessed via .ecxr.
1MbPause+0x201011:
00000001`40201011 cc              int     3

||1:1:001> kL
Child-SP          RetAddr           Call Site
00000000`0012ff58 00000000`7704be3d 1MbPause+0x201011
00000000`0012ff60 00000000`77256a51 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

||1:1:001> ||0s
1MbNop+0x101011:
00000001`40101011 cc              int     3

||0:0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012ff58 00000000`7704be3d 1MbNop+0x101011
00000000`0012ff60 00000000`77256a51 kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

- Dmitry Vostokov @ DumpAnalysis.org -

When looking at complete memory dumps and switching between process contexts we are usually interested in loaded user space modules, their timestamps, version information, vendor and description. lmt and lmv WinDbg commands show both user space and kernel space modules and it becomes annoying to see the same kernel modules over and over again in the output. The following commands list only user and kernel space modules separately: lmu and lmk. Here is an example:

lkd> !process
PROCESS fffffa80056b0c10
SessionId: 1  Cid: 0f6c    Peb: 7fffffdf000  ParentCid: 0a10
DirBase: 34b2b000  ObjectTable: fffff88009796470  HandleCount:  80.
Image: windbg.exe
VadRoot fffffa80056000a0 Vads 82 Clone 0 Private 5884. Modified 2109. Locked 1.
DeviceMap fffff88008c7e9c0
Token                             fffff8800bb3fa70
ElapsedTime                       00:01:27.560
UserTime                          00:00:00.000
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         176696
QuotaPoolUsage[NonPagedPool]      7984
Working Set Sizes (now,min,max)  (7860, 50, 345) (31440KB, 200KB, 1380KB)
PeakWorkingSetSize                7860
VirtualSize                       110 Mb
PeakVirtualSize                   110 Mb
PageFaultCount                    10117
MemoryPriority                    BACKGROUND
BasePriority                      8
CommitCharge                      6293

THREAD fffffa8005648790  Cid 0f6c.096c  Teb: 000007fffffdd000 Win32Thread: fffff900c21ef450 WAIT: (WrUserRequest) UserMode Non-Alertable
   fffffa80056692a0  SynchronizationEvent

THREAD fffffa800557c300  Cid 0f6c.0eb8  Teb: 000007fffffdb000 Win32Thread: fffff900c06402a0 RUNNING on processor 1

lkd> lmu
start             end                 module name
00000000`6d660000 00000000`6d943000   ext       
00000000`6d950000 00000000`6ddac000   dbgeng    
00000000`6e120000 00000000`6e191000   exts      
00000000`6e1a0000 00000000`6e309000   dbghelp   
00000000`6ed20000 00000000`6ed6c000   symsrv    
00000000`6ed70000 00000000`6edb4000   kext      
00000000`76da0000 00000000`76ecd000   kernel32  
00000000`76ed0000 00000000`76f9d000   USER32    
00000000`76fa0000 00000000`77126000   ntdll     
00000001`3f530000 00000001`3f5de000   windbg    
000007fe`f3f00000 000007fe`f40c3000   kdexts    
000007fe`f4600000 000007fe`f46b8000   MSFTEDIT  
000007fe`fb760000 000007fe`fb7b1000   UxTheme   
000007fe`fc200000 000007fe`fc24f000   OLEACC    
000007fe`fc250000 000007fe`fc289000   WINMM     
000007fe`fc4e0000 000007fe`fc6d9000   COMCTL32  
000007fe`fc6f0000 000007fe`fc6fb000   VERSION   
000007fe`fd370000 000007fe`fd389000   MPR       
000007fe`fd4f0000 000007fe`fd5c3000   OLEAUT32  
000007fe`fd5d0000 000007fe`fd7a8000   ole32     
000007fe`fda00000 000007fe`fda2d000   IMM32     
000007fe`fda30000 000007fe`fda3d000   LPK       
000007fe`fda40000 000007fe`fdb83000   RPCRT4    
000007fe`fdb90000 000007fe`fdc03000   SHLWAPI   
000007fe`fde10000 000007fe`fde9c000   COMDLG32  
000007fe`fdea0000 000007fe`feaf3000   SHELL32   
000007fe`feb00000 000007fe`feb64000   GDI32     
000007fe`fecf0000 000007fe`fedf8000   ADVAPI32  
000007fe`fee00000 000007fe`fee9c000   msvcrt    
000007fe`fefc0000 000007fe`ff0c2000   MSCTF     
000007fe`ff180000 000007fe`ff21a000   USP10  

lkd> lmk
start             end                 module name
fffff800`0181c000 fffff800`01d34000   nt        
fffff800`01d34000 fffff800`01d7a000   hal       
fffff960`00020000 fffff960`002d4000   win32k    
fffff960`00420000 fffff960`0043e000   dxg       
fffff960`006e0000 fffff960`006ea000   TSDDD     
fffff960`008e0000 fffff960`008e9000   framebuf  
fffffa60`00602000 fffffa60`0060c000   kdcom     
fffffa60`0060c000 fffffa60`00647000   mcupdate_GenuineIntel
fffffa60`00647000 fffffa60`0065b000   PSHED     
fffffa60`0065b000 fffffa60`006b8000   CLFS      
fffffa60`006b8000 fffffa60`0076a000   CI        
fffffa60`00786000 fffffa60`007d6000   msrpc     
fffffa60`007d6000 fffffa60`007fa000   ataport   
fffffa60`00800000 fffffa60`009c3000   NDIS      
fffffa60`009c3000 fffffa60`009d3000   PCIIDEX   
fffffa60`009d3000 fffffa60`009e6000   mountmgr  
fffffa60`00a0d000 fffffa60`00a66000   NETIO     
fffffa60`00a66000 fffffa60`00b40000   Wdf01000  
fffffa60`00b40000 fffffa60`00b4e000   WDFLDR    
fffffa60`00b4e000 fffffa60`00b7e000   pci       
fffffa60`00b7e000 fffffa60`00b92000   volmgr    
fffffa60`00b92000 fffffa60`00bf8000   volmgrx   
fffffa60`00c00000 fffffa60`00ced000   HDAudBus  
fffffa60`00ced000 fffffa60`00d09000   cdrom     
fffffa60`00d09000 fffffa60`00d42000   msiscsi   
fffffa60`00d43000 fffffa60`00d4c000   WMILIB    
fffffa60`00d4c000 fffffa60`00d7a000   SCSIPORT  
fffffa60`00d7a000 fffffa60`00dd0000   acpi      
fffffa60`00dd0000 fffffa60`00dda000   msisadrv  
fffffa60`00dda000 fffffa60`00def000   partmgr   
fffffa60`00def000 fffffa60`00df7000   intelide  
fffffa60`00df7000 fffffa60`00dff000   atapi     
fffffa60`00e0b000 fffffa60`00e52000   fltmgr    
fffffa60`00e52000 fffffa60`00ed9000   ksecdd    
fffffa60`00ed9000 fffffa60`00ee5000   tunnel    
fffffa60`00ee5000 fffffa60`00ef8000   intelppm  
fffffa60`00ef8000 fffffa60`00f06000   vgapnp    
fffffa60`00f06000 fffffa60`00f2b000   VIDEOPRT  
fffffa60`00f2b000 fffffa60`00f3b000   watchdog  
fffffa60`00f3b000 fffffa60`00f47000   usbuhci   
fffffa60`00f47000 fffffa60`00f8d000   USBPORT   
fffffa60`00f8d000 fffffa60`00f9e000   usbehci   
fffffa60`00f9e000 fffffa60`00fd7000   b57nd60a  
fffffa60`00fd7000 fffffa60`00ff3000   parport   
fffffa60`0100b000 fffffa60`01181000   tcpip     
fffffa60`01181000 fffffa60`011ad000   fwpkclnt  
fffffa60`011ad000 fffffa60`011c1000   disk      
fffffa60`011c1000 fffffa60`011ed000   CLASSPNP  
fffffa60`0120f000 fffffa60`0138f000   Ntfs      
fffffa60`0138f000 fffffa60`013d3000   volsnap   
fffffa60`013d3000 fffffa60`013db000   spldr     
fffffa60`013db000 fffffa60`013ed000   mup       
fffffa60`013ed000 fffffa60`013f7000   crcdisk   
fffffa60`02209000 fffffa60`02266000   storport  
fffffa60`02266000 fffffa60`02273000   TDI       
fffffa60`02273000 fffffa60`02296000   rasl2tp   
fffffa60`02296000 fffffa60`022a2000   ndistapi  
fffffa60`022a2000 fffffa60`022d3000   ndiswan   
fffffa60`022d3000 fffffa60`022e3000   raspppoe  
fffffa60`022e3000 fffffa60`02301000   raspptp   
fffffa60`02301000 fffffa60`02319000   rassstp   
fffffa60`02319000 fffffa60`023b3000   rdpdr     
fffffa60`023b3000 fffffa60`023c6000   termdd    
fffffa60`023c6000 fffffa60`023d4000   kbdclass  
fffffa60`023d4000 fffffa60`023e0000   mouclass  
fffffa60`023e0000 fffffa60`023e1480   swenum    
fffffa60`023e2000 fffffa60`023fd000   smb       
fffffa60`0240b000 fffffa60`0243f000   ks        
fffffa60`0243f000 fffffa60`0244a000   mssmbios  
fffffa60`0244a000 fffffa60`0245a000   umbus     
fffffa60`0245a000 fffffa60`024a2000   usbhub    
fffffa60`024a2000 fffffa60`024b6000   NDProxy   
fffffa60`024b6000 fffffa60`024ff000   HdAudio   
fffffa60`024ff000 fffffa60`0253a000   portcls   
fffffa60`0253a000 fffffa60`0255d000   drmk      
fffffa60`0255d000 fffffa60`02562180   ksthunk   
fffffa60`02563000 fffffa60`0256d000   Fs_Rec    
fffffa60`0256d000 fffffa60`02576000   Null      
fffffa60`02581000 fffffa60`02588b80   HIDPARSE  
fffffa60`02589000 fffffa60`02597000   vga       
fffffa60`02597000 fffffa60`025a0000   RDPCDD    
fffffa60`025a0000 fffffa60`025a9000   rdpencdd  
fffffa60`025a9000 fffffa60`025b4000   Msfs      
fffffa60`025b4000 fffffa60`025c5000   Npfs      
fffffa60`025c5000 fffffa60`025ce000   rasacd    
fffffa60`025ce000 fffffa60`025eb000   tdx       
fffffa60`02e0d000 fffffa60`02e78000   afd       
fffffa60`02e78000 fffffa60`02ebc000   netbt     
fffffa60`02ebc000 fffffa60`02eda000   pacer     
fffffa60`02eda000 fffffa60`02ee9000   netbios   
fffffa60`02f06000 fffffa60`02f21000   wanarp    
fffffa60`02f21000 fffffa60`02f6e000   rdbss     
fffffa60`02f6e000 fffffa60`02f7a000   nsiproxy  
fffffa60`02f7a000 fffffa60`02f97000   dfsc      
fffffa60`02f97000 fffffa60`02fa0000   hidusb    
fffffa60`02fa0000 fffffa60`02fb2000   HIDCLASS  
fffffa60`02fb2000 fffffa60`02fb3e00   USBD      
fffffa60`02fb4000 fffffa60`02fbf000   kbdhid    
fffffa60`02fbf000 fffffa60`02fca000   mouhid    
fffffa60`02fca000 fffffa60`02fd8000   crashdmp  
fffffa60`02fd8000 fffffa60`02fe4000   dump_dumpata
fffffa60`02fe4000 fffffa60`02fec000   dump_atapi
fffffa60`02fec000 fffffa60`02ff8000   Dxapi     
fffffa60`03a00000 fffffa60`03a22000   luafv     
fffffa60`03a22000 fffffa60`03a2a000   psxdrv    
fffffa60`03a2a000 fffffa60`03a3e000   lltdio    
fffffa60`03a3e000 fffffa60`03a56000   rspndr    
fffffa60`03a56000 fffffa60`03a74000   bowser    
fffffa60`03a74000 fffffa60`03a8e000   mpsdrv    
fffffa60`03a8e000 fffffa60`03ab5000   mrxdav    
fffffa60`03ab5000 fffffa60`03ade000   mrxsmb    
fffffa60`03ade000 fffffa60`03b27000   mrxsmb10  
fffffa60`03b27000 fffffa60`03b46000   mrxsmb20  
fffffa60`03b46000 fffffa60`03be9000   HTTP      
fffffa60`04400000 fffffa60`0449a000   spsys     
fffffa60`0449a000 fffffa60`04550000   peauth    
fffffa60`04550000 fffffa60`0455b000   secdrv    
fffffa60`0455b000 fffffa60`04584000   srvnet    
fffffa60`04584000 fffffa60`04594000   tcpipreg  
fffffa60`04594000 fffffa60`045c6000   srv2      
fffffa60`0480d000 fffffa60`0489e000   srv       
fffffa60`0489e000 fffffa60`048a9000   asyncmac  
fffffa60`048a9000 fffffa60`048c5000   cdfs      
fffffa60`048c5000 fffffa60`048cd000   kldbgdrv

Unloaded modules:
fffffa60`01200000 fffffa60`0120e000   crashdmp.sys
fffffa60`011ed000 fffffa60`011f9000   dump_ataport.sys
fffffa60`013f7000 fffffa60`013ff000   dump_atapi.sys
fffffa60`02ee9000 fffffa60`02f06000   serial.sys
fffffa60`02576000 fffffa60`02581000   kbdhid.sys
fffffa60`00c0f000 fffffa60`00d43000   sptd.sys
fffffa60`0076a000 fffffa60`00786000   sacdrv.sys

Of course, verbose equivalents also work: lmuv and lmkv.

- Dmitry Vostokov @ DumpAnalysis.org -

Google Analytics shows the following crash dump analysis pattern frequencies to be fully analyzed later next week:

- Dmitry Vostokov @ DumpAnalysis.org -

According to Google Analytics the number of visits / year increased by 34% since 2008 with almost 150,000 unique visitors (20% increase) from 183 countries (180 in 2008) and more than 35% of them are coming back (2% increase). Here are the top 100 network locations out of 27,300 (16% increase):

Almost 75,000 Google search keywords (more than 13% increase since 2008) pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to almost 1,100 web sites (16% increase) that mention the portal and this blog with the first top 100:

Top 25 visiting countries:

More than 10,000 portal and blog pages were viewed a total of almost 450,000 times with top 100 content pages:

Thanks to Sonny Mir who pointed to !filecache WinDbg command to diagnose low VACB (Virtual Address Control Block or View Address Control Block) conditions I was able to discern another Insufficient Memory pattern for control blocks in general. Certain system and subsystem architectures and designs may put a hard limit on the amount of data structures created to manage resources. If there is a dependency on such resources from other subsystems there could be starvation and blockage conditions resulting in a sluggish system behaviour, absence of a functional response and even in some cases a perceived system, service or application freeze.

7: kd> !filecache
***** Dump file cache******
  Reading and sorting VACBs ...
  Removed 0 nonactive VACBs, processing 1907 active VACBs …
File Cache Information
  Current size 408276 kb
  Peak size    468992 kb
  1907 Control Areas
[…]

I plan to add more insufficient control block case studies including user space.

- Dmitry Vostokov @ DumpAnalysis.org -

Main topics of Debugged! MZ/PE magazine issues for June and September 2010:

Debugged! MZ/PE: Multiplatform Software Defects, June, 2010 (Paperback, ISBN: 978-1906717902)

Debugged! MZ/PE: Software Defect Visualization and Annotation, September, 2010 (Paperback, ISBN: 978-1906717919)

Here are draft front covers designed today:

- Dmitry Vostokov @ DumpAnalysis.org -

See the greeting card on the portal together with New Year’s Eve code analysis puzzle:

DumpAnalysis.org Wishes Happy New Year 7DA!

- Dmitry Vostokov @ DumpAnalysis.org -

a mad day - a day spent doing memory (dump) analysis and/or debugging

Examples: What a mad day! We had several blokes at a bobo address. Those events were rather sad.

- Dmitry Vostokov @ DumpAnalysis.org -

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

I propose to celebrate it on 08.08 every year starting from The Year of Dump Analysis, 2010, 7DA at 8:00 (I prefer 8pm for MAD Day, moderation is important in debugging too).

What do you think? If you count things from 0 or favour user space there is an alternative date: 07.07

- Dmitry Vostokov @ DumpAnalysis.org -

A kernel dump from a frozen system shows an executive resource wait chain:

0: kd> !locks
[...] 
Resource @ driverA!Resource (0xf58de4e0)    Exclusively owned
    Contention Count = 4411
    NumberOfExclusiveWaiters = 11
     Threads: 86d14ae8-01<*>
     Threads Waiting On Exclusive Access:
              8a788db0       8750e970       86c568a0       897ed428      
              86e34db0       86ca8ac0       86b22020       86fef5d8      
              872abdb0       86d16750       87b55830      
[…]

The blocking thread 86d14ae8 had been blocked waiting for a notification event for more than 2 hours:

0: kd> !thread 86d14ae8 1f
THREAD 86d14ae8  Cid 0004.29c4  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
    b81e7adc  NotificationEvent
Not impersonating
DeviceMap                 e1001830
Owning Process            8a78b020       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      8378144        Ticks: 503606 (0:02:11:08.843)
Context Switch Count      1016            
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Start Address driverA!WorkerThreadDispatcher (0xf596ea0e)
Stack Init b81e8000 Current b81e7a2c Base b81e8000 Limit b81e5000 Call 0
Priority 14 BasePriority 10 PriorityDecrement 4
ChildEBP RetAddr 
b81e7a44 8083d5b1 nt!KiSwapContext+0×26
b81e7a70 8083df9e nt!KiSwapThread+0×2e5
b81e7ab8 f59d374d nt!KeWaitForSingleObject+0×346
[…]
b81e7b48 f59b9289 driverB!TcpDisconnect+0×42
[…]
b81e7c40 f595a8a5 nt!IofCallDriver+0×45
b81e7c48 f595ba1e driverA!SubmitTdiRequestNoWait+0×28
[…]
b81e7dac 80920833 driverA!WorkerThreadDispatcher+0×1a
b81e7ddc 8083fe9f nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

We see that the wait happens after requesting a TCP disconnect so we check the list of IRP to see if there is any distribution anomaly among pending IRP:

0: kd> !irpfind
  Irp    [ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]         MDL Process
[...]
86c68d98 [88d2bdb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6a5c0 [89b118c0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6b008 [87564b40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c6caf0 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c7bb28 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c7bd98 [8753ddb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c80008 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c80590 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c845a8 [89d2b400] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c84b80 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86008 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86688 [86d9a788] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c86d98 [88d2bdb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c87990 [88e1c368] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8b640 [8757c3f0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8f368 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c8f650 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c92590 [87625c30] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c92bc8 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c94008 [8757c3f0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c94318 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9a308 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9e008 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86c9e308 [89d2b400] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0350 [87638020] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0870 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0b28 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca0d98 [86db0db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca4918 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86ca6878 [87564b40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86caa458 [88d7b378] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cacc20 [86d4fb40] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb0818 [89c75bb0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb3658 [87638020] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
86cb9d98 [88d66db0] irpStack: ( f, 6)  89cb5ea8 [ \Driver\Tcpip]
[…]

Indeed, we see a high disproportion of TCP I/O requests (many hundreds) after exporting command output to Excel:

We check all stack traces and see one system thread trying to clean TCP connection blocked for almost the same time (more than 2 hours):

0: kd> !stacks
Proc.Thread  .Thread  Ticks   ThreadState Blocker
                            [8a78b020 System]
[...]
   4.00268c  870cf768 00765bd Blocked    tcpip!TCPCleanup+0xcf
[…]

0: kd> !whattime  00765bd
484797 Ticks in Standard Time:   2:06:14.953s

0: kd> !thread 870cf768 1f
THREAD 870cf768  Cid 0004.268c  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
    870a01f4  SynchronizationEvent
IRP List:
    8726fb00: (0006,0268) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 e1001830
Owning Process            8a78b020       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      8396953        Ticks: 484797 (0:02:06:14.953)
Context Switch Count      537            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address nt!ExpWorkerThread (0×8082da4b)
Stack Init b87b0000 Current b87afa18 Base b87b0000 Limit b87ad000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
ChildEBP RetAddr 
b87afa30 8083d5b1 nt!KiSwapContext+0×26
b87afa5c 8083df9e nt!KiSwapThread+0×2e5
b87afaa4 f5a9f9a6 nt!KeWaitForSingleObject+0×346
b87afaf0 f5a96a9d tcpip!TCPCleanup+0xcf
b87afb2c 80840153 tcpip!TCPDispatch+0×10c
b87afb40 f75eb817 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
b87afb64 f75e8698 driverC!DispatchPassThrough+0×4c
[…]
b87afbcc 8092ec0a nt!IofCallDriver+0×45
b87afbfc 8092b6af nt!IopCloseFile+0×2ae
b87afc2c 8092b852 nt!ObpDecrementHandleCount+0xcc
b87afc54 8092b776 nt!ObpCloseHandleTableEntry+0×131
b87afc98 8092b7c1 nt!ObpCloseHandle+0×82
b87afca8 80833bdf nt!NtClose+0×1b
b87afca8 8083b00c nt!KiFastCallEntry+0xfc (TrapFrame @ b87afcb4)
b87afd24 f59d3a3a nt!ZwClose+0×11
b87afd3c f59b78a1 driverB!TdiCloseConnection+0×38
[…]
b87afdac 80920833 nt!ExpWorkerThread+0xeb
b87afddc 8083fe9f nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

- Dmitry Vostokov @ DumpAnalysis.org -

Previously announced Software Maintenance Institute was finally registered in Ireland (Reg. No. 400906) and its certificate was received yesterday.

Here is the current component structure of various institutions (depicted in UML):

Interface Tags:

IIP Interface of Iterative Publishing
IRD Interface of Research and Development
IDR Interface of Defect Research
IIR Interface of Information Repository
IME Interface of Memetic Engineering

- Dmitry Vostokov @ DumpAnalysis.org -

Finally, after the long delay, the issue is available in print on Amazon and through other sellers:

Debugged! MZ/PE: Software Tracing

- Dmitry Vostokov @ DumpAnalysis.org -

Advanced .NET Debugging (Addison-Wesley Microsoft Technology Series)

I’ve just started reading this book (see my notes on Software Generalist blog) and this review is written from the perspective of an unmanaged and native software engineer (the last phrase sounds funny). Being a member of a software support of a large software company I analyze crash dumps that have mscorwks.dll on their stack traces. So if you see them too this book helps you to understand what this DLL is all about and how to dig inside the hidden world of .NET it manages. I’m on page 26 and will update this review as soon as I finish the book in a few months. Please also see my review of the previous Mario’s (co-authored with Daniel Pravat) book: Advanced Windows Debugging. It is of great importance to know .NET world for Windows maintenance engineers and I originally planned a similar book Unmanaged Code: Escaping the Matrix of .NET but didn’t have time to finish it yet

- Dmitry Vostokov @ DumpAnalysis.org -

Some debugger commands or commands they invoke can be context-sensitive and their diagnostic output can depend on a current thread or a process set in a debugger, not to mention loaded debugger extensions and even their load order. Therefore, it is advisable to be context-conscious about or at least to know about context sensitivity. For example, in one mmc.exe process memory dump a default analysis command in x64 WinDbg doesn’t show any managed stack trace reported by a user who had seen it in a failure dialog box:

0:000> !analyze -v

[...]

MANAGED_BITNESS_MISMATCH:
Managed code needs matching platform of sos.dll for proper analysis. Use 'x86' debugger.

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

STACK_TEXT: 
0007fc98 7c827d19 77e6202c 00000002 0007fce8 ntdll!KiFastSystemCallRet
0007fc9c 77e6202c 00000002 0007fce8 00000001 ntdll!NtWaitForMultipleObjects+0xc
0007fd44 7739bbd1 00000002 0007fd6c 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
0007fda0 6c296601 00000001 0007fdd4 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
0007fdc0 6c29684b 000004ff ffffffff 00000001 duser!CoreSC::Wait+0x3a
0007fdf4 6c29693d 0007fe34 00000000 00000000 duser!CoreSC::xwProcessNL+0xab
0007fe14 773b0c02 0007fe34 00000000 00000000 duser!MphProcessMessage+0x2e
0007fe5c 7c828556 0007fe74 00000014 0007ffb0 user32!__ClientGetMessageMPH+0x30
0007fe84 7739c811 7739c844 01116894 00000000 ntdll!KiUserCallbackDispatcher+0x2e
0007fea4 7f072fd6 01116894 00000000 00000000 user32!NtUserGetMessage+0xc
0007fec0 010080ef 01116894 01116860 00000002 mfc42u!CWinThread::PumpMessage+0x16
0007fef0 7f072dda 01116860 01116860 ffffffff mmc!CAMCApp::PumpMessage+0x37
0007ff08 7f044d5b ffffffff 00000002 7ffd9000 mfc42u!CWinThread::Run+0x4a
0007ff1c 01034e19 01000000 00000000 00020710 mfc42u!AfxWinMain+0x7b
0007ffc0 77e6f23b 00000000 00000000 7ffd9000 mmc!wWinMainCRTStartup+0x19d
0007fff0 00000000 01034cb0 00000000 78746341 kernel32!BaseProcessStart+0x23

Instead of concluding that the dump file wasn’t saved at the time of the failure we pay attention to all aspects of the default analysis and see that we need a platform-specific debugger. We load the same dump file into x86 WinDbg: 

0:000> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

Now we see that we need to load SOS extension explicitly and retry: 

0:000> .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll

0:000> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x4ec (0)
Current frame:
ChildEBP RetAddr  Caller,Callee

Managed stack trace is empty here but look at all threads (we list full traces in order not to miss any module) we find one that shows a dialog box reporting a failure:

0:000> ~*kL 100

[...]

  17  Id: 658.7e4 Suspend: 1 Teb: 7ff48000 Unfrozen
ChildEBP RetAddr 
06b4f498 7739bf53 ntdll!KiFastSystemCallRet
06b4f4d0 7738965e user32!NtUserWaitMessage+0xc
06b4f4f8 773896a0 user32!InternalDialogBox+0xd0
06b4f518 773896e8 user32!DialogBoxIndirectParamAorW+0x37
06b4f53c 4afde2e1 user32!DialogBoxParamW+0×3f
06b4f584 4b05c4bc mmcndmgr!IsolationAwareDialogBoxParamW+0×4e
06b4f5a4 4b05c6eb mmcndmgr!ATL::CDialogImpl<CSnapInFailureReportDialog, ATL::CWindow>::DoModal+0×4f
06b4f68c 77c80193 mmcndmgr!CSnapInFailureReporter::ReportSnapInFailure+0×195
06b4f6b8 77ce33e1 rpcrt4!Invoke+0×30
06b4fab8 77ce2ed5 rpcrt4!NdrStubCall2+0×299
06b4fb10 7778d01b rpcrt4!CStdStubBuffer_Invoke+0xc6
06b4fb54 7778cfc8 ole32!SyncStubInvoke+0×37
06b4fb9c 776c120b ole32!StubInvoke+0xa7
06b4fc78 776c0bf5 ole32!CCtxComChnl::ContextInvoke+0xec
06b4fc94 776bc455 ole32!MTAInvoke+0×1a
06b4fcc0 7778ced5 ole32!STAInvoke+0×48
06b4fcf4 7778cd66 ole32!AppInvoke+0xa3
06b4fdc8 7778c24d ole32!ComInvokeWithLockAndIPID+0×2c5
06b4fdf0 776bc344 ole32!ComInvoke+0xca
06b4fe04 776bc30f ole32!ThreadDispatch+0×23
06b4fe1c 7739b6e3 ole32!ThreadWndProc+0xfe
06b4fe48 7739b874 user32!InternalCallWinProc+0×28
06b4fec0 7739ba92 user32!UserCallWinProcCheckWow+0×151
06b4ff28 7739bad0 user32!DispatchMessageWorker+0×327
06b4ff38 7768ffdc user32!DispatchMessageW+0xf
06b4ff6c 7768f366 ole32!CDllHost::STAWorkerLoop+0×5c
06b4ff88 7768f2a2 ole32!CDllHost::WorkerThread+0xc8
06b4ff90 776bbab4 ole32!DLLHostThreadEntry+0xd
06b4ffac 776b1704 ole32!CRpcThread::WorkerLoop+0×26
06b4ffb8 77e6482f ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×20
06b4ffec 00000000 kernel32!BaseThreadStart+0×34

The previous thread #16 is a CLR thread loading an assembly:

  16  Id: 658.28c Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr 
0714d024 7c827d29 ntdll!KiFastSystemCallRet
0714d028 77e61d1e ntdll!ZwWaitForSingleObject+0xc
0714d098 73ca790b kernel32!WaitForSingleObjectEx+0xac
0714d0c4 73ca485a cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x12f
0714d0f0 73ca37ce cryptnet!CryptRetrieveObjectByUrlW+0x9b
0714d168 73ca4a60 cryptnet!RetrieveObjectByUrlValidForSubject+0x5b
0714d1b8 73ca3525 cryptnet!RetrieveTimeValidObjectByUrl+0xbc
0714d220 73ca3473 cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0xc2
0714d2d0 73ca3314 cryptnet!CTVOAgent::GetTimeValidObject+0x2f1
0714d300 73ca2c00 cryptnet!FreshestCrlFromCrlGetTimeValidObject+0x2d
0714d344 73ca43a4 cryptnet!CryptGetTimeValidObject+0x58
0714d3a0 73ca3122 cryptnet!GetTimeValidCrl+0x1e0
0714d3e4 73ca3080 cryptnet!GetBaseCrl+0x34
0714d484 761d9033 cryptnet!MicrosoftCertDllVerifyRevocation+0x128
0714d514 761d8eef crypt32!I_CryptRemainingMilliseconds+0x21b
0714d584 761cf39f crypt32!CertVerifyRevocation+0xb7
0714d604 761c6966 crypt32!CChainPathObject::CalculateRevocationStatus+0x1f2
0714d64c 761c6771 crypt32!CChainPathObject::CalculateAdditionalStatus+0x147
0714d708 761c78bc crypt32!CCertChainEngine::CreateChainContextFromPathGraph+0x227
0714d738 761c783f crypt32!CCertChainEngine::GetChainContext+0x44
0714d760 76bb6d8f crypt32!CertGetCertificateChain+0x60
0714d7c4 76bb6bbc wintrust!_WalkChain+0x1a8
0714d800 76bb39ef wintrust!WintrustCertificateTrust+0xb7
0714d8f4 76bb31e2 wintrust!_VerifyTrust+0x144
0714d918 64025b1b wintrust!WinVerifyTrust+0x4e
0714d9bc 7a117c85 mscorsec!GetPublisher+0xe4
0714da14 79ebeccb mscorwks!PEFile::CheckSecurity+0xcb
0714da3c 79ebec14 mscorwks!PEAssembly::DoLoadSignatureChecks+0x3a
0714da64 79ebf05a mscorwks!PEAssembly::PEAssembly+0x109
0714dd00 79ebf155 mscorwks!PEAssembly::DoOpen+0x103
0714dd94 79eb8ff2 mscorwks!PEAssembly::Open+0x79
0714def8 79eb6a5e mscorwks!AppDomain::BindAssemblySpec+0x247
0714df90 79eb691c mscorwks!PEFile::LoadAssembly+0×95
0714e030 79eb68c0 mscorwks!Module::LoadAssembly+0xee
0714e06c 79e92873 mscorwks!Assembly::FindModuleByTypeRef+0×113
0714e0d8 79fc3dc8 mscorwks!ClassLoader::ResolveTokenToTypeDefThrowing+0×88
0714e12c 79fc953d mscorwks!CEEInfo::AddDependencyOnClassToken+0×103
0714e158 79fc61cf mscorwks!CEEInfo::ScanForModuleDependencies+0xa3
0714e1fc 7908bce1 mscorwks!CEEInfo::getArgType+0×256
0714e214 7908bc5b mscorjit!Compiler::eeGetArgType+0×23
0714e25c 79067745 mscorjit!Compiler::impInlineInitVars+0×3c3
0714e4fc 790673d5 mscorjit!Compiler::fgInvokeInlineeCompiler+0×95
0714e518 79067400 mscorjit!Compiler::fgMorphCallInline+0×41
0714e52c 79065272 mscorjit!Compiler::fgInline+0×30
0714e534 7906513e mscorjit!Compiler::fgMorph+0×45
0714e544 79065b8e mscorjit!Compiler::compCompile+0×83
0714e590 79065d33 mscorjit!Compiler::compCompile+0×44f
0714e618 79066448 mscorjit!jitNativeCode+0xef
0714e63c 79fc7198 mscorjit!CILJit::compileMethod+0×25
0714e6a8 79fc722d mscorwks!invokeCompileMethodHelper+0×72
0714e6ec 79fc72a0 mscorwks!invokeCompileMethod+0×31
0714e740 79fc7019 mscorwks!CallCompileMethodWithSEHWrapper+0×5b
0714eae8 79fc6ddb mscorwks!UnsafeJitFunction+0×31b
0714eb8c 79e811a3 mscorwks!MethodDesc::MakeJitWorker+0×1a8
0714ebe4 79e81363 mscorwks!MethodDesc::DoPrestub+0×41b
0714ec34 01c01efe mscorwks!PreStubWorker+0xf3
WARNING: Frame IP not in any known module. Following frames may be wrong.
0714ec4c 06b08f29 0×1c01efe
0714ecb4 06b088dc 0×6b08f29
0714edf0 79e71b4c 0×6b088dc
0714edf4 79e7e45d mscorwks!CallDescrWorker+0×33
0714eeac 79e968b0 mscorwks!MethodDesc::IsSharedByGenericInstantiations+0×1c
0714ef2c 79e9eeb2 mscorwks!MetaSig::MetaSig+0×3a
0714f258 00000000 mscorwks!JIT_MonReliableEnter+0×120

If we switch to it we get a managed stack:

0:000> ~16s
eax=0714ce90 ebx=048bb528 ecx=77e63d5b edx=7ffd4000 esi=0000073c edi=00000000
eip=7c82860c esp=0714d028 ebp=0714d098 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297
ntdll!KiFastSystemCallRet:
7c82860c c3              ret

0:016> !analyze -v

[...]

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x28c (16)
Current frame:
ChildEBP RetAddr  Caller,Callee
[...]
0714f264 792e0c3a (MethodDesc 0x79104344 +0xa System.Reflection.RuntimeMethodInfo.GetParametersNoCopy())
0714f28c 792d52d8 (MethodDesc 0x790c5058 +0x48 System.RuntimeMethodHandle.InvokeMethodFast(System.Object, System.Object[], System.Signature, System.Reflection.MethodAttributes, System.RuntimeTypeHandle))
0714f2dc 792d5086 (MethodDesc 0x791043a4 +0x106 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean))
0714f314 792d4f6e (MethodDesc 0x7910439c +0x1e System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo))
0714f338 7928ea4b (MethodDesc 0x79108798 +0x82b System.RuntimeType.InvokeMember(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object, System.Object[], System.Reflection.ParameterModifier[], System.Globalization.CultureInfo, System.String[]))
0714f478 7973ea9d (MethodDesc 0x79108264 +0x1d System.Type.InvokeMember(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object, System.Object[]))
[...]
0714f588 792d6cf6 (MethodDesc 0x791939dc +0x66 System.Threading.ThreadHelper.ThreadStart_Context(System.Object))
0714f594 792e019f (MethodDesc 0x7910276c +0x6f System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
0714f5a8 792d6c74 (MethodDesc 0x790fbde4 +0x44 System.Threading.ThreadHelper.ThreadStart())

- Dmitry Vostokov @ DumpAnalysis.org -

In cases where we don’t see managed code exceptions or managed stack traces by default, we need to identify CLR threads in order to try various SOS commands and start digging into a managed realm. These threads are easily distinguished by mscorwks module on their stack traces (don’t forget to list full stack traces):

0:000> ~*kL 100

.  0  Id: 658.4ec Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr 
0007fc98 7c827d19 ntdll!KiFastSystemCallRet
0007fc9c 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0007fd44 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
0007fda0 6c296601 user32!RealMsgWaitForMultipleObjectsEx+0x141
0007fdc0 6c29684b duser!CoreSC::Wait+0x3a
0007fdf4 6c29693d duser!CoreSC::xwProcessNL+0xab
0007fe14 773b0c02 duser!MphProcessMessage+0x2e
0007fe5c 7c828556 user32!__ClientGetMessageMPH+0x30
0007fe84 7739c811 ntdll!KiUserCallbackDispatcher+0x2e
0007fea4 7f072fd6 user32!NtUserGetMessage+0xc
0007fec0 010080ef mfc42u!CWinThread::PumpMessage+0x16
0007fef0 7f072dda mmc!CAMCApp::PumpMessage+0x37
0007ff08 7f044d5b mfc42u!CWinThread::Run+0x4a
0007ff1c 01034e19 mfc42u!AfxWinMain+0x7b
0007ffc0 77e6f23b mmc!wWinMainCRTStartup+0x19d
0007fff0 00000000 kernel32!BaseProcessStart+0x23

   1  Id: 658.82c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr 
003afea0 7c827d19 ntdll!KiFastSystemCallRet
003afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
003aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
003affb8 77e6482f ntdll!EtwpEventPump+0x27f
003affec 00000000 kernel32!BaseThreadStart+0x34

   2  Id: 658.648 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr 
00f3fe18 7c827859 ntdll!KiFastSystemCallRet
00f3fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
00f3ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
00f3ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd
00f3ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0x9d
00f3ffb8 77e6482f rpcrt4!ThreadStartRoutine+0x1b
00f3ffec 00000000 kernel32!BaseThreadStart+0x34

   3  Id: 658.640 Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr 
0156fdb4 7c827d19 ntdll!KiFastSystemCallRet
0156fdb8 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0156fe60 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
0156febc 6c296601 user32!RealMsgWaitForMultipleObjectsEx+0x141
0156fedc 6c29684b duser!CoreSC::Wait+0x3a
0156ff10 6c28f9e6 duser!CoreSC::xwProcessNL+0xab
0156ff30 6c28bce1 duser!GetMessageExA+0x44
0156ff84 77bcb530 duser!ResourceManager::SharedThreadProc+0xb6
0156ffb8 77e6482f msvcrt!_endthreadex+0xa3
0156ffec 00000000 kernel32!BaseThreadStart+0x34

   4  Id: 658.e74 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr 
01d1fe30 7c827d19 ntdll!KiFastSystemCallRet
01d1fe34 77e6202c ntdll!NtWaitForMultipleObjects+0xc
01d1fedc 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
01d1fef8 79f02541 kernel32!WaitForMultipleObjects+0x18
01d1ff58 79f0249e mscorwks!DebuggerRCThread::MainLoop+0xe9
01d1ff88 79f023c5 mscorwks!DebuggerRCThread::ThreadProc+0xe5
01d1ffb8 77e6482f mscorwks!DebuggerRCThread::ThreadProcStatic+0×9c
01d1ffec 00000000 kernel32!BaseThreadStart+0×34

   5  Id: 658.4d4 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr 
03dffcc4 7c827d19 ntdll!KiFastSystemCallRet
03dffcc8 77e6202c ntdll!NtWaitForMultipleObjects+0xc
03dffd70 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03dffd8c 79f92bcb kernel32!WaitForMultipleObjects+0x18
03dffdac 79f97028 mscorwks!WKS::WaitForFinalizerEvent+0×77
03dffdc0 79e9845f mscorwks!WKS::GCHeap::FinalizerThreadWorker+0×49
03dffdd4 79e983fb mscorwks!Thread::DoADCallBack+0×32a
03dffe68 79e98321 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
03dffea4 79eef6cc mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
03dffecc 79eef6dd mscorwks!ManagedThreadBase_NoADTransition+0×32
03dffedc 79f3c63c mscorwks!ManagedThreadBase::FinalizerBase+0xd
03dfff14 79f92015 mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb
03dfffb8 77e6482f mscorwks!Thread::intermediateThreadProc+0×49
03dfffec 00000000 kernel32!BaseThreadStart+0×34

   6  Id: 658.f54 Suspend: 1 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr 
040afec4 7c826f69 ntdll!KiFastSystemCallRet
040afec8 77e41ed5 ntdll!NtDelayExecution+0xc
040aff30 79fd8a41 kernel32!SleepEx+0x68
040affac 79fd88ef mscorwks!ThreadpoolMgr::TimerThreadFire+0×6d
040affb8 77e6482f mscorwks!ThreadpoolMgr::TimerThreadStart+0×57
040affec 00000000 kernel32!BaseThreadStart+0×34

   7  Id: 658.988 Suspend: 1 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr 
0410fc2c 7c827d29 ntdll!KiFastSystemCallRet
0410fc30 77e61d1e ntdll!ZwWaitForSingleObject+0xc
0410fca0 79e8c5f9 kernel32!WaitForSingleObjectEx+0xac
0410fce4 79e8c52f mscorwks!PEImage::LoadImage+0×1af
0410fd34 79e8c54e mscorwks!CLREvent::WaitEx+0×117
0410fd48 79ee3f35 mscorwks!CLREvent::Wait+0×17
0410fe14 79f92015 mscorwks!AppDomain::ADUnloadThreadStart+0×308
0410ffb8 77e6482f mscorwks!Thread::intermediateThreadProc+0×49
0410ffec 00000000 kernel32!BaseThreadStart+0×34

   8  Id: 658.e0 Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr 
0422fcec 7c827d19 ntdll!KiFastSystemCallRet
0422fcf0 7c83c7be ntdll!NtWaitForMultipleObjects+0xc
0422ffb8 77e6482f ntdll!RtlpWaitThread+0x161
0422ffec 00000000 kernel32!BaseThreadStart+0x34

   9  Id: 658.db4 Suspend: 1 Teb: 7ff4e000 Unfrozen
ChildEBP RetAddr 
0447fec0 7c827d19 ntdll!KiFastSystemCallRet
0447fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0447ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
0447ff88 76929e35 kernel32!WaitForMultipleObjects+0x18
0447ffb8 77e6482f userenv!NotificationThread+0x5f
0447ffec 00000000 kernel32!BaseThreadStart+0x34

  10  Id: 658.e7c Suspend: 1 Teb: 7ff4c000 Unfrozen
ChildEBP RetAddr 
0550ff7c 7c8277f9 ntdll!KiFastSystemCallRet
0550ff80 71b25914 ntdll!NtRemoveIoCompletion+0xc
0550ffb8 77e6482f mswsock!SockAsyncThread+0x69
0550ffec 00000000 kernel32!BaseThreadStart+0x34

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

The art of DA+TA analysis now acquires a Latin name: Ars Recordatio (from Latin recordatio: recollection, memory, recall and ars (artis): art, skill, method, technique, knowledge, method, science, way, conduct, character). Other variations I rejected, Ars Memoria, Ars Indicium and Ars Animus, are either widely used in a different sense or presuppose some kind of intelligence inside memory dumps and traces (animus).

- Dmitry Vostokov @ DumpAnalysis.org -