Software Diagnostics Library

Memory Dump Analysis Services plans to introduce Mac OS X memory dump analysis training this year based on the success of its Windows variant. For details and how to register please visit this page:

http://www.dumpanalysis.com/accelerated-mac-os-x-core-dump-analysis

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services starts providing debugging training and assistance in addition to (based on) memory dump and software trace analysis. It has recently registered www.debugging.pro domain and is working on extending its courses to cover live debugging and additional OS platforms this year.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced training sessions time may not suitable due to different geographic time zones. So I have decided to publish this training in a book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

• Title: Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes
• Description: The full transcript of Memory Dump Analysis Services Training with 10 step-by-step exercises, notes, and selected Q&A.
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Publisher: OpenTask (January 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 180 pages
• ISBN-13: 978-1908043344

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services started working on ₂CARE² system (Crash Analysis Report Environment, 2nd generation) and asked me to design a trademark. My Chemistry background (I like Organic Chemistry most) and imagination led me to represent client and server parts holistically as an aromatic-like compound:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having done in the past with user space raw stack data analysis for 32-bit complete memory dumps I found today the need to look at kernel raw stack data from all threads and created this fast script:

!for_each_thread "!thread @#Thread; r? $t1 = ((nt!_KTHREAD *) @#Thread )->StackLimit; r? $t2 = ((nt!_KTHREAD *) @#Thread )->InitialStack; dps @$t1 @$t2"

It can be run for kernel and complete memory dumps from both x86 and x64 systems. If you need to have correct symbolic mapping for user space in kernel space data you need to modify it a bit and it will be slower to run.

!for_each_thread "!thread @#Thread ff; .thread /r /p @#Thread; r? $t1 = ((nt!_KTHREAD *) @#Thread )->StackLimit; r? $t2 = ((nt!_KTHREAD *) @#Thread )->InitialStack; dps @$t1 @$t2"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

3 bugtations in a column:

Every debugger I know has trouble debugging.
Talent is helpful in debugging, but guts are absolutely necessary.
With failure comes a dump.

Joseph Heller

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

According to Google Analytics the number of visits / year increased by 4% since 2010 with almost 160,000 unique visitors (2% increase) from 180 countries and 34% of them are coming back. 2,725 visits were via 12 mobile operating systems (106% increase). Here are the top 100 network locations out of 28,932:

Top 25 visiting countries:

More than 8,000 portal and blog pages were viewed a total of more than 392,000 times with top 100 content pages:

More than 70,000 Google search keywords pointed to the portal and this blog with 100 most frequent (some are in Russian):

Special thanks to more  1,000 web sites that mention the portal and this blog with the first top 100:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to stack trace collections for threads (unmanaged, managed and predicate) we introduce an additional pattern for I/O requests. Such requests are implemented via the so called I/O request packets (IRP) that “travel” from a device driver to a device driver similar to a C++ class method to another C++ class method (where a device object address is similar to a C++ object instance address). An IRP stack is used to keep a track of the current driver which is processing an IRP that is reused between device drivers. Its is basically an array of structures describing how a particular driver function was called with appropriate parameters similar to a call frame on an execution thread stack. Long time ago I created an UML diagram depicting the flow of an IRP through the driver (device) stack (diagram #3). An I/O stack location pointer is decremented (from the bottom to the top) like a thread stack pointer (ESP or RSP). We can list active and completed I/O requests with their stack traces using !irpfind -v WinDbg command:

1: kd> !irpfind -v

Scanning large pool allocation table for Tag: Irp? (832c7000 : 833c7000)

Irp    [ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]         MDL Process
8883dc18: Irp is active with 1 stacks 1 is current (= 0x8883dc88)
No Mdl: No System Buffer: Thread 888f8950:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  d, 0]   5  1 88515ae8 888f82f0 00000000-00000000    pending
\FileSystem\Npfs
Args: 00000000 00000000 00110008 00000000

891204c8: Irp is active with 1 stacks 1 is current (= 0x89120538)
No Mdl: No System Buffer: Thread 889635b0:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 84752028 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000

89120ce8: Irp is active with 1 stacks 1 is current (= 0x89120d58)
No Mdl: No System Buffer: Thread 89212030:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 8921be00 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000
Searching NonPaged pool (80000000 : ffc00000) for Tag: Irp?

[...]

892cbe48: Irp is active with 9 stacks 9 is current (= 0x892cbfd8)
No Mdl: No System Buffer: Thread 892add78:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
>[  c, 2]   0  1 8474a020 892c8c80 00000000-00000000    pending
\FileSystem\Ntfs
Args: 00000800 00000002 00000000 00000000

892daa88: Irp is active with 4 stacks 4 is current (= 0x892dab64)
No Mdl: System buffer=831559c8: Thread 8322c8e8:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
>[  e,2d]   5  1 884ba750 83190c40 00000000-00000000    pending
\Driver\AFD
Args: 890cbc44 890cbc44 88e55297 8943b6c8

892ea4e8: Irp is active with 4 stacks 4 is current (= 0x892ea5c4)
No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  2 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 c0000185
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  2 83a34bb0 00000000 84d779ed-88958050
\Driver\atapi CLASSPNP!ClasspMediaChangeDetectionCompletion
Args: 88958050 00000000 00000000 83992d10
>[  0, 0]   2  0 891ee030 00000000 00000000-00000000
\Driver\cdrom
Args: 00000000 00000000 00000000 00000000

8933fcb0: Irp is active with 1 stacks 1 is current (= 0x8933fd20)
No Mdl: No System Buffer: Thread 84753d78:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 84759f40 00000000-00000000    pending
\FileSystem\Npfs
Args: 0000022a 00000000 00000000 00000000

893cf550: Irp is active with 1 stacks 1 is current (= 0x893cf5c0)
No Mdl: No System Buffer: Thread 888fd3b8:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 88515ae8 834d30d0 00000000-00000000    pending
\FileSystem\Npfs
Args: 00000400 00000000 00000000 00000000

893da468: Irp is active with 6 stacks 7 is current (= 0x893da5b0)
Mdl=892878f0: No System Buffer: Thread 00000000:  Irp is completed.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  0 84b3e028 00000000 9747fcd0-00000000
\Driver\usbehci USBSTOR!USBSTOR_CswCompletion
Args: 00000000 00000000 00000000 00000000
[  f, 0]   0  0 892ba8f8 00000000 84d780ce-8328e0f0
\Driver\USBSTOR CLASSPNP!TransferPktComplete
Args: 00000000 00000000 00000000 00000000

893efb00: Irp is active with 10 stacks 11 is current (= 0x893efcd8)
Mdl=83159378: No System Buffer: Thread 82b7f828:  Irp is completed.  Pending has been returned
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 885a55b8 00000000 81614138-00000000
\Driver\disk partmgr!PmReadWriteCompletion
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 89257c90 00000000 8042e4d4-831caab0
\Driver\partmgr volmgr!VmpReadWriteCompletionRoutine
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 831ca9f8 00000000 84dad0be-00000000
\Driver\volmgr ecache!EcDispatchReadWriteCompletion
Args: 00000000 00000000 00000000 00000000
[  3, 0]   0  0 8319c020 00000000 84dcc4d4-8576f8ac
\Driver\Ecache volsnap!VspSignalCompletion
Args: 00000000 00000000 00000000 00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

An advice to succeed in Software Technical Support:

They said : The dump came from the wrong site
He’s a page from the end of the dump
I said : The dump came from the right site
Took me just one look
<…>
With a little debugging
You will survive
With a little debugging
You will get by
Do what you want
Go your own way
in Technical Support.

Modern Talking

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The beginning of a debugging tale:

He was debugging for 7 years and when he stopped and looked around he saw only 2 people left from the team of 50 strong technical support engineers.

Folktale

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have memory leaks related to the growing number of page tables. One reason for that could be the growing number of zombie processes (noticeable with tens of thousands of them).

1: kd> !process 0 0
[...] 
PROCESS fffffa80266bd6f0
    SessionId: 0  Cid: 0a6c    Peb: 7fffffdc000  ParentCid: 03ac
    DirBase: 9d35a000  ObjectTable: fffff8a00170ac80  HandleCount: 152.
    Image: svchost.exe
[…] 
PROCESS fffffa8027de9b30
    SessionId: 0  Cid: 21d0    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 37881000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…] 
PROCESS fffffa8028eb0600
    SessionId: 0  Cid: ab88    Peb: 7fffffdf000  ParentCid: 02e0
    DirBase: 27a2f000  ObjectTable: 00000000  HandleCount:   0.
    Image: conhost.exe
[…]

Even zombies have at least one remaining page (page directory) from the former page tables of their virtual to physical memory mapping (!dd is the same as dd command but for physical memory):

1: kd> !dd 9d35a000
#9d35a000 9dd62867 03c00000 00000000 00000000
#9d35a010 00000000 00000000 00000000 00000000
#9d35a020 00000000 00000000 00000000 00000000
#9d35a030 00000000 00000000 00000000 00000000
#9d35a040 00000000 00000000 00000000 00000000
#9d35a050 00000000 00000000 00000000 00000000
#9d35a060 00000000 00000000 00000000 00000000
#9d35a070 00000000 00000000 9d45e867 49500000

1: kd> !dd 37881000
#37881000 00000000 00000000 00000000 00000000
#37881010 00000000 00000000 00000000 00000000
#37881020 00000000 00000000 00000000 00000000
#37881030 00000000 00000000 00000000 00000000
#37881040 00000000 00000000 00000000 00000000
#37881050 00000000 00000000 00000000 00000000
#37881060 00000000 00000000 00000000 00000000
#37881070 00000000 00000000 00000000 00000000

1: kd> !dd 27a2f000
#27a2f000 00000000 00000000 00000000 00000000
#27a2f010 00000000 00000000 00000000 00000000
#27a2f020 00000000 00000000 00000000 00000000
#27a2f030 00000000 00000000 00000000 00000000
#27a2f040 00000000 00000000 00000000 00000000
#27a2f050 00000000 00000000 00000000 00000000
#27a2f060 00000000 00000000 00000000 00000000
#27a2f070 00000000 00000000 00000000 00000000

We also see that 2 conhost.exe processes have identical physical to virtual mapping because their user space mappings are no longer valid (zeroed) and svchost.exe process has user space mapping (in blue italics):

1: kd> !ptov 27a2f000
Amd64PtoV: pagedir 27a2f000
27a2f000 fffff6fb`7dbed000
71530000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000 
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 37881000
Amd64PtoV: pagedir 37881000
37881000 fffff6fb`7dbed000
8d482000 fffff6fb`7dbee000
19d000 fffff6fb`7dbef000
199000 fffff6fb`7dbf0000
b6a04000 fffff6fb`7dbf1000
b1f57000 fffff6fb`7dbf2000
29c4000 fffff6fb`7dbf3000
1c53000 fffff6fb`7dbf5000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

1: kd> !ptov 9d35a000
Amd64PtoV: pagedir 9d35a000
9e587000 10000
6871e000 20000
af5aa000 30000
af5ab000 31000
afaac000 32000
afbad000 33000
af2f5000 40000
9d66b000 50000
22199000 60000
9d962000 e5000
9d261000 e6000
9dc60000 e7000
9d256000 ea000
9d84f000 eb000
9e4ec000 ec000
9e081000 ed000
9d876000 ee000
9e271000 ef000
b8bfd000 f0000
b8efe000 f1000
b86ff000 f2000
b5302000 f3000
b5202000 f4000
b5502000 f5000
b7f03000 f6000
b8404000 f7000
b8415000 100000
b8b16000 101000
b1b17000 102000
[…]
2cd4000 77512000
5d7000 77515000
5d8000 77516000
4d9000 77517000
b358f000 77590000
aef04000 77591000
68624000 77592000
64b26000 77593000
af4c6000 77595000
b2042000 7efe0000
b2143000 7efe1000
b1a56000 7efe2000
b1a57000 7efe3000
b1b58000 7efe4000
1ba000 7ffe0000
9da69000 bfeb0000
aeeae000 ffea0000
af191000 ffea1000
9d76a000 ffea2000
ae793000 ffea3000
9dc8e000 ffea5000
b7eb7000 ffea6000
9dffc000 ffea7000
[…]
2e4d8000 fffffa80`28f2d000
2c3d7000 fffffa80`28f2e000
30ed6000 fffffa80`28f2f000
2efd5000 fffffa80`28f30000
2ded4000 fffffa80`28f31000
2a5d3000 fffffa80`28f32000
bb400000 fffffa80`29600000 (large page)
bb200000 fffffa80`29800000 (large page)
100000 ffffffff`ffd00000
105000 ffffffff`ffd01000
101000 ffffffff`ffd02000
102000 ffffffff`ffd03000
103000 ffffffff`ffd04000
104000 ffffffff`ffd05000
fec00000 ffffffff`ffd06000
1000 ffffffff`ffd07000
106000 ffffffff`ffd08000
123000 ffffffff`ffd09000
0 ffffffff`ffd0a000
124000 ffffffff`ffd0b000
2000 ffffffff`ffd0c000
e00c7000 ffffffff`ffd0d000
e0080000 ffffffff`ffd0e000
107000 ffffffff`ffd25000
108000 ffffffff`ffd26000
109000 ffffffff`ffd27000
10a000 ffffffff`ffd28000
10b000 ffffffff`ffd29000
10c000 ffffffff`ffd2a000
10d000 ffffffff`ffd2b000
10e000 ffffffff`ffd2c000
10f000 ffffffff`ffd2d000
110000 ffffffff`ffd2e000
111000 ffffffff`ffd2f000
112000 ffffffff`ffd30000
113000 ffffffff`ffd31000
114000 ffffffff`ffd32000
115000 ffffffff`ffd33000
116000 ffffffff`ffd34000
117000 ffffffff`ffd35000
118000 ffffffff`ffd36000
119000 ffffffff`ffd37000
11a000 ffffffff`ffd38000
11b000 ffffffff`ffd39000
11c000 ffffffff`ffd3a000
11d000 ffffffff`ffd3b000
11e000 ffffffff`ffd3c000
11f000 ffffffff`ffd3d000
120000 ffffffff`ffd3e000
121000 ffffffff`ffd3f000
122000 ffffffff`ffd40000
fee00000 ffffffff`fffe0000

In order to check user space virtual addresses we have to switch to the corresponding process context:

1: kd> !pte fffffa80`28f2d000
                                           VA fffffa8028f2d000
PXE at FFFFF6FB7DBEDFA8    PPE at FFFFF6FB7DBF5000    PDE at FFFFF6FB7EA00A38    PTE at FFFFF6FD40147968
contains 0000000001C53863  contains 0000000001C54863  contains 0000000049320863  contains 000000002E4D8963
pfn 1c53      —DA–KWEV  pfn 1c54      —DA–KWEV  pfn 49320     —DA–KWEV  pfn 2e4d8     -G-DA–KWEV

1: kd> .process /r /p fffffa80266bd6f0
Implicit process is now fffffa80`266bd6f0
Loading User Symbols

1: kd> !pte 10000
                                           VA 0000000000010000
PXE at FFFFF6FB7DBED000    PPE at FFFFF6FB7DA00000    PDE at FFFFF6FB40000000    PTE at FFFFF68000000080
contains 03C000009DD62867  contains 031000009D865867  contains 7C2000009DD66867  contains 9CB000009E587867
pfn 9dd62     —DA–UWEV  pfn 9d865     —DA–UWEV  pfn 9dd66     —DA–UWEV  pfn 9e587     —DA–UW-V

This pattern came to our attention after several customers complained about the growing number of memory allocated for page tables which exceeded a gigabyte after several days.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes debugging information is absent from module info in memory dumps and a debugger can’t recognize and automatically load symbol files. For example, we see this stack trace without loaded component symbols:

THREAD 8a17c6d8  Cid 02ec.02f0  Teb: 7ffdf000 Win32Thread: e17b4420 WAIT: (UserRequest) UserMode Non-Alertable
89873d00  SynchronizationEvent
IRP List:
89d9fd20: (0006,0094) Flags: 00000800  Mdl: 00000000
Not impersonating
DeviceMap                 e10086c8
Owning Process            0       Image:         <Unknown>
Attached Process          8a17cda0       Image:         ApplicationA.exe
Wait Start TickCount      8164394        Ticks: 2884 (0:00:00:45.062)
Context Switch Count      1769160                 LargeStack
UserTime                  00:00:55.250
KernelTime                00:01:56.109
Start Address 0×0103e5e1
Stack Init ba390000 Current ba38fca0 Base ba390000 Limit ba38b000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
*** ERROR: Module load completed but symbols could not be loaded for ModuleA.dll
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0×2f
ba38fcc4 804fb068 nt!KiSwapThread+0×8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0×1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0×9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc (TrapFrame @ ba38fd64)
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0×132
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0×46
WARNING: Stack unwind information not available. Following frames may be wrong.
0006f7a4 0132b785 ModuleA+0×53df7
0006f7cc 0132c728 ModuleA+0xb785
0006f7e4 01346426 ModuleA+0xc728
0006f848 7e418734 ModuleA+0×26426
0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf

0: kd> .process /r /p 8a17cda0
Implicit process is now 8a17cda0
Loading User Symbols

0: kd> lmv m ModuleA
start    end        module name
01320000 013bb000   ModuleA   (deferred)
Image path: C:\Program Files\VendorA\ModuleA.dll
Image name: ModuleA.dll
Timestamp:        Thu Aug 11 21:42:08 2011 (4E4484F0)
CheckSum:         000A9C8B
ImageSize:        0009B000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

0: kd> !lmi ModuleA
Loaded Module Info: [ModuleA]
Module: ModuleA
Base Address: 01320000
Image Name: ModuleA.dll
Machine Type: 332 (I386)
Time Stamp: 4e4484f0 Thu Aug 11 21:42:08 2011
Size: 9b000
CheckSum: a9c8b
Characteristics: 2102
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    5e, 830a0,   830a0 [Debug data not mapped] - can’t validate symbols, if present.
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

However, in a stack trace collection (!process 0 ff WinDbg command) we find another stack trace from a different process but with loaded symbol files for ModuleA:

THREAD 89703020  Cid 1068.1430  Teb: 7ffdf000 Win32Thread: e34d43a8 WAIT: (UserRequest) UserMode Non-Alertable
89a3ac58  NotificationEvent
89703110  NotificationTimer
IRP List:
899ab488: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e10086c8
Owning Process            0       Image:         <Unknown>
Attached Process          89825020       Image:         ApplicationB.exe
Wait Start TickCount      8164457        Ticks: 2821 (0:00:00:44.078)
Context Switch Count      552                 LargeStack
UserTime                  00:00:00.296
KernelTime                00:00:00.890
Start Address 0×0103e5e1
Stack Init b8796000 Current b8795ca0 Base b8796000 Limit b8791000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr
b8795cb8 80503836 nt!KiSwapContext+0×2f
b8795cc4 804fb068 nt!KiSwapThread+0×8a
b8795cec 805c0750 nt!KeWaitForSingleObject+0×1c2
b8795d50 8054161c nt!NtWaitForSingleObject+0×9a
b8795d50 7c90e4f4 nt!KiFastCallEntry+0xfc (TrapFrame @ b8795d64)
0006fa1c 7c90df3c ntdll!KiFastSystemCallRet
0006fa20 7c8025db ntdll!NtWaitForSingleObject+0xc
0006fa84 010ae96a kernel32!WaitForSingleObjectEx+0xa8
0006fafc 010aeaaf ModuleA!Wait+0xaa
0006fb38 010b84ce ModuleA!Read+0×6f
[…]

0: kd> !lmi ModuleA
Loaded Module Info: [ModuleA]
Module: ModuleA
Base Address: 01090000
Image Name: ModuleA.dll
Machine Type: 332 (I386)
Time Stamp: 4e4484f0 Thu Aug 11 21:42:08 2011
Size: 9b000
CheckSum: a9c8b
Characteristics: 2102
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    5e, 830a0,   830a0 RSDS - GUID: {C14E734A-367F-4DD0-974D-FA47C1194F28}
Age: 1, Pdb: Y:\src\…\ModuleA.pdb
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

0: kd> lmv m ModuleA
start    end        module name
01090000 0112b000   ModuleA   (deferred)
Image path: C:\Program Files\VendorA\ModuleA.dll
Image name: ModuleA.dll
Timestamp:        Thu Aug 11 21:42:08 2011 (4E4484F0)
CheckSum:         000A9C8B
ImageSize:        0009B000
File version:     1.3.0.0
Product version:  1.3.0.0
File flags:       8 (Mask 3F) Private
File OS:          40004 NT Win32
File type:        2.0 Dll
File date:        00000000.00000000
Translations:     0409.04b0
CompanyName:      VendorA
ProductName:      VendorA
InternalName:     ModuleA.dll
OriginalFilename: ModuleA.dll
ProductVersion:   1.3
FileVersion:      1.3.0.0
FileDescription:  ModuleA GUI
LegalCopyright:   Copyright VendorA

So we switch to that thread (with the new process context) to get the needed symbol path:

0: kd> .thread /r /p 89703020
Implicit thread is now 89703020
Implicit process is now 89825020
Loading User Symbols

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
b8795cb8 80503836 nt!KiSwapContext+0x2f
b8795cc4 804fb068 nt!KiSwapThread+0x8a
b8795cec 805c0750 nt!KeWaitForSingleObject+0x1c2
b8795d50 8054161c nt!NtWaitForSingleObject+0x9a
b8795d50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006fa1c 7c90df3c ntdll!KiFastSystemCallRet
0006fa20 7c8025db ntdll!NtWaitForSingleObject+0xc
0006fa84 010ae96a kernel32!WaitForSingleObjectEx+0xa8
0006fafc 010aeaaf ModuleA!Wait+0xaa
0006fb38 010b84ce ModuleA!Read+0×6f
[…]

0: kd> lmv m ModuleA
start    end        module name
01090000 0112b000   ModuleA   (private pdb symbols)  c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281\ModuleA.pdb
Loaded symbol image file: ModuleA.dll
[…]

Now we switch back to our problem stack trace and set the found symbol path explicitly:

0: kd> .thread /r /p 8a17c6d8
Implicit thread is now 8a17c6d8
Implicit process is now 8a17cda0
Loading User Symbols

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0x2f
ba38fcc4 804fb068 nt!KiSwapThread+0x8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0x1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0x9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
*** ERROR: Module load completed but symbols could not be loaded for ModuleA.dll
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0x46
WARNING: Stack unwind information not available. Following frames may be wrong.
0006f7a4 0132b785 ModuleA+0×53df7
0006f7cc 0132c728 ModuleA+0xb785
0006f7e4 01346426 ModuleA+0xc728
0006f848 7e418734 ModuleA+0×26426
0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf
[…]

0: kd> .sympath+ c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281
Symbol search path is: SRV*c:\mss*http://msdl.microsoft.com/download/symbols; c:\sym\ModuleA.pdb\C14E734A367F4DD0974DFA47C1194F281
[…]

0: kd> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
ba38fcb8 80503836 nt!KiSwapContext+0x2f
ba38fcc4 804fb068 nt!KiSwapThread+0x8a
ba38fcec 805c0750 nt!KeWaitForSingleObject+0x1c2
ba38fd50 8054161c nt!NtWaitForSingleObject+0x9a
ba38fd50 7c90e4f4 nt!KiFastCallEntry+0xfc
0006f648 7c90df3c ntdll!KiFastSystemCallRet
0006f64c 7c91b22b ntdll!NtWaitForSingleObject+0xc
0006f6d4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0006f6dc 01373df7 ntdll!RtlEnterCriticalSection+0x46
0006f6e4 0132b22e ModuleA!CSLock+0×7
0006f7a4 0132b785 ModuleA!SignalEvent+0×5e
[…]
0006f848 7e418734 ModuleA!WindowProc+0×136
0006f874 7e418816 USER32!InternalCallWinProc+0×28
0006f8dc 7e4189cd USER32!UserCallWinProcCheckWow+0×150
0006f93c 7e418a10 USER32!DispatchMessageWorker+0×306
0006f94c 0084367e USER32!DispatchMessageW+0xf
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When looking at network packet pools using NDIS WinDbg extension we might see increased number of allocated blocks possibly correlated with network problems, for example:

0: kd> !ndiskd.pktpools * normal
Pool     Allocator BlocksAllocated BlockSize PktsPerBlock PacketLength
8a467e20 b9090f96  0×1             0×1000    0×14         0xc8   tcpip!ARPRegister+119
8a491460 ba4eea56  0×1             0×1000    0×14         0xc8   wanarp!WanpInitializeNdis+a8
8a466508 b905d368  0×1             0×1000    0xd          0×138   tcpip!InitForwardingPools+53
8a373578 b905becb  0×3             0×1000    0×11         0xe8   tcpip!AllocIPPacketList+59
8a466580 b9095ac5  0×1             0×1000    0xe          0×118   tcpip!IPInit+e0
8a460958 bac40a97  0xb             0×1000    0×14         0xc8   vmxnet+a97

0: kd> !ndiskd.pktpools * no sent packets
Pool     Allocator BlocksAllocated BlockSize PktsPerBlock PacketLength
8a467e20 b9090f96  0×1             0×1000    0×14         0xc8 tcpip!ARPRegister+119
8a491460 ba4eea56  0×1             0×1000    0×14         0xc8 wanarp!WanpInitializeNdis+a8
8a466508 b905d368  0×1             0×1000    0xd          0×138 tcpip!InitForwardingPools+53
8a373578 b905becb  0xa3            0×1000    0×11         0xe8 tcpip!AllocIPPacketList+59
8a466580 b9095ac5  0×1             0×1000    0xe          0×118 tcpip!IPInit+e0
8a460958 bac40a97  0×9b            0×1000    0×14         0xc8 vmxnet+a97

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we need to check network adapters (miniports) to see whether they are up, down, connected or disconnected. This can be done using ndiskd WinDbg extension and its commands. For example (a kernel memory dump):

1: kd> !ndiskd.miniports
raspptp.sys, v0.0
  88453360 NetLuidIndex  1, IfIndex  3,  WAN Miniport (PPTP)
raspppoe.sys, v0.0
  884860e8 NetLuidIndex  0, IfIndex  4,  WAN Miniport (PPPOE)
ndiswan.sys, v0.0
  8842f0e8 NetLuidIndex  0, IfIndex  5,  WAN Miniport (IPv6)
  8842e0e8 NetLuidIndex  3, IfIndex  6,  WAN Miniport (IP)
rasl2tp.sys, v0.0
  8842b0e8 NetLuidIndex  0, IfIndex  2,  WAN Miniport (L2TP)
E1G60I32.sys, v8.1
  84b730e8 NetLuidIndex  4, IfIndex  8,  Intel(R) PRO/1000 MT Network Connection
tunnel.sys, v1.0
  84b370e8 NetLuidIndex  2, IfIndex  9,  isatap.{0DC6D9AD-70DC-41CE-9798-F71D1A8C899F}

1: kd> !ndiskd.miniport 84b730e8

MINIPORT

    Intel(R) PRO/1000 MT Network Connection

    Ndis Handle        84b730e8
    Ndis API Version   v6.0
    Adapter Context    88460008
    Miniport Driver    84b44938 - E1G60I32.sys  v8.1
    Ndis Verifier      [No flags set]

    Media Type         802.3
    Physical Medium    802.3
    Device Path        \??\PCI#VEN_8086&DEV_100F&SUBSYS_075015AD&REV_01#4&b70f118&0&0888#{ad498944-762f-11d0-8dcb-00c04fc3358c}\{0DC6D9AD-70DC-41CE-9798-F71D1A8C899F}
    Device Object      84b73030
    MAC Address        00-0c-29-b1-7d-39

STATE

    Miniport           Running
    Device PnP         Started
    Datapath           00000002          ← DIVERTED_BECAUSE_MEDIA_DISCONNECTED
    NBL Status         NDIS_STATUS_MEDIA_DISCONNECTED
    Operational status DOWN
    Operational flags  00000002          ← DOWN_NOT_CONNECTED
    Admin status       ADMIN_UP
    Media              MediaDisconnected
    Power              D0
    References         6
    User Handles       0
    Total Resets       0
    Pending OID        None
    Flags              0c452218
        ↑ BUS_MASTER, 64BIT_DMA, SG_DMA, DEFAULT_PORT_ACTIVATED,
        SUPPORTS_MEDIA_SENSE, DOES_NOT_DO_LOOPBACK, NOT_MEDIA_CONNECTED
    PnPFlags           00210021
        ↑ PM_SUPPORTED, DEVICE_POWER_ENABLED, RECEIVED_START, HARDWARE_DEVICE

BINDINGS

    Filter List        Filter              Filter Driver      Context          _
    QoS Packet Scheduler-0000
                       88e453d8            88e18938           88e1ed60

    Open List          Open                Protocol           Context          _
    RSPNDR             8bcbb470            8bd23ac8           8bcbb820
    LLTDIO             8bcb8c00            8bd15980           8bd153f8
    TCPIP6             88e528e8            88e02350           88e52c98
    TCPIP              88e1c078            88e02aa8           88e1e6a8

MORE INFORMATION

     → Driver handlers                      → Task offloads
     → Power management
     → Pending OIDs                         → Timers
                                            → Receive Side Throttling
     → Wake-on-LAN (WoL)                    → Packet filter
     → NDIS ports

Another example from a different complete memory dump: 

STATE

    Device PnP         Started
    Datapath           00000002          ← DIVERTED_BECAUSE_MEDIA_DISCONNECTED
    Packet Status      NDIS_STATUS_NO_CABLE
    Media              Not Connected
  […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

150 bugtations so far…

Program history has two sides, a computational and a human.

Philip Schaff

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Making Software A Better World.

Dmitry Vostokov, Memory Dump Analysis Services

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes Problem Module pattern can help in troubleshooting. Problem modules (including process names) are components that due to their value adding behaviour might break normal software behaviour and therefore require some troubleshooting workarounds from minor configuration changes to complete removal. Typical examples include memory optimization services for terminal services environments or hooksware. Typically you can see main process modules in the output of !vm or !process 0 0 commands. lm command will list module names such as DLLs from a process memory dump, lmk command can give you the list of kernel space modules (for example, drivers) from kernel and complete memory dumps, and the following command lists all user space modules for each process in a complete memory dump:

!for_each_process ".process /r /p @#Process; lmu"

Of course you can also try various lm command variants if you are interested in timestamps and module information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another stack trace related pattern that we call Empty Stack Trace. Here we might need to do manual stack trace reconstruction like in the following example:

0:002> ~2s
eax=00000070 ebx=0110fb94 ecx=00000010 edx=005725d8 esi=0110fe58 edi=00000d80
eip=7c82847c esp=0110efe0 ebp=0110eff0 iopl=0  nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000      efl=00000246
ntdll!KiFastSystemCallRet:
7c82847c c3              ret

0:002> kL
ChildEBP RetAddr
0110efdc 00000000 ntdll!KiFastSystemCallRet

0:002> !teb
TEB at 7ffdc000
ExceptionList:        0110f980
StackBase:            01110000
StackLimit:           0110d000
SubSystemTib:         00000000
FiberData:            00001e00
ArbitraryUserPointer: 00000000
Self:                 7ffdc000
EnvironmentPointer:   00000000
ClientId:             00000b04 . 00000bd0
RpcHandle:            00000000
Tls Storage:          00000000
PEB Address:          7ffda000
LastErrorValue:       87
LastStatusValue:      c000000d
Count Owned Locks:    0
HardErrorMode:        0

0:002> dps 0110d000 01110000
0110d000  00000000
0110d004  00000000
[...]
0110f63c  00001000
0110f640  0110f64c
0110f644  02b91ea8
0110f648  00001000
0110f64c  00000004
0110f650  0110f6f0
0110f654  0374669d DbgHelp!WriteFullMemory+0×3cd
0110f658  ffffffff
0110f65c  0110d000
0110f660  00000000
0110f664  0480f5c0
0110f668  00003000
0110f66c  0110f7b0
0110f670  0110d000
0110f674  00000000
0110f678  00000065
0110f67c  00003000
0110f680  0110d000
0110f684  00000000
0110f688  01010000
0110f68c  00000000
0110f690  00000004
0110f694  00060002
0110f698  00003000
0110f69c  00000000
0110f6a0  00001000
0110f6a4  00000004
0110f6a8  00020000
0110f6ac  00040004
0110f6b0  7ffe0000 SharedUserData
0110f6b4  00000000
0110f6b8  00001000
0110f6bc  00000000
0110f6c0  0480f5c0
0110f6c4  00000000
0110f6c8  04c4a000
0110f6cc  00000000
0110f6d0  000003c7
0110f6d4  00000000
0110f6d8  00023b17
0110f6dc  00000000
0110f6e0  01110000
0110f6e4  00000000
0110f6e8  0099f000
0110f6ec  00000000
0110f6f0  0110f704
0110f6f4  037469d6 DbgHelp!WriteDumpData+0×206
0110f6f8  0110f738
0110f6fc  0110f7b0
0110f700  00000000
0110f704  0110f868
0110f708  03747449 DbgHelp!MiniDumpProvideDump+0×359
0110f70c  0110f738
0110f710  0110f7b0
0110f714  02b91fb0
0110f718  00000000
0110f71c  00000000
0110f720  00000000
0110f724  02b91fb0
0110f728  00000000
0110f72c  00000000
[…]
0110ff1c  00000001
0110ff20  00000008
0110ff24  0000000a
0110ff28  33017f51 ModuleA!Run+0xde
0110ff2c  00000001
0110ff30  0110ff74
0110ff34  00f08898
0110ff38  00000000
0110ff3c  00f082a8
0110ff40  00000000
0110ff44  00000001
0110ff48  33017e33 ModuleA!ThreadProc+0×2c
0110ff4c  a9b21e1e
0110ff50  00000000
0110ff54  00000000
0110ff58  00f08898
0110ff5c  0110ff4c
0110ff60  0110ffac
0110ff64  0110ff9c
0110ff68  33054245
0110ff6c  9ba52ad2
0110ff70  00000000
0110ff74  0110ffac
0110ff78  78543433 msvcr90!_endthreadex+0×44
0110ff7c  00f082a8
0110ff80  a9b2b0d3
0110ff84  00000000
0110ff88  00000000
0110ff8c  00f08898
0110ff90  0110ff80
0110ff94  0110ff80
0110ff98  0110ffdc
0110ff9c  0110ffdc
0110ffa0  7858cf5e msvcr90!_except_handler4
0110ffa4  d0f887df
0110ffa8  00000000
0110ffac  0110ffb8
0110ffb0  785434c7 msvcr90!_endthreadex+0xd8
0110ffb4  00000000
0110ffb8  0110ffec
0110ffbc  77e6482f kernel32!BaseThreadStart+0×34
0110ffc0  00f08898
0110ffc4  00000000
0110ffc8  00000000
0110ffcc  00f08898
0110ffd0  00000000
0110ffd4  0110ffc4
0110ffd8  80833bcc
0110ffdc  ffffffff
0110ffe0  77e61a60 kernel32!_except_handler3
0110ffe4  77e64838 kernel32!`string’+0×98
0110ffe8  00000000
0110ffec  00000000
0110fff0  00000000
0110fff4  7854345e msvcr90!_endthreadex+0×6f
0110fff8  00f08898
0110fffc  00000000
01110000  00000130

0:002> k L=0110f650 0110f650  0110f650
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0110f650 0374669d 0x110f650
0110f6f0 037469d6 DbgHelp!WriteFullMemory+0x3cd
0110f704 03747449 DbgHelp!WriteDumpData+0x206
0110f868 03747662 DbgHelp!MiniDumpProvideDump+0x359
0110f8dc 33050dd9 DbgHelp!MiniDumpWriteDump+0x1b2
[...]
0110fdfc 33031726 ModuleA!WriteExceptionMiniDump+0x50
0110fea0 33018c81 ModuleA!ThreadHung+0x6c
[...]
0110ff44 33017e33 ModuleA!Run+0xde
00000000 00000000 ModuleA!ThreadProc+0x2c

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you are impatient with !analyze -v you can always use a replacement command that shows and sets the context for the current exception so you can quickly get to the possible crashing point (signature):

0:000> .ecxr
eax=00000000 ebx=00000001 ecx=00000000 edx=0018fe40 esi=00426310 edi=00000111
eip=0041ff21 esp=0018f81c ebp=0018f850 iopl=0  nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00010246
*** ERROR: Module load completed but symbols could not be loaded for TestWER.exe
TestWER+0x1ff21:
0041ff21 c7050000000000000000 mov dword ptr ds:[0],0  ds:002b:00000000=????????

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0018f850 00403620 TestWER+0x1ff21
0018f860 0040382f TestWER+0x3620
0018f890 00402df6 TestWER+0x382f
0018f8b4 00409ef8 TestWER+0x2df6
0018f904 0040a792 TestWER+0x9ef8
0018f9a0 00406dea TestWER+0xa792
0018f9c0 00409713 TestWER+0x6dea
0018fa28 004097a2 TestWER+0x9713
0018fa48 76f66238 TestWER+0x97a2
0018fa74 76f668ea user32!InternalCallWinProc+0x23
0018faec 76f6cd1a user32!UserCallWinProcCheckWow+0x109
0018fb30 76f6cd81 user32!SendMessageWorker+0x581
0018fb54 74fb4e95 user32!SendMessageW+0x7f
0018fb74 74fb4ef7 comctl32!Button_NotifyParent+0x3d
0018fb90 74fb4d89 comctl32!Button_ReleaseCapture+0x113
0018fbf0 76f66238 comctl32!Button_WndProc+0xa18
0018fc1c 76f668ea user32!InternalCallWinProc+0x23
0018fc94 76f67d31 user32!UserCallWinProcCheckWow+0x109
0018fcf4 76f67dfa user32!DispatchMessageWorker+0x3bc
0018fd04 76f82292 user32!DispatchMessageW+0xf
0018fd30 0040618c user32!IsDialogMessageW+0x5f6
0018fd44 004071e2 TestWER+0x618c
0018fd50 00402dd3 TestWER+0x71e2
0018fd64 00408dc1 TestWER+0x2dd3
0018fd78 00403f35 TestWER+0x8dc1
0018fd90 00404090 TestWER+0x3f35
0018fd9c 00403f80 TestWER+0x4090
0018fda8 004040dd TestWER+0x3f80
0018fde0 00403440 TestWER+0x40dd
0018fe2c 004204ee TestWER+0x3440
0018fee4 0041fdf5 TestWER+0x204ee
0018fef8 0040fc3e TestWER+0x1fdf5
0018ff88 76ce3677 TestWER+0xfc3e
0018ff94 77b89f02 kernel32!BaseThreadInitThunk+0xe
0018ffd4 77b89ed5 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

However, in case of multiple exceptions you still need to do stack trace collection analysis:

0:000> .ecxr
eax=00000030 ebx=7efde000 ecx=750d2dd9 edx=00000000 esi=00000000 edi=00000000
eip=770d280c esp=0037f828 ebp=0037f870 iopl=0  nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b      efl=00000202
KERNELBASE!DebugBreak+0x2:
770d280c cc              int     3

0:000> ~*k 6

.  0  Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0037f1a4 770d0bdd ntdll!NtWaitForMultipleObjects+0x15
0037f240 7529162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0037f288 75291921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0037f2a4 752b9b2d kernel32!WaitForMultipleObjects+0x18
0037f310 752b9bca kernel32!WerpReportFaultInternal+0x186
0037f324 752b98f8 kernel32!WerpReportFault+0×70

1  Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
0080f9ac 770d31bb ntdll!NtDelayExecution+0x15
0080fa14 770d3a8b KERNELBASE!SleepEx+0x65
0080fa24 752d28dd KERNELBASE!Sleep+0xf
0080fa38 752b98f8 kernel32!WerpReportFault+0×3f
0080fa48 752b9875 kernel32!BasepReportFault+0×20
0080fad4 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

2  Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
00abf640 770d31bb ntdll!NtDelayExecution+0x15
00abf6a8 770d3a8b KERNELBASE!SleepEx+0x65
00abf6b8 752d28dd KERNELBASE!Sleep+0xf
00abf6cc 752b98f8 kernel32!WerpReportFault+0×3f
00abf6dc 752b9875 kernel32!BasepReportFault+0×20
00abf768 77b10df7 kernel32!UnhandledExceptionFilter+0×1af

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

More than 4 years passed since I provided a longer structuralist definition. Recently I came to recognize a pattern-driven iterative and incremental nature of memory and software trace analysis and post-construction software problem solving in general and therefore a one sentence definition became necessary:

“Recognition and interpretation of patterns of software behavior”

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -