Archive for September, 2010

Adjoint Threading in Process Monitor

Thursday, September 30th, 2010

Another tool that supports adjoint threading in addition to Citrix CDFAnalyzer (see also Debugging Experts magazine article for pictorial description of this concept) is Process Monitor. We can view adjoint threads having common attributes like TID (ordinary threads), PID, operation (function), process name, etc. by using this right click context menu:

For example, this adjoint thread having RegOpenKey as its ATID (Adjoint Thread ID) where we excluded Path, Result and Detail fields for viewing clarity (together these fields can constitute an analogous Message field in TMF traces):

Time of Day      Process Name PID  TID  Operation
[…]
09:33:25.9545410 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9548650 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9550234 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9551656 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9692456 WFICA32.EXE  3588 3496 RegOpenKey
09:33:25.9761325 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9761912 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9762295 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9984547 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0023831 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0074675 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0087191 Explorer.EXE 1292 1032 RegOpenKey
09:33:26.1618595 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1625697 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1632745 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1633924 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1639209 iexplore.exe 1348 2228 RegOpenKey
[…]

So if someone writes a converter from TMF to PML format…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

An Exposé of the Debugging Industry (Part 0)

Thursday, September 30th, 2010

The title of these blog post series was motivated by a book I enjoyed reading this summer:

The Altenberg 16: An Exposé of the Evolution Industry

Finally, after thinking and keeping silence (this blog post was in draft folder for several months) I plan an interview next month for a start.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Inherit a fortune (Debugging Slang, Part 16)

Thursday, September 30th, 2010

Inherit a fortune - To get a postmortem artifact like a crash dump.

Examples:

- My program died!
- Did you inherit a fortune?
- Oh, yeah!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Front Cover Glitch

Thursday, September 30th, 2010

While browsing architecture books on Amazon I found one with a glitch when you use look inside feature (at the time of this writing):

All this reminds me of fragments I see in naturally visualized computer memory that prompts me to conjecture that most all (if not all) computer glitches stem from memory restructuring (a postmodern term for memory corruption).

The book with search inside glitch: Programs and Manifestoes on 20th-Century Architecture

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Microsoft Silently Introduces Micro Dumps

Thursday, September 30th, 2010

My April fool’s joke about the 5th dump type partially came true. I’ve just noticed the new tab “Silent Process Exit” in gflags.exe on my W2K8 R2 server:

The registry keys corresponding to settings are:

HKEY_LOCAL_MACHINE \ SOFTWARE \ Microsoft \ Windows NT \CurrentVersion \ SilentProcessExit
HKEY_LOCAL_MACHINE \ SOFTWARE \ Microsoft \ Windows NT \ CurrentVersion \ SilentProcessExit \ TestDefaultDebugger64
DumpType (DWORD) 0x88

I continue my investigation and report more later.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Icons for Memory Dump Analysis Patterns (Part 77)

Wednesday, September 29th, 2010

Today we introduce an icon for No Process Dumps pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Reading Notebook: 20-September-10

Tuesday, September 28th, 2010

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

I/O Completion Ports (pp. 592 - 598) - my own architectural investigation from a complete memory dump perspective: http://www.dumpanalysis.org/blog/index.php/2007/11/27/understanding-io-completion-ports/

Lock contention (p. 594) - some patterns: http://www.dumpanalysis.org/blog/index.php/2010/09/21/contention-patterns/

Concurrency value may exceed concurrently limit for I/O CP (p. 595)

KeRemoveQueueEx (p. 596) - see also Passive System Thread pattern: http://www.dumpanalysis.org/blog/index.php/2007/11/20/crash-dump-analysis-patterns-part-31a/

I/O priority queues and strategies for IRP (p. 599) - priority fields in _EPROCESS and _ETHREAD structures from x64 W2K8 R2:

1: kd> dt _EPROCESS
ntdll!_EPROCESS
+0x000 Pcb              : _KPROCESS
[...]
+0x438 DefaultIoPriority : Pos 27, 3 Bits
[...]

1: kd> dt _ETHREAD
ntdll!_ETHREAD
+0x000 Tcb              : _KTHREAD
[...]
+0x448 ThreadIoPriority : Pos 10, 3 Bits
[...]

Driver Verifier (pp. 604 - 606) - see also Instrumentation Information pattern: http://www.dumpanalysis.org/blog/index.php/2010/09/27/crash-dump-analysis-patterns-part-107/ 

WDF book (p. 607) - there is also another book coming soon: http://www.dumpanalysis.org/blog/index.php/2010/08/19/windows-7-device-driver-book/

Listing KMDF drivers (p. 608) - here’s the output from x64 W2K8 R2 system:

1: kd> !wdfkd.wdfldr
LoadedModuleList      0xfffff8800115a2d8
----------------------------------
LIBRARY_MODULE  fffffa8003bc8d10
Version       v1.9 build(7600)
Service       \Registry\Machine\System\CurrentControlSet\Services\Wdf01000
ImageName     Wdf01000.sys
ImageAddress  0xfffff880010ae000
ImageSize     0xa4000
Associated Clients: 10

  ImageName      Version    WdfGlobals         FxGlobals          ImageAddress       ImageSize
peauth.sys     v1.7(6001) 0xfffffa8004bf6510 0xfffffa8004bf63c0 0xfffff88004600000 0x000a6000
monitor.sys    v1.9(7600) 0xfffffa80048f55d0 0xfffffa80048f5480 0xfffff88003752000 0x0000e000
umbus.sys      v1.9(7600) 0xfffffa8004371160 0xfffffa8004371010 0xfffff88002db0000 0x00012000
CompositeBus.sys v1.9(7600) 0xfffffa8004440800 0xfffffa80044406b0 0xfffff88002a45000 0x00010000
HDAudBus.sys   v1.7(6001) 0xfffffa80043c9160 0xfffffa80043c9010 0xfffff88002b48000 0x00024000
intelppm.sys   v1.9(7600) 0xfffffa8004271dd0 0xfffffa8004271c80 0xfffff88002ab0000 0x00016000
cdrom.sys      v1.9(7600) 0xfffffa80041f3fc0 0xfffffa80041f3e70 0xfffff88001400000 0x0002a000
vmstorfl.sys   v1.5(6000) 0xfffffa80040129e0 0xfffffa8004012890 0xfffff88001750000 0x00010000
msisadrv.sys   v1.9(7600) 0xfffffa8003ebb910 0xfffffa8003ebb7c0 0xfffff880012c6000 0x0000a000
vdrvroot.sys   v1.9(7600) 0xfffffa8003d3fa00 0xfffffa8003d3f8b0 0xfffff88001262000 0x0000d000
----------------------------------
Total: 1 library loaded

Extension of device extension extension into object context in KMDF (pp. 611 - 612)

UMDF reflectors (p. 617)

WUDFHost.exe (p. 618) - here’s its stack trace collection from x64 W2K8 R2 after I inserted an USB flash drive and attached WinDbg non-invasilvely:

0:000> ~*k

.  0  Id: 58c.12f4 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0018f988 000007fe`fd8510ac ntdll!ZwWaitForSingleObject+0xa
00000000`0018f990 00000000`ff3bba44 KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`0018fa30 00000000`ff3b8ce7 WUDFHost!CLpcNotification::Run+0x1c
00000000`0018fa60 00000000`ff3d2cb1 WUDFHost!wmain+0xc7b
00000000`0018fc60 00000000`7746f56d WUDFHost!ConvertStringSidToSidW+0x19b
00000000`0018fca0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0018fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   1  Id: 58c.1304 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00c4f918 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00c4f920 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00c4f990 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00c4f9e0 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00c4fa70 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00c4fc20 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00c4fc70 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00c4fca0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00c4fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   2  Id: 58c.6e8 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00dfe988 000007fe`fd853ef8 ntdll!NtQueryAttributesFile+0xa
00000000`00dfe990 000007fe`f3be9970 KERNELBASE!GetFileAttributesW+0x78
00000000`00dfea30 000007fe`f27ce8c9 WpdFs!COperationGetFastBasicProperties::OnImpersonate+0x1c0
00000000`00dfea70 000007fe`f3be9734 WUDFx!CWdfIoRequest::Impersonate+0x151
00000000`00dfeae0 000007fe`f3bda26b WpdFs!COperationGetFastBasicProperties::Invoke+0x2c4
00000000`00dfeb50 000007fe`f3bd8837 WpdFs!WpdObjectProperties::GetValues+0x3f7
00000000`00dfecd0 000007fe`f3bd8344 WpdFs!WpdObjectProperties::OnGetValues+0x10b
00000000`00dfed50 000007fe`f3bcf974 WpdFs!WpdObjectProperties::DispatchWpdMessage+0x1a0
00000000`00dfee10 000007fe`f3bcd51a WpdFs!WpdBaseDriver::DispatchWpdMessage+0x4c0
00000000`00dfef60 000007fe`f3bcdd6c WpdFs!CQueue::ProcessWpdMessage+0x29a
00000000`00dff010 000007fe`f27bf610 WpdFs!CQueue::OnDeviceIoControl+0x494
00000000`00dff160 000007fe`f27c0b5a WUDFx!CWdfIoQueue::SubmitRequest+0x358
00000000`00dff1f0 000007fe`f27c0955 WUDFx!CWdfIoQueue::DispatchRequestToDriver+0x86
00000000`00dff240 000007fe`f27bff83 WUDFx!CWdfIoQueue::DispatchEvents+0x3cd
00000000`00dff2b0 000007fe`f27b61b5 WUDFx!CWdfIoQueue::QueueRequest+0x2c3
00000000`00dff300 000007fe`f27b6f20 WUDFx!CWdfDevice::DispatchRequest+0x149
00000000`00dff350 00000000`ff3ccbb6 WUDFx!CWdfDevice::DeviceControl+0x1a8
00000000`00dff3c0 00000000`ff3c2f92 WUDFHost!CWudfIoIrp::Dispatch+0x13e
00000000`00dff420 00000000`ff3bad47 WUDFHost!CWudfDeviceStack::Forward+0x41a
00000000`00dff490 000007fe`fb87da6a WUDFHost!CLpcNotification::Message+0xd9b
00000000`00dff6c0 000007fe`fb87c848 WUDFPlatform!WdfLpcPort::ProcessMessage+0x3be
00000000`00dff760 000007fe`fb87b299 WUDFPlatform!WdfLpcCommPort::ProcessMessage+0x214
00000000`00dff7b0 000007fe`fb87b900 WUDFPlatform!WdfLpcConnPort::ProcessMessage+0xf9
00000000`00dff830 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x178
00000000`00dff880 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00dff8b0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00dff8e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   3  Id: 58c.2e4 Suspend: 1 Teb: 000007ff`fffd8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00d7f5e8 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00d7f5f0 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00d7f660 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00d7f6b0 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00d7f740 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00d7f8f0 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00d7f940 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00d7f970 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00d7f9a0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   4  Id: 58c.12b4 Suspend: 1 Teb: 000007ff`fffd6000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00f8fa58 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00f8fa60 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00f8fad0 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00f8fb20 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00f8fbb0 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00f8fd60 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00f8fdb0 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00f8fde0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00f8fe10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   5  Id: 58c.106c Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00f0f958 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00f0f960 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00f0f9d0 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00f0fa20 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00f0fab0 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00f0fc60 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00f0fcb0 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00f0fce0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00f0fd10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   6  Id: 58c.8fc Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0136f8c8 00000000`7758c95e USER32!NtUserGetMessage+0xa
00000000`0136f8d0 000007fe`f3bd26e5 USER32!GetMessageW+0x34
00000000`0136f900 00000000`7746f56d WpdFs!CDiskNotifier::NotificationThreadWorker+0x245
00000000`0136fa50 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0136fa80 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   7  Id: 58c.520 Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0152f6f8 00000000`77689bd7 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0152f700 00000000`7746f56d ntdll!EtwTraceMessageVa+0xe07
00000000`0152f9a0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0152f9d0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   8  Id: 58c.89c Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`012df9b8 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`012df9c0 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`012dfcc0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`012dfcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   9  Id: 58c.1394 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0140f498 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0140f4a0 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0140f7a0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0140f7d0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  10  Id: 58c.1294 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0182f758 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0182f760 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0182fa60 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0182fa90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  11  Id: 58c.a98 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0170f708 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0170f710 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0170fa10 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0170fa40 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  12  Id: 58c.121c Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0179fd68 000007fe`fd851203 ntdll!NtDelayExecution+0xa
00000000`0179fd70 000007fe`fe2cea00 KERNELBASE!SleepEx+0xb3
00000000`0179fe10 000007fe`fe2d2046 ole32!CROIDTable::WorkerThreadLoop+0x10
00000000`0179fe40 000007fe`fe2d358a ole32!CRpcThread::WorkerLoop+0x1e
00000000`0179fe80 00000000`7746f56d ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a
00000000`0179feb0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0179fee0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Reading Notebook: 20-September-10

Tuesday, September 28th, 2010

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

I/O Completion Ports (pp. 592 - 598) - my own architectural investigation from a complete memory dump perspective: http://www.dumpanalysis.org/blog/index.php/2007/11/27/understanding-io-completion-ports/

Lock contention (p. 594) - some patterns: http://www.dumpanalysis.org/blog/index.php/2010/09/21/contention-patterns/

Concurrency value may exceed concurrently limit for I/O CP (p. 595)

KeRemoveQueueEx (p. 596) - see also Passive System Thread pattern: http://www.dumpanalysis.org/blog/index.php/2007/11/20/crash-dump-analysis-patterns-part-31a/

I/O priority queues and strategies for IRP (p. 599) - priority fields in _EPROCESS and _ETHREAD structures from x64 W2K8 R2:

1: kd> dt _EPROCESS
ntdll!_EPROCESS
+0x000 Pcb              : _KPROCESS
[...]
+0x438 DefaultIoPriority : Pos 27, 3 Bits
[...]

1: kd> dt _ETHREAD
ntdll!_ETHREAD
+0x000 Tcb              : _KTHREAD
[...]
+0x448 ThreadIoPriority : Pos 10, 3 Bits
[...]

Driver Verifier (pp. 604 - 606) - see also Instrumentation Information pattern: http://www.dumpanalysis.org/blog/index.php/2010/09/27/crash-dump-analysis-patterns-part-107/ 

WDF book (p. 607) - there is also another book coming soon: http://www.dumpanalysis.org/blog/index.php/2010/08/19/windows-7-device-driver-book/

Listing KMDF drivers (p. 608) - here’s the output from x64 W2K8 R2 system:

1: kd> !wdfkd.wdfldr
LoadedModuleList      0xfffff8800115a2d8
----------------------------------
LIBRARY_MODULE  fffffa8003bc8d10
Version       v1.9 build(7600)
Service       \Registry\Machine\System\CurrentControlSet\Services\Wdf01000
ImageName     Wdf01000.sys
ImageAddress  0xfffff880010ae000
ImageSize     0xa4000
Associated Clients: 10

  ImageName      Version    WdfGlobals         FxGlobals          ImageAddress       ImageSize
peauth.sys     v1.7(6001) 0xfffffa8004bf6510 0xfffffa8004bf63c0 0xfffff88004600000 0x000a6000
monitor.sys    v1.9(7600) 0xfffffa80048f55d0 0xfffffa80048f5480 0xfffff88003752000 0x0000e000
umbus.sys      v1.9(7600) 0xfffffa8004371160 0xfffffa8004371010 0xfffff88002db0000 0x00012000
CompositeBus.sys v1.9(7600) 0xfffffa8004440800 0xfffffa80044406b0 0xfffff88002a45000 0x00010000
HDAudBus.sys   v1.7(6001) 0xfffffa80043c9160 0xfffffa80043c9010 0xfffff88002b48000 0x00024000
intelppm.sys   v1.9(7600) 0xfffffa8004271dd0 0xfffffa8004271c80 0xfffff88002ab0000 0x00016000
cdrom.sys      v1.9(7600) 0xfffffa80041f3fc0 0xfffffa80041f3e70 0xfffff88001400000 0x0002a000
vmstorfl.sys   v1.5(6000) 0xfffffa80040129e0 0xfffffa8004012890 0xfffff88001750000 0x00010000
msisadrv.sys   v1.9(7600) 0xfffffa8003ebb910 0xfffffa8003ebb7c0 0xfffff880012c6000 0x0000a000
vdrvroot.sys   v1.9(7600) 0xfffffa8003d3fa00 0xfffffa8003d3f8b0 0xfffff88001262000 0x0000d000
----------------------------------
Total: 1 library loaded

Extension of device extension extension into object context in KMDF (pp. 611 - 612)

UMDF reflectors (p. 617)

WUDFHost.exe (p. 618) - here’s its stack trace collection from x64 W2K8 R2 after I inserted an USB flash drive and attached WinDbg non-invasilvely:

0:000> ~*k

.  0  Id: 58c.12f4 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0018f988 000007fe`fd8510ac ntdll!ZwWaitForSingleObject+0xa
00000000`0018f990 00000000`ff3bba44 KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`0018fa30 00000000`ff3b8ce7 WUDFHost!CLpcNotification::Run+0x1c
00000000`0018fa60 00000000`ff3d2cb1 WUDFHost!wmain+0xc7b
00000000`0018fc60 00000000`7746f56d WUDFHost!ConvertStringSidToSidW+0x19b
00000000`0018fca0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0018fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   1  Id: 58c.1304 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00c4f918 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00c4f920 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00c4f990 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00c4f9e0 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00c4fa70 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00c4fc20 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00c4fc70 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00c4fca0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00c4fcd0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   2  Id: 58c.6e8 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00dfe988 000007fe`fd853ef8 ntdll!NtQueryAttributesFile+0xa
00000000`00dfe990 000007fe`f3be9970 KERNELBASE!GetFileAttributesW+0x78
00000000`00dfea30 000007fe`f27ce8c9 WpdFs!COperationGetFastBasicProperties::OnImpersonate+0x1c0
00000000`00dfea70 000007fe`f3be9734 WUDFx!CWdfIoRequest::Impersonate+0x151
00000000`00dfeae0 000007fe`f3bda26b WpdFs!COperationGetFastBasicProperties::Invoke+0x2c4
00000000`00dfeb50 000007fe`f3bd8837 WpdFs!WpdObjectProperties::GetValues+0x3f7
00000000`00dfecd0 000007fe`f3bd8344 WpdFs!WpdObjectProperties::OnGetValues+0x10b
00000000`00dfed50 000007fe`f3bcf974 WpdFs!WpdObjectProperties::DispatchWpdMessage+0x1a0
00000000`00dfee10 000007fe`f3bcd51a WpdFs!WpdBaseDriver::DispatchWpdMessage+0x4c0
00000000`00dfef60 000007fe`f3bcdd6c WpdFs!CQueue::ProcessWpdMessage+0x29a
00000000`00dff010 000007fe`f27bf610 WpdFs!CQueue::OnDeviceIoControl+0x494
00000000`00dff160 000007fe`f27c0b5a WUDFx!CWdfIoQueue::SubmitRequest+0x358
00000000`00dff1f0 000007fe`f27c0955 WUDFx!CWdfIoQueue::DispatchRequestToDriver+0x86
00000000`00dff240 000007fe`f27bff83 WUDFx!CWdfIoQueue::DispatchEvents+0x3cd
00000000`00dff2b0 000007fe`f27b61b5 WUDFx!CWdfIoQueue::QueueRequest+0x2c3
00000000`00dff300 000007fe`f27b6f20 WUDFx!CWdfDevice::DispatchRequest+0x149
00000000`00dff350 00000000`ff3ccbb6 WUDFx!CWdfDevice::DeviceControl+0x1a8
00000000`00dff3c0 00000000`ff3c2f92 WUDFHost!CWudfIoIrp::Dispatch+0x13e
00000000`00dff420 00000000`ff3bad47 WUDFHost!CWudfDeviceStack::Forward+0x41a
00000000`00dff490 000007fe`fb87da6a WUDFHost!CLpcNotification::Message+0xd9b
00000000`00dff6c0 000007fe`fb87c848 WUDFPlatform!WdfLpcPort::ProcessMessage+0x3be
00000000`00dff760 000007fe`fb87b299 WUDFPlatform!WdfLpcCommPort::ProcessMessage+0x214
00000000`00dff7b0 000007fe`fb87b900 WUDFPlatform!WdfLpcConnPort::ProcessMessage+0xf9
00000000`00dff830 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x178
00000000`00dff880 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00dff8b0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00dff8e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   3  Id: 58c.2e4 Suspend: 1 Teb: 000007ff`fffd8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00d7f5e8 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00d7f5f0 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00d7f660 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00d7f6b0 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00d7f740 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00d7f8f0 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00d7f940 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00d7f970 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00d7f9a0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   4  Id: 58c.12b4 Suspend: 1 Teb: 000007ff`fffd6000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00f8fa58 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00f8fa60 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00f8fad0 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00f8fb20 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00f8fbb0 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00f8fd60 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00f8fdb0 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00f8fde0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00f8fe10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   5  Id: 58c.106c Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00f0f958 000007fe`fd8753d6 ntdll!NtDeviceIoControlFile+0xa
00000000`00f0f960 00000000`7746610f KERNELBASE!WaitNamedPipeW+0x16c6
00000000`00f0f9d0 000007fe`fb87dd94 kernel32!DeviceIoControlImplementation+0x7f
00000000`00f0fa20 000007fe`fb87e6cd WUDFPlatform!WPP_SF_ssd+0x1e4
00000000`00f0fab0 000007fe`fb87b8af WUDFPlatform!WdfLpcCorePortInterface::GetMessageW+0x119
00000000`00f0fc60 00000000`ff3bd7de WUDFPlatform!WdfWorkerThread::WorkerThread+0x127
00000000`00f0fcb0 00000000`7746f56d WUDFHost!LpcWorkerThreadThunk+0x62
00000000`00f0fce0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`00f0fd10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   6  Id: 58c.8fc Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0136f8c8 00000000`7758c95e USER32!NtUserGetMessage+0xa
00000000`0136f8d0 000007fe`f3bd26e5 USER32!GetMessageW+0x34
00000000`0136f900 00000000`7746f56d WpdFs!CDiskNotifier::NotificationThreadWorker+0x245
00000000`0136fa50 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0136fa80 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   7  Id: 58c.520 Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0152f6f8 00000000`77689bd7 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0152f700 00000000`7746f56d ntdll!EtwTraceMessageVa+0xe07
00000000`0152f9a0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0152f9d0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   8  Id: 58c.89c Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`012df9b8 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`012df9c0 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`012dfcc0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`012dfcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

   9  Id: 58c.1394 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0140f498 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0140f4a0 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0140f7a0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0140f7d0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  10  Id: 58c.1294 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0182f758 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0182f760 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0182fa60 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0182fa90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  11  Id: 58c.a98 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0170f708 00000000`7768914b ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0170f710 00000000`7746f56d ntdll!EtwTraceMessageVa+0x37b
00000000`0170fa10 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0170fa40 00000000`00000000 ntdll!RtlUserThreadStart+0x21

  12  Id: 58c.121c Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0179fd68 000007fe`fd851203 ntdll!NtDelayExecution+0xa
00000000`0179fd70 000007fe`fe2cea00 KERNELBASE!SleepEx+0xb3
00000000`0179fe10 000007fe`fe2d2046 ole32!CROIDTable::WorkerThreadLoop+0x10
00000000`0179fe40 000007fe`fe2d358a ole32!CRpcThread::WorkerLoop+0x1e
00000000`0179fe80 00000000`7746f56d ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a
00000000`0179feb0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0179fee0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Icons for Memory Dump Analysis Patterns (Part 76)

Tuesday, September 28th, 2010

Today we introduce an icon for Dispatch Level Spin pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Bugtation No.125

Tuesday, September 28th, 2010

Who’s your BOSS (Basic Operating Support System)?

I report to Memory……………………………………………………….

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 107)

Monday, September 27th, 2010

You are probably aware of Application and Driver Verifiers (including gflags.exe tool from Debugging Tools for Windows). These tools set flags that modify the behaviour of the system that is reflected in additional information being collected such as memory allocation history and in WinDbg output changes such as stack traces. These tools belong to a broad class of instrumentation tools and I call the analysis pattern Instrumentation Information. To check in a minidump, kernel and complete memory dumps whether Driver Verifier was enabled we use !verifier WinDbg command:

1: kd> !verifier

Verify Level 0 ... enabled options are:

Summary of All Verifier Statistics

RaiseIrqls                             0x0
AcquireSpinLocks                       0x0
Synch Executions                       0x0
Trims                                  0x0

Pool Allocations Attempted             0x0
Pool Allocations Succeeded             0x0
Pool Allocations Succeeded SpecialPool 0x0
Pool Allocations With NO TAG           0x0
Pool Allocations Failed                0x0
Resource Allocations Failed Deliberately   0x0

Current paged pool allocations         0x0 for 00000000 bytes
Peak paged pool allocations            0x0 for 00000000 bytes
Current nonpaged pool allocations      0x0 for 00000000 bytes
Peak nonpaged pool allocations         0x0 for 00000000 bytes

0: kd> !verifier

Verify Level 3 ... enabled options are:
       Special pool
       Special irql

Summary of All Verifier Statistics

RaiseIrqls                             0xdea5
AcquireSpinLocks                       0x87b5c
Synch Executions                       0x17b5
Trims                                  0xab36

Pool Allocations Attempted             0x8990e
Pool Allocations Succeeded             0x8990e
Pool Allocations Succeeded SpecialPool 0x29c0
Pool Allocations With NO TAG           0x1
Pool Allocations Failed                0x0
Resource Allocations Failed Deliberately   0x0

Current paged pool allocations         0x0 for 00000000 bytes
Peak paged pool allocations            0x0 for 00000000 bytes
Current nonpaged pool allocations      0x0 for 00000000 bytes
Peak nonpaged pool allocations         0x0 for 00000000 bytes
 

To check in a process user dump that Application Verifier (and gflags) was enabled use !avrf and !gflags WinDbg extension commands:

0:001> !avrf
Application verifier is not enabled for this process.
Page heap has been enabled separately.

0:001> !gflag
Current NtGlobalFlag contents: 0x02000000
    hpa - Place heap allocations at ends of pages

Here is an example of an instrumented stack trace:

68546e88 verifier!AVrfpDphFindBusyMemoryNoCheck+0xb8
68546f95 verifier!AVrfpDphFindBusyMemory+0×15
68547240 verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0×20
68549080 verifier!AVrfDebugPageHeapFree+0×90

77190aac ntdll!RtlDebugFreeHeap+0×2f
7714a8ff ntdll!RtlpFreeHeap+0×5d
770f2a32 ntdll!RtlFreeHeap+0×142
75fb14d1 kernel32!HeapFree+0×14
748d4c39 msvcr80!free+0xcd
[…]
00a02bb2 ServiceA!ServiceMain+0×302
767175a8 sechost!ScSvcctrlThreadA+0×21
75fb3677 kernel32!BaseThreadInitThunk+0xe
770f9d42 ntdll!__RtlUserThreadStart+0×70
770f9d15 ntdll!_RtlUserThreadStart+0×1b

Another example that shows instrumentation difference. We run double free fault modeling application and see its stack trace from a crash dump:

0:000> !gflag
Current NtGlobalFlag contents: 0x00000000

0:000> kL 100
Child-SP          RetAddr           Call Site
00000000`002dec38 00000000`77735ce2 ntdll!NtWaitForSingleObject+0xa
00000000`002dec40 00000000`77735e85 ntdll!RtlReportExceptionEx+0x1d2
00000000`002ded30 00000000`77735eea ntdll!RtlReportException+0xb5
00000000`002dedb0 00000000`77736d25 ntdll!RtlpTerminateFailureFilter+0x1a
00000000`002dede0 00000000`77685148 ntdll!RtlReportCriticalFailure+0x96
00000000`002dee10 00000000`776a554d ntdll!_C_specific_handler+0x8c
00000000`002dee80 00000000`77685d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`002deeb0 00000000`776862ee ntdll!RtlDispatchException+0x3cb
00000000`002df590 00000000`77736cd2 ntdll!RtlRaiseException+0x221
00000000`002dfbd0 00000000`77737396 ntdll!RtlReportCriticalFailure+0x62
00000000`002dfca0 00000000`777386c2 ntdll!RtlpReportHeapFailure+0x26
00000000`002dfcd0 00000000`7773a0c4 ntdll!RtlpHeapHandleError+0x12
00000000`002dfd00 00000000`776dd1cd ntdll!RtlpLogHeapFailure+0xa4
00000000`002dfd30 00000000`77472c7a ntdll! ?? ::FNODOBFM::`string'+0x123b4
00000000`002dfdb0 00000000`6243c7bc kernel32!HeapFree+0xa
00000000`002dfde0 00000001`3f8f1033 msvcr90!free+0x1c
00000000`002dfe10 00000001`3f8f11f2 InstrumentedApp!wmain+0x33
00000000`002dfe50 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0x11a
00000000`002dfe80 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`002dfeb0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Then we enable Application Verifier and full page heap in gflags.exe GUI. Actually 2 crash dumps are saved at the same time (we’d set up LocalDumps registry key on x64 W2K8 R2) with slightly different stack traces:

0:000> !gflag
Current NtGlobalFlag contents: 0x02000100
    vrf - Enable application verifier
    hpa - Place heap allocations at ends of pages

0:000> kL 100
Child-SP          RetAddr           Call Site
00000000`0022e438 00000000`77735ce2 ntdll!NtWaitForSingleObject+0xa
00000000`0022e440 00000000`77735e85 ntdll!RtlReportExceptionEx+0x1d2
00000000`0022e530 000007fe`f3ed26fb ntdll!RtlReportException+0xb5
00000000`0022e5b0 00000000`77688a8f verifier!AVrfpVectoredExceptionHandler+0×26b
00000000`0022e640 00000000`776859b2 ntdll!RtlpCallVectoredHandlers+0xa8
00000000`0022e6b0 00000000`776bfe48 ntdll!RtlDispatchException+0×22
00000000`0022ed90 000007fe`f3eca668 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0022f350 000007fe`f3ec931d verifier!VerifierStopMessage+0×1f0
00000000`0022f400 000007fe`f3ec9736 verifier!AVrfpDphReportCorruptedBlock+0×155
00000000`0022f4c0 000007fe`f3ec99cd verifier!AVrfpDphCheckNormalHeapBlock+0xce
00000000`0022f530 000007fe`f3ec873a verifier!AVrfpDphNormalHeapFree+0×29
00000000`0022f560 00000000`7773c415 verifier!AVrfDebugPageHeapFree+0xb6

00000000`0022f5c0 00000000`776dd0fe ntdll!RtlDebugFreeHeap+0×35
00000000`0022f620 00000000`776c2075 ntdll! ?? ::FNODOBFM::`string’+0×122e2
00000000`0022f960 000007fe`f3edf4e1 ntdll!RtlFreeHeap+0×1a2
00000000`0022f9e0 00000000`77472c7a verifier!AVrfpRtlFreeHeap+0xa5
00000000`0022fa80 000007fe`f3ee09ae kernel32!HeapFree+0xa
00000000`0022fab0 00000000`642bc7bc verifier!AVrfpHeapFree+0xc6
00000000`0022fb40 00000001`3fac1033 msvcr90!free+0×1c
00000000`0022fb70 00000001`3fac11f2 InstrumentedApp!wmain+0×33
00000000`0022fbb0 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0×11a
00000000`0022fbe0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0022fc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> kL 100
Child-SP          RetAddr           Call Site
00000000`0022e198 000007fe`f3ee0f82 ntdll!NtWaitForMultipleObjects+0xa
00000000`0022e1a0 000007fe`fd8513a6 verifier!AVrfpNtWaitForMultipleObjects+0×4e
00000000`0022e1e0 000007fe`f3ee0e2d KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0022e2e0 000007fe`f3ee0edd verifier!AVrfpWaitForMultipleObjectsExCommon+0xad
00000000`0022e320 00000000`77473143 verifier!AVrfpKernelbaseWaitForMultipleObjectsEx+0×2d

00000000`0022e370 00000000`774e9025 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0022e400 00000000`774e91a7 kernel32!WerpReportFaultInternal+0×215
00000000`0022e4a0 00000000`774e91ff kernel32!WerpReportFault+0×77
00000000`0022e4d0 00000000`774e941c kernel32!BasepReportFault+0×1f
00000000`0022e500 00000000`7770573c kernel32!UnhandledExceptionFilter+0×1fc
00000000`0022e5e0 00000000`77685148 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`0022e610 00000000`776a554d ntdll!_C_specific_handler+0×8c
00000000`0022e680 00000000`77685d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`0022e6b0 00000000`776bfe48 ntdll!RtlDispatchException+0×3cb
00000000`0022ed90 000007fe`f3eca668 ntdll!KiUserExceptionDispatcher+0×2e
00000000`0022f350 000007fe`f3ec931d verifier!VerifierStopMessage+0×1f0
00000000`0022f400 000007fe`f3ec9736 verifier!AVrfpDphReportCorruptedBlock+0×155
00000000`0022f4c0 000007fe`f3ec99cd verifier!AVrfpDphCheckNormalHeapBlock+0xce
00000000`0022f530 000007fe`f3ec873a verifier!AVrfpDphNormalHeapFree+0×29
00000000`0022f560 00000000`7773c415 verifier!AVrfDebugPageHeapFree+0xb6

00000000`0022f5c0 00000000`776dd0fe ntdll!RtlDebugFreeHeap+0×35
00000000`0022f620 00000000`776c2075 ntdll! ?? ::FNODOBFM::`string’+0×122e2
00000000`0022f960 000007fe`f3edf4e1 ntdll!RtlFreeHeap+0×1a2
00000000`0022f9e0 00000000`77472c7a verifier!AVrfpRtlFreeHeap+0xa5
00000000`0022fa80 000007fe`f3ee09ae kernel32!HeapFree+0xa
00000000`0022fab0 00000000`642bc7bc verifier!AVrfpHeapFree+0xc6
00000000`0022fb40 00000001`3fac1033 msvcr90!free+0×1c
00000000`0022fb70 00000001`3fac11f2 InstrumentedApp!wmain+0×33
00000000`0022fbb0 00000000`7746f56d InstrumentedApp!__tmainCRTStartup+0×11a
00000000`0022fbe0 00000000`776a3281 kernel32!BaseThreadInitThunk+0xd
00000000`0022fc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We also see above that enabling instrumentation triggers debug functions of runtime heap (RtlDebugFreeHeap).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

General Memory Analysis

Monday, September 27th, 2010

General Memory Analysis is another name for Memoretics, a discipline that studies memory snapshots including their similarities and differences on different system platforms such as Windows, Linus, Mac OS X, embedded and mobile systems, historical architectures, etc. The analysis of memory helps solve problems in various domains such as software troubleshooting and debugging, computer forensic analysis, etc.

The current focus of interdisciplinary research is to build a unified memory pattern language that covers both behavioral and structural patterns and also to study the possibility of building memory systems from below, not from requirements -> architecture -> design -> implementation -> compilation -> linking -> loading -> execution but from directly modeling and assembling memory systems using memory patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Structural Memory Patterns (Part 3)

Sunday, September 26th, 2010

The next pattern is called Snapshot Collection. This is a collection of files combined from either linear memory snapshots or aggregate snapshots saved as separate files at different times. Typical examples include:

  • Several process memory dump files saved sequentially from a growing heap leaking process
  • Several software traces from working and non-working scenarios for comparative analysis

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Structural Memory Patterns (Part 2)

Sunday, September 26th, 2010

The next pattern is called Aggregate Snapshot. It is any memory dump or software trace file that is combined from Memory Snapshots. Typical examples include:

  • A minidump file where only specific memory ranges are included
  • A software trace file combined from structured memory snapshots

Note. I’ve also created a dedicated page to list all structural patterns: 

Structural Memory Analysis Patterns

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dumps and Password Exposure

Sunday, September 26th, 2010

Sending process crash dumps can lead to the exposure of passwords and other sensitive information especially if they were saved before a process was trying to send entered user data over a secure protocol. Here’s an incident that happened to me this morning. I was trying to login to an online banking system to check my balances and when I entered my user id and password in IE and clicked Continue button the system experienced a small delay and then a WER dialog box appeared asking me to either check online for a solution, debug or close the program. I chose Close the program and a full process memory dump was saved because I have already set up LocalDumps on my old Vista system (the problem was also reproducible).

I opened the crash dump and found that it was a heap corruption:

0:004> kL 100
ChildEBP RetAddr 
02c9cb18 77815620 ntdll!KiFastSystemCallRet
02c9cb1c 77843c62 ntdll!NtWaitForSingleObject+0xc
02c9cba0 77843d4b ntdll!RtlReportExceptionEx+0x14b
02c9cbe0 7785fa87 ntdll!RtlReportException+0x3c
02c9cbf4 7785fb0d ntdll!RtlpTerminateFailureFilter+0x14
02c9cc00 777b9bdc ntdll!RtlReportCriticalFailure+0x6b
02c9cc14 777b4067 ntdll!_EH4_CallFilterFunc+0x12
02c9cc3c 77815f79 ntdll!_except_handler4+0x8e
02c9cc60 77815f4b ntdll!ExecuteHandler2+0x26
02c9cd10 77815dd7 ntdll!ExecuteHandler+0x24
02c9cd10 7785faf8 ntdll!KiUserExceptionDispatcher+0xf
02c9d084 77860704 ntdll!RtlReportCriticalFailure+0x5b
02c9d094 778607f2 ntdll!RtlpReportHeapFailure+0×21
02c9d0c8 7782b1a5 ntdll!RtlpLogHeapFailure+0xa1
02c9d110 7781730a ntdll!RtlpCoalesceFreeBlocks+0×4b9
02c9d208 77817545 ntdll!RtlpFreeHeap+0×1e2
02c9d224 76277e4b ntdll!RtlFreeHeap+0×14e
02c9d26c 760f7277 kernel32!GlobalFree+0×47

02c9d280 76594a1f ole32!ReleaseStgMedium+0×124
02c9d294 765f7feb urlmon!ReleaseBindInfo+0×4c
02c9d2a4 765b9a87 urlmon!CINet::ReleaseCNetObjects+0×3d
02c9d2bc 765b93f0 urlmon!CINetHttp::OnWininetRequestHandleClosing+0×60
02c9d2d0 77582078 urlmon!CINet::CINetCallback+0×2de
02c9d418 77588f5d wininet!InternetIndicateStatus+0xfc
02c9d448 7758937a wininet!HANDLE_OBJECT::~HANDLE_OBJECT+0xc9
02c9d464 7758916b wininet!INTERNET_CONNECT_HANDLE_OBJECT::~INTERNET_CONNECT_HANDLE_OBJECT+0×209
02c9d470 77588d5e wininet!HTTP_REQUEST_HANDLE_OBJECT::`vector deleting destructor’+0xd
02c9d480 77584e72 wininet!HANDLE_OBJECT::Dereference+0×22
02c9d48c 77589419 wininet!DereferenceObject+0×21
02c9d4b4 77589114 wininet!_InternetCloseHandle+0×9d
02c9d4d4 0004aaaf wininet!InternetCloseHandle+0×11e
WARNING: Frame IP not in any known module. Following frames may be wrong.
02c9d4e0 765a5d25 0×4aaaf
02c9d4fc 765a5c1b urlmon!CINet::TerminateRequest+0×82
02c9d50c 765a5a3c urlmon!CINet::MyTerminate+0×7b
02c9d51c 765a5998 urlmon!CINetProtImpl::Terminate+0×13
02c9d538 765a5b92 urlmon!CINetEmbdFilter::Terminate+0×17
02c9d548 765b9bc1 urlmon!CINet::Terminate+0×23
02c9d55c 765979f2 urlmon!CINetHttp::Terminate+0×48
02c9d574 7659766b urlmon!COInetProt::Terminate+0×1d
02c9d598 765979c0 urlmon!CTransaction::Terminate+0×12d
02c9d5b8 76597a2d urlmon!CBinding::ReportResult+0×92
02c9d5d0 76596609 urlmon!COInetProt::ReportResult+0×1a
02c9d5f8 76596322 urlmon!CTransaction::DispatchReport+0×1d9
02c9d624 7659653e urlmon!CTransaction::DispatchPacket+0×31
02c9d644 765a504b urlmon!CTransaction::OnINetCallback+0×92
02c9d65c 7741fd72 urlmon!TransactionWndProc+0×28
02c9d688 7741fe4a user32!InternalCallWinProc+0×23
02c9d700 7742018d user32!UserCallWinProcCheckWow+0×14b
02c9d764 7742022b user32!DispatchMessageWorker+0×322
02c9d774 7094c1d5 user32!DispatchMessageW+0xf
02c9f87c 708f337e ieframe!CTabWindow::_TabWindowThreadProc+0×54c
02c9f934 7647426d ieframe!LCIETab_ThreadProc+0×2c1
02c9f944 7627d0e9 iertutil!CIsoScope::RegisterThread+0xab
02c9f950 777f19bb kernel32!BaseThreadInitThunk+0xe
02c9f990 777f198e ntdll!__RtlUserThreadStart+0×23
02c9f9a8 00000000 ntdll!_RtlUserThreadStart+0×1b

So I quicky enable full page heap for iexpolore.exe and tried to login again. The crash happened after the same GUI sequence and the new dump was saved again with the following stack trace:

0:004> kL 100
ChildEBP RetAddr 
04c590cc 77815610 ntdll!KiFastSystemCallRet
04c590d0 7627a5d7 ntdll!NtWaitForMultipleObjects+0xc
04c5916c 7627a6f0 kernel32!WaitForMultipleObjectsEx+0x11d
04c59188 762ee2a5 kernel32!WaitForMultipleObjects+0x18
04c591f4 762ee4d1 kernel32!WerpReportFaultInternal+0x16d
04c59208 762cff4d kernel32!WerpReportFault+0x70
04c59294 77827fc1 kernel32!UnhandledExceptionFilter+0x1b5
04c5929c 777b9bdc ntdll!__RtlUserThreadStart+0x6f
04c592b0 777b4067 ntdll!_EH4_CallFilterFunc+0x12
04c592d8 77815f79 ntdll!_except_handler4+0x8e
04c592fc 77815f4b ntdll!ExecuteHandler2+0x26
04c593ac 77815dd7 ntdll!ExecuteHandler+0x24
04c593ac 0004a058 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
04c596b4 0004a12e 0x4a058
04c596d4 765bb7b1 0×4a12e
04c59714 765bb32b urlmon!CINetHttp::INetAsyncSendRequest+0×347
04c59f34 765bb4c8 urlmon!CINetHttp::INetAsyncOpenRequest+0×2cf
04c59f48 765bac97 urlmon!CINet::INetAsyncConnect+0×24b
04c59f60 765a6af9 urlmon!CINet::INetAsyncOpen+0×11b
04c59f70 765a6aaa urlmon!CINet::INetAsyncStart+0×1a
04c59f8c 765a693f urlmon!CINet::StartCommon+0×198
04c59fa8 765a6b5e urlmon!CINet::StartEx+0×1c
04c59fdc 76598e84 urlmon!COInetProt::StartEx+0xc2
04c5a02c 76599411 urlmon!CTransaction::StartEx+0×3e1
04c5a0b4 76599022 urlmon!CBinding::StartBinding+0×602
04c5a0f8 76599fc0 urlmon!CUrlMon::StartBinding+0×169
04c5a120 6ca4eac6 urlmon!CUrlMon::BindToStorage+0×90
04c5a14c 6ca4e9cb mshtml!CStreamProxy::Bind+0xce
04c5a3ec 6ca4b277 mshtml!CDwnBindData::Bind+0×74b
04c5a414 6ca4b118 mshtml!NewDwnBindData+0×15f
04c5a464 6c9cf0aa mshtml!CDwnLoad::Init+0×121
04c5a4b8 6ca4aa61 mshtml!CHtmLoad::Init+0×1fe
04c5a4dc 6ca4a967 mshtml!CDwnInfo::SetLoad+0×119
04c5a4fc 6c9ce021 mshtml!CDwnCtx::SetLoad+0×7a
04c5a514 6c9cec7b mshtml!CHtmCtx::SetLoad+0×13
04c5a534 6c9c25c9 mshtml!CMarkup::Load+0×167
04c5a738 6cb6f395 mshtml!CMarkup::LoadFromInfo+0xb5a
04c5a910 6cb6f532 mshtml!CDoc::DoNavigate+0×1508
04c5aa30 6cde557e mshtml!CDoc::FollowHyperlink2+0xda7
04c5aaf8 6cde5170 mshtml!CFormElement::DoSubmit+0×405
04c5ab0c 6ca01bc5 mshtml!CFormElement::submit+0×11
04c5ab28 6ca8adc3 mshtml!Method_void_void+0×75
04c5ab9c 6ca96e11 mshtml!CBase::ContextInvokeEx+0×5d1
04c5abec 6cb89057 mshtml!CElement::ContextInvokeEx+0×9d
04c5ac28 6ca8a7c1 mshtml!CFormElement::VersionedInvokeEx+0xf0
04c5ac78 6d1f392a mshtml!PlainInvokeEx+0xea
04c5acb8 6d1f3876 jscript!IDispatchExInvokeEx2+0xf8
04c5acf4 6d1f4db6 jscript!IDispatchExInvokeEx+0×6a
04c5adb4 6d1f4d10 jscript!InvokeDispatchEx+0×98
04c5ade8 6d1f2bfd jscript!VAR::InvokeByName+0×135
04c5ae34 6d1f40c5 jscript!VAR::InvokeDispName+0×7a
04c5ae64 6d1f4e23 jscript!VAR::InvokeByDispID+0xce
04c5b000 6d1f123b jscript!CScriptRuntime::Run+0×2abe
04c5b0e8 6d1f1175 jscript!ScrFncObj::CallWithFrameOnStack+0xff
04c5b134 6d1f493c jscript!ScrFncObj::Call+0×8f
04c5b1b8 6d1f2755 jscript!NameTbl::InvokeInternal+0×137
04c5b1ec 6d1f2fa4 jscript!VAR::InvokeByDispID+0×17c
04c5b388 6d1f123b jscript!CScriptRuntime::Run+0×29e0
04c5b470 6d1f1175 jscript!ScrFncObj::CallWithFrameOnStack+0xff
04c5b4bc 6d1f0fa3 jscript!ScrFncObj::Call+0×8f
04c5b538 6d1d3ea3 jscript!CSession::Execute+0×175
04c5b584 6d1d552f jscript!COleScript::ExecutePendingScripts+0×1c0
04c5b5e8 6d1d5345 jscript!COleScript::ParseScriptTextCore+0×29a
04c5b610 6c9ca304 jscript!COleScript::ParseScriptText+0×30
04c5b668 6cb954c2 mshtml!CScriptCollection::ParseScriptText+0×219
04c5d700 6cb7a568 mshtml!CWindow::ExecuteScriptUri+0×19f
04c5d748 6cb95810 mshtml!CWindow::NavigateEx+0×5a
04c5d7b4 6cb956b5 mshtml!CDoc::ExecuteScriptUri+0×262
04c5d7f0 6cc66b68 mshtml!CDoc::ExecuteScriptURL+0×4e
04c5d844 6cad41a7 mshtml!CHyperlink::ClickAction+0×269
04c5d854 6cad416e mshtml!CAnchorElement::ClickAction+0×10
04c5d888 6cb296c5 mshtml!CElement::DoClick+0×155
04c5d8b8 6cad01ff mshtml!CAnchorElement::DoClick+0×6d
04c5d954 6cbae941 mshtml!CDoc::PumpMessage+0xf63
04c5dacc 6cad4408 mshtml!CDoc::OnMouseMessage+0×55d
04c5dbf8 6caa9241 mshtml!CDoc::OnWindowMessage+0×9d9
04c5dc24 7741fd72 mshtml!CServer::WndProc+0×78
04c5dc50 7741fe4a user32!InternalCallWinProc+0×23
04c5dcc8 7742018d user32!UserCallWinProcCheckWow+0×14b
04c5dd2c 7742022b user32!DispatchMessageWorker+0×322
04c5dd3c 7094c1d5 user32!DispatchMessageW+0xf
04c5fe44 708f337e ieframe!CTabWindow::_TabWindowThreadProc+0×54c
04c5fefc 7647426d ieframe!LCIETab_ThreadProc+0×2c1
04c5ff0c 7627d0e9 iertutil!CIsoScope::RegisterThread+0xab
04c5ff18 777f19bb kernel32!BaseThreadInitThunk+0xe
04c5ff58 777f198e ntdll!__RtlUserThreadStart+0×23
04c5ff70 00000000 ntdll!_RtlUserThreadStart+0×1b

We see that IE was trying to send an HTTP request:

 0:004> ub 765bb7b1
urlmon!CINetHttp::INetAsyncSendRequest+0x31f:
765bb799 8bce            mov     ecx,esi
765bb79b e8ef000000      call    urlmon!CINetHttp::SetOptionUserAgent (765bb88f)
765bb7a0 ff75f0          push    dword ptr [ebp-10h]
765bb7a3 ff75ec          push    dword ptr [ebp-14h]
765bb7a6 53              push    ebx
765bb7a7 53              push    ebx
765bb7a8 ff767c          push    dword ptr [esi+7Ch]
765bb7ab ff1544a06576    call    dword ptr [urlmon!_imp__HttpSendRequestW (7665a044)]

From MSDN we get the following function prototype:

BOOL HttpSendRequest(
  __in  HINTERNET hRequest,
  __in  LPCTSTR lpszHeaders,
  __in  DWORD dwHeadersLength,
  __in  LPVOID lpOptional,
  __in  DWORD dwOptionalLength
);

So we check raw stack for parameters:

0:004> dps 04c596d4
04c596d4  04c59714
04c596d8  765bb7b1 urlmon!CINetHttp::INetAsyncSendRequest+0x347
04c596dc  00cc000c ; hRequest
04c596e0  1122cd58 ; lpszHeaders
04c596e4  ffffffff ; dwHeadersLength (-1)
04c596e8  11152e88 ; lpOptional
04c596ec  00000179 ; dwOptionalLength

04c596f0  00000001
04c596f4  77583302 wininet!InternetSetOptionA
04c596f8  110f6d68
04c596fc  0000000b
04c59700  11152e88
04c59704  00000178
04c59708  00000000
04c5970c  11230fe8
04c59710  00000000
04c59714  04c59f34
04c59718  765bb32b urlmon!CINetHttp::INetAsyncOpenRequest+0×2cf
04c5971c  00cc0008
04c59720  110f6d68
04c59724  00000000
04c59728  112d2fe8
04c5972c  112d4fe8
04c59730  112d6fe0

lpszHeaders points to this string:

0:004> du 1122cd58
1122cd58  "Referer: https://www.[...XXX...].ie/o"
1122cd98  "nline/login.aspx..Accept-Languag"
1122cdd8  "e: en-ie..User-Agent: Mozilla/4."
1122ce18  "0 (compatible; MSIE 8.0; Windows"
1122ce58  " NT 6.0; Trident/4.0; MathPlayer"
1122ce98  " 2.10d; SLCC1; .NET CLR 2.0.5072"
1122ced8  "7; Media Center PC 5.0; .NET CLR"
1122cf18  " 3.5.30729; .NET CLR 3.0.30729)."
1122cf58  ".Content-Type: application/x-www"
1122cf98  "-form-urlencoded..Accept-Encodin"
1122cfd8  "g: gzip, deflate"

But lpOptional points to a string that contains the login id and password:

0:004> da 11152e88
11152e88  "__EVENTTARGET=lbtnContinue&__EVE"
11152ea8  "NTARGUMENT=&__VIEWSTATE=%2FwEPDw"
[...]
11152fc8  "u7j7pXFuOFg1%2B&txtLogin=0123456
11152fe8  “789&txtPassword=password???????”

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Icons for Memory Dump Analysis Patterns (Part 75)

Friday, September 24th, 2010

Today we introduce an icon for Corrupt Dump pattern (the motivation is that we can’t open such dumps):

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Structural Memory Patterns (Part 1)

Friday, September 24th, 2010

Now it’s time to divide memory analysis patterns discerned so far as mostly abnormal software behavior memory dump and software trace patterns into behavioral and structural catalogues. The goal is to account for normal system-independent structural entities and relationships visible in memory like modules, threads, processes and so on.

The first pattern (and also a super-pattern) we discuss in this part is called Memory Snapshot. It is further subdivided into Structured Memory Snapshot and BLOB Memory Snapshot. Structured sub-pattern includes:

- Contiguous memory dump files with artificially generated headers (for example, physical or process virtual space memory dump)

- Software trace messages with imposed internal structure

BLOB sub-pattern variety includes address range snapshots without any externally imposed structure, for example, saved by .writemem WinDbg command or ReadProcessMemory API and contiguous buffer and raw memory dumps saved by various memory acquisition tools.

Behavioral patterns that relate to Memory Snapshot pattern are:

I strive initially to publish at least one such pattern every day to fill the gap of normal patterns in memory analysis and later add more multi-platform details and examples from other platforms like Linux, Mac OS X, embedded and selected important historical architectures.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Personal Roots of Memory Dump Analysis

Thursday, September 23rd, 2010

When I was a child I experienced dreams where I was carried by a huge wave that was transforming to a torus completely absorbing me up to a breakpoint of my wake up. A year ago I got the book Memory Evolutive Systems because of my interest in applying category theory to memory analysis and debugging and immediately recalled my long-time forgotten childhood dreams while staring at its front cover:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Icons for Memory Dump Analysis Patterns (Part 74)

Thursday, September 23rd, 2010

Today we introduce an icon for Wait Chain (RPC) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Case Study: Extremely Inconsitent Dump and CPU Spike

Wednesday, September 22nd, 2010

100% CPU consumption was reported for one system and a complete memory dump was generated. Unfortunately, it was very inconsistent:

0: kd> !process 0 0
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
GetContextState failed, 0xD0000147
Unable to read selector for PCR for processor 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8b57f648  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: bffd0020  ObjectTable: e1000e10  HandleCount: 3801.
    Image: System

[...]

PROCESS 8a33fd88  SessionId: 4294963440  Cid: 1508    Peb: 7ffdb000  ParentCid: 3a74
    DirBase: bffd2760  ObjectTable: e653c110  HandleCount: 1664628019.
    Image: explorer.exe

[...]

PROCESS 87bd9d88  SessionId: 4294963440  Cid: 3088    Peb: 7ffda000  ParentCid: 1508
    DirBase: bffd23e0  ObjectTable: e4e73d30  HandleCount: 1717711416.
    Image: iexplore.exe

[...]

PROCESS 88c741a0  SessionId: 0  Cid: 46b0    Peb: 7ffd9000  ParentCid: 01f8
    DirBase: bffd2ac0  ObjectTable: e8b60c58  HandleCount: 1415935346.
    Image: csrss.exe

[...]

!process 0 ff command was looping through the same system thread forever. Fortunately !running command was functional:

0: kd> !running
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    ffdff120  888ab360            …………….
  1    f7727120  880d1db0            …………….

GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147

Curiously !ready command showed a different thread running on the same processor 0 before infinitely looping (it was aborted):

0: kd> !ready
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
Processor 0: Ready Threads at priority 6
    THREAD 88fe2b30  Cid 3b8c.232c  Teb: 7ffdf000 Win32Thread: bc6b38f0 RUNNING on processor 0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
TYPE mismatch for thread object at ffdffaf0
[…]

The both “running” threads were showing signs of a spiking thread:

0: kd> !thread 88fe2b30 1f
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
THREAD 88fe2b30  Cid 3b8c.232c  Teb: 7ffdf000 Win32Thread: bc6b38f0 RUNNING on processor 0
Not impersonating
DeviceMap                 e3899900
Owning Process            8862ead8       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      1980369     
Context Switch Count      239076                 LargeStack
UserTime                  00:01:33.187
KernelTime                00:01:49.734

Win32 Start Address 0×0066c181
Start Address 0×77e617f8
Stack Init b97bfbd0 Current b97bf85c Base b97c0000 Limit b97b9000 Call b97bfbd8
Priority 8 BasePriority 8 PriorityDecrement 0
Unable to get context for thread running on processor 0, HRESULT 0×80004002

GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002
GetContextState failed, 0x80004002

0: kd> !thread 888ab360 1f
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
Unable to get program counter
THREAD 888ab360  Cid 2a3c.4260  Teb: 7ffde000 Win32Thread: bc190570 WAIT: (Unknown) UserMode Non-Alertable
    88e4d8d8  SynchronizationEvent
Not impersonating
DeviceMap                 e62a50e0
Owning Process            8a1a5d88       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      1979505     
Context Switch Count      167668                 LargeStack
UserTime                  00:01:03.468
KernelTime                00:01:21.875

Win32 Start Address ApplicationA (0×0066c181)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba884000 Current ba883bac Base ba884000 Limit ba87d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
ChildEBP RetAddr 
ba883c14 bf8a1305 win32k!RGNOBJ::UpdateUserRgn+0×5d
ba883c38 bf8a2a1a win32k!xxxSendMessage+0×1b
ba883c64 bf8a2ac3 win32k!xxxUpdateWindow2+0×79
ba883c84 bf8a1a6a win32k!xxxInternalUpdateWindow+0×6f
ba883cc8 bf8a291b win32k!xxxInternalInvalidate+0×148
ba883cf4 bf858314 win32k!xxxRedrawWindow+0×103
ba883d4c 8088b41c win32k!NtUserRedrawWindow+0xac
ba883d4c 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ ba883d64)
0012fd10 7739b82a ntdll!KiFastSystemCallRet
0012fd98 78a3ed73 USER32!UserCallWinProcCheckWow+0×5c
0012fdb8 78a3f68b mfc90u!CWnd::DefWindowProcW+0×44
0012fdd4 78a3e29a mfc90u!CWnd::WindowProc+0×3b
0012fe58 78585f1a mfc90u!AfxCallWndProc+0xa3
7739ceb8 c25d008b MSVCR90!_msize+0xf8
WARNING: Frame IP not in any known module. Following frames may be wrong.
7739cec0 9090f8eb 0xc25d008b
7739cec4 8b909090 0×9090f8eb
7739cec8 ec8b55ff 0×8b909090
7739cecc e8084d8b 0xec8b55ff
7739ced0 ffffe838 0xe8084d8b
7739ced4 0374c085 0xffffe838
7739ced8 5d78408b 0×374c085
7739cedc 900004c2 0×5d78408b
7739cee0 90909090 0×900004c2
7739cee4 8b55ff8b 0×90909090
7739cee8 18a164ec 0×8b55ff8b
7739ceec 83000000 0×18a164ec
7739cef0 0f004078 0×83000000
7739cef4 fe87d484 0xf004078
7739cef8 087d83ff 0xfe87d484
7739cefc 2c830f20 0×87d83ff
7739cf00 64ffff94 0×2c830f20
7739cf04 0018158b 0×64ffff94
7739cf08 828b0000 0×18158b
7739cf0c 00000000 0×828b0000

GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147
GetContextState failed, 0xD0000147

We see that both threads belong to 2 process instances of ApplicationA.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -