Archive for February, 2011

Bugtation No.138

Thursday, February 24th, 2011

Learning from Philip Marlowe, a detective:

“I like you,” she said suddenly. “You believe in miracles.”

Raymond Chandler, The Big Sleep

Do you believe in miracles from a driver modifying an arbitrary user space? Or in a miracle of suddenly disappearing software incidents?

- Dmitry Vostokov @ + -

Icons for Memory Dump Analysis Patterns (Part 93)

Thursday, February 24th, 2011

Today we introduce an icon for Hardware Error pattern:



- Dmitry Vostokov @ + -

Reading Notebook: 23-February-11

Thursday, February 24th, 2011

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

The distinction between class, port and miniport components in storage stack (pp. 646 - 647)

Example: disk.sys as a class driver, ataport.sys and atapi.sys as port and miniport drivers (pp. 647 - 448)

MPIO (multi path I/O), DSM (device-specific modules) and storage stack (pp. 649 - 650)

Old and new naming convention (DRX) for disk device objects (p. 650)

Win32 API disk drive naming (p. 651)

Partition device objects (p. 652)

Volume manager as a bus driver (p. 655)

System vs. boot volume (p. 660)

Volmgr.sys vs. Volmgrx.sys (p. 661)

The advantages of storing volume metadata in a file (p. 662)

Spanned, striped (RAID-0), mirrored (RAID-1), RAID-5 (striped with rotated parity) (pp. 662 - 667)

Reading Notebook: 21-February-11

Monday, February 21st, 2011

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Differences between driver and service loading (p. 623)

Tag value precedence redefinition (p. 624)

Verbose !devnode command options (pp. 627 - 628)


Hybrid sleep (pp. 637-638)

Power dispatch routine (p. 639) - Here’s a dispatch routine for a PCI driver from my x64 W2K8R2 system:

0: kd> !devnode 0 3
Dumping IopRootDeviceNode (= 0xfffffa8003c1ed90)
DevNode 0xfffffa8003c1ed90 for PDO 0xfffffa8003c1db10
InstancePath is "HTREE\ROOT\0"
State = DeviceNodeStarted (0x308)
Previous State = DeviceNodeEnumerateCompletion (0x30d)


        DevNode 0xfffffa8003e91b10 for PDO 0xfffffa8003e40a20
InstancePath is “PCI\VEN_8086&DEV_2810&SUBSYS_00000000&REV_02\3&172e68dd&0&F8″
ServiceName is “msisadrv”
State = DeviceNodeStarted (0×308)
Previous State = DeviceNodeEnumerateCompletion (0×30d)


0: kd> !devobj 0xfffffa8003e40a20
Device object (fffffa8003e40a20) is for:
NTPNP_PCI0013 \Driver\pci DriverObject fffffa8003cfe270
Current Irp 00000000 RefCount 0 Type 00000022 Flags 00001040
Dacl fffff9a10008b231 DevExt fffffa8003e40b70 DevObjExt fffffa8003e40f90 DevNode fffffa8003e91b10
ExtensionFlags (0×00000800)
Unknown flags 0×00000800
AttachedDevice (Upper) fffffa8003e3f800
Device queue is not busy.

0: kd> !drvobj fffffa8003cfe270 f
Driver object (fffffa8003cfe270) is for:
Driver Extension List: (id , addr)

Device Object list:
fffffa8003e9da20  fffffa8003e9a060  fffffa8003e99a20  fffffa8003e939f0
fffffa8003e93040  fffffa8003e92660  fffffa8003e92cb0  fffffa8003e42060
fffffa8003e41a20  fffffa8003e41060  fffffa8003e40a20  fffffa8003e40060
fffffa8003e3fa20  fffffa8003e3f060  fffffa8003e3ea20  fffffa8003e3e060
fffffa8003e3da20  fffffa8003e3d060  fffffa8003e3ca20  fffffa8003e3c060
fffffa8003e3ba20  fffffa8003e3b060  fffffa8003e3aa20  fffffa8003e3a060

DriverEntry:   fffff880013ae1a0 pci!GsDriverEntry
DriverStartIo: 00000000�
DriverUnload:  fffff880013a2fec pci!PciDriverUnload
AddDevice:     fffff8800139ae54 pci!PciAddDevice

Dispatch routines:
[00] IRP_MJ_CREATE                      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[01] IRP_MJ_CREATE_NAMED_PIPE           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[02] IRP_MJ_CLOSE                       fffff80001ab5cfc nt!IopInvalidDeviceRequest
[03] IRP_MJ_READ                        fffff80001ab5cfc nt!IopInvalidDeviceRequest
[04] IRP_MJ_WRITE                       fffff80001ab5cfc nt!IopInvalidDeviceRequest
[05] IRP_MJ_QUERY_INFORMATION           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[06] IRP_MJ_SET_INFORMATION             fffff80001ab5cfc nt!IopInvalidDeviceRequest
[07] IRP_MJ_QUERY_EA                    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[08] IRP_MJ_SET_EA                      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[09] IRP_MJ_FLUSH_BUFFERS               fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0a] IRP_MJ_QUERY_VOLUME_INFORMATION    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0b] IRP_MJ_SET_VOLUME_INFORMATION      fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0c] IRP_MJ_DIRECTORY_CONTROL           fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0d] IRP_MJ_FILE_SYSTEM_CONTROL         fffff80001ab5cfc nt!IopInvalidDeviceRequest
[0e] IRP_MJ_DEVICE_CONTROL              fffff8800139e6d0 pci!PciDispatchDeviceControl
[0f] IRP_MJ_INTERNAL_DEVICE_CONTROL     fffff80001ab5cfc nt!IopInvalidDeviceRequest
[10] IRP_MJ_SHUTDOWN                    fffff80001ab5cfc nt!IopInvalidDeviceRequest
[11] IRP_MJ_LOCK_CONTROL                fffff80001ab5cfc nt!IopInvalidDeviceRequest
[12] IRP_MJ_CLEANUP                     fffff80001ab5cfc nt!IopInvalidDeviceRequest
[13] IRP_MJ_CREATE_MAILSLOT             fffff80001ab5cfc nt!IopInvalidDeviceRequest
[14] IRP_MJ_QUERY_SECURITY              fffff80001ab5cfc nt!IopInvalidDeviceRequest
[15] IRP_MJ_SET_SECURITY                fffff80001ab5cfc nt!IopInvalidDeviceRequest
[16] IRP_MJ_POWER                       fffff880013848fc pci!PciDispatchPnpPower
[17] IRP_MJ_SYSTEM_CONTROL              fffff8800139e66c pci!PciDispatchSystemControl
[18] IRP_MJ_DEVICE_CHANGE               fffff80001ab5cfc nt!IopInvalidDeviceRequest
[19] IRP_MJ_QUERY_QUOTA                 fffff80001ab5cfc nt!IopInvalidDeviceRequest
[1a] IRP_MJ_SET_QUOTA                   fffff80001ab5cfc nt!IopInvalidDeviceRequest
[1b] IRP_MJ_PNP                         fffff880013848fc pci!PciDispatchPnpPower

!pocaps and !popolicy WinDbg commands (pp. 641 - 643)

Unlike other PnP operations like normal eject power cannot be vetoed by drivers and apps (pp. 643 - 644)

Icons for Memory Dump Analysis Patterns (Part 92)

Monday, February 21st, 2011

Today we introduce an icon for Wild Code pattern:



- Dmitry Vostokov @ + -

Moving to ARM

Monday, February 21st, 2011

With Windows moving to ARM platforms I’m learning ARM assembly language and system programming. Being mostly exposed to PDP-11, VAX-11 and Intel architectures my interest in ARM systems began with porting my old project to Windows Mobile and writing a tool called WindowHistory Mobile to assist me in debugging. I remember how I was surprised when WinDbg was able to open a crash dump and show ARM code! Then I put it aside. Now with Windows Mobile 7 and Windows 8 approaching with its support of ARM I decided to resume my ARM studies and highly recommend this book:

ARM System Developer’s Guide: Designing and Optimizing System Software

My renewed interest in assembly language also coincides with joining Linkedin Assembly language coders group.

I also plan to create a separate version of my popular Windows Debugging: Practical Foundations book with ARM examples and an ARM section on Asmpedia. If time and resources permit, of course.

- Dmitry Vostokov @ + -

Trace Analysis Patterns (Part 38)

Sunday, February 20th, 2011

Sometimes, we look at a trace and say it’s Impossible Trace. For example, this fragment shows that the function foo had been called:

#     Module  PID  TID  Message
1001  ModuleA 202  404  foo: start
1002  ModuleA 202  404  foo: end

However, if we look at the corresponding source code (PLOT) we would see that something is missing: the function bar must have been called with its own set of trace messages we don’t see in the trace:

void foo()
    TRACE("foo: start");
    TRACE("foo: end");

void bar()
    TRACE("bar: start");
    // some code ... 
    TRACE("bar: end");

We suspect the runtime code being modified, perhaps by patching. In other cases of missing messages we can also suspect thrown exceptions or local buffer overflows that led to wrong return address skipping the code with expected tracing statements. The mismatch between the trace and the source code we are looking at is also possible if the old source code didn’t have bar function called.

Note: I’m grateful for this pattern idea to Gary Barton.

- Dmitry Vostokov @ + -

Crash Dump Analysis Patterns (Part 130a)

Saturday, February 19th, 2011

This part starts the block of patterns called Semantic Structures. These structures are fragments of memory which have meaning helping us in troubleshooting and debugging. The first pattern in this block deals with PID.TID structures of the form DWORD : DWORD or QWORD : QWORD. Such memory fragments are useful for wait chain analysis, for example, by looking at the excution residue left on a raw stack to find a target or an origin of RPC or (A)LPC calls. RPC target example can be found in the post: In Search of Lost CID. Here we look at another example, this time to find the originator of an ALPC call.

A ServiceA was executing some undesired functionality and a breakpoint was set on ModuleA code to trigger it under unreproducible conditions. Then a complete memory dump was saved for offline analysis. There we see an ALPC server thread that triggered the breakpoint but don’t see the message information in the output of WinDbg !thread command that can help us finding an ALPC client thread easily:

THREAD fffffa8005e6b060  Cid 0cc0.1838  Teb: 000007fffff8e000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
SuspendCount 1
    fffff880094ad0a0  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a001aba3c0
Owning Process            fffffa8004803b30       Image:         ServiceA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      1441562        Ticks: 106618 (0:00:27:43.251)
Context Switch Count      414            
UserTime                  00:00:00.000
KernelTime                00:00:00.031
Win32 Start Address ntdll!TppWorkerThread (0×0000000077c88f00)
Stack Init fffff880094addb0 Current fffff880094acdb0
Base fffff880094ae000 Limit fffff880094a8000 Call 0
Priority 12 BasePriority 10 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`094acdf0 fffff800`01678992 nt!KiSwapContext+0×7a
fffff880`094acf30 fffff800`0167acff nt!KiCommitThreadWait+0×1d2
fffff880`094acfc0 fffff800`01a150e8 nt!KeWaitForSingleObject+0×19f
fffff880`094ad060 fffff800`01a1546c nt!DbgkpQueueMessage+0×2a8
fffff880`094ad230 fffff800`019b9116 nt!DbgkpSendApiMessage+0×5c
fffff880`094ad270 fffff800`016abb96 nt! ?? ::NNGAKEGL::`string’+0×3463d
fffff880`094ad3b0 fffff800`01670d82 nt!KiDispatchException+0×316
fffff880`094ada40 fffff800`0166ebb4 nt!KiExceptionDispatch+0xc2
fffff880`094adc20 000007fe`f79365d1 nt!KiBreakpointTrap+0xf4 (TrapFrame @ fffff880`094adc20)
00000000`035ee568 000007fe`f80670b5 ModuleA+0×38611
00000000`035ee5d0 000007fe`ff4bc7f5 ModuleB!Start+0×6e1
00000000`035ee770 000007fe`ff56b62e RPCRT4!Invoke+0×65
00000000`035ee7c0 000007fe`ff4bf1f6 RPCRT4!Ndr64StubWorker+0×61b
00000000`035eed80 000007fe`ffedf223 RPCRT4!NdrStubCall3+0xb5
00000000`035eede0 000007fe`ffedfc0d ole32!CStdStubBuffer_Invoke+0×5b
00000000`035eee10 000007fe`ffedfb83 ole32!SyncStubInvoke+0×5d
00000000`035eee80 000007fe`ffd7fd60 ole32!StubInvoke+0xdb
00000000`035eef30 000007fe`ffedfa22 ole32!CCtxComChnl::ContextInvoke+0×190
00000000`035ef0c0 000007fe`ffedf76b ole32!AppInvoke+0xc2
00000000`035ef130 000007fe`ffeded6d ole32!ComInvokeWithLockAndIPID+0×52b
00000000`035ef2c0 000007fe`ff4b9c24 ole32!ThreadInvoke+0×30d
00000000`035ef360 000007fe`ff4b9d86 RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`035ef390 000007fe`ff4bc44b RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
00000000`035ef4b0 000007fe`ff4bc38b RPCRT4!RPC_INTERFACE::DispatchToStub+0×9b
00000000`035ef4f0 000007fe`ff4bc322 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0×5b
00000000`035ef570 000007fe`ff4ba11d RPCRT4!LRPC_SCALL::DispatchRequest+0×422
00000000`035ef650 000007fe`ff4c7ddf RPCRT4!LRPC_SCALL::HandleRequest+0×20d
00000000`035ef780 000007fe`ff4c7995 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
00000000`035ef8c0 00000000`77c8b43b RPCRT4!LrpcIoComplete+0xa5
00000000`035ef950 00000000`77c8923f ntdll!TppAlpcpExecuteCallback+0×26b
00000000`035ef9e0 00000000`77a6f56d ntdll!TppWorkerThread+0×3f8
00000000`035efce0 00000000`77ca3281 kernel32!BaseThreadInitThunk+0xd
00000000`035efd10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We inspect the raw stack starting from the first top Child-SP value for RPCRT4 subtrace and find NNN:NNN data there resembling PID:TID pair:

1: kd> dpp 00000000`035ef360 l100
00000000`035ef698  00000000`00000000
00000000`035ef6a0  00000000`00000001
00000000`035ef6a8  00000000`00000000
00000000`035ef6b0  00000000`00000000
00000000`035ef6b8  00000000`00000118
00000000`035ef6c0  00000000`0000048c
00000000`035ef6c8  00000000`00495e50 000007fe`ff57d920 RPCRT4!LRPC_ADDRESS::`vftable’
00000000`035ef6d0  00000000`00000000

We find such CID in the stack trace collection and see a wait for an ALPC message reply:

THREAD fffffa8003d49b60  Cid 0118.048c  Teb: 000007fffffaa000 Win32Thread: fffff900c01e4c30 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa8003d49f20  Semaphore Limit 0×1
Waiting for reply to ALPC Message fffff8a000bdb6c0 : queued at port fffffa80042f8090 : owned by process fffffa8004803b30
Not impersonating
DeviceMap                 fffff8a000008600
Owning Process            fffffa8003cf15d0       Image:         ServiceB.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      1441554        Ticks: 106626 (0:00:27:43.376)
Context Switch Count      23180                 LargeStack
UserTime                  00:00:00.468
KernelTime                00:00:03.057
Win32 Start Address ntdll!TppWorkerThread (0×0000000077c88f00)
Stack Init fffff88004ffcdb0 Current fffff88004ffc620
Base fffff88004ffd000 Limit fffff88004ff7000 Call 0
Priority 6 BasePriority 6 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`04ffc660 fffff800`01678992 nt!KiSwapContext+0×7a
fffff880`04ffc7a0 fffff800`0167acff nt!KiCommitThreadWait+0×1d2
fffff880`04ffc830 fffff800`0168fd1f nt!KeWaitForSingleObject+0×19f
fffff880`04ffc8d0 fffff800`01977ac6 nt!AlpcpSignalAndWait+0×8f
fffff880`04ffc980 fffff800`01975a50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`04ffc9e0 fffff800`01972fcb nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`04ffcb00 fffff800`01670993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`04ffcbb0 00000000`77cc070a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`04ffcc20)
00000000`018ce308 000007fe`ff4caa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`018ce310 000007fe`ff4bf802 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`018ce3d0 000007fe`ffee0900 RPCRT4!I_RpcSendReceive+0×42
00000000`018ce400 000007fe`ffee05ef ole32!ThreadSendReceive+0×40
00000000`018ce450 000007fe`ffee041b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3
00000000`018ce4f0 000007fe`ffd819c6 ole32!CRpcChannelBuffer::SendReceive2+0×11b
00000000`018ce6b0 000007fe`ffd81928 ole32!CAptRpcChnl::SendReceive+0×52
00000000`018ce780 000007fe`ffedfcf5 ole32!CCtxComChnl::SendReceive+0×68
00000000`018ce830 000007fe`ff56ba3b ole32!NdrExtpProxySendReceive+0×45
00000000`018ce860 000007fe`ffee02d0 RPCRT4!NdrpClientCall3+0×2e2
00000000`018ceb20 000007fe`ffd818a2 ole32!ObjectStublessClient+0×11d
00000000`018ceeb0 00000000`ff5afe64 ole32!ObjectStubless+0×42
00000000`018cf7a0 00000000`77c8f8eb ServiceB!Worker+0×366
00000000`018cf800 00000000`77c89d9f ntdll!RtlpTpWorkCallback+0×16b
00000000`018cf8e0 00000000`77a6f56d ntdll!TppWorkerThread+0×5ff
00000000`018cfbe0 00000000`77ca3281 kernel32!BaseThreadInitThunk+0xd
00000000`018cfc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Inspection of that message shows that it was directed to our server thread that triggered the breakpoint:

1: kd> !alpc /m fffff8a000bdb6c0

Message @ fffff8a000bdb6c0
  MessageID             : 0x0600 (1536)
  CallbackID            : 0x2D910D (2986253)
  SequenceNumber        : 0x0002CB50 (183120)
  Type                  : LPC_REQUEST
  DataLength            : 0x0068 (104)
  TotalLength           : 0x0090 (144)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8004823a80 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa8003d49b60
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa80042f8090 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa8004803b30 (ServiceA.exe)
  ServerThread          : fffffa8005e6b060
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 000000000020f0c0
  ServerContext         : 0000000000000000
  PortContext           : 0000000000416990
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

- Dmitry Vostokov @ + -

.NET and Java Debugging

Saturday, February 19th, 2011

This is the main topic of the forthcoming March Debugged! MZ/PE magazine issue:

Debugged! MZ/PE: .NET and Java Debugging, March, 2011 (Paperback, ISBN: 978-1908043139)

The draft front cover is an allegory of a class hierarchy and was designed by my daughter, Alexandra Vostokova, a coauthor of Baby Turing book.

If you have an article idea or if you’d like to write an article for this issue please use the following contact form:

The deadline is the 31st of March. Articles will first appear in the online version and then in print.

- Dmitry Vostokov @ + -

Join Facebook DA+TA Group: Special Logo

Friday, February 18th, 2011

This specially designed logo explores the concept of a face as a combination of dump and trace artifacts:

Join Facebook DATA (Dump Analysis + Trace Analysis) group:

- Dmitry Vostokov @ + -

Icons for Memory Dump Analysis Patterns (Part 91)

Friday, February 18th, 2011

Today we introduce an icon for Wild Pointer pattern:



- Dmitry Vostokov @ + -

Encyclopedia of Debugging

Thursday, February 17th, 2011

This is the ultimate multi- and cross-platform debugging reference project dubbed internally as “Memory Dump of Debugging”. Useful for years to come, this encyclopedia would cover all aspects of debugging with articles from leading experts from both industry and academia. Preliminary product information:

  • Title: Encyclopedia of Debugging
  • Editor: Dmitry Vostokov
  • Hardcover: 1200 pages
  • Publisher: OpenTask (March 2014)
  • ISBN-13: 978-1-908043-12-2

The title is a successor to both Encyclopedia of Crash Dump Analysis Patterns (ISBN: 978-1906717216) and Dictionary of Debugging (ISBN: 978-1906717353) the publisher plans to release in the mean time.

- Dmitry Vostokov @ + -

Cloud Crash Dump Analysis

Wednesday, February 16th, 2011

Memory Dump Analysis Services plans a Webinar about crash and hang memory dump analysis in cloud computing environments:

- Dmitry Vostokov @ + -

Dictionary of Debugging: Orbifold Memory Space

Wednesday, February 16th, 2011

Orbifold Memory Space

A multiple virtual/physical memory space view taking into account multiple computers:

The picture can be much more complex if we glue different manifold memory spaces. The space name comes from a mathematical orbifold, a generalization of manifold.

Synonyms: cloud memory space


Also: memory space, memory region, physical memory, virtual memory, manifold memory space, memory mapping.

- Dmitry Vostokov @ + -

Debugger’s Dream

Tuesday, February 15th, 2011

- Dmitry Vostokov @ + -

The Ultimate Debugger’s Desk

Monday, February 14th, 2011

My recent desire is to have the ultimate human debugger’s desk: monitors joined together around me to have 360-degree memory surveillance. I haven’t yet been talking about spherical coordinate system for memory analysis, only cylindrical one. Any sponsor? :-)

Source: 360-Degree Memory Analysis

- Dmitry Vostokov @ + -

Notes on Memoidealism (1.11)

Monday, February 14th, 2011

Alexander of Aphrodisias considers soul as a composition of bodily elements and immortal active intellect (god) comprehensible by our own intellects. This view excludes individual immortality but allows the temporary one by actively contemplating god. In memoidalism we can also consider soul as a composition of memory elements and Memory (as whole totality of memories) as immortal supra-intellect and through its contemplation (including other memories) we attain immortality.

- Dmitry Vostokov @ +

Icons for Memory Dump Analysis Patterns (Part 90)

Monday, February 14th, 2011

Today we introduce an icon for Zombie Processes pattern:



- Dmitry Vostokov @ + -

Defect in Defect

Friday, February 11th, 2011

- Dmitry Vostokov @ + -

Crash Dump Analysis Patterns (Part 129)

Wednesday, February 9th, 2011

Sometimes added instrumentation via gflags, application and driver verifier options affect system, service or application performance and resources. I call this pattern Instrumentation Side Effect. For example, after enabling full page heap, one process on an x64 machine was growing up to 24GB and its user memory dump shows that every heap allocation was recorded in a stack trace database:

0:055> !gflag
Current NtGlobalFlag contents: 0x02000000
    hpa - Place heap allocations at ends of pages

0:055> ~*kc


  48  Id: 117fc.c164 Suspend: 1 Teb: 000007ff`fff52000 Unfrozen
Call Site

ntdll! ?? ::FNODOBFM::`string’

  49  Id: 117fc.de80 Suspend: 1 Teb: 000007ff`fff54000 Unfrozen
Call Site


ntdll! ?? ::FNODOBFM::`string’

  50  Id: 117fc.3700 Suspend: 1 Teb: 000007ff`fff4e000 Unfrozen
Call Site

ntdll! ?? ::FNODOBFM::`string’

0:055> !runaway
 User Mode Time
  Thread       Time
  38:d090      0 days 0:02:28.793
  44:ca48      0 days 0:01:04.459
  48:c164      0 days 0:00:56.909
  43:4458      0 days 0:00:54.475
  50:3700      0 days 0:00:43.992
  45:6f98      0 days 0:00:38.953
  49:de80      0 days 0:00:24.211

   1:391c      0 days 0:00:00.639
   0:7e90      0 days 0:00:00.109
  55:a300      0 days 0:00:00.046
  34:10c9c      0 days 0:00:00.015
  21:d054      0 days 0:00:00.015
  56:b0a0      0 days 0:00:00.000
  54:8b78      0 days 0:00:00.000
  53:155b8      0 days 0:00:00.000
  52:b444      0 days 0:00:00.000

Top modules ModuleA(B, C) from the spiking and heap intensive threads are from the same vendor.

I was able to get a 200×27349 slice from that dump using ImageMagick and it shows almost all virtual memory space filled with traces of this pictorial form (magnified by x8):


- Dmitry Vostokov @ + -