Software Diagnostics Library

OpenTask, the publisher of my books, announces restructuring:

http://www.opentask.com/restructuring-2009

- Dmitry Vostokov @ DumpAnalysis.org -

Here is further analysis of a memory dump used to illustrate Swarm of Shared Locks pattern. In that dump there were also exclusively held locks with many blocked threads:

Resource @ 0x8a04c408    Exclusively owned
    Contention Count = 344875
    NumberOfExclusiveWaiters = 6
     Threads: 87eb3db0-01<*>
     Threads Waiting On Exclusive Access:
              88573db0       87b90378       86a49db0       891f4610      
              8662b020       87127db0 

Resource @ 0x89678e80    Exclusively owned
    Contention Count = 10261
    NumberOfExclusiveWaiters = 2
     Threads: 87eb3db0-01<*>
     Threads Waiting On Exclusive Access:
              89131bf0       88d12db0      

Resource @ 0x88d099d8    Exclusively owned
    Contention Count = 562811
    NumberOfExclusiveWaiters = 4
     Threads: 873611d8-01<*>
     Threads Waiting On Exclusive Access:
              88b8bb88       88a72c50       89359af0       88d865e8      

Resource @ 0x86db9248    Exclusively owned
    Contention Count = 1382269
    NumberOfSharedWaiters = 2
    NumberOfExclusiveWaiters = 11
     Threads: 86ab2020-01<*> 8769cdb0-01    880c77b8-01   
     Threads Waiting On Exclusive Access:
              87bf4020       890dc020       874c01c0       884ef020      
              86913af8       875bab10       88e8a0d8       8923cdb0      
              894eca18       86aa6830       86f293a8     

Resource @ 0x873a88d0    Exclusively owned
    Contention Count = 719758
    NumberOfExclusiveWaiters = 8
     Threads: 88d5f990-01<*>
     Threads Waiting On Exclusive Access:
              8759ea88       871b6db0       88117710       87cb4718      
              883eb638       87239020       881ad020       891b9188      

Resource @ 0x88c379a0    Exclusively owned
    Contention Count = 126686
    NumberOfSharedWaiters = 1
    NumberOfExclusiveWaiters = 8
     Threads: 882b8020-01<*> 88951520-01   
     Threads Waiting On Exclusive Access:
              877d34a8       8939fdb0       87fc5668       8851fdb0      
              86fad850       87f1f450       8a1749f0       876a78d0      

Resource @ 0x88ca9250    Exclusively owned
    Contention Count = 319721
    NumberOfExclusiveWaiters = 4
     Threads: 88607908-01<*>
     Threads Waiting On Exclusive Access:
              86829370       892ae8e8       87205208       87b6d7e0      

Resource @ 0x86a90ef8    Exclusively owned
    Contention Count = 852830
    NumberOfExclusiveWaiters = 12
     Threads: 87571640-01<*>
     Threads Waiting On Exclusive Access:
              88a9c9b0       88a50db0       87117928       890e4c50      
              874ffb30       88b540f8       8705d020       8687edb0      
              87143188       8703e430       885b6aa0       8842bc50      

Resource @ 0x88954538    Exclusively owned
    Contention Count = 40708
    NumberOfExclusiveWaiters = 1
     Threads: 87571640-01<*>
     Threads Waiting On Exclusive Access:
              878ee980      

Resource @ 0x88617eb8    Exclusively owned
    Contention Count = 43531
    NumberOfExclusiveWaiters = 2
     Threads: 87571640-01<*>
     Threads Waiting On Exclusive Access:
              88851db0       87382c50      

Resource @ 0x87288bc8    Exclusively owned
    Contention Count = 644675
    NumberOfExclusiveWaiters = 2
     Threads: 874e4508-01<*>
     Threads Waiting On Exclusive Access:
              88863b08       89479650      

Resource @ 0x87c3d8b0    Exclusively owned
    Contention Count = 335064
    NumberOfExclusiveWaiters = 8
     Threads: 87f44520-01<*>
     Threads Waiting On Exclusive Access:
              88277190       88eceb48       87f0d308       8694d460      
              88461db0       876734a8       871721b0       88c2adb0      

All threads owning various locks exclusively are stuck in processing page fault code, for example:

0: kd> !thread 87eb3db0 1f
THREAD 87eb3db0  Cid 47ac.57c8  Teb: 7ffd7000 Win32Thread: bc151230 WAIT: (Unknown) KernelMode Non-Alertable
    8743e4e0  NotificationEvent
IRP List:
    8660c900: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 e1003890
Owning Process            88e49918       Image:         csrss.exe
Wait Start TickCount      15420972       Ticks: 2527 (0:00:00:39.484)
Context Switch Count      1430991                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:02.734
Start Address 0×75a8e96c
Stack Init a3cf7000 Current a3cf6430 Base a3cf7000 Limit a3cf4000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr 
a3cf6448 8083d5b1 nt!KiSwapContext+0×26
a3cf6474 8083df9e nt!KiSwapThread+0×2e5
a3cf64bc 8082629e nt!KeWaitForSingleObject+0×346
a3cf64e4 80826480 nt!MiWaitForInPageComplete+0×1f
a3cf656c 8084790e nt!MiDispatchFault+0xda3
a3cf65c8 80836c2a nt!MmAccessFault+0×64a
a3cf65c8 bfa38de0 nt!KiTrap0E+0xdc (TrapFrame @ a3cf65e0)
a3cf6a24 bf854a72 win32k!vSetPointer+0×36f
a3cf6a50 bf8b1b74 win32k!GreSetPointer+0×66
a3cf6a7c bf883183 win32k!zzzUpdateCursorImage+0×1cc
a3cf6a8c bf884b06 win32k!zzzSetFMouseMoved+0xd5
a3cf6ad4 bf81530a win32k!ProcessQueuedMouseEvents+0×1c4
a3cf6d30 bf86fd25 win32k!RawInputThread+0×5b4
a3cf6d40 bf898a52 win32k!xxxCreateSystemThreads+0×60
a3cf6d54 80833bef win32k!NtUserCallOneParam+0×23
a3cf6d54 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ a3cf6d64)

0: kd> !thread 87571640 1f
THREAD 87571640  Cid 49f4.65b4  Teb: 7ffdf000 Win32Thread: bc011680 WAIT: (Unknown) KernelMode Non-Alertable
    8870db90  NotificationEvent
Not impersonating
DeviceMap                 e24f6570
Owning Process            87be4a00       Image:         ApplicationC.EXE
Wait Start TickCount      15420974       Ticks: 2525 (0:00:00:39.453)
Context Switch Count      25640                 LargeStack
UserTime                  00:00:00.921
KernelTime                00:00:03.859
Win32 Start Address 0×30002658
Start Address 0×77e617f8
Stack Init 9a318600 Current 9a317b70 Base 9a319000 Limit 9a314000 Call 9a31860c
Priority 14 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
9a317b88 8083d5b1 nt!KiSwapContext+0×26
9a317bb4 8083df9e nt!KiSwapThread+0×2e5
9a317bfc 8082629e nt!KeWaitForSingleObject+0×346
9a317c24 80826480 nt!MiWaitForInPageComplete+0×1f
9a317cac 8084790e nt!MiDispatchFault+0xda3
9a317d08 80836c2a nt!MmAccessFault+0×64a
9a317d08 bf8b5485 nt!KiTrap0E+0xdc (TrapFrame @ 9a317d20)
9a317db4 bf8b526c win32k!vSolidFillRect1+0xb0
9a317f58 bf8ad7d2 win32k!vDIBSolidBlt+0×102
9a317fc4 bfa285d1 win32k!EngBitBlt+0xe1
9a3180c8 bf899b57 win32k!GrePatBltLockedDC+0×1ea
9a318160 bf8b32bb win32k!GrePolyPatBltInternal+0×17c
9a31819c bf8bd71c win32k!GrePolyPatBlt+0×45
9a31822c bf85e3d5 win32k!DrawEdge+0×23a
9a318274 bf8ae338 win32k!xxxDrawWindowFrame+0×170
9a3182d4 bf8847d1 win32k!xxxRealDefWindowProc+0×7a7
9a3182ec bf884801 win32k!xxxWrapRealDefWindowProc+0×16
9a318308 bf8c1769 win32k!NtUserfnDWORD+0×27
9a318340 80833bef win32k!NtUserMessageCall+0xc0
9a318340 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ 9a318364)

We also see that their waiting time is almost the same, 39 seconds. This means that the problem with paging probably started at that time before the crash dump was forced.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes there are so many shared locks on the system that it might point to some problems in subsystems that own them. For example, there are two large swarms of them in this memory dump from a system running 90 user sessions:

0: kd> !session
Sessions on machine: 90

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks....

Resource @ nt!CmpRegistryLock (0x808ad4c0)    Shared 210 owning threads
    Contention Count = 1432
     Threads: 88bf1590-01<*> 8a78a660-01<*> 8a787660-01<*> 8825a3a8-01<*>
              89003358-01<*> 86723b90-01<*> 865bbb00-01<*> 89634638-01<*>
              888d9508-01<*> 88da6b48-01<*> 87db9db0-01<*> 86a9e610-01<*>
              89ff7410-01<*> 87450db0-01<*> 86bdedb0-01<*> 86d604c8-01<*>
              88d465d8-01<*> 86c3b6a0-01<*> 87c89020-01<*> 88e73db0-01<*>
              865fe5b0-01<*> 88450020-01<*> 86bd9db0-01<*> 8a73e838-01<*>
              88dc3db0-01<*> 88035708-01<*> 8833a2f0-01<*> 88608350-01<*>
              87aca020-01<*> 87e007c0-01<*> 86ec39b8-01<*> 893be1b8-01<*>
              8671ddb0-01<*> 8679a718-01<*> 89fe34c8-01<*> 86ccd720-01<*>
              881b1db0-01<*> 86771b20-01<*> 86d71db0-01<*> 89574db0-01<*>
              87dfac50-01<*> 86597020-01<*> 874b3488-01<*> 873b59b0-01<*>
              88e792f8-01<*> 878d2430-01<*> 8853d480-01<*> 889e2020-01<*>
              88c36db0-01<*> 8824f990-01<*> 8719b830-01<*> 884ba020-01<*>
              88e1d768-01<*> 89523db0-01<*> 896529f8-01<*> 887e2870-01<*>
              8a022db0-01<*> 867253a0-01<*> 865f0448-01<*> 87d35640-01<*>
              8715d968-01<*> 87ce0c50-01<*> 87d44730-01<*> 86d69aa8-01<*>
              88e5b020-01<*> 88734410-01<*> 898f2b40-01<*> 8a00a510-01<*>
              87e69db0-01<*> 8722b860-01<*> 86d8e308-01<*> 87263c50-01<*>
              8706ddb0-01<*> 892136e8-01<*> 8875b020-01<*> 8833ca48-01<*>
              8a100db0-01<*> 86b77590-01<*> 888bc020-01<*> 865c3db0-01<*>
              89fba910-01<*> 8a789660-01<*> 8670b2a8-01<*> 868737a8-01<*>
              868326d0-01<*> 871cdaf0-01<*> 8852edb0-01<*> 882b23b8-01<*>
              877e29e0-01<*> 8774f558-01<*> 876aa020-01<*> 89187518-01<*>
              8664b8e0-01<*> 865b4478-01<*> 88135020-01<*> 8686f020-01<*>
              866a0190-01<*> 87316758-01<*> 894dab18-01<*> 87938560-01<*>
              8658f5f0-01<*> 88e54020-01<*> 867f6350-01<*> 89246af8-01<*>
              86801430-01<*> 86db2af0-01<*> 865cf588-01<*> 86ab64f8-01<*>
              8a4a61e8-01<*> 885f3020-01<*> 86ea9af0-01<*> 8a4a7ba8-01<*>
              8a746b08-01<*> 89fc4790-01<*> 87093b10-01<*> 8659bc50-01<*>
              86681db0-01<*> 87102228-01<*> 866145a0-01<*> 866dddb0-01<*>
              86bda990-01<*> 88257db0-01<*> 8687d590-01<*> 867a9db0-01<*>
              89898848-01<*> 8a49b920-01<*> 86596db0-01<*> 8a0f7db0-01<*>
              866c1b40-01<*> 8754e020-01<*> 87fc1428-01<*> 8658c870-01<*>
              880d6a90-01<*> 88be6c50-01<*> 86bbcdb0-01<*> 8a37b8f8-01<*>
              866a13e0-01<*> 873e33d0-01<*> 87d43db0-01<*> 88a5adb0-01<*>
              884a5440-01<*> 883646f0-01<*> 87128020-01<*> 88e1d020-01<*>
              888e6418-01<*> 875c7c50-01<*> 871dd020-01<*> 890d5838-01<*>
              88d061f0-01<*> 88a09428-01<*> 8972f780-01<*> 87325b08-01<*>
              86deb020-01<*> 878b31b8-01<*> 891ac8a8-01<*> 86b234c0-01<*>
              86dd2190-01<*> 875f9db0-01<*> 87bbf200-01<*> 8a1a9c40-01<*>
              88628020-01<*> 87919020-01<*> 87c2a660-01<*> 877dc7c0-01<*>
              8a08adb0-01<*> 87c0f628-01<*> 87ca9a28-01<*> 8880a210-01<*>
              86ec0020-01<*> 88571020-01<*> 8a01edb0-01<*> 88115db0-01<*>
              87a9adb0-01<*> 879ecdb0-01<*> 8868ddb0-01<*> 872bcb58-01<*>
              884a0100-01<*> 8929f020-01<*> 87087020-01<*> 886e75a8-01<*>
              885a5908-01<*> 8762c020-01<*> 89550db0-01<*> 8a554768-01<*>
              89f10680-01<*> 87b322e8-01<*> 87cc74d0-01<*> 883ee2d0-01<*>
              8956caf8-01<*> 8788f330-01<*> 87d5c320-01<*> 86b99db0-01<*>
              876f42e0-01<*> 88e812d0-01<*> 8687cdb0-01<*> 8677a310-01<*>
              89711b40-01<*> 89b013a8-01<*> 86abcdb0-01<*> 89fd7bb0-01<*>
              877c22b0-01<*> 883fc850-01<*> 889e11f8-01<*> 892ff0e0-01<*>
              878ac490-01<*> 86de5c50-01<*> 87741db0-01<*> 8679f020-01<*>
              880ac6d0-01<*> 86d8fb00-01<*>
KD: Scanning for held locks….

Resource @ Ntfs!NtfsData (0xf71665b0)    Shared 1 owning threads
     Threads: 8a78d660-01<*>
KD: Scanning for held locks.

Resource @ 0x8a5c7734    Shared 1 owning threads
    Contention Count = 507565
    NumberOfSharedWaiters = 128
    NumberOfExclusiveWaiters = 1
     Threads: 894b4db0-01    87c773e0-01    88de7020-01    891c9db0-01   
              894d2020-01    865af5f8-01    87867340-01    88c964a0-01<*>
              88e57c98-01    87ae3020-01    86dbe730-01    88343790-01   
              871102e8-01    8855f020-01    87c99920-01    8796a318-01   
              88028db0-01    88ad6610-01    88b73db0-01    89fba3f0-01   
              87d8bc00-01    86f4c5c8-01    8a028608-01    88c783f0-01   
              88c138e0-01    89236910-01    896fbb78-01    88523600-01   
              8926f3b0-01    88a49a48-01    87c19750-01    86c88c50-01   
              88adfad8-01    872b0020-01    87ecab18-01    88b02020-01   
              875f9b10-01    8755e020-01    86f9fdb0-01    86a1cab8-01   
              86816858-01    881eedb0-01    894a99f0-01    87c97740-01   
              8a3bf4b0-01    867765a8-01    8a787660-01    86810330-01   
              876ad268-01    87af3320-01    865fedb0-01    88eb8230-01   
              86b0c438-01    881c0230-01    888b67c8-01    883e3210-01   
              87acbc50-01    873d6648-01    86ed0db0-01    88e2d020-01   
              89fdadb0-01    8934e830-01    870f89f0-01    8756c5e0-01   
              878c88d0-01    86fec608-01    88fdb420-01    87fa0628-01   
              87cad8d8-01    88ee3978-01    86fc49a0-01    875d5020-01   
              871a5020-01    89667a60-01    87170db0-01    88254ae0-01   
              8775e408-01    88204db0-01    87989890-01    873b89a8-01   
              888e6bf8-01    88cc3db0-01    88bf1590-01    879565a0-01   
              86773db0-01    8731a020-01    88aa7a78-01    8759cdb0-01   
              87e555f8-01    86de5678-01    86e28020-01    86ec9320-01   
              86871af0-01    8719cba0-01    8723f820-01    884dac20-01   
              89249020-01    889da168-01    8900b810-01    8a78d660-01   
              88cac758-01    892984c8-01    87d0c020-01    87ecec50-01   
              87ad8c90-01    88109aa8-01    86ef5bf0-01    8a78d3f0-01   
              88d2b020-01    88640db0-01    86fec878-01    895b12d8-01   
              86dd6708-01    87386930-01    888e34e0-01    86a56c50-01   
              8815f768-01    886c42a0-01    898f2020-01    87ca3610-01   
              886dd448-01    86ada210-01    8a37adb0-01    8896c940-01   
              8800e898-01    8733d4b8-01    865fa358-01    88ae1af0-01   
              868dd020-01   
     Threads Waiting On Exclusive Access:
              8a78b020     

Both swarms are grouped around NTFS structures as can be seen from thread stack traces but also have another module in common: PGPsdk

0: kd> !thread 88bf1590 1f
THREAD 88bf1590  Cid 4354.2338  Teb: 7ffdf000 Win32Thread: bc3e88f8 WAIT: (Unknown) KernelMode Non-Alertable
    8a7a73d8  Semaphore Limit 0x7fffffff
    88bf1608  NotificationTimer
IRP List:
    86fb39d0: (0006,0268) Flags: 00000004  Mdl: 00000000
Not impersonating
DeviceMap                 e13c9ca0
Owning Process            869a6d88       Image:         ApplicationA.exe
Wait Start TickCount      15423469       Ticks: 30 (0:00:00:00.468)
Context Switch Count      6465                 LargeStack
UserTime                  00:00:00.343
KernelTime                00:00:01.062
Win32 Start Address 0x0056f122
Start Address 0x77e617f8
Stack Init 97e9d000 Current 97e9c788 Base 97e9d000 Limit 97e98000 Call 0
Priority 14 BasePriority 8 PriorityDecrement 6
ChildEBP RetAddr 
97e9c7a0 8083d5b1 nt!KiSwapContext+0x26
97e9c7cc 8083df9e nt!KiSwapThread+0x2e5
97e9c814 8081e05b nt!KeWaitForSingleObject+0x346
97e9c850 80824ba8 nt!ExpWaitForResource+0xd5
97e9c870 f718a07d nt!ExAcquireResourceSharedLite+0xf5
97e9c884 f717b2eb Ntfs!NtfsAcquireSharedVcb+0×23
97e9c8f0 f717a2e2 Ntfs!NtfsCommonFlushBuffers+0xf5
97e9c954 80840153 Ntfs!NtfsFsdFlushBuffers+0×92
97e9c968 f7272c45 nt!IofCallDriver+0×45
97e9c990 80840153 fltmgr!FltpDispatch+0×6f
97e9c9a4 f6fb1835 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
97e9c9b8 f6fad69a PGPsdk+0×5835
97e9c9c4 80840153 PGPsdk+0×169a
86fb39d0 00000000 nt!IofCallDriver+0×45

0: kd> !thread 88c964a0 1f
THREAD 88c964a0  Cid 323c.43f0  Teb: 7ffad000 Win32Thread: bc2ceea8 WAIT: (Unknown) KernelMode Non-Alertable
    88268338  SynchronizationEvent
    88c96518  NotificationTimer
IRP List:
    86dad430: (0006,0268) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 e16c8eb0
Owning Process            8886ac88       Image:         ApplicationB.EXE
Wait Start TickCount      15423352       Ticks: 147 (0:00:00:02.296)
Context Switch Count      1660                 LargeStack
UserTime                  00:00:00.078
KernelTime                00:00:00.109
Win32 Start Address 0x14225c34
Start Address 0x77e617ec
Stack Init 96835000 Current 96834640 Base 96835000 Limit 96832000 Call 0
Priority 14 BasePriority 8 PriorityDecrement 6
ChildEBP RetAddr 
96834658 8083d5b1 nt!KiSwapContext+0x26
96834684 8083df9e nt!KiSwapThread+0x2e5
968346cc 8081e05b nt!KeWaitForSingleObject+0x346
96834708 8082e012 nt!ExpWaitForResource+0xd5
96834728 f714b89b nt!ExAcquireResourceExclusiveLite+0x8d
96834738 f718b194 Ntfs!NtfsAcquirePagingResourceExclusive+0×20
9683493c f718b8d9 Ntfs!NtfsCommonCleanup+0×193
96834aac 80840153 Ntfs!NtfsFsdCleanup+0xcf
96834ac0 f7272c45 nt!IofCallDriver+0×45
96834ae8 80840153 fltmgr!FltpDispatch+0×6f
96834afc f6fb196c nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
96834b10 f6fad69a PGPsdk+0×596c
96834b1c 80840153 PGPsdk+0×169a
86dad430 00000000 nt!IofCallDriver+0×45

Because no processors are busy except the one that processes crash dump request via NMI interrupt and there are no ready threads it would be natural to assume that the problem with paging started some time ago and some checks for 3rd-party volume encryption software are necessary as PGP name of the module suggests:

0: kd> lmv m PGPsdk
start    end        module name
f6fac000 f6fb7000   PGPsdk     (no symbols)          
    Loaded symbol image file: PGPsdk.sys
    Image path: \SystemRoot\System32\Drivers\PGPsdk.sys
    Image name: PGPsdk.sys
    Timestamp:        Wed Jun 09 11:44:04 2004 (40C6E9F4)
    CheckSum:         00010F72
    ImageSize:        0000B000
    Translations:     0000.04b0 0000.04e0 0409.04b0 0409.04e0

0: kd> !running

System Processors f (affinity mask)
  Idle Processors e

     Prcb      Current   Next   
  0  ffdff120  808a68c0  86841588  ................

0: kd> !thread 808a68c0 1f
THREAD 808a68c0  Cid 0000.0000  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
Owning Process            808a6b40       Image:         Idle
Wait Start TickCount      0              Ticks: 15423499 (2:18:56:32.171)
Context Switch Count      100782385            
UserTime                  00:00:00.000
KernelTime                2 Days 12:18:49.343
Stack Init 808a38b0 Current 808a35fc Base 808a38b0 Limit 808a08b0 Call 0
Priority 0 BasePriority 0 PriorityDecrement 0
ChildEBP RetAddr 
808a07bc 80a84df7 nt!KeBugCheckEx+0x1b
808a080c 80834b83 hal!HalHandleNMI+0x1a5
808a080c 80a80853 nt!KiTrap02+0x136 (TrapFrame @ 808a0820)
808a3570 f7659ca2 hal!HalpClockInterrupt+0xff (TrapFrame @ 808a3570)
808a3600 80839b12 intelppm!AcpiC1Idle+0x12
808a3604 00000000 nt!KiIdleLoop+0xa

0: kd> !ready
Processor 0: No threads in READY state
Processor 1: No threads in READY state
Processor 2: No threads in READY state
Processor 3: No threads in READY state

- Dmitry Vostokov @ DumpAnalysis.org -

Here is the message from John R. Ingram:

http://www.lightningsource.com/podnow.aspx

All my books are printed on demand. For me it is the most economical way to publish and I look forward to seeing POD offering more formats.

- Dmitry Vostokov @ DumpAnalysis.org -

Andrzej Dyjak features Linux application core dumps converted to bitmap images:

Dumb Gallery

- Dmitry Vostokov @ DumpAnalysis.org -

In the past I was not able or didn’t know how to view 32-bit process thread stacks when looking at a complete memory dump from x64 Windows. So I had to request user dumps. Now I want to share a technique a reader of my blog (Yuhong Bao) suggested: to use .thread WinDbg command with /w option. Here are additional steps that I found necessary when playing with my test complete memory dump from x64 Windows Server 2003 SP2 (I used the latest version of WinDbg from 64-bit Debugging Tools for Windows):

0. Find a 32-bit process of interest:

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
    SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 0014a000  ObjectTable: fffffa8000000c10  HandleCount: 736.
    Image: System

PROCESS fffffadfe6b14040
    SessionId: none  Cid: 0130    Peb: 7fffffd8000  ParentCid: 0004
    DirBase: 353c0000  ObjectTable: fffffa80009104a0  HandleCount:  19.
    Image: smss.exe

PROCESS fffffadfe65cec20
    SessionId: 0  Cid: 0160    Peb: 7fffffde000  ParentCid: 0130
    DirBase: 30210000  ObjectTable: fffffa80006a4d80  HandleCount: 732.
    Image: csrss.exe

PROCESS fffffadfe73b7040
    SessionId: 0  Cid: 0270    Peb: 7fffffdc000  ParentCid: 0130
    DirBase: 302b6000  ObjectTable: fffffa8000520710  HandleCount: 751.
    Image: winlogon.exe

PROCESS fffffadfe737d040
    SessionId: 0  Cid: 02a0    Peb: 7fffffd7000  ParentCid: 0270
    DirBase: 0060d000  ObjectTable: fffffa80008df6a0  HandleCount: 339.
    Image: services.exe

PROCESS fffffadfe6574040
    SessionId: 0  Cid: 02ac    Peb: 7fffffd5000  ParentCid: 0270
    DirBase: 0070d000  ObjectTable: fffffa80008e16a0  HandleCount: 510.
    Image: lsass.exe

PROCESS fffffadfe7860040
    SessionId: 0  Cid: 0364    Peb: 7fffffd7000  ParentCid: 02a0
    DirBase: 0935e000  ObjectTable: fffffa8000969710  HandleCount:  87.
    Image: svchost.exe

[...]

PROCESS fffffadfe751d040
    SessionId: 0  Cid: 0bcc    Peb: 7efdf000  ParentCid: 0abc
    DirBase: 18861000  ObjectTable: fffffa8001ecbc30  HandleCount: 326.
    Image: Application32.exe

[...]

1. Switch to the process context:

kd> .process /r /p fffffadfe751d040
Implicit process is now fffffadf`e751d040
Loading User Symbols

Stacks traces are 64-bit:

kd> !process fffffadfe751d040
PROCESS fffffadfe751d040
    SessionId: 0  Cid: 0bcc    Peb: 7efdf000  ParentCid: 0abc
    DirBase: 18861000  ObjectTable: fffffa8001ecbc30  HandleCount: 326.
    Image: Application32.exe
    VadRoot fffffadfe7550ae0 Vads 160 Clone 0 Private 1616. Modified 1675. Locked 0.
    DeviceMap fffffa800210e600
    Token                             fffffa80028ef060
    ElapsedTime                       21:57:59.125
    UserTime                          00:00:00.718
    KernelTime                        00:00:00.953
    QuotaPoolUsage[PagedPool]         185704
    QuotaPoolUsage[NonPagedPool]      20080
    Working Set Sizes (now,min,max)  (3021, 50, 345) (12084KB, 200KB, 1380KB)
    PeakWorkingSetSize                3696
    VirtualSize                       93 Mb
    PeakVirtualSize                   104 Mb
    PageFaultCount                    12097
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      2051

THREAD fffffadfe664e040  Cid 0bcc.0bdc  Teb: 000000007efdb000 Win32Thread: fffff97ff4898bd0 WAIT: (Unknown) UserMode Non-Alertable
    fffffadfe73bac40  SynchronizationEvent
    fffffadfe6b69790  SynchronizationEvent
Not impersonating
DeviceMap                 fffffa800210e600
Owning Process            fffffadfe751d040       Image:         Application32.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4153935        Ticks: 912354 (0:03:57:35.531)
Context Switch Count      8088                 LargeStack
UserTime                  00:00:00.343
KernelTime                00:00:00.593
Win32 Start Address Application32 (0x00000000004077ec)
Start Address 0x0000000077d59620
Stack Init fffffadfdede7e00 Current fffffadfdede7250
Base fffffadfdede8000 Limit fffffadfdede2000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffffadf`dede7290 fffff800`0103b0a3 nt!KiSwapContext+0x85
fffffadf`dede7410 fffff800`0103af8a nt!KiSwapThread+0xc3
fffffadf`dede7450 fffff800`012b9958 nt!KeWaitForMultipleObjects+0x5ec
fffffadf`dede74f0 fffff800`012e63ec nt!ObpWaitForMultipleObjects+0x325
fffffadf`dede79b0 fffff800`0104113d nt!NtWaitForMultipleObjects32+0xcc
fffffadf`dede7c00 00000000`78b83d44 nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`dede7c70)
00000000`0012edc8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0x3a
00000000`0012ee70 00000000`6b005e0d wow64!RunCpuSimulation+0xa
00000000`0012eea0 00000000`77ed8030 wow64!Wow64LdrpInitialize+0x2ed
00000000`0012f6d0 00000000`77ed582f ntdll!LdrpInitializeProcess+0x1538
00000000`0012f9d0 00000000`77ef30a5 ntdll!LdrpInitialize+0x18f
00000000`0012fab0 00000000`77d59620 ntdll!KiUserApcDispatcher+0x15 (TrapFrame @ 00000000`0012fe18)
[...]

2. Load WOW64 extension

kd> .load wow64exts

3. Set the current thread and switch to x86 context:

kd> .thread /w fffffadfe664e040
Implicit thread is now fffffadf`e664e040
x86 context set

4. Sometimes reloading symbols is necessary:

kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

5. Now we can get our stack trace (it is a bit rough because Application32.exe symbols were not available)

kd:x86> kv 100
ChildEBP          RetAddr           Args to Child                                        
002cfd94 7d4e286c 00000002 002cfde0 00000001 ntdll_7d600000!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
002cfe3c 7d94d299 00000002 002cfe64 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
002cfe98 7d94d327 00000001 002d8148 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x152 (FPO: [5,13,0])
*** ERROR: Module load completed but symbols could not be loaded for Application32.exe
002cfeb4 00408081 00000001 002d8148 00000000 USER32!MsgWaitForMultipleObjects+0x1f (FPO: [5,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
002cff00 00407d4b ffffffff 00408b78 004010ee Application32+0x8081
002cff08 00408b78 004010ee 004352e0 004352e0 Application32+0x7d4b
002cff0c 004010ee 004352e0 004352e0 0042f004 Application32+0x8b78
00408b78 90909090 90c3c033 90909090 90909090 Application32+0x10ee
00408b7c 90c3c033 90909090 90909090 90909090 0x90909090
00408b80 90909090 90909090 90909090 433aa0a1 0x90c3c033
[...]

6. We can also access raw stack trace if we need to see 32-bit execution residue and reconstruct partial stack traces:  

kd:x86> !teb
Wow64 TEB32 at 000000007efdd000
[...]
Wow64 TEB at 000000007efdb000
    ExceptionList:        000000007efdd000
    StackBase:            0000000000130000
    StackLimit:           000000000012a000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efdb000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000bcc . 0000000000000bdc
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       6
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

kd:x86> dds 000000000012a000 0000000000130000
[...]

- Dmitry Vostokov @ DumpAnalysis.org -

I forced a complete memory dump of Windows 7 Beta running under VMWare Fusion on my MacBook Air laptop using SystemDump. In WinDbg I see kernel32 API refactoring. It looks like common API was factored out into KERNELBASE.dll. For example, a new session 1 process taskhost.exe has the following highlighted changes (the rest of stack trace layout looks the same as in Vista except nt!KiCommitThreadWait in kernel stack trace counterpart):

kd> vertarget
Windows Kernel Version 7000 UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7000.0.x86fre.winmain_win7beta.081212-1400
Kernel base = 0x82639000 PsLoadedModuleList = 0x82790830
Debug session time: Thu Feb  5 12:21:31.765 2009 (GMT+0)
System Uptime: 0 days 0:14:43.078

kd> .process /r /p 85471598
Implicit process is now 85471598
Loading User Symbols

kd> !process 85471598
PROCESS 85471598  SessionId: 1  Cid: 0750    Peb: 7ffd5000  ParentCid: 01a4
    DirBase: 1efb2320  ObjectTable: 90282990  HandleCount: 176.
    Image: taskhost.exe
    VadRoot 8547c480 Vads 93 Clone 0 Private 410. Modified 107. Locked 0.
    DeviceMap 8f909fc8
    Token                             9025d980
    ElapsedTime                       00:13:41.390
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.125
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (1276, 50, 345) (5104KB, 200KB, 1380KB)
    PeakWorkingSetSize                1278
    VirtualSize                       38 Mb
    PeakVirtualSize                   38 Mb
    PageFaultCount                    2040
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      669

THREAD 85471af0  Cid 0750.0754  Teb: 7ffdf000 Win32Thread: fe823598 WAIT: (UserRequest) UserMode Non-Alertable
    8543f778  SynchronizationEvent
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      4012           Ticks: 52505 (0:00:13:40.390)
Context Switch Count      53            
UserTime                  00:00:00.000
KernelTime                00:00:00.078
Win32 Start Address taskhost!wWinMainCRTStartup (0x006b2e64)
Stack Init 8a3ebfd0 Current 8a3ebb30 Base 8a3ec000 Limit 8a3e9000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr 
8a3ebb48 8268951d nt!KiSwapContext+0x26
8a3ebb8c 826cf460 nt!KiSwapThread+0x57b
8a3ebbe0 8268ccaf nt!KiCommitThreadWait+0×340
8a3ebcb8 828ad5bc nt!KeWaitForSingleObject+0×3ee
8a3ebd20 8269066a nt!NtWaitForSingleObject+0xc6
8a3ebd20 771e5704 nt!KiFastCallEntry+0×12a
001dfac0 771d429c ntdll!KiFastSystemCallRet
001dfac4 7543182c ntdll!NtWaitForSingleObject+0xc
001dfb30 76f54f23 KERNELBASE!WaitForSingleObjectEx+0×98
001dfb48 76f54ed2 kernel32!WaitForSingleObjectExStub+0×75
001dfb5c 006b3400 kernel32!WaitForSingleObject+0×12
001dfbbc 006b36c9 taskhost!UbpmpTaskHostSendResponseReceiveCommand+0×6c
001dfc10 006b2b52 taskhost!UbpmTaskHostWaitForCommands+0xf5
001dfc1c 006b2d0c taskhost!wWinMain+0xd
001dfcb0 76f536d6 taskhost!_initterm_e+0×1b1
001dfcbc 771c883c kernel32!BaseThreadInitThunk+0xe
001dfcfc 771c880f ntdll!__RtlUserThreadStart+0×70
001dfd14 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 8547dab0  Cid 0750.075c  Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
    8547dd98  SynchronizationTimer
    8547de60  SynchronizationTimer
    85431df0  SynchronizationEvent
    85444500  SynchronizationTimer
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      5989           Ticks: 50528 (0:00:13:09.500)
Context Switch Count      9            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWaiterpThread (0x771ccb5e)
Stack Init 8f698fd0 Current 8f698688 Base 8f699000 Limit 8f696000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr 
8f6986a0 8268951d nt!KiSwapContext+0x26
8f6986e4 826cf460 nt!KiSwapThread+0x57b
8f698738 826cbb81 nt!KiCommitThreadWait+0×340
8f698940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f698bcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f698d18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f698d18 771e5704 nt!KiFastCallEntry+0×12a
0068fa44 771d427c ntdll!KiFastSystemCallRet
0068fa48 771ccc8a ntdll!NtWaitForMultipleObjects+0xc
0068fbdc 76f536d6 ntdll!TppWaiterpThread+0×33d
0068fbe8 771c883c kernel32!BaseThreadInitThunk+0xe
0068fc28 771c880f ntdll!__RtlUserThreadStart+0×70
0068fc40 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 8547d3f8  Cid 0750.0760  Teb: 7ffdd000 Win32Thread: fe81f888 WAIT: (UserRequest) UserMode Non-Alertable
    8546dff0  NotificationEvent
    8542a490  SynchronizationEvent
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      43362          Ticks: 13155 (0:00:03:25.546)
Context Switch Count      43            
UserTime                  00:00:00.000
KernelTime                00:00:00.078
Win32 Start Address taskhost!ComTaskMgrWnd::MsgPumpThreadProc (0x006b69f6)
Stack Init 8f6a3fd0 Current 8f6a3688 Base 8f6a4000 Limit 8f6a1000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr 
8f6a36a0 8268951d nt!KiSwapContext+0x26
8f6a36e4 826cf460 nt!KiSwapThread+0x57b
8f6a3738 826cbb81 nt!KiCommitThreadWait+0×340
8f6a3940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f6a3bcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f6a3d18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f6a3d18 771e5704 nt!KiFastCallEntry+0×12a
0130f93c 771d427c ntdll!KiFastSystemCallRet
0130f940 75436e4d ntdll!NtWaitForMultipleObjects+0xc
0130f9dc 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
0130fa24 76fd9c0d kernel32!WaitForMultipleObjectsExStub+0xe0
0130fa78 76fda24f USER32!RealMsgWaitForMultipleObjectsEx+0×13c
0130fa94 006b6a46 USER32!MsgWaitForMultipleObjects+0×1f
0130fadc 76f536d6 taskhost!ComTaskMgrWnd::MsgPumpThreadProc+0×50
0130fae8 771c883c kernel32!BaseThreadInitThunk+0xe
0130fb28 771c880f ntdll!__RtlUserThreadStart+0×70
0130fb40 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 854b66a8  Cid 0750.0788  Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
    85394928  QueueObject
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      4060           Ticks: 52457 (0:00:13:39.640)
Context Switch Count      7            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0x771c8ede)
Stack Init 904e5fd0 Current 904e5b00 Base 904e6000 Limit 904e3000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr 
904e5b18 8268951d nt!KiSwapContext+0x26
904e5b5c 826cf460 nt!KiSwapThread+0x57b
904e5bb0 826d2e5c nt!KiCommitThreadWait+0×340
904e5c38 828ad62d nt!KeRemoveQueueEx+0×7df
904e5c90 826d95cb nt!IoRemoveIoCompletion+0×23
904e5d24 8269066a nt!NtWaitForWorkViaWorkerFactory+0×1a1
904e5d24 771e5704 nt!KiFastCallEntry+0×12a
0148fc54 771d42ac ntdll!KiFastSystemCallRet
0148fc58 771cce31 ntdll!NtWaitForWorkViaWorkerFactory+0xc
0148fdbc 76f536d6 ntdll!TppWorkerThread+0×223
0148fdc8 771c883c kernel32!BaseThreadInitThunk+0xe
0148fe08 771c880f ntdll!__RtlUserThreadStart+0×70
0148fe20 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 85491658  Cid 0750.07b8  Teb: 7ffd3000 Win32Thread: fe4afbb8 WAIT: (UserRequest) UserMode Non-Alertable
    8540c280  NotificationEvent
    85494a08  NotificationEvent
    85494980  NotificationEvent
    854948f8  NotificationEvent
    85494870  NotificationEvent
    854947e8  NotificationEvent
    85494760  NotificationEvent
    854946d8  NotificationEvent
    85494650  NotificationEvent
    854945c8  NotificationEvent
    85494540  NotificationEvent
    8544ba30  NotificationEvent
    85145480  NotificationEvent
    84a27448  SynchronizationEvent
    85459e50  SynchronizationEvent
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      53222          Ticks: 3295 (0:00:00:51.484)
Context Switch Count      738            
UserTime                  00:00:00.000
KernelTime                00:00:00.125
Win32 Start Address MsCtfMonitor!MsCtfMonitor::ThreadProc (0x702c208d)
Stack Init 89f1efd0 Current 89f1e688 Base 89f1f000 Limit 89f1c000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr 
89f1e6a0 8268951d nt!KiSwapContext+0x26
89f1e6e4 826cf460 nt!KiSwapThread+0x57b
89f1e738 826cbb81 nt!KiCommitThreadWait+0×340
89f1e940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
89f1ebcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
89f1ed18 8269066a nt!NtWaitForMultipleObjects+0xcc
89f1ed18 771e5704 nt!KiFastCallEntry+0×12a
0142f864 771d427c ntdll!KiFastSystemCallRet
0142f868 75436e4d ntdll!NtWaitForMultipleObjects+0xc
0142f904 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
0142f94c 76fd9c0d kernel32!WaitForMultipleObjectsExStub+0xe0
0142f9a0 76fda24f USER32!RealMsgWaitForMultipleObjectsEx+0×13c
0142f9bc 702c1435 USER32!MsgWaitForMultipleObjects+0×1f
0142fb7c 702c20e1 MsCtfMonitor!DoMsCtfMonitor+0×2b8
0142fd9c 76f536d6 MsCtfMonitor!MsCtfMonitor::ThreadProc+0×5d
0142fda8 771c883c kernel32!BaseThreadInitThunk+0xe
0142fde8 771c880f ntdll!__RtlUserThreadStart+0×70
0142fe00 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 85491370  Cid 0750.07bc  Teb: 7ffda000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
    85492ff0  NotificationEvent
    853bc030  NotificationEvent
IRP List:
    85492408: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492568: (0006,0094) Flags: 00060070  Mdl: 00000000
    854926c8: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492828: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492988: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492ae8: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492c48: (0006,0094) Flags: 00060070  Mdl: 00000000
    85492da8: (0006,0094) Flags: 00060070  Mdl: 00000000
    8544e4b8: (0006,0094) Flags: 00060070  Mdl: 00000000
    853cf470: (0006,0094) Flags: 00060070  Mdl: 00000000
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      4060           Ticks: 52457 (0:00:13:39.640)
Context Switch Count      2            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address PlaySndSrv!CBeepRedirector::WorkThread (0x70271c6c)
Stack Init 8f65dfd0 Current 8f65d688 Base 8f65e000 Limit 8f65b000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr 
8f65d6a0 8268951d nt!KiSwapContext+0x26
8f65d6e4 826cf460 nt!KiSwapThread+0x57b
8f65d738 826cbb81 nt!KiCommitThreadWait+0×340
8f65d940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f65dbcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f65dd18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f65dd18 771e5704 nt!KiFastCallEntry+0×12a
01c6f6d4 771d427c ntdll!KiFastSystemCallRet
01c6f6d8 75436e4d ntdll!NtWaitForMultipleObjects+0xc
01c6f774 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
01c6f7bc 70271cdd kernel32!WaitForMultipleObjectsExStub+0xe0
01c6f93c 76f536d6 PlaySndSrv!CBeepRedirector::WorkThread+0×266
01c6f948 771c883c kernel32!BaseThreadInitThunk+0xe
01c6f988 771c880f ntdll!__RtlUserThreadStart+0×70
01c6f9a0 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 84a01370  Cid 0750.07c8  Teb: 7ffd9000 Win32Thread: fe4afde0 WAIT: (WrLpcReceive) UserMode Non-Alertable
    84a015a4  Semaphore Limit 0x1
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      53222          Ticks: 3295 (0:00:00:51.484)
Context Switch Count      890            
UserTime                  00:00:00.000
KernelTime                00:00:00.031
Win32 Start Address MSCTF!CCtfServerPort::StaticServerThread (0x76bea423)
Stack Init 89e4ffd0 Current 89e4fa78 Base 89e50000 Limit 89e4d000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr 
89e4fa90 8268951d nt!KiSwapContext+0x26
89e4fad4 826cf460 nt!KiSwapThread+0x57b
89e4fb28 8268ccaf nt!KiCommitThreadWait+0×340
89e4fc04 828b9a5a nt!KeWaitForSingleObject+0×3ee
89e4fc34 828ba1c9 nt!AlpcpReceiveMessagePort+0×245
89e4fcb4 828ba489 nt!AlpcpReceiveMessage+0×1b8
89e4fd0c 8269066a nt!NtAlpcSendWaitReceivePort+0×11b
89e4fd0c 771e5704 nt!KiFastCallEntry+0×12a
005feb10 771d2c8c ntdll!KiFastSystemCallRet
005feb14 76bd5b34 ntdll!NtAlpcSendWaitReceivePort+0xc
005ffb9c 76bea53c MSCTF!CCtfServerPort::ServerLoop+0×136
005ffe2c 76bea441 MSCTF!CCtfServerPort::ServerThread+0xde
005ffe3c 76f536d6 MSCTF!CCtfServerPort::StaticServerThread+0×22
005ffe48 771c883c kernel32!BaseThreadInitThunk+0xe
005ffe88 771c880f ntdll!__RtlUserThreadStart+0×70
005ffea0 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 854cc5f0  Cid 0750.0114  Teb: 7ffd8000 Win32Thread: fe4bb008 WAIT: (WrUserRequest) UserMode Non-Alertable
    854cc488  SynchronizationEvent
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      53850          Ticks: 2667 (0:00:00:41.671)
Context Switch Count      301            
UserTime                  00:00:00.000
KernelTime                00:00:00.218
Win32 Start Address WINMM!mciwindow (0x73942761)
Stack Init 904c6fd0 Current 904c6a60 Base 904c7000 Limit 904c4000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr 
904c6a78 8268951d nt!KiSwapContext+0x26
904c6abc 826cf460 nt!KiSwapThread+0x57b
904c6b10 8268ccaf nt!KiCommitThreadWait+0×340
904c6be8 8e50c768 nt!KeWaitForSingleObject+0×3ee
904c6c44 8e50c575 win32k!xxxRealSleepThread+0×1d7
904c6c60 8e508379 win32k!xxxSleepThread+0×2d
904c6cb8 8e50cf9a win32k!xxxRealInternalGetMessage+0×4b2
904c6d1c 8269066a win32k!NtUserGetMessage+0×3f
904c6d1c 771e5704 nt!KiFastCallEntry+0×12a
0169f7d8 76fdbb29 ntdll!KiFastSystemCallRet
0169f7dc 76fd3f49 USER32!NtUserGetMessage+0xc
0169f800 739427e0 USER32!GetMessageA+0×8d
0169f838 76f536d6 WINMM!mciwindow+0×102
0169f844 771c883c kernel32!BaseThreadInitThunk+0xe
0169f884 771c880f ntdll!__RtlUserThreadStart+0×70
0169f89c 00000000 ntdll!_RtlUserThreadStart+0×1b

THREAD 83bafd48  Cid 0750.09f8  Teb: 7ffdb000 Win32Thread: fe569198 WAIT: (WrQueue) UserMode Alertable
    8547dfd0  QueueObject
    83bafdd8  NotificationTimer
Not impersonating
DeviceMap                 8f909fc8
Owning Process            85471598       Image:         taskhost.exe
Wait Start TickCount      53850          Ticks: 2667 (0:00:00:41.671)
Context Switch Count      102            
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Win32 Start Address ntdll!TppWorkerThread (0x771c8ede)
Stack Init 8bff3fd0 Current 8bff3b00 Base 8bff4000 Limit 8bff1000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr 
8bff3b18 8268951d nt!KiSwapContext+0x26
8bff3b5c 826cf460 nt!KiSwapThread+0x57b
8bff3bb0 826d2e5c nt!KiCommitThreadWait+0×340
8bff3c38 828ad62d nt!KeRemoveQueueEx+0×7df
8bff3c90 826d95cb nt!IoRemoveIoCompletion+0×23
8bff3d24 8269066a nt!NtWaitForWorkViaWorkerFactory+0×1a1
8bff3d24 771e5704 nt!KiFastCallEntry+0×12a
0184f9f4 771d42ac ntdll!KiFastSystemCallRet
0184f9f8 771cce31 ntdll!NtWaitForWorkViaWorkerFactory+0xc
0184fb5c 76f536d6 ntdll!TppWorkerThread+0×223
0184fb68 771c883c kernel32!BaseThreadInitThunk+0xe
0184fba8 771c880f ntdll!__RtlUserThreadStart+0×70
0184fbc0 00000000 ntdll!_RtlUserThreadStart+0×1b

kd> lmv m taskhost
start    end        module name
006b0000 006be000   taskhost   (deferred)            
    Image path: C:\Windows\system32\taskhost.exe
    Image name: taskhost.exe
    Timestamp:        Sat Dec 13 02:02:54 2008 (494317CE)
    CheckSum:         00011C71
    ImageSize:        0000E000
    File version:     6.1.7000.0
    Product version:  6.1.7000.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     taskhost.exe
    OriginalFilename: taskhost.exe
    ProductVersion:   6.1.7000.0
    FileVersion:      6.1.7000.0 (winmain_win7beta.081212-1400)
    FileDescription:  Host Process for Windows Tasks
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

Functions that previously called kernel32 API now call their stub equivalents in kernel32 (function names affixed with Stub) and then stubs call KERNELBASE functions having previous kernel32 function names.

It can be seen from dumping contents of import directories of USER32, ADVAPI32, and GDI32 modules that they also depend on KERNELBASE. For example, for GDI32 we have:

kd> !dh 75e70000

File Type: DLL
FILE HEADER VALUES
     14C machine (i386)
       4 number of sections
49433CCD time date stamp Sat Dec 13 04:40:45 2008

       0 file pointer to symbol table
       0 number of symbols
      E0 size of optional header
    2102 characteristics
            Executable
            32 bit word machine
            DLL

OPTIONAL HEADER VALUES
     10B magic #
    9.00 linker version
   46600 size of code
    3A00 size of initialized data
       0 size of uninitialized data
    CF7C address of entry point
    1000 base of code
         ----- new -----
75e70000 image base
    1000 section alignment
     200 file alignment
       3 subsystem (Windows CUI)
    6.01 operating system version
    6.01 image version
    6.01 subsystem version
   4D000 size of image
     800 size of headers
   4D765 checksum
00040000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
    1284 [    4BB4] address [size] of Export Directory
   46308 [     1B8] address [size] of Import Directory
   4A000 [     3D0] address [size] of Resource Directory
       0 [       0] address [size] of Exception Directory
       0 [       0] address [size] of Security Directory
   4B000 [    1920] address [size] of Base Relocation Directory
   474F0 [      38] address [size] of Debug Directory
       0 [       0] address [size] of Description Directory
       0 [       0] address [size] of Special Directory
       0 [       0] address [size] of Thread Storage Directory
   2A000 [      40] address [size] of Load Configuration Directory
     280 [     3E4] address [size] of Bound Import Directory
    1000 [     284] address [size] of Import Address Table Directory
       0 [       0] address [size] of Delay Import Directory
       0 [       0] address [size] of COR20 Header Directory
       0 [       0] address [size] of Reserved Directory
[…]

kd> dds 75e70000+1000 75e70000+1000+284
75e71000  771d3da0 ntdll!ZwSecureConnectPort
75e71004  771d3bb0 ntdll!ZwRegisterThreadTerminatePort
75e71008  771d38b0 ntdll!ZwQueryInformationProcess
75e7100c  771ab232 ntdll!RtlUnwind
75e71010  771d3680 ntdll!NtOpenThreadToken
75e71014  771d3600 ntdll!ZwOpenProcessToken
75e71018  771d38e0 ntdll!NtQueryInformationToken
75e7101c  771adecf ntdll!RtlLengthSid
75e71020  771adeeb ntdll!RtlCopySid
75e71024  771d3cd0 ntdll!ZwRequestWaitReplyPort
75e71028  771bb080 ntdll!_vsnwprintf
75e7102c  771aca7c ntdll!_strnicmp
75e71030  771b75a8 ntdll!_stricmp
75e71034  771b30f4 ntdll!RtlCreateUnicodeStringFromAsciiz
75e71038  771d59c0 ntdll!strncpy
75e7103c  771d4230 ntdll!ZwUnmapViewOfSection
75e71040  771f3b4b ntdll!RtlMultiByteToUnicodeN
75e71044  771c9339 ntdll!RtlDosPathNameToNtPathName_U
75e71048  771d3490 ntdll!NtMapViewOfSection
75e7104c  771d2f50 ntdll!NtCreateSection
75e71050  771d3880 ntdll!ZwQueryInformationFile
75e71054  771d5580 ntdll!memset
75e71058  771d5240 ntdll!memmove
75e7105c  771f1f7e ntdll!RtlUnicodeToMultiByteN
75e71060  771f221b ntdll!RtlUnicodeToMultiByteSize
75e71064  771b069d ntdll!RtlInitializeCriticalSection
75e71068  771b77b7 ntdll!RtlEncodePointer
75e7106c  771c5093 ntdll!RtlDeleteCriticalSection
75e71070  771d43b0 ntdll!RtlInitUnicodeString
75e71074  771d3570 ntdll!NtOpenKey
75e71078  771d3ab0 ntdll!NtQueryValueKey
75e7107c  771d2d30 ntdll!ZwClose
75e71080  771d3540 ntdll!ZwOpenFile
75e71084  771cf682 ntdll!_wcsnicmp
75e71088  771cc1cd ntdll!RtlNtStatusToDosError
75e7108c  771f2a11 ntdll!RtlFreeAnsiString
75e71090  771c2fe5 ntdll!RtlNtPathNameToDosPathName
75e71094  771a3e05 ntdll!RtlpEnsureBufferSize
75e71098  771b3cf0 ntdll!_wcsicmp
75e7109c  771b13db ntdll!wcschr
75e710a0  771cf0ea ntdll!wcsrchr
75e710a4  771d5e00 ntdll!RtlCompareMemory
75e710a8  771bd9e4 ntdll!RtlDecodePointer
75e710ac  771d4240 ntdll!NtVdmControl
75e710b0  771f0ea0 ntdll!RtlAllocateHeap
75e710b4  771f0fb0 ntdll!RtlFreeHeap
75e710b8  771d4f00 ntdll!memcpy
75e710bc  771f1068 ntdll!RtlLeaveCriticalSection
75e710c0  771f10a6 ntdll!RtlEnterCriticalSection
75e710c4  00000000
75e710c8  75440220 KERNELBASE!IsDBCSLeadByte
75e710cc  7544f8b9 KERNELBASE!IsDBCSLeadByteEx
75e710d0  00000000
75e710d4  75436dec KERNELBASE!GetLastError
75e710d8  7545f842 KERNELBASE!UnhandledExceptionFilter
75e710dc  7544c2b3 KERNELBASE!SetUnhandledExceptionFilter
75e710e0  771f1412 ntdll!RtlSetLastWin32Error
75e710e4  00000000
75e710e8  76f465cc kernel32!GetDriveTypeWStub
75e710ec  76f55685 kernel32!WriteFileStub
75e710f0  76f55169 kernel32!CreateFileWStub
75e710f4  76f466b8 kernel32!GetFullPathNameWStub
75e710f8  76f40808 kernel32!DeleteFileWStub
75e710fc  76f354aa kernel32!SetFilePointerExStub
75e71100  76f4a269 kernel32!SetFilePointerStub
75e71104  76f40c4d kernel32!GetFileSizeExStub
75e71108  76f370ed kernel32!GetTempFileNameWStub
75e7110c  00000000
75e71110  76f55137 kernel32!CloseHandleStub
75e71114  00000000
75e71118  75436d3a KERNELBASE!InterlockedCompareExchange
75e7111c  00000000
75e71120  7543ab61 KERNELBASE!FreeLibrary
75e71124  754436f1 KERNELBASE!SizeofResource
75e71128  754376d8 KERNELBASE!GetModuleHandleA
75e7112c  7543bb5a KERNELBASE!LoadLibraryExW
75e71130  75438116 KERNELBASE!SetHandleCount
75e71134  7544367e KERNELBASE!LoadResource
75e71138  7543cad6 KERNELBASE!DisableThreadLibraryCalls
75e7113c  7543762d KERNELBASE!GetProcAddress
75e71140  00000000
75e71144  7543810b KERNELBASE!GetACP
75e71148  75444dee KERNELBASE!GetLocaleInfoW
75e7114c  7544c484 KERNELBASE!GetOEMCP
75e71150  00000000
75e71154  7543d213 KERNELBASE!RegOpenKeyExA
75e71158  75439771 KERNELBASE!RegCloseKey
75e7115c  7543d379 KERNELBASE!RegQueryValueExA
75e71160  75439549 KERNELBASE!RegOpenKeyExW
75e71164  75449b64 KERNELBASE!RegEnumValueW
75e71168  00000000
75e7116c  754373cc KERNELBASE!UnmapViewOfFile
75e71170  7543fc4c KERNELBASE!CreateFileMappingW
75e71174  7543fbc8 KERNELBASE!MapViewOfFile
75e71178  00000000
75e7117c  75438854 KERNELBASE!GlobalFree
75e71180  75437256 KERNELBASE!lstrlenW
75e71184  7543cec7 KERNELBASE!LocalReAlloc
75e71188  754388d1 KERNELBASE!LocalAlloc
75e7118c  7543d9a9 KERNELBASE!GlobalAlloc
75e71190  75438e61 KERNELBASE!lstrlenA
75e71194  75438854 KERNELBASE!GlobalFree
75e71198  00000000
75e7119c  75449d05 KERNELBASE!SearchPathW
75e711a0  00000000
75e711a4  75436d30 KERNELBASE!GetCurrentThreadId
75e711a8  75436e20 KERNELBASE!GetCurrentProcessId
75e711ac  7543771a KERNELBASE!ProcessIdToSessionId
75e711b0  754370bf KERNELBASE!GetCurrentThread
75e711b4  75459f89 KERNELBASE!TerminateProcess
75e711b8  75436dfb KERNELBASE!GetCurrentProcess
75e711bc  00000000
75e711c0  771f145a ntdll!RtlQueryPerformanceCounter
75e711c4  00000000
75e711c8  7545a887 KERNELBASE!IsWellKnownSid
75e711cc  00000000
75e711d0  75437e76 KERNELBASE!MultiByteToWideChar
75e711d4  7543839a KERNELBASE!WideCharToMultiByte
75e711d8  00000000
75e711dc  771c5093 ntdll!RtlDeleteCriticalSection
75e711e0  771f1068 ntdll!RtlLeaveCriticalSection
75e711e4  771b069d ntdll!RtlInitializeCriticalSection
75e711e8  771f10a6 ntdll!RtlEnterCriticalSection
75e711ec  00000000
75e711f0  75438eb9 KERNELBASE!GetTickCount64+0×4
75e711f4  7543f6ea KERNELBASE!GetWindowsDirectoryW
75e711f8  7543f67b KERNELBASE!GetSystemWindowsDirectoryW
75e711fc  7543aa71 KERNELBASE!GetSystemInfo
75e71200  754387b0 KERNELBASE!GetLocalTime
75e71204  75436cc3 KERNELBASE!GetTickCount+0×4
75e71208  7543712d KERNELBASE!GetSystemTimeAsFileTime
75e7120c  00000000
75e71210  76f351d4 kernel32!CopyFileW
75e71214  76f526c8 kernel32!GlobalLock
75e71218  76f54be0 kernel32!MulDiv
75e7121c  76f4662d kernel32!LoadLibraryW
75e71220  76f3b86c kernel32!GlobalSize
75e71224  76f3a5c0 kernel32!GetTempPathW
75e71228  76f40c2f kernel32!FindResourceW
75e7122c  76f45a27 kernel32!LoadLibraryA
75e71230  76f37015 kernel32!VirtualUnlock
75e71234  76f5018b kernel32!GlobalUnlock
75e71238  00000000
75e7123c  76fd89ed USER32!GetAppCompatFlags2
75e71240  76fd68f6 USER32!InitializeLpkHooks
75e71244  76fda345 USER32!NtUserGetDC
75e71248  76ff21c7 USER32!UserRealizePalette
75e7124c  76fd34f2 USER32!GetAppCompatFlags
75e71250  76fd7c23 USER32!CharUpperBuffA
75e71254  76fe17ff USER32!IsThreadDesktopComposited
75e71258  76fda409 USER32!GetWindowRect
75e7125c  76fe1766 USER32!IntersectRect
75e71260  76fd7ce4 USER32!CharLowerBuffW
75e71264  76fda31a USER32!ReleaseDC
75e71268  00000000
75e7126c  772e1bbf LPK!LpkUseGDIWidthCache
75e71270  772e4e3e LPK!LpkGetCharacterPlacement
75e71274  772e167a LPK!LpkExtTextOut
75e71278  772e1df6 LPK!LpkGetTextExtentExPoint
75e7127c  772e1898 LPK!LpkInitialize
75e71280  00000000
75e71284  00000000

- Dmitry Vostokov @ DumpAnalysis.org -

After an upgrade to the new version of a productivity software package one unrelated application started to crash frequently. A crash dump was collected and the following stack trace pointed to a NULL code pointer:

0:000> r
eax=09680104 ebx=0013aefc ecx=0968a710 edx=0cdc0c0c esi=16a19058 edi=00000001
eip=00000000 esp=0013aea8 ebp=0013aeb8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200202
00000000 ??              ???

0:000> k 100
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0013aea4 096e28a0 0×0
0013aeb8 32e688bd dllC!Abort+0×10
0013aec8 32c82395 dllB+0×589e
0013aed8 32865718 dllB+0×18f1
[…]
0013b0c0 314de1ff dllB+0×4c6
0013b154 31293494 dllA!DllGetLCID+0×46d2d
0013b178 312af217 dllA!DllGetClassObject+0×4e896
[…]
0013f3d0 300e8721 dllA!DllGetClassObject+0×69e42
0013f578 300e7f5a application+0xcff5
[…]
0013ffc0 7c816ff7 application+0×51d5
0013fff0 00000000 kernel32!BaseProcessStart+0×23

To see if changed environment somehow affected this application the presence of any DLL hooks was checked. The following  hooked functions were found in user32.dll:

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\USER32.dll\45F02D7C90000\USER32.dll
No range specified

Scanning section:    .text
Size: 389095
Range to scan: 7e411000-7e46ffe7
    7e4188a6-7e4188aa  5 bytes - USER32!GetWindowLongW
 [ 6a 08 68 e0 88:e9 55 77 a4 01 ]
    7e418f9c-7e418fa0  5 bytes - USER32!GetSystemMetrics (+0×6f6)
 [ 6a 10 68 00 90:e9 5f 70 b5 01 ]
    7e41945d-7e419461  5 bytes - USER32!GetWindowLongA (+0×4c1)
 [ 6a 10 68 78 9f:e9 f5 60 a2 01 ]
    7e41b6ae-7e41b6b2  5 bytes - USER32!GetClientRect (+0×17a8)
 [ 8b ff 55 8b ec:e9 4d 49 9f 01 ]
    7e41b6d4-7e41b6d8  5 bytes - USER32!GetWindowRect (+0×26)
 [ b8 74 11 00 00:e9 98 30 af 01 ]
    7e41d60d-7e41d611  5 bytes - USER32!SetWindowLongA (+0×6aa)
 [ 8b ff 55 8b ec:e9 ee 29 a5 01 ]
    7e41d62b-7e41d62f  5 bytes - USER32!SetWindowLongW (+0×1e)
 [ 6a 08 68 28 f5:e9 0e 0b b2 01 ]
    7e41fc25-7e41fc29  5 bytes - USER32!CreateWindowExW (+0×738)
 [ 8b ff 55 8b ec:e9 d6 03 b6 01 ]
    7e41ff33-7e41ff37  5 bytes - USER32!CreateWindowExA (+0×30e)
Total bytes compared: 389095(100%)
Number of errors: 52
52 errors : !user32 (7e4188a6-7e42e8d5)

The hooking DLL was found to be from that upgraded package:

0:000> u 7e4188a6
USER32!GetWindowLongW:
7e4188a6 e95577a401      jmp     7fe60000
7e4188ab 41              inc     ecx
7e4188ac 7ee8            jle     USER32!_GetWindowLong+0xda (7e418896)
7e4188ae 0e              push    cs
7e4188af fd              std
7e4188b0 ff              ???
7e4188b1 ff8b4d08e816    dec     dword ptr [ebx+16E8084Dh]
7e4188b7 fc              cld

0:000> u 7fe60000
7fe60000 e9bb62b080      jmp     hookA+0×62c0 (009662c0)
7fe60005 6a08            push    8
7fe60007 68e088417e      push    offset USER32!`string’+0×34 (7e4188e0)
7fe6000c e99c885bfe      jmp     USER32!GetWindowLongW+0×7 (7e4188ad)
7fe60011 0000            add     byte ptr [eax],al
7fe60013 0000            add     byte ptr [eax],al
7fe60015 0000            add     byte ptr [eax],al
7fe60017 0000            add     byte ptr [eax],al

0:000> lmv m hookA
start    end        module name
00960000 00976000   hookA     (no symbols)          
    Loaded symbol image file: hookA.dll
    Image path: C:\Program Files\CompanyA\hookA.dll
    Image name: hookA.dll
    […]
    ProductName:      ProductA
    […]

Execution residue from hookA module was also found on the problem thread raw stack and it looks like real code (not a coincidental symbolic information):

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0013f02c
    StackBase:            00140000
    StackLimit:           0010c000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             00000c38 . 00000840
    RpcHandle:            00000000
    Tls Storage:          00163268
    PEB Address:          7ffdb000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 0010c000 00140000
0010c000  00000000
0010c004  00000000
0010c008  00000000
[…]
00121f50  0096d7d1*** ERROR: Module load completed but symbols could not be loaded for hookA.dll
 hookA+0xd7d1
00121f54  00009924
00121f58  00121fb4
00121f5c  00000000
00121f60  00121f9c
00121f64  0096d895 hookA+0xd895
00121f68  00121f78
00121f6c  00000000
00121f70  00122008
00121f74  00000000
00121f78  00000000
00121f7c  00000000
00121f80  00000000
00121f84  00121f6c
00121f88  000001fe
00121f8c  001220fc
00121f90  0096ec98 hookA+0xec98
00121f94  00970e48 hookA+0×10e48
00121f98  00121fb4
00121f9c  7e41b6a3 USER32!_GetClientRect+0×6e
00121fa0  00122008
00121fa4  fffffa01
[…]

0:000> u 0096d895
hookA+0xd895:
0096d895 8bc6            mov     eax,esi
0096d897 8b4df0          mov     ecx,dword ptr [ebp-10h]
0096d89a 64890d00000000  mov     dword ptr fs:[0],ecx
0096d8a1 5f              pop     edi
0096d8a2 5e              pop     esi
0096d8a3 5b              pop     ebx
0096d8a4 8be5            mov     esp,ebp
0096d8a6 5d              pop     ebp

0:000> ub 0096d895
hookA+0xd876:
0096d876 8b65e8          mov     esp,dword ptr [ebp-18h]
0096d879 be0d000000      mov     esi,0Dh
0096d87e c745fcffffffff  mov     dword ptr [ebp-4],0FFFFFFFFh
0096d885 eb05            jmp     hookA+0xd88c (0096d88c)
0096d887 be72000000      mov     esi,72h
0096d88c 8d55dc          lea     edx,[ebp-24h]
0096d88f 52              push    edx
0096d890 e8fbfeffff      call    hookA+0xd790 (0096d790)

As was found the upgraded application had special DLL hooks to improve productivity and ease of use of GUI applications. Fortunately it was possible to disable that hook on a per-application basis and application crashes disappeared.

- Dmitry Vostokov @ DumpAnalysis.org -

The number of visits / month increased by 50% by the end of the year with more than 125,000 unique visitors in 2008 from 180 countries (150 in 2007) and almost 34% of them coming back. Here are the top 100 network locations out of almost 24,000:

Almost 66,000 Google search keywords (more than 100% increase since 2007) pointed to the portal and this blog with 100 most frequent:

Special thanks to 950 web sites that mention the portal and this blog with the first top 100:

Top 25 visiting countries:

- Dmitry Vostokov @ DumpAnalysis.org -

What is the source of our intuition about ∞, or ∞^∞, more powers of ∞, and even ∞ number of powers? I believe that the underlying structure of our Universe or at least a universe as a model of Universe, Infinite Memory, with perceived processes as limits and Time Arrow as a bundle of sequences of memory pointers, provides basis for our intuition about infinite.

- Dmitry Vostokov @ DumpAnalysis.org - 

Listening to étude No. 1 in C major written by Frédéric Chopin (Op. 10) an idea came to me about writing 16 debugging études (ISBN: 978-1906717575). It is surprising that there are many programming études out there but there are no debugging ones. Stay tuned and be in touch with this blog.

Draft definition:

Debugging étude is a composition of software with intentional defects (bugs) of considerable difficulty to find and fix, designed to provide practice material to perfect debugging techniques and problem-solving skills.

The idea actually came to me long time ago to create some sort of debugging excersises for training purposes. 

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org announces forthcoming 2010 as The Year of Dump Analysis.

Q&A 

Q. Why 2010?

A. Two reasons: 1) To do dump analysis effectively and efficiently an engineer needs some experience in debugging acquired in the previous year of debugging (perhaps after 7 debugging nights, 2009, 0×7D9); 2) 2010 is 0×7DA.

Q. What is the meaning of 7?

A. It is interpreted as Dump Analysis 7 days a week. Like what I do. Or from kernel pool tag perspective it is AD7: Analysis of Dumps 7 days a week. 

Q. What about the year 2011, 2012, 2013? 0×7DB, 0×7DC and 0×7DD?

A. Hmm, sounds like WinDbg commands db, dc and dd…

- Dmitry Vostokov @ DumpAnalysis.org -

Last week had some fearetical features. What’s it all about you should wait until my memoirs are published:

Crash Dump: A Software Engineering Autobiography, ISBN: 978-1906717193

If we break down fearetical linguistically we come with the following free morphemes:

fear e tical

According to Wikipedia, the last one is a currency unit subdivided into into 64, 32, 8 and 4. A coin weighing 15g (0xFg). It was replaced by another currency unit, the franc.

- Dmitry Vostokov @ DumpAnalysis.org -

I very proud to announce that after 3 weeks of final work the book has been released in both paperback and PDF format. In a week or so it should also appear on Amazon and other booksellers around the world. The book information and how to buy it can be found on the portal:

Windows Debugging: Practical Foundations

- Dmitry Vostokov @ DumpAnalysis.org -

Draft Table of Contents is available for the forthcoming Windows Debugging: Practical Foundations book to be released next week:

Draft Table of Contents

- Dmitry Vostokov @ DumpAnalysis.org -

What if you are given a universal memory dump and want to find some intelligence artifacts in it? I think the problem is similar to searching for software artifacts in a computer memory dump out of quadrimemorillion of them in the absence of symbol files and suitable memory dump reader. Perhaps memory visualization techniques provide a direction to solving extraterrestrial problems too. This SETI association probably came to my mind when one of the readers of my memory religion post recalled his job application to SETI institute.

- Dmitry Vostokov @ DumpAnalysis.org -

One question that people often ask is to how to simulate a process hang. One method that I found is to attach WinDbg noninvasively, freeze all threads by executing the following command:

~*n

and then quit by using q command. This leaves an application or a service process in a total hang state.

- Dmitry Vostokov @ DumpAnalysis.org -

I noticed yesterday that my home Vista computer suddenly became slower than usual so I brought Task Manager, sorted processes by CPU usage and discovered an instance of IE7 with 50% - 60% of CPU consumption. Dumping processes in Vista is easier than ever, so I did the right click on that process and selected Create Dump File menu option. The dump was saved and I killed the process. The size of the dump file was 1.2Gb and that definitely indicated a memory leak. Examining process heap showed large heap segments amounting to 800Mb and therefore pointing to the possible heap leak:

0:000> !heap 0 0
Index   Address  Name      Debugging options enabled
  1:   00370000
    Segment at 00370000 to 00470000 (00100000 bytes committed)
    Segment at 04990000 to 04a90000 (00100000 bytes committed)
    Segment at 063e0000 to 065e0000 (00200000 bytes committed)
    Segment at 08440000 to 08840000 (00400000 bytes committed)
    Segment at 0ce80000 to 0d680000 (00800000 bytes committed)
    Segment at 160b0000 to 17080000 (00fd0000 bytes committed)
    Segment at 19b00000 to 1aad0000 (00fd0000 bytes committed)
    Segment at 1c8c0000 to 1d890000 (00fd0000 bytes committed)
    Segment at 27870000 to 28840000 (00fd0000 bytes committed)
    Segment at 29870000 to 2a840000 (00fd0000 bytes committed)
    Segment at 2d1f0000 to 2e1c0000 (00fd0000 bytes committed)
    Segment at 31fb0000 to 32f80000 (00fd0000 bytes committed)
    Segment at 384c0000 to 39490000 (00fd0000 bytes committed)
    Segment at 3c040000 to 3d010000 (00fd0000 bytes committed)
    Segment at 41cf0000 to 42cc0000 (00fd0000 bytes committed)
    Segment at 43c90000 to 44c60000 (00fd0000 bytes committed)
    Segment at 44c60000 to 45c30000 (00fd0000 bytes committed)
    Segment at 473f0000 to 483c0000 (00fd0000 bytes committed)
    Segment at 4a390000 to 4b360000 (00fd0000 bytes committed)
    Segment at 4b360000 to 4c330000 (00fd0000 bytes committed)
    Segment at 4d300000 to 4e2d0000 (00fd0000 bytes committed)
    Segment at 4e2d0000 to 4f2a0000 (00fd0000 bytes committed)
    Segment at 50480000 to 51450000 (00fd0000 bytes committed)
    Segment at 51450000 to 52420000 (00fd0000 bytes committed)
    Segment at 533f0000 to 543c0000 (00fd0000 bytes committed)
    Segment at 54810000 to 557e0000 (00fd0000 bytes committed)
    Segment at 567b0000 to 57780000 (00fd0000 bytes committed)
    Segment at 57c80000 to 58c50000 (00fc1000 bytes committed)
    Segment at 59c20000 to 5abf0000 (00fc6000 bytes committed)
    Segment at 5b0f0000 to 5c0c0000 (00fc1000 bytes committed)
    Segment at 5c0c0000 to 5d090000 (00fc1000 bytes committed)
    Segment at 5d090000 to 5e060000 (00fc1000 bytes committed)
    Segment at 5f030000 to 60000000 (00fc1000 bytes committed)
    Segment at 60000000 to 60fd0000 (00fc1000 bytes committed)
    Segment at 60fd0000 to 61fa0000 (00fd0000 bytes committed)
    Segment at 61fa0000 to 62f70000 (00e26000 bytes committed)
  2:   00010000
    Segment at 00010000 to 00020000 (00003000 bytes committed)
  3:   00d80000
    Segment at 00d80000 to 00d90000 (00010000 bytes committed)
    Segment at 00050000 to 00150000 (00014000 bytes committed)
  4:   00190000
    Segment at 00190000 to 001a0000 (00010000 bytes committed)
    Segment at 00d90000 to 00e90000 (00100000 bytes committed)
    Segment at 0a430000 to 0a630000 (00200000 bytes committed)
    Segment at 0d8d0000 to 0dcd0000 (00400000 bytes committed)
    Segment at 0ecc0000 to 0f4c0000 (00800000 bytes committed)
    Segment at 18690000 to 19660000 (00fd0000 bytes committed)
    Segment at 24fe0000 to 25fb0000 (00fd0000 bytes committed)
    Segment at 2bf40000 to 2cf10000 (00fd0000 bytes committed)
    Segment at 303b0000 to 31380000 (00fd0000 bytes committed)
    Segment at 33370000 to 34340000 (00fd0000 bytes committed)
    Segment at 39490000 to 3a460000 (00fd0000 bytes committed)
    Segment at 40d20000 to 41cf0000 (00fd0000 bytes committed)
    Segment at 483c0000 to 49390000 (00fd0000 bytes committed)
    Segment at 557e0000 to 567b0000 (00452000 bytes committed)
  5:   00330000
    Segment at 00330000 to 00340000 (00010000 bytes committed)
    Segment at 00c10000 to 00d10000 (00100000 bytes committed)
    Segment at 0c910000 to 0cb10000 (00200000 bytes committed)
    Segment at 18280000 to 18680000 (00400000 bytes committed)
    Segment at 2ec20000 to 2f420000 (00800000 bytes committed)
    Segment at 42cc0000 to 43c90000 (00fc7000 bytes committed)
    Segment at 4c330000 to 4d300000 (00d45000 bytes committed)
    Segment at 52420000 to 533f0000 (00d39000 bytes committed)
    Segment at 58c50000 to 59c20000 (00ddc000 bytes committed)
    Segment at 5e060000 to 5f030000 (00dd1000 bytes committed)
  6:   00e90000
    Segment at 00e90000 to 00ea0000 (00010000 bytes committed)
    Segment at 06780000 to 06880000 (00026000 bytes committed)
  7:   00170000
    Segment at 00170000 to 00180000 (00010000 bytes committed)
    Segment at 06880000 to 06980000 (00026000 bytes committed)
  8:   01bf0000
    Segment at 01bf0000 to 01c00000 (00010000 bytes committed)
    Segment at 03bb0000 to 03cb0000 (00100000 bytes committed)
    Segment at 0e610000 to 0e810000 (00200000 bytes committed)
  9:   00bf0000
    Segment at 00bf0000 to 00c00000 (00001000 bytes committed)
 10:   00b70000
    Segment at 00b70000 to 00b80000 (00003000 bytes committed)
 11:   01b60000
    Segment at 01b60000 to 01ba0000 (00040000 bytes committed)
 12:   03650000
    Segment at 03650000 to 03690000 (00009000 bytes committed)
 13:   039c0000
    Segment at 039c0000 to 039d0000 (00008000 bytes committed)
    Segment at 07e30000 to 07f30000 (00012000 bytes committed)
 14:   00b20000
    Segment at 00b20000 to 00b30000 (00003000 bytes committed)
 15:   01b00000
    Segment at 01b00000 to 01b40000 (00040000 bytes committed)
    Segment at 22b80000 to 22c80000 (00032000 bytes committed)
 16:   00b30000
    Segment at 00b30000 to 00b70000 (00040000 bytes committed)
    Segment at 08f00000 to 09000000 (00100000 bytes committed)
    Segment at 376f0000 to 378f0000 (000e3000 bytes committed)
 17:   03700000
    Segment at 03700000 to 03740000 (00040000 bytes committed)
 18:   03a70000
    Segment at 03a70000 to 03ab0000 (00040000 bytes committed)
 19:   00be0000
    Segment at 00be0000 to 00bf0000 (00010000 bytes committed)
    Segment at 0a630000 to 0a730000 (000a8000 bytes committed)
 20:   04df0000
    Segment at 04df0000 to 04ef0000 (00100000 bytes committed)
 21:   044d0000
    Segment at 044d0000 to 044e0000 (00010000 bytes committed)
    Segment at 04390000 to 04490000 (00028000 bytes committed)
 22:   04730000
    Segment at 04730000 to 04740000 (00010000 bytes committed)
    Segment at 04620000 to 04720000 (00100000 bytes committed)
    Segment at 23fb0000 to 241b0000 (001f6000 bytes committed)
 23:   055e0000
    Segment at 055e0000 to 056e0000 (00100000 bytes committed)
 24:   05ce0000
    Segment at 05ce0000 to 05cf0000 (00010000 bytes committed)
    Segment at 06bb0000 to 06cb0000 (00012000 bytes committed)
 25:   05e20000
    Segment at 05e20000 to 05e60000 (00020000 bytes committed)
 26:   04860000
    Segment at 04860000 to 04870000 (00010000 bytes committed)
    Segment at 0df60000 to 0e060000 (00024000 bytes committed)
 27:   04dc0000
    Segment at 04dc0000 to 04dd0000 (00010000 bytes committed)
    Segment at 062e0000 to 063e0000 (00100000 bytes committed)
    Segment at 26d70000 to 26f70000 (001eb000 bytes committed)
 28:   06aa0000
    Segment at 06aa0000 to 06ab0000 (00010000 bytes committed)
    Segment at 06980000 to 06a80000 (00100000 bytes committed)
    Segment at 1ede0000 to 1efe0000 (00200000 bytes committed)
    Segment at 1efe0000 to 1f3e0000 (00322000 bytes committed)
    Segment at 1f3e0000 to 1fbe0000 (00800000 bytes committed)
    Segment at 205e0000 to 215b0000 (001c7000 bytes committed)
 29:   05420000
    Segment at 05420000 to 05430000 (00010000 bytes committed)
    Segment at 06ab0000 to 06bb0000 (00053000 bytes committed)
 30:   05980000
    Segment at 05980000 to 05990000 (00010000 bytes committed)
    Segment at 17d90000 to 17e90000 (00012000 bytes committed)
 31:   07c20000
    Segment at 07c20000 to 07c60000 (00040000 bytes committed)
    Segment at 08cc0000 to 08dc0000 (00100000 bytes committed)
    Segment at 1fbe0000 to 1fde0000 (001fd000 bytes committed)
    Segment at 241b0000 to 245b0000 (003fa000 bytes committed)
    Segment at 2a840000 to 2b040000 (0007c000 bytes committed)
 32:   07be0000
    Segment at 07be0000 to 07c20000 (0003a000 bytes committed)
    Segment at 17900000 to 17a00000 (000fd000 bytes committed)
    Segment at 3b2b0000 to 3b4b0000 (001fe000 bytes committed)
    Segment at 45c30000 to 46030000 (00289000 bytes committed)
 33:   07df0000
    Segment at 07df0000 to 07e30000 (0003a000 bytes committed)
    Segment at 22810000 to 22910000 (0001c000 bytes committed)
 34:   08000000
    Segment at 08000000 to 08040000 (00001000 bytes committed)
 35:   07da0000
    Segment at 07da0000 to 07de0000 (00001000 bytes committed)
 36:   04b60000
    Segment at 04b60000 to 04b70000 (00002000 bytes committed)
 37:   08990000
    Segment at 08990000 to 089a0000 (00010000 bytes committed)
    Segment at 06cb0000 to 06db0000 (00024000 bytes committed)
 38:   051f0000
    Segment at 051f0000 to 05200000 (00010000 bytes committed)
    Segment at 050c0000 to 051c0000 (00100000 bytes committed)
    Segment at 0c110000 to 0c310000 (00200000 bytes committed)
    Segment at 0c310000 to 0c710000 (003f6000 bytes committed)
    Segment at 1bd00000 to 1c500000 (00529000 bytes committed)
    Segment at 216c0000 to 22690000 (00376000 bytes committed)
 39:   0ac10000
    Segment at 0ac10000 to 0ac20000 (00010000 bytes committed)
    Segment at 0aa80000 to 0ab80000 (000c4000 bytes committed)
 40:   12ed0000
    Segment at 12ed0000 to 12ee0000 (00010000 bytes committed)
    Segment at 199e0000 to 19ae0000 (00022000 bytes committed)
 41:   15450000
    Segment at 15450000 to 15490000 (00001000 bytes committed)
 42:   17ad0000
    Segment at 17ad0000 to 17b10000 (00001000 bytes committed)
 43:   1b2f0000
    Segment at 1b2f0000 to 1b300000 (00010000 bytes committed)
    Segment at 1ad30000 to 1ae30000 (0002c000 bytes committed)
 44:   232b0000
    Segment at 232b0000 to 232f0000 (00015000 bytes committed)
 45:   21680000
    Segment at 21680000 to 216c0000 (00001000 bytes committed)
 46:   23490000
    Segment at 23490000 to 234d0000 (00001000 bytes committed)
 47:   23670000
    Segment at 23670000 to 236b0000 (00001000 bytes committed)
 48:   17ed0000
    Segment at 17ed0000 to 17f10000 (00001000 bytes committed)
 49:   247f0000
    Segment at 247f0000 to 24830000 (00040000 bytes committed)
 50:   28c40000
    Segment at 28c40000 to 28c80000 (00040000 bytes committed)
 51:   2ffd0000
    Segment at 2ffd0000 to 2ffe0000 (00006000 bytes committed)
 52:   376b0000
    Segment at 376b0000 to 376f0000 (00040000 bytes committed)
 53:   2ff90000
    Segment at 2ff90000 to 2ffd0000 (00040000 bytes committed)
 54:   26260000
    Segment at 26260000 to 262a0000 (00040000 bytes committed)
 55:   3a530000
    Segment at 3a530000 to 3a570000 (00040000 bytes committed)

However I concentrated on CPU spike and !runaway WinDbg command showed the following distribution of thread user mode times:

0:000> !runaway
 User Mode Time
  Thread       Time
 117:10a0      0 days 3:09:13.643
  13:ca4       0 days 2:18:41.311
  61:16c4      0 days 0:25:46.515
  33:1690      0 days 0:25:25.954
   4:fb0       0 days 0:22:20.797
  29:840       0 days 0:21:25.385
  23:1614      0 days 0:21:08.194
  77:3e0       0 days 0:18:57.434
  45:11f4      0 days 0:17:13.647
  71:1314      0 days 0:17:10.667
  31:1198      0 days 0:16:48.374
  39:156c      0 days 0:16:40.980
  59:d1c       0 days 0:16:37.610
 115:3e8       0 days 0:16:32.384
  57:170c      0 days 0:16:30.746
  47:1364      0 days 0:16:18.360
  84:12a8      0 days 0:15:56.145
 112:a10       0 days 0:15:52.089
 106:1374      0 days 0:15:51.652
  89:b58       0 days 0:15:47.768
 125:115c      0 days 0:15:41.122
 101:1100      0 days 0:15:30.748
 104:1294      0 days 0:15:16.147
  99:d00       0 days 0:15:15.008
  96:9b4       0 days 0:15:13.604
 123:1624      0 days 0:15:12.247
  86:1444      0 days 0:15:11.654
 131:1728      0 days 0:14:35.914
 135:100c      0 days 0:14:16.414
 133:1530      0 days 0:14:04.963
 137:a30       0 days 0:13:41.360
 139:dd8       0 days 0:13:40.674
 142:1098      0 days 0:12:51.284
   0:efc       0 days 0:02:43.005
   1:f44       0 days 0:01:34.536
  19:8d0       0 days 0:00:42.557
  98:54c       0 days 0:00:28.282
 114:138c      0 days 0:00:26.598
  83:1060      0 days 0:00:22.354
  88:17ec      0 days 0:00:22.027
 103:da8       0 days 0:00:20.404
 141:15c8      0 days 0:00:19.843
  10:b14       0 days 0:00:12.526
   8:5b8       0 days 0:00:02.246
  21:cfc       0 days 0:00:00.795
  12:10c       0 days 0:00:00.561
  11:8d4       0 days 0:00:00.312
  65:b0c       0 days 0:00:00.202
  22:ae8       0 days 0:00:00.187
  17:744       0 days 0:00:00.124
  28:168c      0 days 0:00:00.093
   6:5a8       0 days 0:00:00.046
   2:f90       0 days 0:00:00.031
 130:fa4       0 days 0:00:00.015
 113:17c4      0 days 0:00:00.015
  76:1a4       0 days 0:00:00.015
  70:10a8      0 days 0:00:00.015
  32:df0       0 days 0:00:00.015
  18:ee0       0 days 0:00:00.015
   7:3f4       0 days 0:00:00.015
 148:11cc      0 days 0:00:00.000
 147:132c      0 days 0:00:00.000
 146:1458      0 days 0:00:00.000
 145:133c      0 days 0:00:00.000
 144:1268      0 days 0:00:00.000
 143:838       0 days 0:00:00.000
 140:1168      0 days 0:00:00.000
 138:f48       0 days 0:00:00.000
 136:1f0       0 days 0:00:00.000
 134:17ac      0 days 0:00:00.000
 132:119c      0 days 0:00:00.000
 129:fc4       0 days 0:00:00.000
 128:bd8       0 days 0:00:00.000
 127:1528      0 days 0:00:00.000
 126:1058      0 days 0:00:00.000
 124:16a4      0 days 0:00:00.000
 122:1518      0 days 0:00:00.000
 121:7c        0 days 0:00:00.000
 120:103c      0 days 0:00:00.000
 119:a2c       0 days 0:00:00.000
 118:1524      0 days 0:00:00.000
 116:1240      0 days 0:00:00.000
 111:1248      0 days 0:00:00.000
 110:de8       0 days 0:00:00.000
 109:dc8       0 days 0:00:00.000
 108:17e8      0 days 0:00:00.000
 107:994       0 days 0:00:00.000
 105:162c      0 days 0:00:00.000
 102:112c      0 days 0:00:00.000
 100:1764      0 days 0:00:00.000
  97:1548      0 days 0:00:00.000
  95:1334      0 days 0:00:00.000
  94:1024      0 days 0:00:00.000
  93:1170      0 days 0:00:00.000
  92:12f0      0 days 0:00:00.000
  91:12d4      0 days 0:00:00.000
  90:1264      0 days 0:00:00.000
  87:12d8      0 days 0:00:00.000
  85:153c      0 days 0:00:00.000
  82:14c4      0 days 0:00:00.000
  81:834       0 days 0:00:00.000
  80:17f4      0 days 0:00:00.000
  79:1784      0 days 0:00:00.000
  78:530       0 days 0:00:00.000
  75:1320      0 days 0:00:00.000
  74:15fc      0 days 0:00:00.000
  73:16e4      0 days 0:00:00.000
  72:17b0      0 days 0:00:00.000
  69:af0       0 days 0:00:00.000
  68:83c       0 days 0:00:00.000
  67:b78       0 days 0:00:00.000
  66:cc4       0 days 0:00:00.000
  64:14fc      0 days 0:00:00.000
  63:14dc      0 days 0:00:00.000
  62:16b0      0 days 0:00:00.000
  60:1130      0 days 0:00:00.000
  58:1504      0 days 0:00:00.000
  56:1160      0 days 0:00:00.000
  55:16c0      0 days 0:00:00.000
  54:bfc       0 days 0:00:00.000
  53:f70       0 days 0:00:00.000
  52:1178      0 days 0:00:00.000
  51:1448      0 days 0:00:00.000
  50:15e8      0 days 0:00:00.000
  49:1410      0 days 0:00:00.000
  48:10c0      0 days 0:00:00.000
  46:14e4      0 days 0:00:00.000
  44:1150      0 days 0:00:00.000
  43:1454      0 days 0:00:00.000
  42:131c      0 days 0:00:00.000
  41:8cc       0 days 0:00:00.000
  40:17bc      0 days 0:00:00.000
  38:17c0      0 days 0:00:00.000
  37:15a4      0 days 0:00:00.000
  36:1048      0 days 0:00:00.000
  35:143c      0 days 0:00:00.000
  34:1384      0 days 0:00:00.000
  30:fa0       0 days 0:00:00.000
  27:1688      0 days 0:00:00.000
  26:1684      0 days 0:00:00.000
  25:1680      0 days 0:00:00.000
  24:161c      0 days 0:00:00.000
  20:500       0 days 0:00:00.000
  16:1a0       0 days 0:00:00.000
  15:a18       0 days 0:00:00.000
  14:c44       0 days 0:00:00.000
   9:6c4       0 days 0:00:00.000
   5:ec8       0 days 0:00:00.000
   3:fa8       0 days 0:00:00.000

Threads 117 and 13 were waiting for a critical section 6e1876c4:

0:000> ~117kv
ChildEBP RetAddr  Args to Child             
35f0e468 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet
35f0e46c 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
35f0e4d0 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155
35f0e4f8 6e16ac32 6e1876c4 00071370 35f0e59c ntdll!RtlEnterCriticalSection+0x152
35f0e510 6e16b4cc 6e16e2f1 00000000 35f0e59c AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22
35f0e524 6e16bb90 00071370 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3
35f0e544 6e16bbce 00071370 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a
35f0e574 6e16e327 00071370 00000002 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for PDFCreator_Toolbar.dll -
35f0e620 05729772 00071370 00000002 35f0e690 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
35f0e6a4 76b60528 c02193db 00000128 00000000 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
35f0e6dc 73207be1 000319f8 00000128 00030001 user32!DefWindowProcW+0×86
76b60528 90909090 fffffffe 00000000 ffffffd0 comctl32!ToolbarWndProc+0×14f7
76b60528 00000000 fffffffe 00000000 ffffffd0 0×90909090

0:000> ~13kv
ChildEBP RetAddr  Args to Child             
0c90e5ec 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet
0c90e5f0 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
0c90e654 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155
0c90e67c 6e16ac32 6e1876c4 00071348 0c90e720 ntdll!RtlEnterCriticalSection+0x152
0c90e694 6e16b4cc 6e16e2f1 00000000 0c90e720 AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22
0c90e6a8 6e16bb90 00071348 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3
0c90e6c8 6e16bbce 00071348 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a
0c90e6f8 6e16e327 00071348 0000000c 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26
0c90e7a4 05729772 00071348 0000000c 0c90e814 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
0c90e858 76b60528 73207be1 000205e2 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
0c90e8d4 76b5f8d2 626f6441 44502065 00200046 user32!DefWindowProcW+0×86
0c90e978 76b60817 0041fecc 73207ae0 000205e2 user32!InternalCallWinProc+0×23
00030ad4 0031002e 00300038 00350036 006e005f user32!DispatchClientMessage+0xda
00030ad4 00000000 00300038 00350036 006e005f 0×31002e

Examining critical section locks showed this section to be the only one locked and having high contention:

0:000> !locks

CritSec AcRedir!NS_RedirectRegistry::RedirectorRegistry::ClassLock+0 at 6e1876c4
WaiterWoken        No
LockCount          32
RecursionCount     1
OwningThread       d1c
EntryCount         0
ContentionCount    c74ad4
*** Locked

Scanned 22054 critical sections

There were 32 threads waiting on it. Examining its owning thread d1c showed similar stack trace pattern:

0:000> ~~[d1c]kv
ChildEBP RetAddr  Args to Child             
269ae72c 6e16f1da 269ae808 31f4a7e8 269ae75c AcRedir!NS_RedirectRegistry::OwnedRegistryKeyPair::Match+0×14
269ae73c 6e16f40c 269ae7ec 269ae808 269ae808 AcRedir!NS_RedirectRegistry::MergedRegistryKey::Match+0×22
269ae75c 6e16bc11 269ae7ec 269ae808 269ae784 AcRedir!NS_RedirectRegistry::MergedRegistryKeyList::FindItem+0×25
269ae790 6e16e327 00c211b0 00000008 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×69
269ae83c 05729772 000714a4 00000008 269ae8ac AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
269ae8f0 76b60528 73207be1 00050cf8 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
269ae96c 76b5f8d2 00000001 00070598 00040582 user32!DefWindowProcW+0×86
269aea10 76b60817 0041fecc 73207ae0 00050cf8 user32!InternalCallWinProc+0×23
00030ad4 0031002e 00300038 00350036 006e005f user32!DispatchClientMessage+0xda
00030ad4 00000000 00300038 00350036 006e005f 0×31002e

Two components immediately came to suspicion, AcRedir.dll and PDFCreator_Toolbar.dll:

0:000> lmv m AcRedir
start    end        module name
6e150000 6e18e000   AcRedir    (pdb symbols)          c:\mss\AcRedir.pdb\923AF38F594246C99580DC1CFB4B3AE02\AcRedir.pdb
    Loaded symbol image file: AcRedir.dll
    Image path: C:\Windows\AppPatch\AcRedir.dll
    Image name: AcRedir.dll
    Timestamp:        Sat Jan 19 07:26:39 2008 (4791A62F)
    CheckSum:         0003F278
    ImageSize:        0003E000
    File version:     6.0.6001.18000
    Product version:  6.0.6001.18000
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     Microsoft® Windows® Operating System
    OriginalFilename: Microsoft® Windows® Operating System
    ProductVersion:   6.0.6001.18000
    FileVersion:      6.0.6001.18000 (longhorn_rtm.080118-1840)
    FileDescription:  Windows Compatibility DLL
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

0:000> lmv m PDFCreator_Toolbar
start    end        module name
056e0000 057bb000   PDFCreator_Toolbar   (export symbols)       PDFCreator_Toolbar.dll
    Loaded symbol image file: PDFCreator_Toolbar.dll
    Image path: C:\Program Files\PDFCreator Toolbar\v3.3.0.1\PDFCreator_Toolbar.dll
    Image name: PDFCreator_Toolbar.dll
    Timestamp:        Sat Aug 09 08:53:38 2008 (489D4D02)
    CheckSum:         000AA334
    ImageSize:        000DB000
    File version:     3.3.0.1
    Product version:  3.3.0.1
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    ProductName:      PDFCreator Toolbar
    InternalName:     PDFCreator Toolbar
    OriginalFilename: Toolbar.dll
    ProductVersion:   3,3,0,1
    FileVersion:      3,3,0,1
    FileDescription:  PDFCreator Toolbar
    LegalCopyright:   Copyright 2006

Then I decided to examine some heap blocks from leaked segments and found the prolifiration of UNICODE string fragments containing “PDFCreator Toolbar”:

0:000> dc 567b0000 l3000
[...]
567b21a0  00000001 00000008 00000040 00650054  ............T.e.
567b21b0  0070006d 00610044 00610074 00730000  m.p.D.a.t.a...s.
567b21c0  00740069 006f0069 0000006e 00000072  i.t.i.o.n...r...
567b21d0  00000068 005c0067 00440050 00430046  h…g.\.P.D.F.C.
567b21e0  00650072 00740061 0072006f 00540020  r.e.a.t.o.r. .T.
567b21f0  006f006f 0062006c 00720061 002d0000  o.o.l.b.a.r…-.
567b2200  00300031 00300030 00300000 00420025  1.0.0.0…0.%.B.
567b2210  00250030 00310044 00380025 00250031  0.%.D.1.%.8.1.%.
567b2220  00310044 00380025 00000031 00000000  D.1.%.8.1…….
567b2230  52332e04 88000000 00000001 00000013  ..3R…………
567b2240  00000040 00690044 00450064 0061006e  @…D.i.d.E.n.a.
567b2250  006c0062 00410065 00740075 0053006f  b.l.e.A.u.t.o.S.
567b2260  00610065 00630072 00000068 005c0067  e.a.r.c.h…g.\.
567b2270  00440050 00430046 00650072 00740061  P.D.F.C.r.e.a.t.
567b2280  0072006f 00540020 006f006f 0062006c  o.r. .T.o.o.l.b.
567b2290  00720061 002d0000 00300031 00300030  a.r…-.1.0.0.0.
567b22a0  00300000 00420025 00250030 00310044  ..0.%.B.0.%.D.1.
567b22b0  00380025 00250031 00310044 00380025  %.8.1.%.D.1.%.8.
567b22c0  00000031 00000000 52332e1b 88000000  1………3R….
567b22d0  00000001 00000005 00000040 004c0053  …………S.L.
567b22e0  00730069 00000074 00450052 0070005c  i.s.t…R.E.\.p.
567b22f0  00660064 006f0066 00670072 002e0065  d.f.f.o.r.g.e…
567b2300  0072006f 005c0067 00440050 00430046  o.r.g.\.P.D.F.C.
567b2310  00650072 00740061 0072006f 00540020  r.e.a.t.o.r. .T.
567b2320  006f006f 0062006c 00720061 00300000  o.o.l.b.a.r…0.
567b2330  00420025 00250042 00300044 00420025  %.B.B.%.D.0.%.B.
567b2340  00250030 00310044 00380025 00250031  0.%.D.1.%.8.1.%.
567b2350  00310044 00380025 00000031 00000000  D.1.%.8.1…….
567b2360  52332e2e 88000000 00000001 00000005  ..3R…………

Because AcRedir code was called from PDFCreator Toolbar component the final decision was to uninstall PDFCreator Toolbar. Before I quit the debugger I dumped the list of modules and was astonished at the module variety:

0:000> lm
start    end        module name
00850000 008eb000   iexplore   (pdb symbols)         
05430000 054ab000   ssv        (deferred)            
056e0000 057bb000   PDFCreator_Toolbar   (export symbols)
0a7f0000 0aa73000   igdumd32   (deferred)            
10000000 100a4000   swg        (deferred)            
16080000 160a5000   mdnsNSP    (deferred)            
28f90000 28f9a000   icalogon   (deferred)            
29330000 29337000   PScript    (deferred)            
29440000 29462000   ctxmui     (deferred)            
29470000 29476000   TcpPServ   (deferred)            
29480000 29492000   CgpCore    (deferred)            
295a0000 295b0000   confmgr    (deferred)            
295b0000 295b7000   logging    (deferred)            
296c0000 296c7000   icafile    (deferred)            
296d0000 296d6000   cgpcfg     (deferred)            
296e0000 296e5000   ctxmuiUI   (deferred)            
2bd20000 2bd8e000   Wfica      (deferred)            
30000000 303ae000   Flash9e    (export symbols)
63f00000 63f0c000   mscorie    (deferred)            
655e0000 65639000   rpbrowserrecordplugin   (deferred)            
692a0000 69a66000   wmploc     (deferred)            
69a70000 6a4a0000   wmp        (deferred)            
6b220000 6b2e3000   VGX        (deferred)            
6b2f0000 6b3bc000   d3dim700   (deferred)            
6b450000 6b4d3000   AdobeUpdater   (deferred)            
6b4e0000 6b7de000   agcore     (deferred)            
6b7e0000 6ba63000   fastsearch_219B3E1547538286   (deferred)            
6ba70000 6be17000   GoogleToolbarDynamic_F423308312A7B033   (export symbols)   
6be20000 6be89000   vbscript   (deferred)            
6bf90000 6c302000   mshtml     (export symbols)   
6c320000 6c36a000   ntshrui    (deferred)            
6c3d0000 6c447000   mshtmled   (deferred)            
6c4d0000 6c527000   dxtmsft    (deferred)            
6c590000 6c60d000   jscript    (pdb symbols)         
6c610000 6c649000   dxtrans    (pdb symbols)         
6c770000 6c7e0000   dsound     (deferred)            
6c810000 6c839000   msls31     (deferred)            
6ca50000 6ca6b000   cryptnet   (deferred)            
6ca90000 6cada000   rasapi32   (deferred)            
6cca0000 6ccb4000   rasman     (deferred)            
6cd40000 6cd71000   tapi32     (deferred)            
6d0b0000 6d0fc000   Wpc        (deferred)            
6d350000 6d410000   npctrl     (deferred)            
6d450000 6d482000   iepeers    (deferred)            
6d4d0000 6d530000   ieapfltr   (deferred)            
6d5b0000 6d603000   AcroIEFavClient   (deferred)            
6d6b0000 6d795000   ddraw      (deferred)            
6d7a0000 6d818000   AcSpecfc   (deferred)            
6d820000 6d82e000   pngfilt    (deferred)            
6d830000 6d892000   mscms      (deferred)            
6dbc0000 6dc5b000   msvcr80    (deferred)            
6dc60000 6dce7000   msvcp80    (deferred)            
6dd70000 6ddf8000   AcLayers   (deferred)            
6de00000 6de0a000   ddrawex    (deferred)            
6de60000 6de83000   msvfw32    (deferred)            
6dfb0000 6dfc1000   AcroIEHelperShim   (deferred)            
6dff0000 6e036000   GoogleToolbar   (deferred)            
6e060000 6e086000   dssenh     (deferred)            
6e090000 6e0f0000   tiptsf     (deferred)            
6e0f0000 6e11f000   ieui       (pdb symbols)         
6e130000 6e140000   AcroIEHelper   (deferred)            
6e150000 6e18e000   AcRedir    (pdb symbols)         
6e570000 6e57b000   msimtf     (deferred)            
6e580000 6e58f000   davclnt    (deferred)            
6e590000 6e5a3000   ntlanman   (deferred)            
6e610000 6e618000   drprov     (deferred)            
6e620000 6e630000   iebrshim   (deferred)            
6e650000 6e680000   mlang      (deferred)            
6f7b0000 6f7b8000   dispex     (deferred)            
6f8a0000 6f8ab000   cscapi     (deferred)            
6fa70000 6fb4c000   dbghelp    (deferred)            
6fe40000 6fe73000   msrating   (deferred)            
6ff00000 6ff3a000   sqlite     (deferred)            
70530000 70afe000   ieframe    (pdb symbols)      
71260000 71462000   msi        (deferred)            
717c0000 717d2000   pnrpnsp    (deferred)            
71870000 71877000   wsock32    (deferred)            
718a0000 718a7000   msiltcfg   (pdb symbols)      
71920000 71973000   actxprxy   (deferred)            
71980000 7198c000   wshbth     (deferred)            
71990000 71998000   winrnr     (deferred)            
719e0000 71b06000   msxml3     (deferred)            
71b10000 71b1f000   NapiNSP    (deferred)            
71b20000 71b29000   linkinfo   (deferred)            
71c70000 71c76000   SensApi    (deferred)            
71d10000 71e56000   browseui   (deferred)            
71ee0000 71fe7000   shdocvw    (deferred)            
72100000 72109000   snmpapi    (deferred)            
72580000 725c2000   winspool   (deferred)            
725d0000 725d6000   rasadhlp   (deferred)            
72610000 72615000   sfc        (deferred)            
72620000 7262c000   dwmapi     (deferred)            
72640000 72676000   mfplat     (deferred)            
72850000 72857000   midimap    (deferred)            
72860000 72874000   msacm32_72860000   (deferred)            
72880000 72933000   WindowsCodecs   (deferred)            
72940000 729a6000   AudioEng   (deferred)            
729b0000 729d1000   AudioSes   (deferred)            
729e0000 72a0f000   wdmaud     (pdb symbols)      
72a50000 72a59000   msacm32    (deferred)            
72a60000 72a64000   ksuser     (deferred)            
72a70000 72aa2000   winmm      (pdb symbols)      
72b60000 72b6c000   imgutil    (deferred)            
72b80000 72b8d000   sfc_os     (deferred)            
72b90000 72bae000   shimeng    (deferred)            
72bb0000 72bb6000   dciman32   (deferred)            
72c60000 72c8f000   xmllite    (deferred)            
72c90000 72c9c000   rtutils    (deferred)            
72ed0000 72f66000   FWPUCLNT   (deferred)            
73080000 731ca000   msxml6     (deferred)            
731d0000 731d5000   msimg32    (deferred)            
73200000 73285000   comctl32   (pdb symbols)      
73290000 732ef000   winhttp    (deferred)            
73380000 733b9000   oleacc     (deferred)            
733c0000 733ff000   uxtheme    (deferred)            
73400000 73430000   duser      (deferred)            
73430000 735db000   GdiPlus    (deferred)            
738d0000 7398b000   propsys    (deferred)            
74460000 74474000   atl        (deferred)            
74580000 7471e000   comctl32_74580000   (pdb symbols)
74890000 748b7000   MMDevAPI   (deferred)            
74960000 74975000   cabinet    (deferred)            
74980000 749ad000   wintrust   (deferred)            
74a40000 74a4f000   nlaapi     (deferred)            
74a50000 74a5a000   wtsapi32   (deferred)            
74b10000 74b15000   WSHTCPIP   (deferred)            
74b20000 74b27000   avrt       (deferred)            
74b30000 74b4a000   powrprof   (deferred)            
74b50000 74b71000   ntmarta    (deferred)            
74bb0000 74beb000   rsaenh     (deferred)            
74c20000 74c64000   schannel   (deferred)            
74dd0000 74de5000   gpapi      (deferred)            
74ed0000 74f0b000   mswsock    (pdb symbols)         
74f10000 74f55000   bcrypt     (deferred)            
74f60000 74f95000   ncrypt     (deferred)            
74fb0000 74fd1000   dhcpcsvc6   (deferred)            
74fe0000 74fe7000   winnsi     (deferred)            
74ff0000 75025000   dhcpcsvc   (deferred)            
75030000 75049000   IPHLPAPI   (deferred)            
75050000 75090000   wevtapi    (deferred)            
75090000 750ca000   SLC        (deferred)            
750d0000 751c1000   crypt32    (deferred)            
75200000 75214000   mpr        (deferred)            
75260000 75265000   wship6     (deferred)            
75270000 75278000   version    (deferred)            
75280000 75287000   credssp    (deferred)            
752c0000 752d2000   msasn1     (deferred)            
752e0000 752f1000   samlib     (deferred)            
75300000 7532c000   dnsapi     (deferred)            
75360000 753d5000   netapi32   (deferred)            
755a0000 755ff000   sxs        (deferred)            
75660000 7568c000   apphelp    (deferred)            
756c0000 756d4000   secur32    (deferred)            
756e0000 756fe000   userenv    (deferred)            
75820000 75865000   iertutil   (deferred)            
75870000 76380000   shell32    (deferred)            
76380000 763ca000   Wldap32    (deferred)            
763d0000 76428000   shlwapi    (deferred)            
76430000 764b4000   clbcatq    (deferred)            
764c0000 76588000   msctf      (deferred)            
76590000 765b9000   imagehlp   (deferred)            
765c0000 76682000   rpcrt4     (pdb symbols)        
76690000 76760000   wininet    (pdb symbols)        
76760000 767ab000   gdi32      (deferred)            
767b0000 767dd000   ws2_32     (pdb symbols)        
767e0000 76924000   ole32      (pdb symbols)        
76930000 76aba000   setupapi   (deferred)            
76ac0000 76b33000   comdlg32   (deferred)            
76b40000 76bdd000   user32     (pdb symbols)        
76be0000 76cbb000   kernel32   (pdb symbols)    
76cc0000 76d3d000   usp10      (deferred)            
76d40000 76dea000   msvcrt     (pdb symbols)    
76df0000 76f19000   urlmon     (deferred)            
76f20000 76fad000   oleaut32   (deferred)            
76fb0000 770d7000   ntdll      (pdb symbols)    
770e0000 770e7000   psapi      (deferred)            
770f0000 770f9000   lpk        (deferred)            
77100000 7711e000   imm32      (deferred)            
77120000 77126000   nsi        (deferred)            
77130000 77133000   normaliz   (deferred)            
77140000 77206000   advapi32   (deferred)            
79000000 79046000   mscoree    (deferred)            
7c340000 7c396000   msvcr71    (deferred)            
7c3a0000 7c41b000   msvcp71    (deferred)

Actually, before I quit the debugger, I saved a secured stripped version of the dump file using this command: 

0:000> .dump /mrRFt c:\UserDumps\ie7_pattern_cooperation.dmp

The dump file is available on ftp:

ftp://dumpanalysis.org/pub/ie7_pattern_cooperation.zip

Thread times and stack traces are available in it together with module information. However heap data and critical section list was not included in it.

- Dmitry Vostokov @ DumpAnalysis.org -

Applying  a mathematical definition of a memory dump to natural systems we can introduce:

Universal Memory Dump

    - a snapshot of observables describing the system.

Similar to software memory dump analysis we need a suitable reader and a set of:

Universal Symbol Files

    - semantical mappings or NDB (Nature Data Base) files.

Therefore we have these two categories of universal memory dumps:

• - Natural Memory Dumps
• - Software Memory Dumps 

- Dmitry Vostokov @ DumpAnalysis.org -

According to Hex Blog IDA v5.4 will be able to do user and kernel local and remote mode debugging on Windows automatically loading PDB files if necessary. I’m now considering to spend a portion of my book author compensation on purchasing a licence for IDA Pro

- Dmitry Vostokov @ DumpAnalysis.org -