An idea struck me today while I was walking in People’s Park near Dun Laoghaire to formalize various effective intuitive notions in memory dump analysis, debugging and troubleshooting using topos theory. More on this later.
- Dmitry Vostokov @ DumpAnalysis.org -
Noticed today on Amazon that my book becomes more expensive after being used:
- Dmitry Vostokov @ DumpAnalysis.org -
I remember in my old days of PDP-11 system programming I learnt about the system call to spawn processes and wrote a program in assembly language that was spawning itself. This recursive spawning resulted in geometrical progression of running tasks and brought RSX-11M system to halt very quickly. Recently I observed the similar but non-recursive Process Factory pattern in one of memory dumps: explorer was relentlessly creating application.exe processes and at the time some effect was noticed there were more than 5,000 of them:
1: kd> !vm
[...]
5d20 application.exe 212 ( 848 Kb)
5d08 application.exe 212 ( 848 Kb)
5d04 application.exe 212 ( 848 Kb)
5cf8 application.exe 212 ( 848 Kb)
5cf0 application.exe 212 ( 848 Kb)
5ce8 application.exe 212 ( 848 Kb)
5cdc application.exe 212 ( 848 Kb)
5ccc application.exe 212 ( 848 Kb)
5cc8 application.exe 212 ( 848 Kb)
5cc0 application.exe 212 ( 848 Kb)
5ca8 application.exe 212 ( 848 Kb)
5c9c application.exe 212 ( 848 Kb)
5c98 application.exe 212 ( 848 Kb)
5c90 application.exe 212 ( 848 Kb)
5c88 application.exe 212 ( 848 Kb)
5c7c application.exe 212 ( 848 Kb)
5c70 application.exe 212 ( 848 Kb)
5c68 application.exe 212 ( 848 Kb)
5c64 application.exe 212 ( 848 Kb)
5c60 application.exe 212 ( 848 Kb)
5c50 application.exe 212 ( 848 Kb)
5c4c application.exe 212 ( 848 Kb)
5c44 application.exe 212 ( 848 Kb)
5c3c application.exe 212 ( 848 Kb)
5c34 application.exe 212 ( 848 Kb)
5c2c application.exe 212 ( 848 Kb)
5c24 application.exe 212 ( 848 Kb)
5c1c application.exe 212 ( 848 Kb)
5bf8 application.exe 212 ( 848 Kb)
5be0 application.exe 212 ( 848 Kb)
5bd4 application.exe 212 ( 848 Kb)
5bd0 application.exe 212 ( 848 Kb)
5ba4 application.exe 212 ( 848 Kb)
5b58 application.exe 212 ( 848 Kb)
5b50 application.exe 212 ( 848 Kb)
5b44 application.exe 212 ( 848 Kb)
5b38 application.exe 212 ( 848 Kb)
5b30 application.exe 212 ( 848 Kb)
5b04 application.exe 212 ( 848 Kb)
5af4 application.exe 212 ( 848 Kb)
5ad8 application.exe 212 ( 848 Kb)
5ad4 application.exe 212 ( 848 Kb)
5ac8 application.exe 212 ( 848 Kb)
5ac4 application.exe 212 ( 848 Kb)
5ab4 application.exe 212 ( 848 Kb)
5aa4 application.exe 212 ( 848 Kb)
5a9c application.exe 212 ( 848 Kb)
5a94 application.exe 212 ( 848 Kb)
5a8c application.exe 212 ( 848 Kb)
5a88 application.exe 212 ( 848 Kb)
5a74 application.exe 212 ( 848 Kb)
[...]
1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8b57f020 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: cffb3020 ObjectTable: e1003da0 HandleCount: 3932.
Image: System
PROCESS 8a9f8d88 SessionId: none Cid: 01b8 Peb: 7ffdf000 ParentCid: 0004
DirBase: cffb3040 ObjectTable: e13e3f68 HandleCount: 111.
Image: smss.exe
PROCESS 89f0d508 SessionId: 0 Cid: 01f0 Peb: 7ffd8000 ParentCid: 01b8
DirBase: cffb3060 ObjectTable: e16bc370 HandleCount: 1292.
Image: csrss.exe
PROCESS 89eea7c8 SessionId: 0 Cid: 0208 Peb: 7ffde000 ParentCid: 01b8
DirBase: cffb3080 ObjectTable: e14b4160 HandleCount: 564.
Image: winlogon.exe
[...]
PROCESS 8607c020 SessionId: 1 Cid: 44c8 Peb: 7ffdc000 ParentCid: 4cf8
DirBase: cffb7080 ObjectTable: e3c9fd38 HandleCount: 25407.
Image: explorer.exe
[...]
PROCESS 85e1d020 SessionId: 1 Cid: 538c Peb: 7ffda000 ParentCid: 44c8
DirBase: cffb8980 ObjectTable: e8065b20 HandleCount: 39.
Image: application.exe
PROCESS 85c74610 SessionId: 1 Cid: 5394 Peb: 7ffd9000 ParentCid: 44c8
DirBase: cffb89a0 ObjectTable: e6951878 HandleCount: 39.
Image: application.exe
PROCESS 85c81020 SessionId: 1 Cid: 53a4 Peb: 7ffd7000 ParentCid: 44c8
DirBase: cffb89c0 ObjectTable: e6d2f600 HandleCount: 39.
Image: application.exe
PROCESS 85c6fb18 SessionId: 1 Cid: 53a8 Peb: 7ffd7000 ParentCid: 44c8
DirBase: cffb89e0 ObjectTable: e54df078 HandleCount: 39.
Image: application.exe
PROCESS 85c60020 SessionId: 1 Cid: 53bc Peb: 7ffdf000 ParentCid: 44c8
DirBase: cffb8a40 ObjectTable: e1214e90 HandleCount: 39.
Image: application.exe
PROCESS 85c5d380 SessionId: 1 Cid: 53c8 Peb: 7ffde000 ParentCid: 44c8
DirBase: cffb8a60 ObjectTable: e7baf638 HandleCount: 39.
Image: application.exe
PROCESS 85c648b8 SessionId: 1 Cid: 53dc Peb: 7ffde000 ParentCid: 44c8
DirBase: cffb8a80 ObjectTable: e759d060 HandleCount: 39.
Image: application.exe
PROCESS 85c62528 SessionId: 1 Cid: 53e0 Peb: 7ffde000 ParentCid: 44c8
DirBase: cffb8aa0 ObjectTable: e3b8fa00 HandleCount: 39.
Image: application.exe
PROCESS 85c59d88 SessionId: 1 Cid: 53e8 Peb: 7ffdc000 ParentCid: 44c8
DirBase: cffb8ac0 ObjectTable: e31751e0 HandleCount: 39.
Image: application.exe
PROCESS 85c46d88 SessionId: 1 Cid: 542c Peb: 7ffd5000 ParentCid: 4d9c
DirBase: cffb8b00 ObjectTable: e6fbc500 HandleCount: 136.
Image: nlapplication.exe
PROCESS 85c3c020 SessionId: 1 Cid: 5464 Peb: 7ffdc000 ParentCid: 44c8
DirBase: cffb8b40 ObjectTable: e218b948 HandleCount: 39.
Image: application.exe
PROCESS 85c2a020 SessionId: 1 Cid: 546c Peb: 7ffdb000 ParentCid: 44c8
DirBase: cffb8b60 ObjectTable: e639a8d0 HandleCount: 39.
Image: application.exe
PROCESS 85c202c8 SessionId: 1 Cid: 5474 Peb: 7ffd7000 ParentCid: 44c8
DirBase: cffb8b80 ObjectTable: e517caa8 HandleCount: 39.
Image: application.exe
PROCESS 85c1b020 SessionId: 1 Cid: 547c Peb: 7ffd6000 ParentCid: 44c8
DirBase: cffb8ba0 ObjectTable: e6c0cbc0 HandleCount: 39.
Image: application.exe
PROCESS 85c1dd88 SessionId: 1 Cid: 5484 Peb: 7ffd5000 ParentCid: 44c8
DirBase: cffb8bc0 ObjectTable: e4a42f68 HandleCount: 39.
Image: application.exe
PROCESS 85d3ed88 SessionId: 1 Cid: 5488 Peb: 7ffd5000 ParentCid: 44c8
DirBase: cffb8be0 ObjectTable: e68558f0 HandleCount: 39.
Image: application.exe
[...]
We see that all created processes have the same parent process with PID 44c8 and when we inspect it we see many threads inside creating application.exe process:
1: kd> .process /r /p 8607c020
Implicit process is now 8607c020
Loading User Symbols
1: kd> !process 8607c020
PROCESS 8607c020 SessionId: 1 Cid: 44c8 Peb: 7ffdc000 ParentCid: 4cf8
DirBase: cffb7080 ObjectTable: e3c9fd38 HandleCount: 25407.
Image: explorer.exe
VadRoot 88efec98 Vads 3445 Clone 0 Private 30423. Modified 71292. Locked 0.
DeviceMap e3743340
Token e29be5e0
ElapsedTime 00:54:31.359
UserTime 00:00:19.234
KernelTime 00:04:04.828
QuotaPoolUsage[PagedPool] 1075132
QuotaPoolUsage[NonPagedPool] 137800
Working Set Sizes (now,min,max) (15457, 50, 345) (61828KB, 200KB, 1380KB)
PeakWorkingSetSize 48919
VirtualSize 585 Mb
PeakVirtualSize 978 Mb
PageFaultCount 123488
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 49919
[...]
THREAD 84f25300 Cid 44c8.6288 Teb: 7ff8e000 Win32Thread: bc486830 READY
IRP List:
88699110: (0006,0220) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap e3743340
Owning Process 8607c020 Image: explorer.exe
Wait Start TickCount 1327981 Ticks: 29 (0:00:00:00.453)
Context Switch Count 145332 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.093
Win32 Start Address SHLWAPI!SHCreateThread (0x77ec3ea5)
Start Address kernel32!BaseThreadStartThunk (0x7c8217ec)
Stack Init a98e4000 Current a98e3700 Base a98e4000 Limit a98e0000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
a98e3718 80833ec5 nt!KiSwapContext+0x26
a98e3744 80829bc0 nt!KiSwapThread+0x2e5
a98e378c 8087e0d8 nt!KeWaitForSingleObject+0x346
a98e37c4 8087e397 nt!ExpWaitForResource+0x30
a98e37e4 badff32a nt!ExAcquireResourceExclusiveLite+0x8d
a98e3808 badffe35 driverA+0x132a
a98e3824 bae00208 driverA+0x1e35
a98e3868 bae0e45a driverA+0x2208
a98e38a0 8081e095 driverA+0x1045a
a98e38b4 b972c73b nt!IofCallDriver+0x45
[...]
a98e38e8 b9b194e1 nt!IofCallDriver+0x45
[...]
a98e3940 b85cbf08 nt!IofCallDriver+0x45
a98e3968 b85bcfcc driverB!LowerDevicePassThrough+0x48
a98e398c b85bd63d driverB+0x6fcc
a98e3a24 b85cb167 driverB+0x763d
a98e3a34 b85cb1b7 driverB+0x15167
a98e3a5c 8081e095 driverB!DispatchPassThrough+0x48
a98e3a70 808fb13b nt!IofCallDriver+0x45
a98e3b58 80939c6a nt!IopParseDevice+0xa35
a98e3bd8 80935d9e nt!ObpLookupObjectName+0x5b0
a98e3c2c 808ece57 nt!ObOpenObjectByName+0xea
a98e3ca8 808ee0f1 nt!IopCreateFile+0x447
a98e3d04 808f1e31 nt!IoCreateFile+0xa3
a98e3d44 8088ad3c nt!NtOpenFile+0x27
a98e3d44 7c9485ec nt!KiFastCallEntry+0xfc (TrapFrame @ a98e3d64)
03bbda04 7c82bdf6 ntdll!KiFastSystemCallRet
03bbda2c 7c82dd9a kernel32!BasepSxsCreateStreams+0xe2
03bbda9c 7c82d895 kernel32!BasepSxsCreateProcessCsrMessage+0x136
03bbe2c4 7c8024a0 kernel32!CreateProcessInternalW+0x1943
03bbe2fc 7ca36750 kernel32!CreateProcessW+0×2c
03bbed80 7ca36b45 SHELL32!_SHCreateProcess+0×387
03bbedd4 7ca3617b SHELL32!CShellExecute::_DoExecCommand+0xb4
03bbede0 7ca35a76 SHELL32!CShellExecute::_TryInvokeApplication+0×49
03bbedf4 7ca3599f SHELL32!CShellExecute::ExecuteNormal+0xb1
03bbee08 7ca35933 SHELL32!ShellExecuteNormal+0×30
03bbee24 7ca452ff SHELL32!ShellExecuteExW+0×8d
1: kd> .thread 84e6a600
Implicit thread is now 84e6a600
1: kd> kv 100
[...]
03bbda04 7c82bdf6 001200a9 03bbda8c 03bbdb20 ntdll!KiFastSystemCallRet
03bbda2c 7c82dd9a 00000000 00000003 001200a9 kernel32!BasepSxsCreateStreams+0xe2
03bbda9c 7c82d895 00000000 00000000 03bbdc38 kernel32!BasepSxsCreateProcessCsrMessage+0x136
03bbe2c4 7c8024a0 00000000 01dafb9c 01dad904 kernel32!CreateProcessInternalW+0x1943
03bbe2fc 7ca36750 01dafb9c 01dad904 00000000 kernel32!CreateProcessW+0×2c
03bbed80 7ca36b45 00010098 00000000 01daffac SHELL32!_SHCreateProcess+0×387
[…]
1: kd> du /c 100 01dafb9c
01dafb9c “C:\Program Files\App Package\Application.exe”
The difference of this pattern and similar Handle Leak or Zombie Processes is the fact that leaks usually happen when a process forgets to close handles but Process Factory creates active processes which are full resource containers and consume system resources, for example, they all have full handle table or consume GDI resources if they are GUI processes.
- Dmitry Vostokov @ DumpAnalysis.org -
If you complain about heap corruption again or type !locks every day and see pages of output filled with columns the following album from Supertramp reminds us that
Here is my track name interpretation:
1. It’s a Buggy World
2. You Code, I Debug
3. Let’s Debug Together
4. Live to Debug It
5. Some Bugs Never Disappear
6. Read My Bug Report Please
7. Sooner or Later I Fix It
8. Help Me Down that Code Path
9. And the Customer
10. Il Est De Mon Bug!
11. Where There’s a Bug
- Dmitry Vostokov @ DumpAnalysis.org -
This is a continuation of the story of a process hang simulation where I made all threads in IE7 instance frozen. I left that process frozen after my experiments and later tried to reply to one e-mail using PHP-based browser client running in another IE7 process. And I found that the mouse click on ”Reply” button didn’t bring out any GUI response. I tried to close IE7 instance: all tabs closed except the one that was hanging. So I dumped the process and found a blocking thread inside waiting for an RPC call. I made all threads in the first IE7 process unfrozen and the second hanging IE7 process immediately exited. Instead of digging into the dump further I decided to repeat the problem. First I launched the fresh instance of IE7 and opened my e-mail client. After clicking on “Reply” with success I dumped the process using Vista Task Manager and renamed the resulted memory dump as iexplore2.dmp. Then I launched another IE instance and made all threads frozen. Then I came back to the first instance of IE7 and tried to do ”Reply” again. After waiting for about 10 minutes for any response I dumped the process again and renamed the dump file as iexplore3.dmp. Comparing thread stack traces from both dump files showed one difference: the blocked OLE/RPC thread obviously processing some JavaScript code:
[Before hang]
0:000> ~[0n36]k 100
ChildEBP RetAddr
161bfc00 76b60dde ntdll!KiFastSystemCallRet
161bfc04 705e41a1 user32!NtUserWaitMessage+0xc
161bfc68 76c24911 ieframe!CTabWindow::_TabWindowThreadProc+0x2d0
161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe
161bfcb4 76fee489 ntdll!__RtlUserThreadStart+0x23
161bfccc 00000000 ntdll!_RtlUserThreadStart+0x1b
[After hang]
0:000> ~[0n36]k 100
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
161bc27c 76b60208 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 ole32!CCliModalLoop::BlockFn+0×97
161bc320 76907b73 ole32!ModalLoop+0×5b
161bc33c 76908b68 ole32!ThreadSendReceive+0×12c
161bc364 769089d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0×194
161bc444 767fad2e ole32!CRpcChannelBuffer::SendReceive2+0xef
161bc460 767face0 ole32!CCliModalLoop::SendReceive+0×1e
161bc4d8 7681e688 ole32!CAptRpcChnl::SendReceive+0×73
161bc52c 7667264e ole32!CCtxComChnl::SendReceive+0×1c5
161bc544 766726af rpcrt4!NdrProxySendReceive+0×43
161bc9c8 76f3ad86 rpcrt4!NdrProxySendReceive+0xa4
161bc9e0 76f3ad58 oleaut32!IEnumVARIANT_RemoteNext_Proxy+0×19
161bc9fc 6c1f2a7b oleaut32!IEnumVARIANT_Next_Proxy+0×1c
161bca5c 6c1f2b9c mshtml!SearchBrowsersForWindow+0×1bd
161bca84 6c1a2932 mshtml!GetTargetWindow+0×53
161bcabc 6c1b1300 mshtml!CWindow::FindWindowByName+0xe1
161bcad4 706498d4 mshtml!CWindow::FindWindowByName+0×17
161bcaf4 70649e5a ieframe+0×1198d4
161bcb48 70649ff6 ieframe+0×119e5a
161bcbac 70649b82 ieframe+0×119ff6
161bcbe0 6c189f9b ieframe+0×119b82
161bcc10 6c119cba mshtml!COmWindowProxy::FindFrame+0×5c
161bcc44 6c18be8e mshtml!COmWindowProxy::AccessAllowedToFrame+0×7f
161bccb4 6c1c4a2e mshtml!COmWindowProxy::open+0×15b
161bcd1c 6c0371b6 mshtml!Method_IDispatchpp_oDoBSTR_oDoBSTR_oDoBSTR_oDoVARIANTBOOL+0xeb
161bcdb4 6c037493 mshtml!CBase::ContextInvokeEx+0×4ef
161bcde0 6c037607 mshtml!CBase::InvokeEx+0×25
161bce48 6c0374c2 mshtml!COmWindowProxy::InvokeEx+0×297
161bce70 6c5b348e mshtml!COmWindowProxy::subInvokeEx+0×26
161bcea8 6c5b33fe jscript!IDispatchExInvokeEx2+0xac
161bcee0 6c5b3e09 jscript!IDispatchExInvokeEx+0×56
161bcf50 6c5b30eb jscript!InvokeDispatchEx+0×78
161bcf98 6c5b18ab jscript!VAR::InvokeByName+0xba
161bcfd8 6c5b2109 jscript!VAR::InvokeDispName+0×43
161bcffc 6c5b28d8 jscript!VAR::InvokeByDispID+0xb9
161bd0b4 6c5b1019 jscript!CScriptRuntime::Run+0×167f
161bd0cc 6c5b2aa8 jscript!ScrFncObj::Call+0×8d
161bd158 6c5b00f2 jscript!NameTbl::InvokeInternal+0xe0
161bd184 6c5b28d8 jscript!VAR::InvokeByDispID+0xfd
161bd23c 6c5b1019 jscript!CScriptRuntime::Run+0×167f
161bd254 6c5b1b7f jscript!ScrFncObj::Call+0×8d
161bd2c4 6c59f9d2 jscript!CSession::Execute+0xa7
161bd314 6c59fdf7 jscript!COleScript::ExecutePendingScripts+0×147
161bd378 6c59fc46 jscript!COleScript::ParseScriptTextCore+0×243
161bd3a4 6bfcca36 jscript!COleScript::ParseScriptText+0×2b
161bd404 6c1b1931 mshtml!CScriptCollection::ParseScriptText+0×240
161bf48c 6c12adae mshtml!CWindow::ExecuteScriptUri+0×197
161bf4cc 6c1b2f77 mshtml!CWindow::NavigateEx+0×50
161bf530 6c1b3372 mshtml!CDoc::ExecuteScriptUri+0×1f7
161bf560 6c27b8ac mshtml!CDoc::ExecuteScriptURL+0×4b
161bf5a8 6c27a54c mshtml!CHyperlink::ClickAction+0×1a9
161bf5b8 6c121847 mshtml!CAnchorElement::ClickAction+0×10
161bf5e4 6c07a7ef mshtml!CElement::DoClick+0×121
161bf610 6c07a5bd mshtml!CAnchorElement::DoClick+0×4d
161bf69c 6c07f680 mshtml!CDoc::PumpMessage+0xcbd
161bf7e8 6c12a7e0 mshtml!CDoc::OnMouseMessage+0×3d7
161bf90c 6c039a11 mshtml!CDoc::OnWindowMessage+0×8f7
161bf938 76b5f8d2 mshtml!CServer::WndProc+0×78
161bf964 76b5f794 user32!InternalCallWinProc+0×23
161bf9dc 76b606f6 user32!UserCallWinProcCheckWow+0×14b
161bfa0c 76b6069c user32!CallWindowProcAorW+0×97
161bfa2c 6baad980 user32!CallWindowProcW+0×1b
161bfa98 6baa104a GoogleToolbarDynamic_F423308312A7B033+0×3d980
161bfabc 6bb67e57 GoogleToolbarDynamic_F423308312A7B033+0×3104a
161bfae8 76b5f8d2 GoogleToolbarDynamic_F423308312A7B033+0xf7e57
161bfb14 76b5f794 user32!InternalCallWinProc+0×23
161bfb8c 76b60008 user32!UserCallWinProcCheckWow+0×14b
161bfbf0 76b60060 user32!DispatchMessageWorker+0×322
161bfc00 705e42c1 user32!DispatchMessageW+0xf
161bfc68 76c24911 ieframe+0xb42c1
161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe
Upon seeing SendReceive2 on the latter stack trace I recalled that it is possible to know the target process PID: In Search of Lost CID. The same procedure applied here reveals PID = 0xdec:
0:000> ~[0n36]kv 9
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
161bc27c 76b60208 161bc230 161bc2a4 00000000 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 00000000 161bc318 00000000 user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 161bc318 00000000 161bc328 ole32!CCliModalLoop::BlockFn+0x97
161bc320 76907b73 00000000 00000000 161bc42c ole32!ModalLoop+0x5b
161bc33c 76908b68 00000000 161bc440 00000000 ole32!ThreadSendReceive+0x12c
161bc364 769089d4 161bc42c 00000000 161bc488 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x194
161bc444 767fad2e 14f75040 161bc56c 161bc550 ole32!CRpcChannelBuffer::SendReceive2+0xef
161bc460 767face0 161bc56c 161bc550 00000000 ole32!CCliModalLoop::SendReceive+0×1e
161bc4d8 7681e688 14f75040 161bc56c 161bc550 ole32!CAptRpcChnl::SendReceive+0×73
Note: 14f75040 is 00000000 in iexplore3.dmp from ftp because the dumps were stripped from almost all process data and contain only values necessary to reconstruct stack traces. So you won’t be able to extract correct raw stack data from them.
0:000> ddp 14f75040
14f75040 76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface
14f75044 7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface
14f75048 00000003
14f7504c 00000002
14f75050 00000000
14f75054 00000000
14f75058 0046ccd0 0046ce50
14f7505c 0e8de858 00000000
14f75060 1acb7310 00000044
14f75064 1acb3130 76828510 ole32!CStdIdentity::`vftable’
14f75068 7682b098 767f8066 ole32!CDestObject::QueryInterface
14f7506c 00070005 ee0100ed
14f75070 00000000
14f75074 00000000
14f75078 00000d78
14f7507c 00000000
14f75080 76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface
14f75084 7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface
14f75088 00000001
14f7508c 00000024
14f75090 00000000
14f75094 00000000
14f75098 07abd4a8 07aacab0
14f7509c 00000000
14f750a0 00000000
14f750a4 1ae12b10 76828510 ole32!CStdIdentity::`vftable’
14f750a8 7682b098 767f8066 ole32!CDestObject::QueryInterface
14f750ac 00070005 ee0100ed
14f750b0 ffffffff
14f750b4 00001134
14f750b8 00001134
14f750bc 00000000
0:000> dd 0046ccd0 l4
0046ccd0 0046ce50 0046cc50 00000dec 00000000
In Task Manager I found this to be ieuser.exe process so I suspect there is a high degree of process coupling between all launched IE7 processes and ieuser.exe including COM/OLE runtime.
The stripped versions of dumps are available for practice on ftp:
ftp://dumpanalysis.org/pub/ie7_pattern_cooperation2.zip
- Dmitry Vostokov @ DumpAnalysis.org -
If you are interested in mathematical ideas or want to learn serious math you can skip proofs when reading various math textbooks. My speed of math book processing greatly increased after I started to skip proofs of lemmas and theorems. The slow progress through proofs inhibited my reading advance in the past. Even professional mathematicians confess after a few beers how slow they are as Thomas Garrity mentioned in the preface to his book All the Mathematics You Missed. I found that it is more important is to read several books on the same subject to see different explanations and more examples than to concentrate on a one book. By skipping proofs I can now read 2-3 more books in the same amount of time.
- Dmitry Vostokov @ DumpAnalysis.org -
Most of (if not all) debugging is arithmetical. Here I would like to introduce a new kind of debugging and troubleshooting approach that interprets observables as objects in their own spaces, for example, the possible space of various GUI forms. These spaces are not necessarily rational-valued spaces of simulation output or discreet arithmetic spaces of memory locations and values.
This geometrical approach applies modeling and systems theory to debugging and troubleshooting by treating them as mappings (or functions in the case of one-to-one or many-to-one mappings) from the space of all possible software environment states (SE) to the space(s) of observables. Here we have a family of mappings to different spaces:
fi: SE → SOi
Some observables can be found fixed like the list of components and the number of mappings can be reduced (i < j):
fj: SEa,b,c,d,… → SOj
In every system and its environment we have something fixed as parameters (a, b, c, d, …) and this could be the list of components as high level ”genotype” or it could be just specific code (low-level “genotype”), specific data or hardware specification. The whole family of mappings become parametrized. If we want, we can reduce mappings even more to treat them as many-valued (one-to-many or many-to-many) if several observables belong to the same kind of space.
Let me illustrate this by an analogy with modeling of a natural system. The system to be modeled is a falling ball together with its environment (Earth). The system obviously has some internal structure (abstract space of states, E) but we don’t know it. Fortunately, we can observe some measurable values like the ball position at any time (Q). So we have these mappings for balls with different masses:
fm: E → Q
We also find that for any individual ball its mass doesn’t change so we abstract it as a parameter:
f: Em → Q
The same modeling approach can be applied to a software system be it an application or a service running inside an operating system or a software system itself running inside a hardware. The case of pure software system abstracted from hardware is simple. In such a case SE space theoretically could be the space of abstract memory dumps. Practically we deal with the space of observables (universal memory dumps) that approximate SE and spaces of software “phenotypes”, observable behaviour, like distorted GUI, for example, or measured values of memory and CPU consumption or disk I/O throughput.
- Dmitry Vostokov @ DumpAnalysis.org -
The book is available for ordering from Amazon with a significant discount:
Windows Debugging: Practical Foundations
Although listed as temporarily out of stock at the time of this writing it should appear in stock in a few days. Search Inside is also enabled.
- Dmitry Vostokov @ DumpAnalysis.org -
OpenTask, the publisher of my books, announces restructuring:
http://www.opentask.com/restructuring-2009
- Dmitry Vostokov @ DumpAnalysis.org -
Here is further analysis of a memory dump used to illustrate Swarm of Shared Locks pattern. In that dump there were also exclusively held locks with many blocked threads:
Resource @ 0x8a04c408 Exclusively owned
Contention Count = 344875
NumberOfExclusiveWaiters = 6
Threads: 87eb3db0-01<*>
Threads Waiting On Exclusive Access:
88573db0 87b90378 86a49db0 891f4610
8662b020 87127db0
Resource @ 0x89678e80 Exclusively owned
Contention Count = 10261
NumberOfExclusiveWaiters = 2
Threads: 87eb3db0-01<*>
Threads Waiting On Exclusive Access:
89131bf0 88d12db0
Resource @ 0x88d099d8 Exclusively owned
Contention Count = 562811
NumberOfExclusiveWaiters = 4
Threads: 873611d8-01<*>
Threads Waiting On Exclusive Access:
88b8bb88 88a72c50 89359af0 88d865e8
Resource @ 0x86db9248 Exclusively owned
Contention Count = 1382269
NumberOfSharedWaiters = 2
NumberOfExclusiveWaiters = 11
Threads: 86ab2020-01<*> 8769cdb0-01 880c77b8-01
Threads Waiting On Exclusive Access:
87bf4020 890dc020 874c01c0 884ef020
86913af8 875bab10 88e8a0d8 8923cdb0
894eca18 86aa6830 86f293a8
Resource @ 0x873a88d0 Exclusively owned
Contention Count = 719758
NumberOfExclusiveWaiters = 8
Threads: 88d5f990-01<*>
Threads Waiting On Exclusive Access:
8759ea88 871b6db0 88117710 87cb4718
883eb638 87239020 881ad020 891b9188
Resource @ 0x88c379a0 Exclusively owned
Contention Count = 126686
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 8
Threads: 882b8020-01<*> 88951520-01
Threads Waiting On Exclusive Access:
877d34a8 8939fdb0 87fc5668 8851fdb0
86fad850 87f1f450 8a1749f0 876a78d0
Resource @ 0x88ca9250 Exclusively owned
Contention Count = 319721
NumberOfExclusiveWaiters = 4
Threads: 88607908-01<*>
Threads Waiting On Exclusive Access:
86829370 892ae8e8 87205208 87b6d7e0
Resource @ 0x86a90ef8 Exclusively owned
Contention Count = 852830
NumberOfExclusiveWaiters = 12
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
88a9c9b0 88a50db0 87117928 890e4c50
874ffb30 88b540f8 8705d020 8687edb0
87143188 8703e430 885b6aa0 8842bc50
Resource @ 0x88954538 Exclusively owned
Contention Count = 40708
NumberOfExclusiveWaiters = 1
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
878ee980
Resource @ 0x88617eb8 Exclusively owned
Contention Count = 43531
NumberOfExclusiveWaiters = 2
Threads: 87571640-01<*>
Threads Waiting On Exclusive Access:
88851db0 87382c50
Resource @ 0x87288bc8 Exclusively owned
Contention Count = 644675
NumberOfExclusiveWaiters = 2
Threads: 874e4508-01<*>
Threads Waiting On Exclusive Access:
88863b08 89479650
Resource @ 0x87c3d8b0 Exclusively owned
Contention Count = 335064
NumberOfExclusiveWaiters = 8
Threads: 87f44520-01<*>
Threads Waiting On Exclusive Access:
88277190 88eceb48 87f0d308 8694d460
88461db0 876734a8 871721b0 88c2adb0
All threads owning various locks exclusively are stuck in processing page fault code, for example:
0: kd> !thread 87eb3db0 1f
THREAD 87eb3db0 Cid 47ac.57c8 Teb: 7ffd7000 Win32Thread: bc151230 WAIT: (Unknown) KernelMode Non-Alertable
8743e4e0 NotificationEvent
IRP List:
8660c900: (0006,0094) Flags: 00000900 Mdl: 00000000
Not impersonating
DeviceMap e1003890
Owning Process 88e49918 Image: csrss.exe
Wait Start TickCount 15420972 Ticks: 2527 (0:00:00:39.484)
Context Switch Count 1430991 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:02.734
Start Address 0×75a8e96c
Stack Init a3cf7000 Current a3cf6430 Base a3cf7000 Limit a3cf4000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr
a3cf6448 8083d5b1 nt!KiSwapContext+0×26
a3cf6474 8083df9e nt!KiSwapThread+0×2e5
a3cf64bc 8082629e nt!KeWaitForSingleObject+0×346
a3cf64e4 80826480 nt!MiWaitForInPageComplete+0×1f
a3cf656c 8084790e nt!MiDispatchFault+0xda3
a3cf65c8 80836c2a nt!MmAccessFault+0×64a
a3cf65c8 bfa38de0 nt!KiTrap0E+0xdc (TrapFrame @ a3cf65e0)
a3cf6a24 bf854a72 win32k!vSetPointer+0×36f
a3cf6a50 bf8b1b74 win32k!GreSetPointer+0×66
a3cf6a7c bf883183 win32k!zzzUpdateCursorImage+0×1cc
a3cf6a8c bf884b06 win32k!zzzSetFMouseMoved+0xd5
a3cf6ad4 bf81530a win32k!ProcessQueuedMouseEvents+0×1c4
a3cf6d30 bf86fd25 win32k!RawInputThread+0×5b4
a3cf6d40 bf898a52 win32k!xxxCreateSystemThreads+0×60
a3cf6d54 80833bef win32k!NtUserCallOneParam+0×23
a3cf6d54 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ a3cf6d64)
0: kd> !thread 87571640 1f
THREAD 87571640 Cid 49f4.65b4 Teb: 7ffdf000 Win32Thread: bc011680 WAIT: (Unknown) KernelMode Non-Alertable
8870db90 NotificationEvent
Not impersonating
DeviceMap e24f6570
Owning Process 87be4a00 Image: ApplicationC.EXE
Wait Start TickCount 15420974 Ticks: 2525 (0:00:00:39.453)
Context Switch Count 25640 LargeStack
UserTime 00:00:00.921
KernelTime 00:00:03.859
Win32 Start Address 0×30002658
Start Address 0×77e617f8
Stack Init 9a318600 Current 9a317b70 Base 9a319000 Limit 9a314000 Call 9a31860c
Priority 14 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
9a317b88 8083d5b1 nt!KiSwapContext+0×26
9a317bb4 8083df9e nt!KiSwapThread+0×2e5
9a317bfc 8082629e nt!KeWaitForSingleObject+0×346
9a317c24 80826480 nt!MiWaitForInPageComplete+0×1f
9a317cac 8084790e nt!MiDispatchFault+0xda3
9a317d08 80836c2a nt!MmAccessFault+0×64a
9a317d08 bf8b5485 nt!KiTrap0E+0xdc (TrapFrame @ 9a317d20)
9a317db4 bf8b526c win32k!vSolidFillRect1+0xb0
9a317f58 bf8ad7d2 win32k!vDIBSolidBlt+0×102
9a317fc4 bfa285d1 win32k!EngBitBlt+0xe1
9a3180c8 bf899b57 win32k!GrePatBltLockedDC+0×1ea
9a318160 bf8b32bb win32k!GrePolyPatBltInternal+0×17c
9a31819c bf8bd71c win32k!GrePolyPatBlt+0×45
9a31822c bf85e3d5 win32k!DrawEdge+0×23a
9a318274 bf8ae338 win32k!xxxDrawWindowFrame+0×170
9a3182d4 bf8847d1 win32k!xxxRealDefWindowProc+0×7a7
9a3182ec bf884801 win32k!xxxWrapRealDefWindowProc+0×16
9a318308 bf8c1769 win32k!NtUserfnDWORD+0×27
9a318340 80833bef win32k!NtUserMessageCall+0xc0
9a318340 7c8285ec nt!KiFastCallEntry+0xfc (TrapFrame @ 9a318364)
We also see that their waiting time is almost the same, 39 seconds. This means that the problem with paging probably started at that time before the crash dump was forced.
- Dmitry Vostokov @ DumpAnalysis.org -
Sometimes there are so many shared locks on the system that it might point to some problems in subsystems that own them. For example, there are two large swarms of them in this memory dump from a system running 90 user sessions:
0: kd> !session
Sessions on machine: 90
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks....
Resource @ nt!CmpRegistryLock (0x808ad4c0) Shared 210 owning threads
Contention Count = 1432
Threads: 88bf1590-01<*> 8a78a660-01<*> 8a787660-01<*> 8825a3a8-01<*>
89003358-01<*> 86723b90-01<*> 865bbb00-01<*> 89634638-01<*>
888d9508-01<*> 88da6b48-01<*> 87db9db0-01<*> 86a9e610-01<*>
89ff7410-01<*> 87450db0-01<*> 86bdedb0-01<*> 86d604c8-01<*>
88d465d8-01<*> 86c3b6a0-01<*> 87c89020-01<*> 88e73db0-01<*>
865fe5b0-01<*> 88450020-01<*> 86bd9db0-01<*> 8a73e838-01<*>
88dc3db0-01<*> 88035708-01<*> 8833a2f0-01<*> 88608350-01<*>
87aca020-01<*> 87e007c0-01<*> 86ec39b8-01<*> 893be1b8-01<*>
8671ddb0-01<*> 8679a718-01<*> 89fe34c8-01<*> 86ccd720-01<*>
881b1db0-01<*> 86771b20-01<*> 86d71db0-01<*> 89574db0-01<*>
87dfac50-01<*> 86597020-01<*> 874b3488-01<*> 873b59b0-01<*>
88e792f8-01<*> 878d2430-01<*> 8853d480-01<*> 889e2020-01<*>
88c36db0-01<*> 8824f990-01<*> 8719b830-01<*> 884ba020-01<*>
88e1d768-01<*> 89523db0-01<*> 896529f8-01<*> 887e2870-01<*>
8a022db0-01<*> 867253a0-01<*> 865f0448-01<*> 87d35640-01<*>
8715d968-01<*> 87ce0c50-01<*> 87d44730-01<*> 86d69aa8-01<*>
88e5b020-01<*> 88734410-01<*> 898f2b40-01<*> 8a00a510-01<*>
87e69db0-01<*> 8722b860-01<*> 86d8e308-01<*> 87263c50-01<*>
8706ddb0-01<*> 892136e8-01<*> 8875b020-01<*> 8833ca48-01<*>
8a100db0-01<*> 86b77590-01<*> 888bc020-01<*> 865c3db0-01<*>
89fba910-01<*> 8a789660-01<*> 8670b2a8-01<*> 868737a8-01<*>
868326d0-01<*> 871cdaf0-01<*> 8852edb0-01<*> 882b23b8-01<*>
877e29e0-01<*> 8774f558-01<*> 876aa020-01<*> 89187518-01<*>
8664b8e0-01<*> 865b4478-01<*> 88135020-01<*> 8686f020-01<*>
866a0190-01<*> 87316758-01<*> 894dab18-01<*> 87938560-01<*>
8658f5f0-01<*> 88e54020-01<*> 867f6350-01<*> 89246af8-01<*>
86801430-01<*> 86db2af0-01<*> 865cf588-01<*> 86ab64f8-01<*>
8a4a61e8-01<*> 885f3020-01<*> 86ea9af0-01<*> 8a4a7ba8-01<*>
8a746b08-01<*> 89fc4790-01<*> 87093b10-01<*> 8659bc50-01<*>
86681db0-01<*> 87102228-01<*> 866145a0-01<*> 866dddb0-01<*>
86bda990-01<*> 88257db0-01<*> 8687d590-01<*> 867a9db0-01<*>
89898848-01<*> 8a49b920-01<*> 86596db0-01<*> 8a0f7db0-01<*>
866c1b40-01<*> 8754e020-01<*> 87fc1428-01<*> 8658c870-01<*>
880d6a90-01<*> 88be6c50-01<*> 86bbcdb0-01<*> 8a37b8f8-01<*>
866a13e0-01<*> 873e33d0-01<*> 87d43db0-01<*> 88a5adb0-01<*>
884a5440-01<*> 883646f0-01<*> 87128020-01<*> 88e1d020-01<*>
888e6418-01<*> 875c7c50-01<*> 871dd020-01<*> 890d5838-01<*>
88d061f0-01<*> 88a09428-01<*> 8972f780-01<*> 87325b08-01<*>
86deb020-01<*> 878b31b8-01<*> 891ac8a8-01<*> 86b234c0-01<*>
86dd2190-01<*> 875f9db0-01<*> 87bbf200-01<*> 8a1a9c40-01<*>
88628020-01<*> 87919020-01<*> 87c2a660-01<*> 877dc7c0-01<*>
8a08adb0-01<*> 87c0f628-01<*> 87ca9a28-01<*> 8880a210-01<*>
86ec0020-01<*> 88571020-01<*> 8a01edb0-01<*> 88115db0-01<*>
87a9adb0-01<*> 879ecdb0-01<*> 8868ddb0-01<*> 872bcb58-01<*>
884a0100-01<*> 8929f020-01<*> 87087020-01<*> 886e75a8-01<*>
885a5908-01<*> 8762c020-01<*> 89550db0-01<*> 8a554768-01<*>
89f10680-01<*> 87b322e8-01<*> 87cc74d0-01<*> 883ee2d0-01<*>
8956caf8-01<*> 8788f330-01<*> 87d5c320-01<*> 86b99db0-01<*>
876f42e0-01<*> 88e812d0-01<*> 8687cdb0-01<*> 8677a310-01<*>
89711b40-01<*> 89b013a8-01<*> 86abcdb0-01<*> 89fd7bb0-01<*>
877c22b0-01<*> 883fc850-01<*> 889e11f8-01<*> 892ff0e0-01<*>
878ac490-01<*> 86de5c50-01<*> 87741db0-01<*> 8679f020-01<*>
880ac6d0-01<*> 86d8fb00-01<*>
KD: Scanning for held locks….
Resource @ Ntfs!NtfsData (0xf71665b0) Shared 1 owning threads
Threads: 8a78d660-01<*>
KD: Scanning for held locks.
Resource @ 0x8a5c7734 Shared 1 owning threads
Contention Count = 507565
NumberOfSharedWaiters = 128
NumberOfExclusiveWaiters = 1
Threads: 894b4db0-01 87c773e0-01 88de7020-01 891c9db0-01
894d2020-01 865af5f8-01 87867340-01 88c964a0-01<*>
88e57c98-01 87ae3020-01 86dbe730-01 88343790-01
871102e8-01 8855f020-01 87c99920-01 8796a318-01
88028db0-01 88ad6610-01 88b73db0-01 89fba3f0-01
87d8bc00-01 86f4c5c8-01 8a028608-01 88c783f0-01
88c138e0-01 89236910-01 896fbb78-01 88523600-01
8926f3b0-01 88a49a48-01 87c19750-01 86c88c50-01
88adfad8-01 872b0020-01 87ecab18-01 88b02020-01
875f9b10-01 8755e020-01 86f9fdb0-01 86a1cab8-01
86816858-01 881eedb0-01 894a99f0-01 87c97740-01
8a3bf4b0-01 867765a8-01 8a787660-01 86810330-01
876ad268-01 87af3320-01 865fedb0-01 88eb8230-01
86b0c438-01 881c0230-01 888b67c8-01 883e3210-01
87acbc50-01 873d6648-01 86ed0db0-01 88e2d020-01
89fdadb0-01 8934e830-01 870f89f0-01 8756c5e0-01
878c88d0-01 86fec608-01 88fdb420-01 87fa0628-01
87cad8d8-01 88ee3978-01 86fc49a0-01 875d5020-01
871a5020-01 89667a60-01 87170db0-01 88254ae0-01
8775e408-01 88204db0-01 87989890-01 873b89a8-01
888e6bf8-01 88cc3db0-01 88bf1590-01 879565a0-01
86773db0-01 8731a020-01 88aa7a78-01 8759cdb0-01
87e555f8-01 86de5678-01 86e28020-01 86ec9320-01
86871af0-01 8719cba0-01 8723f820-01 884dac20-01
89249020-01 889da168-01 8900b810-01 8a78d660-01
88cac758-01 892984c8-01 87d0c020-01 87ecec50-01
87ad8c90-01 88109aa8-01 86ef5bf0-01 8a78d3f0-01
88d2b020-01 88640db0-01 86fec878-01 895b12d8-01
86dd6708-01 87386930-01 888e34e0-01 86a56c50-01
8815f768-01 886c42a0-01 898f2020-01 87ca3610-01
886dd448-01 86ada210-01 8a37adb0-01 8896c940-01
8800e898-01 8733d4b8-01 865fa358-01 88ae1af0-01
868dd020-01
Threads Waiting On Exclusive Access:
8a78b020
Both swarms are grouped around NTFS structures as can be seen from thread stack traces but also have another module in common: PGPsdk
0: kd> !thread 88bf1590 1f
THREAD 88bf1590 Cid 4354.2338 Teb: 7ffdf000 Win32Thread: bc3e88f8 WAIT: (Unknown) KernelMode Non-Alertable
8a7a73d8 Semaphore Limit 0x7fffffff
88bf1608 NotificationTimer
IRP List:
86fb39d0: (0006,0268) Flags: 00000004 Mdl: 00000000
Not impersonating
DeviceMap e13c9ca0
Owning Process 869a6d88 Image: ApplicationA.exe
Wait Start TickCount 15423469 Ticks: 30 (0:00:00:00.468)
Context Switch Count 6465 LargeStack
UserTime 00:00:00.343
KernelTime 00:00:01.062
Win32 Start Address 0x0056f122
Start Address 0x77e617f8
Stack Init 97e9d000 Current 97e9c788 Base 97e9d000 Limit 97e98000 Call 0
Priority 14 BasePriority 8 PriorityDecrement 6
ChildEBP RetAddr
97e9c7a0 8083d5b1 nt!KiSwapContext+0x26
97e9c7cc 8083df9e nt!KiSwapThread+0x2e5
97e9c814 8081e05b nt!KeWaitForSingleObject+0x346
97e9c850 80824ba8 nt!ExpWaitForResource+0xd5
97e9c870 f718a07d nt!ExAcquireResourceSharedLite+0xf5
97e9c884 f717b2eb Ntfs!NtfsAcquireSharedVcb+0×23
97e9c8f0 f717a2e2 Ntfs!NtfsCommonFlushBuffers+0xf5
97e9c954 80840153 Ntfs!NtfsFsdFlushBuffers+0×92
97e9c968 f7272c45 nt!IofCallDriver+0×45
97e9c990 80840153 fltmgr!FltpDispatch+0×6f
97e9c9a4 f6fb1835 nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
97e9c9b8 f6fad69a PGPsdk+0×5835
97e9c9c4 80840153 PGPsdk+0×169a
86fb39d0 00000000 nt!IofCallDriver+0×45
0: kd> !thread 88c964a0 1f
THREAD 88c964a0 Cid 323c.43f0 Teb: 7ffad000 Win32Thread: bc2ceea8 WAIT: (Unknown) KernelMode Non-Alertable
88268338 SynchronizationEvent
88c96518 NotificationTimer
IRP List:
86dad430: (0006,0268) Flags: 00000404 Mdl: 00000000
Not impersonating
DeviceMap e16c8eb0
Owning Process 8886ac88 Image: ApplicationB.EXE
Wait Start TickCount 15423352 Ticks: 147 (0:00:00:02.296)
Context Switch Count 1660 LargeStack
UserTime 00:00:00.078
KernelTime 00:00:00.109
Win32 Start Address 0x14225c34
Start Address 0x77e617ec
Stack Init 96835000 Current 96834640 Base 96835000 Limit 96832000 Call 0
Priority 14 BasePriority 8 PriorityDecrement 6
ChildEBP RetAddr
96834658 8083d5b1 nt!KiSwapContext+0x26
96834684 8083df9e nt!KiSwapThread+0x2e5
968346cc 8081e05b nt!KeWaitForSingleObject+0x346
96834708 8082e012 nt!ExpWaitForResource+0xd5
96834728 f714b89b nt!ExAcquireResourceExclusiveLite+0x8d
96834738 f718b194 Ntfs!NtfsAcquirePagingResourceExclusive+0×20
9683493c f718b8d9 Ntfs!NtfsCommonCleanup+0×193
96834aac 80840153 Ntfs!NtfsFsdCleanup+0xcf
96834ac0 f7272c45 nt!IofCallDriver+0×45
96834ae8 80840153 fltmgr!FltpDispatch+0×6f
96834afc f6fb196c nt!IofCallDriver+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
96834b10 f6fad69a PGPsdk+0×596c
96834b1c 80840153 PGPsdk+0×169a
86dad430 00000000 nt!IofCallDriver+0×45
Because no processors are busy except the one that processes crash dump request via NMI interrupt and there are no ready threads it would be natural to assume that the problem with paging started some time ago and some checks for 3rd-party volume encryption software are necessary as PGP name of the module suggests:
0: kd> lmv m PGPsdk
start end module name
f6fac000 f6fb7000 PGPsdk (no symbols)
Loaded symbol image file: PGPsdk.sys
Image path: \SystemRoot\System32\Drivers\PGPsdk.sys
Image name: PGPsdk.sys
Timestamp: Wed Jun 09 11:44:04 2004 (40C6E9F4)
CheckSum: 00010F72
ImageSize: 0000B000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
0: kd> !running
System Processors f (affinity mask)
Idle Processors e
Prcb Current Next
0 ffdff120 808a68c0 86841588 ................
0: kd> !thread 808a68c0 1f
THREAD 808a68c0 Cid 0000.0000 Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
Owning Process 808a6b40 Image: Idle
Wait Start TickCount 0 Ticks: 15423499 (2:18:56:32.171)
Context Switch Count 100782385
UserTime 00:00:00.000
KernelTime 2 Days 12:18:49.343
Stack Init 808a38b0 Current 808a35fc Base 808a38b0 Limit 808a08b0 Call 0
Priority 0 BasePriority 0 PriorityDecrement 0
ChildEBP RetAddr
808a07bc 80a84df7 nt!KeBugCheckEx+0x1b
808a080c 80834b83 hal!HalHandleNMI+0x1a5
808a080c 80a80853 nt!KiTrap02+0x136 (TrapFrame @ 808a0820)
808a3570 f7659ca2 hal!HalpClockInterrupt+0xff (TrapFrame @ 808a3570)
808a3600 80839b12 intelppm!AcpiC1Idle+0x12
808a3604 00000000 nt!KiIdleLoop+0xa
0: kd> !ready
Processor 0: No threads in READY state
Processor 1: No threads in READY state
Processor 2: No threads in READY state
Processor 3: No threads in READY state
- Dmitry Vostokov @ DumpAnalysis.org -
Here is the message from John R. Ingram:
http://www.lightningsource.com/podnow.aspx
All my books are printed on demand. For me it is the most economical way to publish and I look forward to seeing POD offering more formats.
- Dmitry Vostokov @ DumpAnalysis.org -
Andrzej Dyjak features Linux application core dumps converted to bitmap images:
- Dmitry Vostokov @ DumpAnalysis.org -
In the past I was not able or didn’t know how to view 32-bit process thread stacks when looking at a complete memory dump from x64 Windows. So I had to request user dumps. Now I want to share a technique a reader of my blog (Yuhong Bao) suggested: to use .thread WinDbg command with /w option. Here are additional steps that I found necessary when playing with my test complete memory dump from x64 Windows Server 2003 SP2 (I used the latest version of WinDbg from 64-bit Debugging Tools for Windows):
0. Find a 32-bit process of interest:
kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 0014a000 ObjectTable: fffffa8000000c10 HandleCount: 736.
Image: System
PROCESS fffffadfe6b14040
SessionId: none Cid: 0130 Peb: 7fffffd8000 ParentCid: 0004
DirBase: 353c0000 ObjectTable: fffffa80009104a0 HandleCount: 19.
Image: smss.exe
PROCESS fffffadfe65cec20
SessionId: 0 Cid: 0160 Peb: 7fffffde000 ParentCid: 0130
DirBase: 30210000 ObjectTable: fffffa80006a4d80 HandleCount: 732.
Image: csrss.exe
PROCESS fffffadfe73b7040
SessionId: 0 Cid: 0270 Peb: 7fffffdc000 ParentCid: 0130
DirBase: 302b6000 ObjectTable: fffffa8000520710 HandleCount: 751.
Image: winlogon.exe
PROCESS fffffadfe737d040
SessionId: 0 Cid: 02a0 Peb: 7fffffd7000 ParentCid: 0270
DirBase: 0060d000 ObjectTable: fffffa80008df6a0 HandleCount: 339.
Image: services.exe
PROCESS fffffadfe6574040
SessionId: 0 Cid: 02ac Peb: 7fffffd5000 ParentCid: 0270
DirBase: 0070d000 ObjectTable: fffffa80008e16a0 HandleCount: 510.
Image: lsass.exe
PROCESS fffffadfe7860040
SessionId: 0 Cid: 0364 Peb: 7fffffd7000 ParentCid: 02a0
DirBase: 0935e000 ObjectTable: fffffa8000969710 HandleCount: 87.
Image: svchost.exe
[...]
PROCESS fffffadfe751d040
SessionId: 0 Cid: 0bcc Peb: 7efdf000 ParentCid: 0abc
DirBase: 18861000 ObjectTable: fffffa8001ecbc30 HandleCount: 326.
Image: Application32.exe
[...]
1. Switch to the process context:
kd> .process /r /p fffffadfe751d040
Implicit process is now fffffadf`e751d040
Loading User Symbols
Stacks traces are 64-bit:
kd> !process fffffadfe751d040
PROCESS fffffadfe751d040
SessionId: 0 Cid: 0bcc Peb: 7efdf000 ParentCid: 0abc
DirBase: 18861000 ObjectTable: fffffa8001ecbc30 HandleCount: 326.
Image: Application32.exe
VadRoot fffffadfe7550ae0 Vads 160 Clone 0 Private 1616. Modified 1675. Locked 0.
DeviceMap fffffa800210e600
Token fffffa80028ef060
ElapsedTime 21:57:59.125
UserTime 00:00:00.718
KernelTime 00:00:00.953
QuotaPoolUsage[PagedPool] 185704
QuotaPoolUsage[NonPagedPool] 20080
Working Set Sizes (now,min,max) (3021, 50, 345) (12084KB, 200KB, 1380KB)
PeakWorkingSetSize 3696
VirtualSize 93 Mb
PeakVirtualSize 104 Mb
PageFaultCount 12097
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 2051
THREAD fffffadfe664e040 Cid 0bcc.0bdc Teb: 000000007efdb000 Win32Thread: fffff97ff4898bd0 WAIT: (Unknown) UserMode Non-Alertable
fffffadfe73bac40 SynchronizationEvent
fffffadfe6b69790 SynchronizationEvent
Not impersonating
DeviceMap fffffa800210e600
Owning Process fffffadfe751d040 Image: Application32.exe
Attached Process N/A Image: N/A
Wait Start TickCount 4153935 Ticks: 912354 (0:03:57:35.531)
Context Switch Count 8088 LargeStack
UserTime 00:00:00.343
KernelTime 00:00:00.593
Win32 Start Address Application32 (0x00000000004077ec)
Start Address 0x0000000077d59620
Stack Init fffffadfdede7e00 Current fffffadfdede7250
Base fffffadfdede8000 Limit fffffadfdede2000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
Child-SP RetAddr Call Site
fffffadf`dede7290 fffff800`0103b0a3 nt!KiSwapContext+0x85
fffffadf`dede7410 fffff800`0103af8a nt!KiSwapThread+0xc3
fffffadf`dede7450 fffff800`012b9958 nt!KeWaitForMultipleObjects+0x5ec
fffffadf`dede74f0 fffff800`012e63ec nt!ObpWaitForMultipleObjects+0x325
fffffadf`dede79b0 fffff800`0104113d nt!NtWaitForMultipleObjects32+0xcc
fffffadf`dede7c00 00000000`78b83d44 nt!KiSystemServiceCopyEnd+0x3 (TrapFrame @ fffffadf`dede7c70)
00000000`0012edc8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0x3a
00000000`0012ee70 00000000`6b005e0d wow64!RunCpuSimulation+0xa
00000000`0012eea0 00000000`77ed8030 wow64!Wow64LdrpInitialize+0x2ed
00000000`0012f6d0 00000000`77ed582f ntdll!LdrpInitializeProcess+0x1538
00000000`0012f9d0 00000000`77ef30a5 ntdll!LdrpInitialize+0x18f
00000000`0012fab0 00000000`77d59620 ntdll!KiUserApcDispatcher+0x15 (TrapFrame @ 00000000`0012fe18)
[...]
2. Load WOW64 extension
kd> .load wow64exts
3. Set the current thread and switch to x86 context:
kd> .thread /w fffffadfe664e040
Implicit thread is now fffffadf`e664e040
x86 context set
4. Sometimes reloading symbols is necessary:
kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols
5. Now we can get our stack trace (it is a bit rough because Application32.exe symbols were not available)
kd:x86> kv 100
ChildEBP RetAddr Args to Child
002cfd94 7d4e286c 00000002 002cfde0 00000001 ntdll_7d600000!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
002cfe3c 7d94d299 00000002 002cfe64 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
002cfe98 7d94d327 00000001 002d8148 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x152 (FPO: [5,13,0])
*** ERROR: Module load completed but symbols could not be loaded for Application32.exe
002cfeb4 00408081 00000001 002d8148 00000000 USER32!MsgWaitForMultipleObjects+0x1f (FPO: [5,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
002cff00 00407d4b ffffffff 00408b78 004010ee Application32+0x8081
002cff08 00408b78 004010ee 004352e0 004352e0 Application32+0x7d4b
002cff0c 004010ee 004352e0 004352e0 0042f004 Application32+0x8b78
00408b78 90909090 90c3c033 90909090 90909090 Application32+0x10ee
00408b7c 90c3c033 90909090 90909090 90909090 0x90909090
00408b80 90909090 90909090 90909090 433aa0a1 0x90c3c033
[...]
6. We can also access raw stack trace if we need to see 32-bit execution residue and reconstruct partial stack traces:
kd:x86> !teb
Wow64 TEB32 at 000000007efdd000
[...]
Wow64 TEB at 000000007efdb000
ExceptionList: 000000007efdd000
StackBase: 0000000000130000
StackLimit: 000000000012a000
SubSystemTib: 0000000000000000
FiberData: 0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self: 000000007efdb000
EnvironmentPointer: 0000000000000000
ClientId: 0000000000000bcc . 0000000000000bdc
RpcHandle: 0000000000000000
Tls Storage: 0000000000000000
PEB Address: 000000007efdf000
LastErrorValue: 6
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
kd:x86> dds 000000000012a000 0000000000130000
[...]
- Dmitry Vostokov @ DumpAnalysis.org -
I forced a complete memory dump of Windows 7 Beta running under VMWare Fusion on my MacBook Air laptop using SystemDump. In WinDbg I see kernel32 API refactoring. It looks like common API was factored out into KERNELBASE.dll. For example, a new session 1 process taskhost.exe has the following highlighted changes (the rest of stack trace layout looks the same as in Vista except nt!KiCommitThreadWait in kernel stack trace counterpart):
kd> vertarget
Windows Kernel Version 7000 UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7000.0.x86fre.winmain_win7beta.081212-1400
Kernel base = 0x82639000 PsLoadedModuleList = 0x82790830
Debug session time: Thu Feb 5 12:21:31.765 2009 (GMT+0)
System Uptime: 0 days 0:14:43.078
kd> .process /r /p 85471598
Implicit process is now 85471598
Loading User Symbols
kd> !process 85471598
PROCESS 85471598 SessionId: 1 Cid: 0750 Peb: 7ffd5000 ParentCid: 01a4
DirBase: 1efb2320 ObjectTable: 90282990 HandleCount: 176.
Image: taskhost.exe
VadRoot 8547c480 Vads 93 Clone 0 Private 410. Modified 107. Locked 0.
DeviceMap 8f909fc8
Token 9025d980
ElapsedTime 00:13:41.390
UserTime 00:00:00.000
KernelTime 00:00:00.125
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (1276, 50, 345) (5104KB, 200KB, 1380KB)
PeakWorkingSetSize 1278
VirtualSize 38 Mb
PeakVirtualSize 38 Mb
PageFaultCount 2040
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 669
THREAD 85471af0 Cid 0750.0754 Teb: 7ffdf000 Win32Thread: fe823598 WAIT: (UserRequest) UserMode Non-Alertable
8543f778 SynchronizationEvent
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 4012 Ticks: 52505 (0:00:13:40.390)
Context Switch Count 53
UserTime 00:00:00.000
KernelTime 00:00:00.078
Win32 Start Address taskhost!wWinMainCRTStartup (0x006b2e64)
Stack Init 8a3ebfd0 Current 8a3ebb30 Base 8a3ec000 Limit 8a3e9000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr
8a3ebb48 8268951d nt!KiSwapContext+0x26
8a3ebb8c 826cf460 nt!KiSwapThread+0x57b
8a3ebbe0 8268ccaf nt!KiCommitThreadWait+0×340
8a3ebcb8 828ad5bc nt!KeWaitForSingleObject+0×3ee
8a3ebd20 8269066a nt!NtWaitForSingleObject+0xc6
8a3ebd20 771e5704 nt!KiFastCallEntry+0×12a
001dfac0 771d429c ntdll!KiFastSystemCallRet
001dfac4 7543182c ntdll!NtWaitForSingleObject+0xc
001dfb30 76f54f23 KERNELBASE!WaitForSingleObjectEx+0×98
001dfb48 76f54ed2 kernel32!WaitForSingleObjectExStub+0×75
001dfb5c 006b3400 kernel32!WaitForSingleObject+0×12
001dfbbc 006b36c9 taskhost!UbpmpTaskHostSendResponseReceiveCommand+0×6c
001dfc10 006b2b52 taskhost!UbpmTaskHostWaitForCommands+0xf5
001dfc1c 006b2d0c taskhost!wWinMain+0xd
001dfcb0 76f536d6 taskhost!_initterm_e+0×1b1
001dfcbc 771c883c kernel32!BaseThreadInitThunk+0xe
001dfcfc 771c880f ntdll!__RtlUserThreadStart+0×70
001dfd14 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 8547dab0 Cid 0750.075c Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
8547dd98 SynchronizationTimer
8547de60 SynchronizationTimer
85431df0 SynchronizationEvent
85444500 SynchronizationTimer
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 5989 Ticks: 50528 (0:00:13:09.500)
Context Switch Count 9
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ntdll!TppWaiterpThread (0x771ccb5e)
Stack Init 8f698fd0 Current 8f698688 Base 8f699000 Limit 8f696000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr
8f6986a0 8268951d nt!KiSwapContext+0x26
8f6986e4 826cf460 nt!KiSwapThread+0x57b
8f698738 826cbb81 nt!KiCommitThreadWait+0×340
8f698940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f698bcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f698d18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f698d18 771e5704 nt!KiFastCallEntry+0×12a
0068fa44 771d427c ntdll!KiFastSystemCallRet
0068fa48 771ccc8a ntdll!NtWaitForMultipleObjects+0xc
0068fbdc 76f536d6 ntdll!TppWaiterpThread+0×33d
0068fbe8 771c883c kernel32!BaseThreadInitThunk+0xe
0068fc28 771c880f ntdll!__RtlUserThreadStart+0×70
0068fc40 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 8547d3f8 Cid 0750.0760 Teb: 7ffdd000 Win32Thread: fe81f888 WAIT: (UserRequest) UserMode Non-Alertable
8546dff0 NotificationEvent
8542a490 SynchronizationEvent
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 43362 Ticks: 13155 (0:00:03:25.546)
Context Switch Count 43
UserTime 00:00:00.000
KernelTime 00:00:00.078
Win32 Start Address taskhost!ComTaskMgrWnd::MsgPumpThreadProc (0x006b69f6)
Stack Init 8f6a3fd0 Current 8f6a3688 Base 8f6a4000 Limit 8f6a1000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr
8f6a36a0 8268951d nt!KiSwapContext+0x26
8f6a36e4 826cf460 nt!KiSwapThread+0x57b
8f6a3738 826cbb81 nt!KiCommitThreadWait+0×340
8f6a3940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f6a3bcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f6a3d18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f6a3d18 771e5704 nt!KiFastCallEntry+0×12a
0130f93c 771d427c ntdll!KiFastSystemCallRet
0130f940 75436e4d ntdll!NtWaitForMultipleObjects+0xc
0130f9dc 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
0130fa24 76fd9c0d kernel32!WaitForMultipleObjectsExStub+0xe0
0130fa78 76fda24f USER32!RealMsgWaitForMultipleObjectsEx+0×13c
0130fa94 006b6a46 USER32!MsgWaitForMultipleObjects+0×1f
0130fadc 76f536d6 taskhost!ComTaskMgrWnd::MsgPumpThreadProc+0×50
0130fae8 771c883c kernel32!BaseThreadInitThunk+0xe
0130fb28 771c880f ntdll!__RtlUserThreadStart+0×70
0130fb40 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 854b66a8 Cid 0750.0788 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
85394928 QueueObject
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 4060 Ticks: 52457 (0:00:13:39.640)
Context Switch Count 7
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0x771c8ede)
Stack Init 904e5fd0 Current 904e5b00 Base 904e6000 Limit 904e3000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr
904e5b18 8268951d nt!KiSwapContext+0x26
904e5b5c 826cf460 nt!KiSwapThread+0x57b
904e5bb0 826d2e5c nt!KiCommitThreadWait+0×340
904e5c38 828ad62d nt!KeRemoveQueueEx+0×7df
904e5c90 826d95cb nt!IoRemoveIoCompletion+0×23
904e5d24 8269066a nt!NtWaitForWorkViaWorkerFactory+0×1a1
904e5d24 771e5704 nt!KiFastCallEntry+0×12a
0148fc54 771d42ac ntdll!KiFastSystemCallRet
0148fc58 771cce31 ntdll!NtWaitForWorkViaWorkerFactory+0xc
0148fdbc 76f536d6 ntdll!TppWorkerThread+0×223
0148fdc8 771c883c kernel32!BaseThreadInitThunk+0xe
0148fe08 771c880f ntdll!__RtlUserThreadStart+0×70
0148fe20 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 85491658 Cid 0750.07b8 Teb: 7ffd3000 Win32Thread: fe4afbb8 WAIT: (UserRequest) UserMode Non-Alertable
8540c280 NotificationEvent
85494a08 NotificationEvent
85494980 NotificationEvent
854948f8 NotificationEvent
85494870 NotificationEvent
854947e8 NotificationEvent
85494760 NotificationEvent
854946d8 NotificationEvent
85494650 NotificationEvent
854945c8 NotificationEvent
85494540 NotificationEvent
8544ba30 NotificationEvent
85145480 NotificationEvent
84a27448 SynchronizationEvent
85459e50 SynchronizationEvent
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 53222 Ticks: 3295 (0:00:00:51.484)
Context Switch Count 738
UserTime 00:00:00.000
KernelTime 00:00:00.125
Win32 Start Address MsCtfMonitor!MsCtfMonitor::ThreadProc (0x702c208d)
Stack Init 89f1efd0 Current 89f1e688 Base 89f1f000 Limit 89f1c000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr
89f1e6a0 8268951d nt!KiSwapContext+0x26
89f1e6e4 826cf460 nt!KiSwapThread+0x57b
89f1e738 826cbb81 nt!KiCommitThreadWait+0×340
89f1e940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
89f1ebcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
89f1ed18 8269066a nt!NtWaitForMultipleObjects+0xcc
89f1ed18 771e5704 nt!KiFastCallEntry+0×12a
0142f864 771d427c ntdll!KiFastSystemCallRet
0142f868 75436e4d ntdll!NtWaitForMultipleObjects+0xc
0142f904 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
0142f94c 76fd9c0d kernel32!WaitForMultipleObjectsExStub+0xe0
0142f9a0 76fda24f USER32!RealMsgWaitForMultipleObjectsEx+0×13c
0142f9bc 702c1435 USER32!MsgWaitForMultipleObjects+0×1f
0142fb7c 702c20e1 MsCtfMonitor!DoMsCtfMonitor+0×2b8
0142fd9c 76f536d6 MsCtfMonitor!MsCtfMonitor::ThreadProc+0×5d
0142fda8 771c883c kernel32!BaseThreadInitThunk+0xe
0142fde8 771c880f ntdll!__RtlUserThreadStart+0×70
0142fe00 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 85491370 Cid 0750.07bc Teb: 7ffda000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
85492ff0 NotificationEvent
853bc030 NotificationEvent
IRP List:
85492408: (0006,0094) Flags: 00060070 Mdl: 00000000
85492568: (0006,0094) Flags: 00060070 Mdl: 00000000
854926c8: (0006,0094) Flags: 00060070 Mdl: 00000000
85492828: (0006,0094) Flags: 00060070 Mdl: 00000000
85492988: (0006,0094) Flags: 00060070 Mdl: 00000000
85492ae8: (0006,0094) Flags: 00060070 Mdl: 00000000
85492c48: (0006,0094) Flags: 00060070 Mdl: 00000000
85492da8: (0006,0094) Flags: 00060070 Mdl: 00000000
8544e4b8: (0006,0094) Flags: 00060070 Mdl: 00000000
853cf470: (0006,0094) Flags: 00060070 Mdl: 00000000
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 4060 Ticks: 52457 (0:00:13:39.640)
Context Switch Count 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address PlaySndSrv!CBeepRedirector::WorkThread (0x70271c6c)
Stack Init 8f65dfd0 Current 8f65d688 Base 8f65e000 Limit 8f65b000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr
8f65d6a0 8268951d nt!KiSwapContext+0x26
8f65d6e4 826cf460 nt!KiSwapThread+0x57b
8f65d738 826cbb81 nt!KiCommitThreadWait+0×340
8f65d940 828ae100 nt!KeWaitForMultipleObjects+0×5e3
8f65dbcc 828ade6b nt!ObpWaitForMultipleObjects+0×264
8f65dd18 8269066a nt!NtWaitForMultipleObjects+0xcc
8f65dd18 771e5704 nt!KiFastCallEntry+0×12a
01c6f6d4 771d427c ntdll!KiFastSystemCallRet
01c6f6d8 75436e4d ntdll!NtWaitForMultipleObjects+0xc
01c6f774 76f5506f KERNELBASE!WaitForMultipleObjectsEx+0×100
01c6f7bc 70271cdd kernel32!WaitForMultipleObjectsExStub+0xe0
01c6f93c 76f536d6 PlaySndSrv!CBeepRedirector::WorkThread+0×266
01c6f948 771c883c kernel32!BaseThreadInitThunk+0xe
01c6f988 771c880f ntdll!__RtlUserThreadStart+0×70
01c6f9a0 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 84a01370 Cid 0750.07c8 Teb: 7ffd9000 Win32Thread: fe4afde0 WAIT: (WrLpcReceive) UserMode Non-Alertable
84a015a4 Semaphore Limit 0x1
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 53222 Ticks: 3295 (0:00:00:51.484)
Context Switch Count 890
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address MSCTF!CCtfServerPort::StaticServerThread (0x76bea423)
Stack Init 89e4ffd0 Current 89e4fa78 Base 89e50000 Limit 89e4d000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr
89e4fa90 8268951d nt!KiSwapContext+0x26
89e4fad4 826cf460 nt!KiSwapThread+0x57b
89e4fb28 8268ccaf nt!KiCommitThreadWait+0×340
89e4fc04 828b9a5a nt!KeWaitForSingleObject+0×3ee
89e4fc34 828ba1c9 nt!AlpcpReceiveMessagePort+0×245
89e4fcb4 828ba489 nt!AlpcpReceiveMessage+0×1b8
89e4fd0c 8269066a nt!NtAlpcSendWaitReceivePort+0×11b
89e4fd0c 771e5704 nt!KiFastCallEntry+0×12a
005feb10 771d2c8c ntdll!KiFastSystemCallRet
005feb14 76bd5b34 ntdll!NtAlpcSendWaitReceivePort+0xc
005ffb9c 76bea53c MSCTF!CCtfServerPort::ServerLoop+0×136
005ffe2c 76bea441 MSCTF!CCtfServerPort::ServerThread+0xde
005ffe3c 76f536d6 MSCTF!CCtfServerPort::StaticServerThread+0×22
005ffe48 771c883c kernel32!BaseThreadInitThunk+0xe
005ffe88 771c880f ntdll!__RtlUserThreadStart+0×70
005ffea0 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 854cc5f0 Cid 0750.0114 Teb: 7ffd8000 Win32Thread: fe4bb008 WAIT: (WrUserRequest) UserMode Non-Alertable
854cc488 SynchronizationEvent
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 53850 Ticks: 2667 (0:00:00:41.671)
Context Switch Count 301
UserTime 00:00:00.000
KernelTime 00:00:00.218
Win32 Start Address WINMM!mciwindow (0x73942761)
Stack Init 904c6fd0 Current 904c6a60 Base 904c7000 Limit 904c4000 Call 0
Priority 12 BasePriority 10 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr
904c6a78 8268951d nt!KiSwapContext+0x26
904c6abc 826cf460 nt!KiSwapThread+0x57b
904c6b10 8268ccaf nt!KiCommitThreadWait+0×340
904c6be8 8e50c768 nt!KeWaitForSingleObject+0×3ee
904c6c44 8e50c575 win32k!xxxRealSleepThread+0×1d7
904c6c60 8e508379 win32k!xxxSleepThread+0×2d
904c6cb8 8e50cf9a win32k!xxxRealInternalGetMessage+0×4b2
904c6d1c 8269066a win32k!NtUserGetMessage+0×3f
904c6d1c 771e5704 nt!KiFastCallEntry+0×12a
0169f7d8 76fdbb29 ntdll!KiFastSystemCallRet
0169f7dc 76fd3f49 USER32!NtUserGetMessage+0xc
0169f800 739427e0 USER32!GetMessageA+0×8d
0169f838 76f536d6 WINMM!mciwindow+0×102
0169f844 771c883c kernel32!BaseThreadInitThunk+0xe
0169f884 771c880f ntdll!__RtlUserThreadStart+0×70
0169f89c 00000000 ntdll!_RtlUserThreadStart+0×1b
THREAD 83bafd48 Cid 0750.09f8 Teb: 7ffdb000 Win32Thread: fe569198 WAIT: (WrQueue) UserMode Alertable
8547dfd0 QueueObject
83bafdd8 NotificationTimer
Not impersonating
DeviceMap 8f909fc8
Owning Process 85471598 Image: taskhost.exe
Wait Start TickCount 53850 Ticks: 2667 (0:00:00:41.671)
Context Switch Count 102
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address ntdll!TppWorkerThread (0x771c8ede)
Stack Init 8bff3fd0 Current 8bff3b00 Base 8bff4000 Limit 8bff1000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr
8bff3b18 8268951d nt!KiSwapContext+0x26
8bff3b5c 826cf460 nt!KiSwapThread+0x57b
8bff3bb0 826d2e5c nt!KiCommitThreadWait+0×340
8bff3c38 828ad62d nt!KeRemoveQueueEx+0×7df
8bff3c90 826d95cb nt!IoRemoveIoCompletion+0×23
8bff3d24 8269066a nt!NtWaitForWorkViaWorkerFactory+0×1a1
8bff3d24 771e5704 nt!KiFastCallEntry+0×12a
0184f9f4 771d42ac ntdll!KiFastSystemCallRet
0184f9f8 771cce31 ntdll!NtWaitForWorkViaWorkerFactory+0xc
0184fb5c 76f536d6 ntdll!TppWorkerThread+0×223
0184fb68 771c883c kernel32!BaseThreadInitThunk+0xe
0184fba8 771c880f ntdll!__RtlUserThreadStart+0×70
0184fbc0 00000000 ntdll!_RtlUserThreadStart+0×1b
kd> lmv m taskhost
start end module name
006b0000 006be000 taskhost (deferred)
Image path: C:\Windows\system32\taskhost.exe
Image name: taskhost.exe
Timestamp: Sat Dec 13 02:02:54 2008 (494317CE)
CheckSum: 00011C71
ImageSize: 0000E000
File version: 6.1.7000.0
Product version: 6.1.7000.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: taskhost.exe
OriginalFilename: taskhost.exe
ProductVersion: 6.1.7000.0
FileVersion: 6.1.7000.0 (winmain_win7beta.081212-1400)
FileDescription: Host Process for Windows Tasks
LegalCopyright: © Microsoft Corporation. All rights reserved.
Functions that previously called kernel32 API now call their stub equivalents in kernel32 (function names affixed with Stub) and then stubs call KERNELBASE functions having previous kernel32 function names.
It can be seen from dumping contents of import directories of USER32, ADVAPI32, and GDI32 modules that they also depend on KERNELBASE. For example, for GDI32 we have:
kd> !dh 75e70000
File Type: DLL
FILE HEADER VALUES
14C machine (i386)
4 number of sections
49433CCD time date stamp Sat Dec 13 04:40:45 2008
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL
OPTIONAL HEADER VALUES
10B magic #
9.00 linker version
46600 size of code
3A00 size of initialized data
0 size of uninitialized data
CF7C address of entry point
1000 base of code
----- new -----
75e70000 image base
1000 section alignment
200 file alignment
3 subsystem (Windows CUI)
6.01 operating system version
6.01 image version
6.01 subsystem version
4D000 size of image
800 size of headers
4D765 checksum
00040000 size of stack reserve
00001000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
1284 [ 4BB4] address [size] of Export Directory
46308 [ 1B8] address [size] of Import Directory
4A000 [ 3D0] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
4B000 [ 1920] address [size] of Base Relocation Directory
474F0 [ 38] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
2A000 [ 40] address [size] of Load Configuration Directory
280 [ 3E4] address [size] of Bound Import Directory
1000 [ 284] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
[…]
kd> dds 75e70000+1000 75e70000+1000+284
75e71000 771d3da0 ntdll!ZwSecureConnectPort
75e71004 771d3bb0 ntdll!ZwRegisterThreadTerminatePort
75e71008 771d38b0 ntdll!ZwQueryInformationProcess
75e7100c 771ab232 ntdll!RtlUnwind
75e71010 771d3680 ntdll!NtOpenThreadToken
75e71014 771d3600 ntdll!ZwOpenProcessToken
75e71018 771d38e0 ntdll!NtQueryInformationToken
75e7101c 771adecf ntdll!RtlLengthSid
75e71020 771adeeb ntdll!RtlCopySid
75e71024 771d3cd0 ntdll!ZwRequestWaitReplyPort
75e71028 771bb080 ntdll!_vsnwprintf
75e7102c 771aca7c ntdll!_strnicmp
75e71030 771b75a8 ntdll!_stricmp
75e71034 771b30f4 ntdll!RtlCreateUnicodeStringFromAsciiz
75e71038 771d59c0 ntdll!strncpy
75e7103c 771d4230 ntdll!ZwUnmapViewOfSection
75e71040 771f3b4b ntdll!RtlMultiByteToUnicodeN
75e71044 771c9339 ntdll!RtlDosPathNameToNtPathName_U
75e71048 771d3490 ntdll!NtMapViewOfSection
75e7104c 771d2f50 ntdll!NtCreateSection
75e71050 771d3880 ntdll!ZwQueryInformationFile
75e71054 771d5580 ntdll!memset
75e71058 771d5240 ntdll!memmove
75e7105c 771f1f7e ntdll!RtlUnicodeToMultiByteN
75e71060 771f221b ntdll!RtlUnicodeToMultiByteSize
75e71064 771b069d ntdll!RtlInitializeCriticalSection
75e71068 771b77b7 ntdll!RtlEncodePointer
75e7106c 771c5093 ntdll!RtlDeleteCriticalSection
75e71070 771d43b0 ntdll!RtlInitUnicodeString
75e71074 771d3570 ntdll!NtOpenKey
75e71078 771d3ab0 ntdll!NtQueryValueKey
75e7107c 771d2d30 ntdll!ZwClose
75e71080 771d3540 ntdll!ZwOpenFile
75e71084 771cf682 ntdll!_wcsnicmp
75e71088 771cc1cd ntdll!RtlNtStatusToDosError
75e7108c 771f2a11 ntdll!RtlFreeAnsiString
75e71090 771c2fe5 ntdll!RtlNtPathNameToDosPathName
75e71094 771a3e05 ntdll!RtlpEnsureBufferSize
75e71098 771b3cf0 ntdll!_wcsicmp
75e7109c 771b13db ntdll!wcschr
75e710a0 771cf0ea ntdll!wcsrchr
75e710a4 771d5e00 ntdll!RtlCompareMemory
75e710a8 771bd9e4 ntdll!RtlDecodePointer
75e710ac 771d4240 ntdll!NtVdmControl
75e710b0 771f0ea0 ntdll!RtlAllocateHeap
75e710b4 771f0fb0 ntdll!RtlFreeHeap
75e710b8 771d4f00 ntdll!memcpy
75e710bc 771f1068 ntdll!RtlLeaveCriticalSection
75e710c0 771f10a6 ntdll!RtlEnterCriticalSection
75e710c4 00000000
75e710c8 75440220 KERNELBASE!IsDBCSLeadByte
75e710cc 7544f8b9 KERNELBASE!IsDBCSLeadByteEx
75e710d0 00000000
75e710d4 75436dec KERNELBASE!GetLastError
75e710d8 7545f842 KERNELBASE!UnhandledExceptionFilter
75e710dc 7544c2b3 KERNELBASE!SetUnhandledExceptionFilter
75e710e0 771f1412 ntdll!RtlSetLastWin32Error
75e710e4 00000000
75e710e8 76f465cc kernel32!GetDriveTypeWStub
75e710ec 76f55685 kernel32!WriteFileStub
75e710f0 76f55169 kernel32!CreateFileWStub
75e710f4 76f466b8 kernel32!GetFullPathNameWStub
75e710f8 76f40808 kernel32!DeleteFileWStub
75e710fc 76f354aa kernel32!SetFilePointerExStub
75e71100 76f4a269 kernel32!SetFilePointerStub
75e71104 76f40c4d kernel32!GetFileSizeExStub
75e71108 76f370ed kernel32!GetTempFileNameWStub
75e7110c 00000000
75e71110 76f55137 kernel32!CloseHandleStub
75e71114 00000000
75e71118 75436d3a KERNELBASE!InterlockedCompareExchange
75e7111c 00000000
75e71120 7543ab61 KERNELBASE!FreeLibrary
75e71124 754436f1 KERNELBASE!SizeofResource
75e71128 754376d8 KERNELBASE!GetModuleHandleA
75e7112c 7543bb5a KERNELBASE!LoadLibraryExW
75e71130 75438116 KERNELBASE!SetHandleCount
75e71134 7544367e KERNELBASE!LoadResource
75e71138 7543cad6 KERNELBASE!DisableThreadLibraryCalls
75e7113c 7543762d KERNELBASE!GetProcAddress
75e71140 00000000
75e71144 7543810b KERNELBASE!GetACP
75e71148 75444dee KERNELBASE!GetLocaleInfoW
75e7114c 7544c484 KERNELBASE!GetOEMCP
75e71150 00000000
75e71154 7543d213 KERNELBASE!RegOpenKeyExA
75e71158 75439771 KERNELBASE!RegCloseKey
75e7115c 7543d379 KERNELBASE!RegQueryValueExA
75e71160 75439549 KERNELBASE!RegOpenKeyExW
75e71164 75449b64 KERNELBASE!RegEnumValueW
75e71168 00000000
75e7116c 754373cc KERNELBASE!UnmapViewOfFile
75e71170 7543fc4c KERNELBASE!CreateFileMappingW
75e71174 7543fbc8 KERNELBASE!MapViewOfFile
75e71178 00000000
75e7117c 75438854 KERNELBASE!GlobalFree
75e71180 75437256 KERNELBASE!lstrlenW
75e71184 7543cec7 KERNELBASE!LocalReAlloc
75e71188 754388d1 KERNELBASE!LocalAlloc
75e7118c 7543d9a9 KERNELBASE!GlobalAlloc
75e71190 75438e61 KERNELBASE!lstrlenA
75e71194 75438854 KERNELBASE!GlobalFree
75e71198 00000000
75e7119c 75449d05 KERNELBASE!SearchPathW
75e711a0 00000000
75e711a4 75436d30 KERNELBASE!GetCurrentThreadId
75e711a8 75436e20 KERNELBASE!GetCurrentProcessId
75e711ac 7543771a KERNELBASE!ProcessIdToSessionId
75e711b0 754370bf KERNELBASE!GetCurrentThread
75e711b4 75459f89 KERNELBASE!TerminateProcess
75e711b8 75436dfb KERNELBASE!GetCurrentProcess
75e711bc 00000000
75e711c0 771f145a ntdll!RtlQueryPerformanceCounter
75e711c4 00000000
75e711c8 7545a887 KERNELBASE!IsWellKnownSid
75e711cc 00000000
75e711d0 75437e76 KERNELBASE!MultiByteToWideChar
75e711d4 7543839a KERNELBASE!WideCharToMultiByte
75e711d8 00000000
75e711dc 771c5093 ntdll!RtlDeleteCriticalSection
75e711e0 771f1068 ntdll!RtlLeaveCriticalSection
75e711e4 771b069d ntdll!RtlInitializeCriticalSection
75e711e8 771f10a6 ntdll!RtlEnterCriticalSection
75e711ec 00000000
75e711f0 75438eb9 KERNELBASE!GetTickCount64+0×4
75e711f4 7543f6ea KERNELBASE!GetWindowsDirectoryW
75e711f8 7543f67b KERNELBASE!GetSystemWindowsDirectoryW
75e711fc 7543aa71 KERNELBASE!GetSystemInfo
75e71200 754387b0 KERNELBASE!GetLocalTime
75e71204 75436cc3 KERNELBASE!GetTickCount+0×4
75e71208 7543712d KERNELBASE!GetSystemTimeAsFileTime
75e7120c 00000000
75e71210 76f351d4 kernel32!CopyFileW
75e71214 76f526c8 kernel32!GlobalLock
75e71218 76f54be0 kernel32!MulDiv
75e7121c 76f4662d kernel32!LoadLibraryW
75e71220 76f3b86c kernel32!GlobalSize
75e71224 76f3a5c0 kernel32!GetTempPathW
75e71228 76f40c2f kernel32!FindResourceW
75e7122c 76f45a27 kernel32!LoadLibraryA
75e71230 76f37015 kernel32!VirtualUnlock
75e71234 76f5018b kernel32!GlobalUnlock
75e71238 00000000
75e7123c 76fd89ed USER32!GetAppCompatFlags2
75e71240 76fd68f6 USER32!InitializeLpkHooks
75e71244 76fda345 USER32!NtUserGetDC
75e71248 76ff21c7 USER32!UserRealizePalette
75e7124c 76fd34f2 USER32!GetAppCompatFlags
75e71250 76fd7c23 USER32!CharUpperBuffA
75e71254 76fe17ff USER32!IsThreadDesktopComposited
75e71258 76fda409 USER32!GetWindowRect
75e7125c 76fe1766 USER32!IntersectRect
75e71260 76fd7ce4 USER32!CharLowerBuffW
75e71264 76fda31a USER32!ReleaseDC
75e71268 00000000
75e7126c 772e1bbf LPK!LpkUseGDIWidthCache
75e71270 772e4e3e LPK!LpkGetCharacterPlacement
75e71274 772e167a LPK!LpkExtTextOut
75e71278 772e1df6 LPK!LpkGetTextExtentExPoint
75e7127c 772e1898 LPK!LpkInitialize
75e71280 00000000
75e71284 00000000
- Dmitry Vostokov @ DumpAnalysis.org -
After an upgrade to the new version of a productivity software package one unrelated application started to crash frequently. A crash dump was collected and the following stack trace pointed to a NULL code pointer:
0:000> r
eax=09680104 ebx=0013aefc ecx=0968a710 edx=0cdc0c0c esi=16a19058 edi=00000001
eip=00000000 esp=0013aea8 ebp=0013aeb8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200202
00000000 ?? ???
0:000> k 100
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0013aea4 096e28a0 0×0
0013aeb8 32e688bd dllC!Abort+0×10
0013aec8 32c82395 dllB+0×589e
0013aed8 32865718 dllB+0×18f1
[…]
0013b0c0 314de1ff dllB+0×4c6
0013b154 31293494 dllA!DllGetLCID+0×46d2d
0013b178 312af217 dllA!DllGetClassObject+0×4e896
[…]
0013f3d0 300e8721 dllA!DllGetClassObject+0×69e42
0013f578 300e7f5a application+0xcff5
[…]
0013ffc0 7c816ff7 application+0×51d5
0013fff0 00000000 kernel32!BaseProcessStart+0×23
To see if changed environment somehow affected this application the presence of any DLL hooks was checked. The following hooked functions were found in user32.dll:
0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\USER32.dll\45F02D7C90000\USER32.dll
No range specified
Scanning section: .text
Size: 389095
Range to scan: 7e411000-7e46ffe7
7e4188a6-7e4188aa 5 bytes - USER32!GetWindowLongW
[ 6a 08 68 e0 88:e9 55 77 a4 01 ]
7e418f9c-7e418fa0 5 bytes - USER32!GetSystemMetrics (+0×6f6)
[ 6a 10 68 00 90:e9 5f 70 b5 01 ]
7e41945d-7e419461 5 bytes - USER32!GetWindowLongA (+0×4c1)
[ 6a 10 68 78 9f:e9 f5 60 a2 01 ]
7e41b6ae-7e41b6b2 5 bytes - USER32!GetClientRect (+0×17a8)
[ 8b ff 55 8b ec:e9 4d 49 9f 01 ]
7e41b6d4-7e41b6d8 5 bytes - USER32!GetWindowRect (+0×26)
[ b8 74 11 00 00:e9 98 30 af 01 ]
7e41d60d-7e41d611 5 bytes - USER32!SetWindowLongA (+0×6aa)
[ 8b ff 55 8b ec:e9 ee 29 a5 01 ]
7e41d62b-7e41d62f 5 bytes - USER32!SetWindowLongW (+0×1e)
[ 6a 08 68 28 f5:e9 0e 0b b2 01 ]
7e41fc25-7e41fc29 5 bytes - USER32!CreateWindowExW (+0×738)
[ 8b ff 55 8b ec:e9 d6 03 b6 01 ]
7e41ff33-7e41ff37 5 bytes - USER32!CreateWindowExA (+0×30e)
Total bytes compared: 389095(100%)
Number of errors: 52
52 errors : !user32 (7e4188a6-7e42e8d5)
The hooking DLL was found to be from that upgraded package:
0:000> u 7e4188a6
USER32!GetWindowLongW:
7e4188a6 e95577a401 jmp 7fe60000
7e4188ab 41 inc ecx
7e4188ac 7ee8 jle USER32!_GetWindowLong+0xda (7e418896)
7e4188ae 0e push cs
7e4188af fd std
7e4188b0 ff ???
7e4188b1 ff8b4d08e816 dec dword ptr [ebx+16E8084Dh]
7e4188b7 fc cld
0:000> u 7fe60000
7fe60000 e9bb62b080 jmp hookA+0×62c0 (009662c0)
7fe60005 6a08 push 8
7fe60007 68e088417e push offset USER32!`string’+0×34 (7e4188e0)
7fe6000c e99c885bfe jmp USER32!GetWindowLongW+0×7 (7e4188ad)
7fe60011 0000 add byte ptr [eax],al
7fe60013 0000 add byte ptr [eax],al
7fe60015 0000 add byte ptr [eax],al
7fe60017 0000 add byte ptr [eax],al
0:000> lmv m hookA
start end module name
00960000 00976000 hookA (no symbols)
Loaded symbol image file: hookA.dll
Image path: C:\Program Files\CompanyA\hookA.dll
Image name: hookA.dll
[…]
ProductName: ProductA
[…]
Execution residue from hookA module was also found on the problem thread raw stack and it looks like real code (not a coincidental symbolic information):
0:000> !teb
TEB at 7ffdf000
ExceptionList: 0013f02c
StackBase: 00140000
StackLimit: 0010c000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffdf000
EnvironmentPointer: 00000000
ClientId: 00000c38 . 00000840
RpcHandle: 00000000
Tls Storage: 00163268
PEB Address: 7ffdb000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 0
0:000> dds 0010c000 00140000
0010c000 00000000
0010c004 00000000
0010c008 00000000
[…]
00121f50 0096d7d1*** ERROR: Module load completed but symbols could not be loaded for hookA.dll
hookA+0xd7d1
00121f54 00009924
00121f58 00121fb4
00121f5c 00000000
00121f60 00121f9c
00121f64 0096d895 hookA+0xd895
00121f68 00121f78
00121f6c 00000000
00121f70 00122008
00121f74 00000000
00121f78 00000000
00121f7c 00000000
00121f80 00000000
00121f84 00121f6c
00121f88 000001fe
00121f8c 001220fc
00121f90 0096ec98 hookA+0xec98
00121f94 00970e48 hookA+0×10e48
00121f98 00121fb4
00121f9c 7e41b6a3 USER32!_GetClientRect+0×6e
00121fa0 00122008
00121fa4 fffffa01
[…]
0:000> u 0096d895
hookA+0xd895:
0096d895 8bc6 mov eax,esi
0096d897 8b4df0 mov ecx,dword ptr [ebp-10h]
0096d89a 64890d00000000 mov dword ptr fs:[0],ecx
0096d8a1 5f pop edi
0096d8a2 5e pop esi
0096d8a3 5b pop ebx
0096d8a4 8be5 mov esp,ebp
0096d8a6 5d pop ebp
0:000> ub 0096d895
hookA+0xd876:
0096d876 8b65e8 mov esp,dword ptr [ebp-18h]
0096d879 be0d000000 mov esi,0Dh
0096d87e c745fcffffffff mov dword ptr [ebp-4],0FFFFFFFFh
0096d885 eb05 jmp hookA+0xd88c (0096d88c)
0096d887 be72000000 mov esi,72h
0096d88c 8d55dc lea edx,[ebp-24h]
0096d88f 52 push edx
0096d890 e8fbfeffff call hookA+0xd790 (0096d790)
As was found the upgraded application had special DLL hooks to improve productivity and ease of use of GUI applications. Fortunately it was possible to disable that hook on a per-application basis and application crashes disappeared.
- Dmitry Vostokov @ DumpAnalysis.org -
The number of visits / month increased by 50% by the end of the year with more than 125,000 unique visitors in 2008 from 180 countries (150 in 2007) and almost 34% of them coming back. Here are the top 100 network locations out of almost 24,000:
|
Network Location |
Visits |
|
microsoft corp |
7849 |
|
insignium llc |
6206 |
|
comcast cable communications inc. |
3603 |
|
road runner holdco llc |
3260 |
|
verizon internet services inc. |
2263 |
|
comite gestor da internet no brasil |
1818 |
|
deutsche telekom ag |
1804 |
|
hewlett-packard company |
1736 |
|
network of citrix systems inc |
1667 |
|
eircom ltd |
1582 |
|
japan network information center |
1569 |
|
reliance communications ltd |
1419 |
|
videsh sanchar nigam ltd - india. |
1414 |
|
chunghwa telecom data communication business group |
1371 |
|
symantec corporation |
1228 |
|
unknown |
1167 |
|
data general corporation |
1098 |
|
microsoft european internet data centres |
1096 |
|
comcast cable communications |
1074 |
|
bellsouth.net inc. |
1048 |
|
ip pools |
1025 |
|
intel corporation |
987 |
|
charter communications |
967 |
|
nib (national internet backbone) |
960 |
|
korea telecom |
895 |
|
cncgroup beijing province network |
885 |
|
cox communications |
875 |
|
comcast cable communications holdings inc |
835 |
|
proxad / free sas |
785 |
|
uunet non-portable customer assignment |
710 |
|
shaw communications inc. |
675 |
|
citrix systems inc. |
662 |
|
xo communications |
660 |
|
cox communications inc. |
655 |
|
comcast cable communications ip services |
646 |
|
qwest communications corporation |
617 |
|
krnic |
571 |
|
rcs & rds s.a. |
566 |
|
jarvis universal purchase company |
551 |
|
chinanet guangdong province network |
545 |
|
optimum online (cablevision systems) |
541 |
|
computer associates international |
540 |
|
telstra internet |
540 |
|
dell computer corporation |
514 |
|
rogers cable communications inc. |
509 |
|
axa-tech |
506 |
|
provider local registry |
492 |
|
chinanet shanghai province network |
487 |
|
performance systems international inc. |
454 |
|
telus communications inc. |
445 |
|
kintiskton llc |
444 |
|
at&t internet services |
443 |
|
arcor ag |
434 |
|
merrill lynch and company inc. |
433 |
|
ntt communications corporation |
428 |
|
easynet ltd |
426 |
|
research in motion limited |
419 |
|
iinet limited |
399 |
|
vmware inc. |
395 |
|
abts (karnataka) |
393 |
|
symantec inc |
390 |
|
hoshin gigamedia center inc. |
389 |
|
zao mtu-intel |
389 |
|
microsoft corporation |
369 |
|
telefonica de espana |
356 |
|
internet service provider |
352 |
|
time warner telecom inc. |
351 |
|
@home network japan |
348 |
|
telekom malaysia berhad |
342 |
|
sympatico hse |
341 |
|
network of ign arch. and design gb |
327 |
|
global crossing |
322 |
|
wipro technologies |
308 |
|
xdsl access and service provider in norway |
301 |
|
chinanet fujian province network |
289 |
|
at&t global network services |
283 |
|
comcast cable communications inc |
277 |
|
neostrada plus |
274 |
|
samtel |
273 |
|
oracle datenbanksysteme gmbh |
272 |
|
provider |
266 |
|
telecom italia net |
265 |
|
sun microsystems inc |
257 |
|
tiscali uk ltd |
254 |
|
starhub cable vision ltd |
251 |
|
telecom italia s.p.a. tin easy lite |
251 |
|
tw telecom holdings inc. |
249 |
|
earthlink inc. |
247 |
|
level 3 communications inc. |
247 |
|
kddi corporation |
245 |
|
comcast business communications inc. |
242 |
|
nvidia |
242 |
|
cisco systems inc. |
240 |
|
hanaro telecom inc. |
239 |
|
chinanet jiangsu province network |
235 |
|
internet provider of donetsk region |
234 |
|
videotron ltee |
230 |
|
xs4all internet bv |
229 |
|
gesti n de direccionamiento uninet |
224 |
|
dynamic pools |
223 |
Almost 66,000 Google search keywords (more than 100% increase since 2007) pointed to the portal and this blog with 100 most frequent:
|
Keyword |
Visits |
|
kifastsystemcallret |
2483 |
|
crash dump analysis |
1933 |
|
crash dump |
1551 |
|
ntdll!kifastsystemcallret |
1072 |
|
dump analysis |
852 |
|
crash dumps |
608 |
|
windbg commands |
560 |
|
dumpanalysis.org/asmpedia |
537 |
|
vista crash dump |
537 |
|
kmode_exception_not_handled |
521 |
|
crashdump |
509 |
|
minidump |
429 |
|
win32 error 0n2 |
412 |
|
memory dump analysis anthology |
395 |
|
dynamicbase aslr |
362 |
|
symbol file could not be found |
357 |
|
system_service_exception |
316 |
|
windbg |
311 |
|
memuon |
298 |
|
windbg analyze |
292 |
|
dmitry vostokov |
289 |
|
warning: frame ip not in any known module. following frames may be wrong. |
272 |
|
kernel32!pnlsuserinfo |
267 |
|
time travel debugging |
258 |
|
crash dump vista |
251 |
|
memory dump analysis |
243 |
|
minidump analysis |
236 |
|
getcontextstate failed, 0×80070026 |
230 |
|
dumpanalysis.org |
225 |
|
dr watson vista |
218 |
|
windbg script |
206 |
|
memory intelligence analysis”" |
205 |
|
crash dump analyzer |
187 |
|
kernel_mode_exception_not_handled |
182 |
|
frame ip not in any known module |
180 |
|
windows crash dump analysis |
179 |
|
calling+kernel+functions+from+userspace |
175 |
|
minidump analyzer |
172 |
|
windows via c/c++ |
170 |
|
dumpanalysis |
169 |
|
the stored exception information can be accessed via .ecxr. |
159 |
|
warning: stack unwind information not available. following frames may be wrong. |
159 |
|
pool corruption |
158 |
|
your debugger is not using the correct symbols |
158 |
|
error: symbol file could not be found |
157 |
|
windbg scripts |
156 |
|
drwtsn32 vista |
143 |
|
windbg cheat sheet |
142 |
|
minidump analyze |
136 |
|
adplus |
134 |
|
memory dump analysis”" download |
132 |
|
www.dump |
128 |
|
ibmsprem.exe |
126 |
|
session_has_valid_views_on_exit (ba) |
125 |
|
what is a crash dump |
125 |
|
bios disassembly ninjutsu uncovered |
122 |
|
the stored exception information can be accessed via .ecxr |
122 |
|
how to use windbg |
121 |
|
memory dump |
121 |
|
trap frame |
121 |
|
gdb teb |
119 |
|
type referenced: kernel32!pnlsuserinfo |
118 |
|
windows dump analysis |
118 |
|
savedump.exe |
115 |
|
bugcheck a |
113 |
|
windbg crash dump |
113 |
|
0×80070026 |
110 |
|
dxg.sys |
110 |
|
dump analyzer |
109 |
|
windbg !analyze |
106 |
|
how to open corrupt memory dump |
105 |
|
kisystemservicecopyend |
104 |
|
exfreepoolwithtag |
103 |
|
dump |
102 |
|
windbg command |
101 |
|
obfreferenceobject |
99 |
|
analyze minidump |
96 |
|
forthcoming windows® debugging: practical foundations |
95 |
|
kiswapcontext |
95 |
|
failure_bucket_id |
93 |
|
ntdll kifastsystemcallret |
91 |
|
regionusageisvad |
91 |
|
c++ dereferencing null debug |
90 |
|
receivelotsacalls |
90 |
|
userdump |
90 |
|
debug_flr_image_timestamp |
89 |
|
kifastsystemcall |
89 |
|
bugcheck 3b |
87 |
|
your debugger is not using the correct symbols”" |
86 |
|
vista dr watson |
84 |
|
windows crash dump |
84 |
|
“flow analysis was incomplete, some code may be missing” |
83 |
|
practical foundations of debugging |
83 |
|
system_thread_exception_not_handled |
83 |
|
warning: frame ip not in any known module. following frames may be wrong |
83 |
|
windbg dump |
83 |
|
dd srvcomputername |
81 |
|
error: symbol file could not be found. |
79 |
|
windows dump analyzer |
78 |
|
crash analyzer |
77 |
Special thanks to 950 web sites that mention the portal and this blog with the first top 100:
|
google.com |
|
blogs.msdn.com |
|
windbg.dumpanalysis.org |
|
rsdn.ru |
|
jasonhaley.com |
|
dumpanalysis.com |
|
dumpanalysis.org |
|
stumbleupon.com |
|
images.google.com |
|
advancedwindowsdebugging.com |
|
nynaeve.net |
|
blog.flexilis.com |
|
blog.not-a-kernel-guy.com |
|
brianmadden.com |
|
voneinem-windbg.blogspot.com |
|
insidewindows.kr |
|
forum.sysinternals.com |
|
caloni.com.br |
|
en.wikipedia.org |
|
debuglab.com |
|
reddit.com |
|
winvistaclub.com |
|
driveronline.org |
|
127.0.0.1:12108 |
|
support.citrix.com |
|
softwareastrology.com |
|
managementbits.com |
|
hanrss.com |
|
opentask.com |
|
msuiche.net |
|
blog.naver.com |
|
bloglines.com |
|
blogs.microsoft.co.il |
|
clausbrod.de |
|
citrixblogger.org |
|
images.google.co.uk |
|
reconstructer.org |
|
advdbg.org |
|
community.citrix.com |
|
google.co.kr |
|
stackoverflow.com |
|
citrite.org |
|
mail.google.com |
|
serious-code.net |
|
shellexecute.wordpress.com |
|
experts-exchange.com |
|
google.co.uk |
|
groups.google.com |
|
wasm.ru |
|
microsoft.com |
|
images.google.co.in |
|
dogpile.com |
|
google.ca |
|
images.google.de |
|
del.icio.us |
|
thinkdigit.com |
|
google.co.in |
|
blog.gamedeff.com |
|
blogs.technet.com |
|
ttoyota.com |
|
goozydumps.wordpress.com |
|
software.rkuster.com |
|
users.livejournal.com |
|
insidekernel.net |
|
insiderim |
|
jpassing.wordpress.com |
|
10.1.12.201 |
|
d.hatena.ne.jp |
|
google.ru |
|
archut.net |
|
isisaka.com |
|
facebook.com |
|
devnote.net |
|
evilcodecave.wordpress.com |
|
google-analytics.com |
|
my.live.com |
|
shm.polar.tw |
|
64.233.183.104 |
|
technorati.com |
|
delicious.com |
|
tarasc0.blogspot.com |
|
literatescientist.com |
|
search.naver.com |
|
linkedin.com |
|
pubforum.info |
|
twitter.com |
|
nyx.cz |
|
hongyver.pe.kr |
|
cnblogs.com |
|
bishop3000.livejournal.com |
|
webmail.dumpanalysis.org |
|
fafeng.blogbus.com |
|
driverentry.com.br |
|
gp32x.com |
|
windowstips.wordpress.com |
|
vahidnasiri.blogspot.com |
|
209.85.173.104 |
|
images.google.ca |
|
kerneldebugging.com |
Top 25 visiting countries:
|
United States |
|
United Kingdom |
|
India |
|
Germany |
|
Canada |
|
China |
|
Russia |
|
Japan |
|
France |
|
South Korea |
|
Ireland |
|
Australia |
|
Taiwan |
|
Netherlands |
|
Israel |
|
Italy |
|
Sweden |
|
Brazil |
|
Singapore |
|
Spain |
|
Ukraine |
|
Romania |
|
Poland |
|
Norway |
|
Belgium |
- Dmitry Vostokov @ DumpAnalysis.org -
What is the source of our intuition about ∞, or ∞∞, more powers of ∞, and even ∞ number of powers? I believe that the underlying structure of our Universe or at least a universe as a model of Universe, Infinite Memory, with perceived processes as limits and Time Arrow as a bundle of sequences of memory pointers, provides basis for our intuition about infinite.
- Dmitry Vostokov @ DumpAnalysis.org -
Listening to étude No. 1 in C major written by Frédéric Chopin (Op. 10) an idea came to me about writing 16 debugging études (ISBN: 978-1906717575). It is surprising that there are many programming études out there but there are no debugging ones. Stay tuned and be in touch with this blog.
Draft definition:
Debugging étude is a composition of software with intentional defects (bugs) of considerable difficulty to find and fix, designed to provide practice material to perfect debugging techniques and problem-solving skills.
The idea actually came to me long time ago to create some sort of debugging excersises for training purposes.
- Dmitry Vostokov @ DumpAnalysis.org -
DumpAnalysis.org announces forthcoming 2010 as The Year of Dump Analysis.
Q&A
Q. Why 2010?
A. Two reasons: 1) To do dump analysis effectively and efficiently an engineer needs some experience in debugging acquired in the previous year of debugging (perhaps after 7 debugging nights, 2009, 0×7D9); 2) 2010 is 0×7DA.
Q. What is the meaning of 7?
A. It is interpreted as Dump Analysis 7 days a week. Like what I do. Or from kernel pool tag perspective it is AD7: Analysis of Dumps 7 days a week.
Q. What about the year 2011, 2012, 2013? 0×7DB, 0×7DC and 0×7DD?
A. Hmm, sounds like WinDbg commands db, dc and dd…
- Dmitry Vostokov @ DumpAnalysis.org -
