Software Diagnostics Library

Today we introduce an icon for Wait Chain (window messaging) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As the new decade is approaching (2011-2020) we would like to make a few previews and predictions:

- Increased complexity of software will bring more methods from biological, social sciences and humanities in addition to existing methods of automated debugging and computer science techniques

- Focus on first fault software problem solving (when aspect)

- Focus on pattern-driven software problem solving (how aspect)

- Fusion of debugging and malware analysis into a unified structural and behavioral pattern framework

- Visual debugging, memory and software trace visualization techniques

- Software maintenance certification

- Focus on domain-driven troubleshooting and debugging tools as a service (debugware TaaS)

- Focus on security issues related to memory dumps and software traces

- New scripting languages and programming language extensions for debugging

- The maturation of the science of memory snapshots and software traces (memoretics)

Imagining is not not limited to the above and more to come and explain in the forthcoming parts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Self-Dump pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another variant of the general Wait Chain pattern where blocked threads are waiting for synchronous window message calls (sent messages). For example, here three threads from different processes are blocked in such a chain where hWnd parameters for SendMessage calls and associated window procedures are marked with corresponding colors: 

0:000> ~*kbL

.  0  Id: 116c.1174 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr  Args to Child             
0034f83c 76261c01 000a0e54 00000111 00000068 USER32!NtUserMessageCall+0x15
0034f87c 7625cd81 011114d0 00000000 00d41190 USER32!SendMessageWorker+0x5e9
0034f8a0 00fa1256 000a0e54 00000111 00000068 USER32!SendMessageW+0×7f
0034f90c 76256238 00040eb0 00000111 00000068 WCM_A!WndProc+0xc6
0034f938 762568ea 00fa1190 00040eb0 00000111 USER32!InternalCallWinProc+0×23
0034f9b0 76257d31 00000000 00fa1190 00040eb0 USER32!UserCallWinProcCheckWow+0×109
0034fa10 76257dfa 00fa1190 00000000 76257d79 USER32!DispatchMessageWorker+0×3bc
0034fa20 00fa10d3 0034fa3c 0034fae8 00000000 USER32!DispatchMessageW+0xf
0034fa54 00fa14b6 00fa0000 00000000 00571bee WCM_A!wWinMain+0xd3
0034fae8 76493677 7efde000 0034fb34 77399d72 WCM_A!__tmainCRTStartup+0×150
0034faf4 77399d72 7efde000 72afcb2e 00000000 kernel32!BaseThreadInitThunk+0xe
0034fb34 77399d45 00fa1625 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0034fb4c 00000000 00fa1625 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> ~*kbL

.  0  Id: 10dc.e14 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr  Args to Child             
0017f7c4 76261c01 000c0ea4 00000111 00000068 USER32!NtUserMessageCall+0x15
0017f804 7625cd81 00ec3ec0 00000000 012e1190 USER32!SendMessageWorker+0x5e9
0017f828 00d41256 000c0ea4 00000111 00000068 USER32!SendMessageW+0×7f
0017f890 76256238 000a0e54 00000111 00000068 WCM_B!WndProc+0xc6
0017f8bc 762568ea 00d41190 000a0e54 00000111 USER32!InternalCallWinProc+0×23
0017f934 76257177 00000000 00d41190 000a0e54 USER32!UserCallWinProcCheckWow+0×109
0017f990 762572f1 00eb14d0 00000000 00000111 USER32!DispatchClientMessage+0xe0
0017f9cc 773700e6 0017f9e4 00000000 0017fae4 USER32!__fnDWORD+0×2b
0017f9e0 00eb14d0 00000000 00000111 00000068 ntdll!KiUserCallbackDispatcher+0×2e
WARNING: Frame IP not in any known module. Following frames may be wrong.
0017fa20 00d410e0 0017fa48 00000000 00000000 0xeb14d0
0017fa60 00d414b6 00d40000 00000000 00601bee WCM_B!wWinMain+0xe0
0017faf4 76493677 7efde000 0017fb40 77399d72 WCM_B!__tmainCRTStartup+0×150
0017fb00 77399d72 7efde000 728cf6de 00000000 kernel32!BaseThreadInitThunk+0xe
0017fb40 77399d45 00d41625 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0017fb58 00000000 00d41625 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> ~*kbL

.  0  Id: e68.fbc Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr  Args to Child             
0017f4c8 76272674 000c0ea4 00000000 00000000 USER32!NtUserWaitMessage+0x15
0017f504 7627288a 00070ee6 000c0ea4 00000000 USER32!DialogBox2+0x222
0017f530 762727b8 012e0000 012efc54 000c0ea4 USER32!InternalDialogBox+0xe5
0017f550 76272aa1 012e0000 012efc54 000c0ea4 USER32!DialogBoxIndirectParamAorW+0x37
0017f574 012e124d 012e0000 00000067 000c0ea4 USER32!DialogBoxParamW+0x3f
0017f5e4 76256238 000c0ea4 00000111 00000068 WCM_C!WndProc+0xbd
0017f610 762568ea 012e1190 000c0ea4 00000111 USER32!InternalCallWinProc+0×23
0017f688 76257177 00000000 012e1190 000c0ea4 USER32!UserCallWinProcCheckWow+0×109
0017f6e4 762572f1 01463ec0 00000000 00000111 USER32!DispatchClientMessage+0xe0
0017f720 773700e6 0017f738 00000000 0017f838 USER32!__fnDWORD+0×2b
0017f734 01463ec0 00000000 00000111 00000068 ntdll!KiUserCallbackDispatcher+0×2e
WARNING: Frame IP not in any known module. Following frames may be wrong.
0017f774 012e10e0 0017f79c 00000000 00000000 0×1463ec0
0017f7b4 012e1496 012e0000 00000000 00471bee WCM_C!wWinMain+0xe0
0017f848 76493677 7efde000 0017f894 77399d72 WCM_C!__tmainCRTStartup+0×150
0017f854 77399d72 7efde000 728ca9cf 00000000 kernel32!BaseThreadInitThunk+0xe
0017f894 77399d45 012e1605 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0017f8ac 00000000 012e1605 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

Detailed modeling example (WCM application) is to follow soon.

Send message calls can also be directed to the same thread:

0: kd> kv 100
ChildEBP RetAddr  Args to Child             
aa839180 80833ed1 8c446b60 8c446c08 00000002 nt!KiSwapContext+0x26
aa8391ac 80829c14 8c446d4c 8c446d20 8c446b60 nt!KiSwapThread+0x2e5
aa8391f4 80921102 8c446d4c 00000011 8c4a8c01 nt!KeWaitForSingleObject+0x346
aa8392b0 8088b41c 000006a8 00172e58 00172e58 nt!NtRequestWaitReplyPort+0x776
aa8392b0 7c82860c 000006a8 00172e58 00172e58 nt!KiFastCallEntry+0xfc
0012f194 7c827899 77c80a6e 000006a8 00172e58 ntdll!KiFastSystemCallRet
0012f198 77c80a6e 000006a8 00172e58 00172e58 ntdll!ZwRequestWaitReplyPort+0xc
0012f1e4 77c7fcf0 0012f220 0012f204 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230
0012f1f0 77c80673 0012f220 771f2918 0012f60c RPCRT4!I_RpcSendReceive+0x24
0012f204 77ce315a 0012f24c 00172ea8 77e63e5f RPCRT4!NdrSendReceive+0x2b
0012f5ec 771f4fbd 771f2918 771f1858 0012f60c RPCRT4!NdrClientCall2+0x22e
[...]
0012f698 7739b6e3 0004001a 00000016 00000001 ApplicationA!WndProc+0xcc
0012f6c4 7739b874 00407440 0004001a 00000016 USER32!InternalCallWinProc+0×28
0012f73c 7739c8b8 00000000 00407440 0004001a USER32!UserCallWinProcCheckWow+0×151
0012f798 7739c9c6 00607890 00000016 00000001 USER32!DispatchClientMessage+0xd9
0012f7c0 7c828556 0012f7d8 00000018 0012f894 USER32!__fnDWORD+0×24
0012f7c0 80831378 0012f7d8 00000018 0012f894 ntdll!KiUserCallbackDispatcher+0×2e
aa83957c 8091fbbb aa839634 aa839638 aa839608 nt!KiCallUserMode+0×4
aa8395d4 bf8a2492 00000002 aa839618 00000018 nt!KeUserModeCallback+0×8f
aa839658 bf8a229d be487890 00000016 00000001 win32k!SfnDWORD+0xb4
aa8396a0 bf8a1249 02487890 00000016 00000001 win32k!xxxSendMessageToClient+0×176
aa8396ec bf8a115e be487890 00000016 00000001 win32k!xxxSendMessageTimeout+0×1a6
aa839710 bf926e0d be487890 00000016 00000001 win32k!xxxSendMessage+0×1b
aa83974c bf926eb5 bc18cbc8 00000016 00000001 win32k!xxxClientShutdown2+0×87
aa839768 bf8ad9fa be487890 80000009 0000029e win32k!xxxClientShutdown+0×47
aa8397c4 bf8845d4 be487890 0000003b 80000009 win32k!xxxRealDefWindowProc+0×364
aa8397dc bf884604 be487890 0000003b 80000009 win32k!xxxWrapRealDefWindowProc+0×16
aa8397f8 bf8c1259 be487890 0000003b 80000009 win32k!NtUserfnNCDESTROY+0×27
aa839830 8088b41c 0004001a 0000003b 80000009 win32k!NtUserMessageCall+0xc0
aa839830 7c82860c 0004001a 0000003b 80000009 nt!KiFastCallEntry+0xfc (TrapFrame @ aa839854)
0012f7c0 7c828556 0012f7d8 00000018 0012f894 ntdll!KiFastSystemCallRet
0012f7c0 80831378 0012f7d8 00000018 0012f894 ntdll!KiUserCallbackDispatcher+0×2e
aa839b08 8091fbbb aa839bc0 aa839bc4 aa839b94 nt!KiCallUserMode+0×4
aa839b60 bf8a2492 00000002 aa839ba4 00000018 nt!KeUserModeCallback+0×8f
aa839be4 bf8a229d be487890 0000003b 80000009 win32k!SfnDWORD+0xb4
aa839c2c bf8c3f77 02487890 0000003b 80000009 win32k!xxxSendMessageToClient+0×176
aa839c9c bf89b88e bc18e838 aa839d64 0012fa38 win32k!xxxReceiveMessage+0×2b5
aa839cec bf89d201 aa839d18 0004001a 00000000 win32k!xxxRealInternalGetMessage+0×2d7
aa839d4c 8088b41c 0012fa5c 0004001a 00000000 win32k!NtUserGetMessage+0×3f
aa839d4c 7c82860c 0012fa5c 0004001a 00000000 nt!KiFastCallEntry+0xfc (TrapFrame @ aa839d64)
0012f9f0 7c828556 0012fa08 00000018 0012ffb0 ntdll!KiFastSystemCallRet
0012fa1c 7739c811 7739c844 0012fa5c 0004001a ntdll!KiUserCallbackDispatcher+0×2e
0012fa3c 0040634e 0012fa5c 0004001a 00000000 USER32!NtUserGetMessage+0xc
0012ff18 00408d9d 00000032 00000000 00142546 ApplicationA!WinMain+0×80f
0012ffc0 77e6f22b 00000000 00000000 7ffdf000 ApplicationA!WinMainCRTStartup+0×185
0012fff0 00000000 00408c18 00000000 78746341 kernel32!BaseProcessStart+0×23

Blocked sent message calls can also be manifested in kernel space and mixed with patterns like Message Box and Main Thread, for example:

1: kd> k250
ChildEBP RetAddr             
8d5d2808 82a7eb15 nt!KiSwapContext+0x26
8d5d2840 82a7d403 nt!KiSwapThread+0x266
8d5d2868 82a772cf nt!KiCommitThreadWait+0x1df
8d5d28e0 82550d75 nt!KeWaitForSingleObject+0x393
8d5d293c 82550e10 win32k!xxxRealSleepThread+0x1d7
8d5d2958 824ff4b0 win32k!xxxSleepThread+0x2d
8d5d29cc 825547e8 win32k!xxxInterSendMsgEx+0xb1c
8d5d2a1c 825546a4 win32k!xxxSendMessageTimeout+0x13b
8d5d2a44 82533843 win32k!xxxSendMessage+0×28
8d5d2b08 824fd865 win32k!xxxCalcValidRects+0xf7
8d5d2b64 82502c98 win32k!xxxEndDeferWindowPosEx+0×100
8d5d2b84 825170c9 win32k!xxxSetWindowPos+0xf6
8d5d2c08 82517701 win32k!xxxActivateThisWindow+0×2b1
8d5d2c38 82517537 win32k!xxxActivateWindow+0×144
8d5d2c4c 824fd9dd win32k!xxxSwpActivate+0×44
8d5d2ca4 82502c98 win32k!xxxEndDeferWindowPosEx+0×278
8d5d2cc4 824fff82 win32k!xxxSetWindowPos+0xf6
8d5d2d10 82a5342a win32k!NtUserSetWindowPos+0×140
8d5d2d10 76ee64f4 nt!KiFastCallEntry+0×12a (TrapFrame @ 8d5d2d34)
01e2cea0 7621358d ntdll!KiFastSystemCallRet
01e2cea4 6a8fa0eb USER32!NtUserSetWindowPos+0xc
01e2cf14 6a894b13 IEFRAME!SHToggleDialogExpando+0×15a
01e2cf28 6a894d5d IEFRAME!EleDlg::ToggleExpando+0×20
01e2d74c 6a895254 IEFRAME!EleDlg::OnInitDlg+0×229
01e2d7b8 762186ef IEFRAME!EleDlg::DlgProcEx+0×189
01e2d7e4 76209eb2 USER32!InternalCallWinProc+0×23
01e2d860 7620b98b USER32!UserCallDlgProcCheckWow+0xd6
01e2d8a8 7620bb7b USER32!DefDlgProcWorker+0xa8
01e2d8c4 762186ef USER32!DefDlgProcW+0×22
01e2d8f0 76218876 USER32!InternalCallWinProc+0×23
01e2d968 76217631 USER32!UserCallWinProcCheckWow+0×14b
01e2d9a8 76209b1d USER32!SendMessageWorker+0×4d0
01e2da64 76235500 USER32!InternalCreateDialog+0xb0d
01e2da94 76235553 USER32!InternalDialogBox+0xa7
01e2dab4 76235689 USER32!DialogBoxIndirectParamAorW+0×37
01e2dad8 6a5d4952 USER32!DialogBoxParamW+0×3f
01e2db00 6a5d5024 IEFRAME!Detour_DialogBoxParamW+0×47
01e2db24 6a8956df IEFRAME!SHFusionDialogBoxParam+0×32
01e2db58 6a8957bb IEFRAME!EleDlg::ShowDialog+0×398
01e2e638 6a8959d3 IEFRAME!ShowDialogBox+0xb6
01e2eb9c 6a9013ed IEFRAME!ShowElevationPrompt+0×1dd
01e2f010 7669fc8f IEFRAME!CIEUserBrokerObject::BrokerCoCreateInstance+0×202
01e2f040 76704c53 RPCRT4!Invoke+0×2a
01e2f448 76d9d936 RPCRT4!NdrStubCall2+0×2d6
01e2f490 76d9d9c6 ole32!CStdStubBuffer_Invoke+0xb6
01e2f4d8 76d9df1f ole32!SyncStubInvoke+0×3c
01e2f524 76cb213c ole32!StubInvoke+0xb9
01e2f600 76cb2031 ole32!CCtxComChnl::ContextInvoke+0xfa
01e2f61c 76d9a754 ole32!MTAInvoke+0×1a
01e2f64c 76d9dcbb ole32!AppInvoke+0xab
01e2f72c 76d9a773 ole32!ComInvokeWithLockAndIPID+0×372
01e2f778 7669f34a ole32!ThreadInvoke+0×302
01e2f7b4 7669f4da RPCRT4!DispatchToStubInCNoAvrf+0×4a
01e2f80c 7669f3c6 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×16c
01e2f834 766a0cef RPCRT4!RPC_INTERFACE::DispatchToStub+0×8b
01e2f86c 7669f882 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xb2
01e2f8b8 7669f7a4 RPCRT4!LRPC_SCALL::DispatchRequest+0×23b
01e2f8d8 7669f763 RPCRT4!LRPC_SCALL::QueueOrDispatchCall+0xbd
01e2f8f4 7669f5ff RPCRT4!LRPC_SCALL::HandleRequest+0×34f
01e2f928 7669f573 RPCRT4!LRPC_SASSOCIATION::HandleRequest+0×144
01e2f960 7669ee4f RPCRT4!LRPC_ADDRESS::HandleRequest+0xbd
01e2f9dc 7669ece7 RPCRT4!LRPC_ADDRESS::ProcessIO+0×50a
01e2f9e8 766a1357 RPCRT4!LrpcServerIoHandler+0×16
01e2f9f8 76ecd3a3 RPCRT4!LrpcIoComplete+0×16
01e2fa20 76ed0748 ntdll!TppAlpcpExecuteCallback+0×1c5
01e2fb88 76e11174 ntdll!TppWorkerThread+0×5a4
01e2fb94 76efb3f5 kernel32!BaseThreadInitThunk+0xe
01e2fbd4 76efb3c8 ntdll!__RtlUserThreadStart+0×70
01e2fbec 00000000 ntdll!_RtlUserThreadStart+0×1b

2: kd> !process 890ff430 1f
PROCESS 890ff430  SessionId: 1  Cid: 18a4    Peb: 7ffdc000  ParentCid: 1fdc
DirBase: 7fbf04e0  ObjectTable: da89fb80  HandleCount: 852.
Image: iexplore.exe

THREAD 89141db0  Cid 18a4.19c8  Teb: 7ffdf000 Win32Thread: bc373d18 WAIT: (Unknown) UserMode Non-Alertable
    8915b020  SynchronizationEvent
Not impersonating
DeviceMap                 da7f9680
Owning Process            890ff430       Image:         iexplore.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      56879          Ticks: 1634 (0:00:00:25.531)
Context Switch Count      12410  NoStackSwap    LargeStack
UserTime                  00:00:00.078
KernelTime                00:00:01.234
Win32 Start Address iexplore!wWinMainCRTStartup (0x004031b9)
Start Address kernel32!BaseProcessStartThunk (0x77e617f8)
Stack Init b5672000 Current b56717c4 Base b5672000 Limit b566c000 Call 0
Priority 4 BasePriority 4 PriorityDecrement 0
ChildEBP RetAddr              
b56717dc 80833ec5 nt!KiSwapContext+0x26
b5671808 80829c14 nt!KiSwapThread+0x2e5
b5671850 bf89ab73 nt!KeWaitForSingleObject+0x346
b56718ac bf8c4ba6 win32k!xxxSleepThread+0x1be
b5671948 bf8a13e0 win32k!xxxInterSendMsgEx+0x798
b5671994 bf8a132f win32k!xxxSendMessageTimeout+0x1f3
b56719b8 bf85ca01 win32k!xxxSendMessage+0×1b
b5671a7c bf85da04 win32k!xxxCalcValidRects+0xea
b5671ad8 bf85de2e win32k!xxxEndDeferWindowPosEx+0xf2
b5671af4 bf861cf2 win32k!xxxSetWindowPos+0xb1
b5671b3c bf882098 win32k!xxxProcessEventMessage+0×232
b5671c7c bf89b89e win32k!xxxScanSysQueue+0×21e
b5671ce4 bf89c529 win32k!xxxRealInternalGetMessage+0×2aa
b5671d48 8088b41c win32k!NtUserPeekMessage+0×42
b5671d48 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b5671d64)
0012e6e8 7739bde5 ntdll!KiFastSystemCallRet
0012e714 7739be5e USER32!NtUserPeekMessage+0xc
0012e740 02935f8c USER32!PeekMessageW+0xab
0012e7b4 02936150 IEUI!DUserRegisterSuper+0×920
0012e7d4 40d2ee98 IEUI!PeekMessageExW+0×42
0012e818 40d2abf4 IEFRAME!CBrowserFrame::FrameMessagePump+0×23
0012e824 40d2bc63 IEFRAME!BrowserThreadProc+0×3f
0012e848 40d2bbb1 IEFRAME!BrowserNewThreadProc+0×7b
0012f8b8 40d2ba61 IEFRAME!SHOpenFolderWindow+0×188
0012fae8 00401484 IEFRAME!IEWinMain+0×2d9
0012ff2c 0040131f iexplore!wWinMain+0×2c6
0012ffc0 77e6f23b iexplore!_initterm_e+0×1b1
0012fff0 00000000 kernel32!BaseProcessStart+0×23

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to hooked functions pattern we should also pay attention to Hooking Level. The latter is the number of patched functions. Often value-added hooksware has configuration options that fine-tune hooking behavior. For example, an application with the less number of patched functions behaved incorrectly and two process user dumps were saved from the working and non-working environment:  

0:000> * problem behavior

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\user32.dll\49E0380E9d000\user32.dll
No range specified

Scanning section:    .text
Size: 422527
Range to scan: 76e31000-76e9827f
    76e3d6f8-76e3d6fc  5 bytes - user32!NtUserSetThreadDesktop
 [ b8 30 12 00 00:e9 03 29 13 09 ]
    76e3dc2a-76e3dc2e  5 bytes - user32!CreateWindowExA (+0x532)
 [ 8b ff 55 8b ec:e9 d1 23 15 09 ]
    76e3f8f8-76e3f8fc  5 bytes - user32!PostMessageA (+0x1cce)
 [ 8b ff 55 8b ec:e9 03 07 fa 08 ]
    76e41305-76e41309  5 bytes - user32!CreateWindowExW (+0x1a0d)
 [ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
    76e435e3-76e435e7  5 bytes - user32!NtUserSetWindowPos (+0x22de)
 [ b8 38 12 00 00:e9 18 ca 11 09 ]
    76e48343-76e48347  5 bytes - user32!PeekMessageA (+0x4d60)
 [ 8b ff 55 8b ec:e9 b8 7c fb 08 ]
    76e48ab3-76e48ab7  5 bytes - user32!GetMessageA (+0x770)
 [ 8b ff 55 8b ec:e9 48 75 fd 08 ]
    76e4a175-76e4a179  5 bytes - user32!PostMessageW (+0x16c2)
 [ 8b ff 55 8b ec:e9 86 5e f8 08 ]
    76e4fef7-76e4fefb  5 bytes - user32!GetMessageW (+0x5d82)
 [ 8b ff 55 8b ec:e9 04 01 fc 08 ]
    76e5045a-76e5045e  5 bytes - user32!PeekMessageW (+0x563)
 [ 8b ff 55 8b ec:e9 a1 fb f9 08 ]
    76e8d37d-76e8d381  5 bytes - user32!MessageBoxTimeoutW (+0x3cf23)
 [ 8b ff 55 8b ec:e9 7e 2c fd 08 ]
    76e8d4d9-76e8d4dd  5 bytes - user32!MessageBoxIndirectA (+0x15c)
 [ 8b ff 55 8b ec:e9 22 2b ff 08 ]
    76e8d5d3-76e8d5d7  5 bytes - user32!MessageBoxIndirectW (+0xfa)
 [ 8b ff 55 8b ec:e9 28 2a fe 08 ]
    76e8d65d-76e8d661  5 bytes - user32!MessageBoxExW (+0x8a)
 [ 8b ff 55 8b ec:e9 9e 29 00 09 ]
Total bytes compared: 422527(100%)
Number of errors: 70
70 errors : !user32 (76e3d6f8-76e8d661)

0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b 8bff            mov     edi,edi
76e3ba5d 55              push    ebp
76e3ba5e 8bec            mov     ebp,esp
76e3ba60 81ec54030000    sub     esp,354h
76e3ba66 a1c090e976      mov     eax,dword ptr [user32!__security_cookie (76e990c0)]
76e3ba6b 33c5            xor     eax,ebp
76e3ba6d 8945fc          mov     dword ptr [ebp-4],eax
76e3ba70 53              push    ebx

0:000> * expected behavior 

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\user32.dll\49E0380E9d000\user32.dll
No range specified

Scanning section:    .text
Size: 422527
Range to scan: 76e31000-76e9827f
    76e39c11-76e39c15  5 bytes - user32!MonitorFromPoint
 [ 6a 08 68 50 9c:e9 ea 63 10 09 ]
    76e3b8ea-76e3b8ee  5 bytes - user32!GetMonitorInfoA (+0x1cd9)
 [ 8b ff 55 8b ec:e9 11 47 12 09 ]
    76e3ba5b-76e3ba5f  5 bytes - user32!EnumDisplayDevicesW (+0×171)
 [ 8b ff 55 8b ec:e9 a0 45 0b 09 ]
    76e3d6f8-76e3d6fa  3 bytes - user32!NtUserSetThreadDesktop (+0×1c9d)
 [ b8 30 12:e9 03 29 ]
    76e3d6fc - user32!NtUserSetThreadDesktop+4 (+0×04)
 [ 00:09 ]
    76e3dc2a-76e3dc2e  5 bytes - user32!CreateWindowExA (+0×52e)
 [ 8b ff 55 8b ec:e9 d1 23 15 09 ]
    76e3e7cd-76e3e7d1  5 bytes - user32!SetWindowLongA (+0xba3)
 [ 8b ff 55 8b ec:e9 2e 18 03 09 ]
    76e3f8f8-76e3f8fc  5 bytes - user32!PostMessageA (+0×112b)
 [ 8b ff 55 8b ec:e9 03 07 e7 08 ]
    76e41305-76e41309  5 bytes - user32!CreateWindowExW (+0×1a0d)
 [ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
    76e413b4-76e413b8  5 bytes - user32!SetWindowLongW (+0xaf)
 [ 8b ff 55 8b ec:e9 47 ec 03 09 ]
    76e41709-76e4170d  5 bytes - user32!MonitorFromRect (+0×355)
 [ 6a 08 68 48 17:e9 f2 e8 0e 09 ]
    76e435e3-76e435e7  5 bytes - user32!NtUserSetWindowPos (+0×1eda)
 [ b8 38 12 00 00:e9 18 ca fe 08 ]
    76e440c5-76e440c9  5 bytes - user32!EnumDisplaySettingsExW (+0xae2)
 [ 8b ff 55 8b ec:e9 36 bf 06 09 ]
    76e441a1-76e441a5  5 bytes - user32!EnumDisplaySettingsW (+0xdc)
 [ 8b ff 55 8b ec:e9 5a be 08 09 ]
    76e46d4a-76e46d4e  5 bytes - user32!EnumDisplayDevicesA (+0×2ba9)
 [ 8b ff 55 8b ec:e9 b1 92 0b 09 ]
    76e46fe6-76e46fea  5 bytes - user32!EnumDisplaySettingsA (+0×29c)
 [ 8b ff 55 8b ec:e9 15 90 09 09 ]
    76e47010-76e47014  5 bytes - user32!EnumDisplaySettingsExA (+0×2a)
 [ 8b ff 55 8b ec:e9 eb 8f 07 09 ]
    76e47d12-76e47d16  5 bytes - user32!GetMonitorInfoW (+0xd02)
 [ 8b ff 55 8b ec:e9 e9 82 10 09 ]
    76e48343-76e48347  5 bytes - user32!PeekMessageA (+0×631)
 [ 8b ff 55 8b ec:e9 b8 7c e8 08 ]
    76e4844c-76e48450  5 bytes - user32!NtUserEnumDisplayMonitors (+0×109)
 [ b8 81 11 00 00:e9 af 7b 0c 09 ]
    76e488d4-76e488d8  5 bytes - user32!MonitorFromWindow (+0×488)
 [ 6a 08 68 28 89:e9 27 77 0d 09 ]
    76e48ab3-76e48ab7  5 bytes - user32!GetMessageA (+0×1df)
 [ 8b ff 55 8b ec:e9 48 75 ea 08 ]
    76e49994-76e49998  5 bytes - user32!GetWindowLongA (+0xee1)
 [ 6a 08 68 d0 99:e9 67 66 00 09 ]
    76e49af1-76e49af5  5 bytes - user32!GetSystemMetrics (+0×15d)
 [ 6a 0c 68 58 9b:e9 0a 65 12 09 ]
    76e4a175-76e4a179  5 bytes - user32!PostMessageW (+0×684)
 [ 8b ff 55 8b ec:e9 86 5e e5 08 ]
    76e4f8bf-76e4f8c3  5 bytes - user32!GetWindowLongW (+0×574a)
 [ 6a 08 68 00 f9:e9 3c 07 01 09 ]
    76e4fef7-76e4fefb  5 bytes - user32!GetMessageW (+0×638)
 [ 8b ff 55 8b ec:e9 04 01 e9 08 ]
    76e5045a-76e5045e  5 bytes - user32!PeekMessageW (+0×563)
 [ 8b ff 55 8b ec:e9 a1 fb e6 08 ]
    76e8d37d-76e8d381  5 bytes - user32!MessageBoxTimeoutW (+0×3cf23)
 [ 8b ff 55 8b ec:e9 7e 2c ea 08 ]
    76e8d4d9-76e8d4dd  5 bytes - user32!MessageBoxIndirectA (+0×15c)
 [ 8b ff 55 8b ec:e9 22 2b ec 08 ]
    76e8d5d3-76e8d5d7  5 bytes - user32!MessageBoxIndirectW (+0xfa)
 [ 8b ff 55 8b ec:e9 28 2a eb 08 ]
    76e8d65d-76e8d661  5 bytes - user32!MessageBoxExW (+0×8a)
 [ 8b ff 55 8b ec:e9 9e 29 ed 08 ]
Total bytes compared: 422527(100%)
Number of errors: 154
154 errors : !user32 (76e39c11-76e8d661)

0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b e9a0450b09      jmp     7fef0000
76e3ba60 81ec54030000    sub     esp,354h
76e3ba66 a1c090e976      mov     eax,dword ptr [user32!__security_cookie (76e990c0)]
76e3ba6b 33c5            xor     eax,ebp
76e3ba6d 8945fc          mov     dword ptr [ebp-4],eax
76e3ba70 53              push    ebx
76e3ba71 56              push    esi
76e3ba72 8b7510          mov     esi,dword ptr [ebp+10h]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Embedded Comments in dump files are useful to record external information like the reason for saving a memory dump, a tool used to do that, and some pre-analysis and monitoring data that might help or guide in the future analysis. Comments are not widely used but some examples include Manual Process Dump, False Positive Dump patterns, and process and thread CPU consumption comments in dump files saved by Sysinternals ProcDump tool. Such comments may not be necessarily saved by IDebugClient2 :: WriteDumpFile2 function but any buffer saved in memory that is accessible later from a dump file will do as can be easily demonstrated by the old Citrix SystemDump tool.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

By analogy with Well-Tested Function we introduce another pattern called Well-Tested Module. This is a module we usually skip when analyzing a stack trace because we suspect it the least. WinDbg can also be customized to skip such modules for the default analysis command as shown in the following example: Minidump Analysis (Part 2)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We now start unifying software behavior analysis patterns with debugging architecture, design, implementation and usage. This is analogous to software construction where a problem analysis leads to various software engineering phases. The important difference here is the addition of debugging usage patterns. Let’s look at an example (we discuss suggested patterns later):

- Analysis Patterns

Shared Buffer Overwrite

- Architectural Patterns

Debug Event Subscription / Notification

- Design Patterns

Punctuated Execution

- Implementation Patterns

Breakpoint (software and hardware)

- Usage Patterns

Kernel vs. user space breakpoints

To differentiate this systematic approach from the various published ad hoc debugging patterns we call it Unified Debugging Pattern Language. ADI parts can also correspond to various DebugWare patterns where we provide a mapping later.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One frequently useful analysis pattern is the presence of String Parameter on a function call stack. The trivial case is when a function parameter is a pointer to an ASCII or a Unicode string (da and du WinDbg commands). More interesting case is when we have a function that takes pointers to a structure that has string fields (dpa and dpu commands), for example:

0:018> kv 100
ChildEBP RetAddr  Args to Child             
00de8c7c 7739bf53 7739610a 07750056 00000000 ntdll!KiFastSystemCallRet
00de8cb4 7738965e 00080126 07750056 00000001 user32!NtUserWaitMessage+0xc
00de8cdc 7739f762 77380000 0012b238 07750056 user32!InternalDialogBox+0xd0
00de8f9c 7739f047 00de90f8 00000000 ffffffff user32!SoftModalMessageBox+0x94b
00de90ec 7739eec9 00de90f8 00000028 07750056 user32!MessageBoxWorker+0x2ba
00de9144 773d7d0d 07750056 0015cd68 00132a60 user32!MessageBoxTimeoutW+0x7a
00de9178 773c42c8 07750056 00de923f 00de91ec user32!MessageBoxTimeoutA+0x9c
00de9198 773c42a4 07750056 00de923f 00de91ec user32!MessageBoxExA+0x1b
00de91b4 6dfcf8c2 07750056 00de923f 00de91ec user32!MessageBoxA+0×45
00de99f0 6dfcfad2 00de9285 00de9a1c 77bc6cd5 compstui!FilterException+0×174
00dead94 7739b6e3 0038010e 00000110 00000000 compstui!CPSUIPageDlgProc+0xf3
00deadc0 77395f82 6dfcf9df 0038010e 00000110 user32!InternalCallWinProc+0×28
00deae3c 77395e22 0015d384 6dfcf9df 0038010e user32!UserCallDlgProcCheckWow+0×147
00deae84 7738aaa4 00000000 00000110 00000000 user32!DefDlgProcWorker+0xa8
00deaeb4 77388c01 004673d0 00461130 00000000 user32!SendMessageWorker+0×43e
00deaf6c 77387910 6dfc0000 004673d0 00000404 user32!InternalCreateDialog+0×9cf
00deaf90 7739fb5b 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamAorW+0×33
00deafb0 774279a5 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamW+0×1b
00deb000 77427abc 02192c78 000ddd08 07750056 comctl32!_CreatePageDialog+0×79
00deb028 77429d12 02192c78 6dff5c30 07750056 comctl32!_CreatePage+0xb1
00deb244 7742b8b6 02192c78 00000001 00290110 comctl32!PageChange+0xcc
00deb604 7742c446 07750056 02192c78 00deb6ec comctl32!InitPropSheetDlg+0xbb8
00deb674 7739b6e3 07750056 00000110 00290110 comctl32!PropSheetDlgProc+0×4cb
00deb6a0 77395f82 7742bf7b 07750056 00000110 user32!InternalCallWinProc+0×28
00deb71c 77395e22 0008c33c 7742bf7b 07750056 user32!UserCallDlgProcCheckWow+0×147
00deb764 7738aaa4 00000000 00000110 00290110 user32!DefDlgProcWorker+0xa8
00deb794 77388c01 004652e0 00461130 00290110 user32!SendMessageWorker+0×43e
00deb84c 77387910 77420000 004652e0 00000100 user32!InternalCreateDialog+0×9cf
00deb870 7739fb5b 77420000 02184be8 00000000 user32!CreateDialogIndirectParamAorW+0×33
00deb890 774ab1c5 77420000 02184be8 00000000 user32!CreateDialogIndirectParamW+0×1b
00deb8d8 7742ca78 77420000 02184be8 00000000 comctl32!SHFusionCreateDialogIndirectParam+0×36
00deb93c 7742ccea 00000000 000000a0 00000000 comctl32!_RealPropertySheet+0×242
00deb954 7742cd05 00deb9b4 00000000 00deb99c comctl32!_PropertySheet+0×146
00deb964 6dfd1178 00deb9b4 000000a0 00deba30 comctl32!PropertySheetW+0xf
00deb99c 6dfcf49b 00deb9b4 0256b3f8 0013fbe0 compstui!PropertySheetW+0×4b
00deba14 6dfd0718 00000000 00134da4 00debae8 compstui!DoComPropSheet+0×2ef
00deba44 6dfd0799 00000000 7307c8da 00debad0 compstui!DoCommonPropertySheetUI+0xe9
00deba5c 730801c5 00000000 7307c8da 00debad0 compstui!CommonPropertySheetUIW+0×17
00debaa4 73080f5d 00000000 7307c8da 00debad0 winspool!CallCommonPropertySheetUI+0×43
00debeec 4f49cdfe 00000000 0218bd84 02277fe8 winspool!PrinterPropertiesNative+0×10c
WARNING: Stack unwind information not available. Following frames may be wrong.
00debf2c 4f4950a5 00deea08 00000002 02277fe8 PrintDriverA!DllGetClassObject+0xdb7e
00deee18 4f4904fb 00ca6ee0 00000003 00000001 PrintDriverA!DllGetClassObject+0×5e25
00deee30 18f60282 02277fe8 00ca6ee0 00000003 PrintDriverA!DllGetClassObject+0×127b
00deee58 18f5abce 001042e4 00ca6ee0 00000003 ps5ui!HComOEMPrinterEvent+0×33
00deee9c 7308218c 00ca6ee0 00000003 00000001 ps5ui!DrvPrinterEvent+0×22e
00deeee8 761543c8 00ca6ee0 00000003 00000001 winspool!SpoolerPrinterEventNative+0×57
00deef04 761560d2 00ca6ee0 00000003 00000000 localspl!SplDriverEvent+0×21
00deef28 761447f9 00cb2160 00000003 00000000 localspl!PrinterDriverEvent+0×46
00def3f0 76144b12 00000000 00000002 00d12020 localspl!SplAddPrinter+0×5f3
00def41c 74070193 00000000 00000002 00d12020 localspl!LocalAddPrinterEx+0×2e
00def86c 7407025c 00000000 00000002 00d12020 spoolss!AddPrinterExW+0×151
00def888 01007a93 00000000 00000002 00d12020 spoolss!AddPrinterW+0×17
00def8a4 01006772 00000000 00ce74b0 021b6278 spoolsv!YAddPrinter+0×75
00def8c8 77c80355 00000000 00ce74b0 021b6278 spoolsv!RpcAddPrinter+0×37
00def8f0 77ce43e1 0100673b 00defae0 00000005 rpcrt4!Invoke+0×30
00defcf8 77ce45c4 00000000 00000000 000e8584 rpcrt4!NdrStubCall2+0×299
00defd14 77c8013a 000e8584 000d63d8 000e8584 rpcrt4!NdrServerCall2+0×19
00defd48 77c805ef 01002c57 000e8584 00defdec rpcrt4!DispatchToStubInCNoAvrf+0×38
00defd9c 77c80515 00000005 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
00defdc0 77c8139e 000e8584 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
00defdfc 77c814b2 000e1c48 000d85b8 02154180 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×42c
00defe20 77c88848 000d85f0 00defe38 000e1c48 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
00deff84 77c88962 00deffac 77c888fd 000d85b8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
00deff8c 77c888fd 000d85b8 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
00deffac 77c7b293 0008b038 00deffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0×9d
00deffb8 77e6482f 000bdba8 00000000 00000000 rpcrt4!ThreadStartRoutine+0×1b
00deffec 00000000 77c7b278 000bdba8 00000000 kernel32!BaseThreadStart+0×34

0:018> da 00de923f
00de923f  “Function address 0×77481456 caus”
00de925f  “ed a protection fault. (exceptio”
00de927f  “n code 0xc0000005).The applicati”
00de929f  “on property sheet page(s) may no”
00de92bf  “t function properly.”

0:018> dpu 00d12020
00d12020  00000000
00d12024  021b6088 “Printer A User B Server C”
00d12028  00000000
00d1202c  021b6124 “Remote Printer Address for User C”
00d12030  021b6190 “Printer Name and Family”
00d12034  021b61c4 “Printer Client Name”
00d12038  021b6228 “Printer Location”
00d1203c  00000000
00d12040  00000000
00d12044  021b6264 “Printer Module Name”
00d12048  00000000
00d1204c  00000000
00d12050  021b628c
00d12054  00008841
00d12058  00000000
00d1205c  00000000
00d12060  00000000
00d12064  00000000
00d12068  00000000
00d1206c  00000000
00d12070  00000000
00d12074  00000000
00d12078  00000000
00d1207c  00000000
00d12080  00000000
00d12084  00000000
00d12088  00000000
00d1208c  00000000
00d12090  00000000
00d12094  00000000
00d12098  00000000
00d1209c  00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

MAaaS includes 2 complementary DA+TA services:

1. Dump Analysis as a Service (DAaaS)
2. Trace Analysis as a Service (TAaaS)

Memory Dump Analysis Services is the first organization to provide such a service at an audit and certification levels.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Invalid Parameter is a general pattern of passing unexpected values to functions. Here we look at invalid heap block parameter specialization. It is different from heap corruption or double free pattern because no corruption happens in heap structures before detection and the parameter value has never been correct before its use. For example, we have this stack trace:

0:003> kL 100
ChildEBP RetAddr 
01b2e6f0 77f27d0c ntdll!ZwWaitForSingleObject+0x15
01b2e774 77f27e3a ntdll!RtlReportExceptionEx+0x14b
01b2e7cc 77f4dc2e ntdll!RtlReportException+0x86
01b2e7e0 77f4dcab ntdll!RtlpTerminateFailureFilter+0x14
01b2e7ec 77ef05c4 ntdll!RtlReportCriticalFailure+0x67
01b2e800 77ef0469 ntdll!_EH4_CallFilterFunc+0x12
01b2e828 77ed8799 ntdll!_except_handler4+0x8e
01b2e84c 77ed876b ntdll!ExecuteHandler2+0x26
01b2e8fc 77e9010f ntdll!ExecuteHandler+0x24
01b2e8fc 77f4dc9b ntdll!KiUserExceptionDispatcher+0xf
01b2ecc4 77f4eba1 ntdll!RtlReportCriticalFailure+0x57
01b2ecd4 77f4ec81 ntdll!RtlpReportHeapFailure+0x21
01b2ed08 77efdda0 ntdll!RtlpLogHeapFailure+0xa1
01b2ed38 76bc14d1 ntdll!RtlFreeHeap+0×64
01b2ed4c 75694c39 kernel32!HeapFree+0×14
01b2ed98 726f167d msvcr80!free+0xcd
01b2eda4 7270613d DllA!FreeData+0xd
[…]
01b2fe38 77eb9d42 kernel32!BaseThreadInitThunk+0xe
01b2fe78 77eb9d15 ntdll!__RtlUserThreadStart+0×70
01b2fe90 00000000 ntdll!_RtlUserThreadStart+0×1b

We see that the failure was detected and logged immediately without any instrumentation information:

0:003> !gflag
Current NtGlobalFlag contents: 0x00000000

If we enable full page heap we get this default analysis output and the following stack trace:

0:003> !gflag
Current NtGlobalFlag contents: 0x02000000
    hpa - Place heap allocations at ends of pages

0:003> !analyze -v

[...]

APPLICATION_VERIFIER_HEAPS_CORRUPTED_HEAP_BLOCK_EXCEPTION_RAISED_FOR_PROBING (c)
Exception raised while verifying the heap block.
This situation happens if we really cannot determine any particular type of corruption for the block. For instance you will get this if during a heap free operation you pass an address that points to a non-accessible memory area.
This can also happen for double free situations if we do not find the block among full page heap blocks and we probe it as a light page heap block.
Arguments:
Arg1: 05eb1000, Heap handle used in the call.
Arg2: 00720071, Heap block involved in the operation.
Arg3: 00000000, Size of the heap block.
Arg4: c0000005, Reserved.

[...]

0:003> kL 100
ChildEBP RetAddr 
0818dca4 75fa0962 ntdll!ZwWaitForMultipleObjects+0x15
0818dd40 76bc162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0818dd88 76bc1921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0818dda4 76be9b0d kernel32!WaitForMultipleObjects+0x18
0818de10 76be9baa kernel32!WerpReportFaultInternal+0x186
0818de24 76be98d8 kernel32!WerpReportFault+0x70
0818de34 76be9855 kernel32!BasepReportFault+0x20
0818dec0 77ef06e7 kernel32!UnhandledExceptionFilter+0x1af
0818dec8 77ef05c4 ntdll!__RtlUserThreadStart+0x62
0818dedc 77ef0469 ntdll!_EH4_CallFilterFunc+0x12
0818df04 77ed8799 ntdll!_except_handler4+0x8e
0818df28 77ed876b ntdll!ExecuteHandler2+0x26
0818dfd8 77e9010f ntdll!ExecuteHandler+0x24
0818dfd8 71a6ba58 ntdll!KiUserExceptionDispatcher+0xf
0818e344 71a69ee0 verifier!VerifierStopMessage+0x1f8
0818e3a8 71a66f11 verifier!AVrfpDphReportCorruptedBlock+0x2b0
0818e3bc 71a819ec verifier!AVrfpDphFindBusyMemoryNoCheck+0x141
0818e3d0 71a8174e verifier!_EH4_CallFilterFunc+0x12
0818e3f8 77ed8799 verifier!_except_handler4+0x8e
0818e41c 77ed876b ntdll!ExecuteHandler2+0x26
0818e4cc 77e9010f ntdll!ExecuteHandler+0x24
0818e4cc 71a66e88 ntdll!KiUserExceptionDispatcher+0xf
0818e868 71a66f95 verifier!AVrfpDphFindBusyMemoryNoCheck+0xb8
0818e88c 71a67240 verifier!AVrfpDphFindBusyMemory+0x15
0818e8a8 71a69080 verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0x20
0818e8c4 77f50aac verifier!AVrfDebugPageHeapFree+0x90
0818e90c 77f0a8ff ntdll!RtlDebugFreeHeap+0x2f
0818ea00 77eb2a32 ntdll!RtlpFreeHeap+0x5d
0818ea20 76bc14d1 ntdll!RtlFreeHeap+0x142
0818ea34 75694c39 kernel32!HeapFree+0x14
0818ea80 726f167d msvcr80!free+0xcd
0818ea8c 7270613d DllA!FreeData+0xd
[...]
0818fb20 77eb9d42 kernel32!BaseThreadInitThunk+0xe
0818fb60 77eb9d15 ntdll!__RtlUserThreadStart+0x70
0818fb78 00000000 ntdll!_RtlUserThreadStart+0x1b

In both examples above we see that 00720071 was passed to free function (we also verify from the code using ub command that there was no parameter optimization):

0:003> kv
ChildEBP RetAddr  Args to Child             
[...]
01b2ed98 726f167d 00720071 01b2edb0 7270613d msvcr80!free+0xcd (FPO: [SEH])
[…]

We recognize that value as Unicode (as an example of a wild pointer but parameters need not be pointers in general case). We can also consider Invalid Handle pattern as another specialization of Invalid Parameter pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services announces the launch of the first Memory Dump Analysis Audit Service:

http://www.dumpanalysis.com/memory-dump-analysis-audit-service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Message Box pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

During repeated execution either on one computer or in parallel on many computers with a uniform software / hardware the given process VM size tends to cluster around some value range, for example, 40 - 60 Mb. If we get a collection of user process memory dumps taken from several production servers, say 20 files, we can either employ scripts to process all of them or compare their file size and look for a bigger ones for a starter, for example, 85 or 110 Mb. For certain processes, for example, a print spooler, after a problem the process size tends to increase compared to normal execution. For other processes, certain error processing modules might be loaded increasing VM size or in case of incoming requests for a hang process certain memory regions like heap could increase as well contributing to dump file size increase. We call this pattern Fat Process Dump. If we have fat and thin clients we should also have thin and fat process dumps as well. A case study is following.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca

The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Product information:

• Title: Memory Dump Analysis Anthology, Volume 5
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-96-4

• Hardcover: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This new pattern is called Blocked Queue and we provide an example of an ALPC port here. If we see an LPC/ALPC wait chain endpoint or just have a message address (and optionally a port address) we can check the port queue length, for example, for a frozen system we have this (WinDbg output was trimmed to save space and paper):

THREAD fffffa8009db7160  Cid 03b0.2ec0  Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa8009db7520  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8a00dbc6650 : queued at port fffffa800577ee60 : owned by process fffffa80056ddb30
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8005691b30       Image:         ServiceA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      39742808       Ticks: 3469954 (0:15:02:11.629)
Context Switch Count      9            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×0000000076cd8e70
Stack Init fffff8800bf60db0 Current fffff8800bf60620
Base fffff8800bf61000 Limit fffff8800bf5b000 Call 0
Priority 10 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`0bf60660 fffff800`016de992 nt!KiSwapContext+0×7a
fffff880`0bf607a0 fffff800`016e0cff nt!KiCommitThreadWait+0×1d2
fffff880`0bf60830 fffff800`016f5d1f nt!KeWaitForSingleObject+0×19f
fffff880`0bf608d0 fffff800`019ddac6 nt!AlpcpSignalAndWait+0×8f
fffff880`0bf60980 fffff800`019dba50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`0bf609e0 fffff800`019d8fcb nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`0bf60b00 fffff800`016d6993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`0bf60bb0 00000000`76d105aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0bf60c20)
00000000`01efe638 000007fe`fec0aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01efe640 000007fe`fecacb64 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`01efe700 000007fe`fecacd55 RPCRT4!NdrpClientCall3+0×244
00000000`01efe9c0 000007fe`fcbf18a1 RPCRT4!NdrClientCall3+0xf2
[…]

0: kd> !alpc /m fffff8a00dbc6650
Message @ fffff8a00dbc6650
  MessageID             : 0x0720 (1824)
  CallbackID            : 0x257C575 (39306613)
  SequenceNumber        : 0x00000002 (2)
  Type                  : LPC_REQUEST
  DataLength            : 0x0044 (68)
  TotalLength           : 0x006C (108)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8006a4bb10 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa8009db7160
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa800577ee60 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa80056ddb30 (ServiceB.exe)
  ServerThread          : fffffa8007ead4d0
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 0000000002a60f40
  ServerContext         : 0000000000000000
  PortContext           : 000000000227a370
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

0: kd> !alpc /p fffffa800577ee60
Port @ fffffa800577ee60
  Type                      : ALPC_CONNECTION_PORT
  CommunicationInfo         : fffff8a0022435d0
    ConnectionPort          : fffffa800577ee60
    ClientCommunicationPort : 0000000000000000
    ServerCommunicationPort : 0000000000000000
  OwnerProcess              : fffffa80056ddb30 (ServiceB.exe)
  SequenceNo                : 0×0000481A (18458)
  CompletionPort            : fffffa8005728e80
  CompletionList            : 0000000000000000
  MessageZone               : 0000000000000000
  ConnectionPending         : No
  ConnectionRefused         : No
  Disconnected              : No
  Closed                    : No
  FlushOnClose              : Yes
  ReturnExtendedInfo        : No
  Waitable                  : No
  Security                  : Static
  Wow64CompletionList       : No

  Main queue is empty.

  Large message queue is empty.

  Pending queue has 698 message(s)

    fffff8a002355aa0 00000404 0000000000001344:0000000000001358 0000000000000000 fffffa8004c0cb60 LPC_REQUEST
    fffff8a00a52f030 00000644 0000000000001078:00000000000024c0 0000000000000000 fffffa80072f1b60 LPC_REQUEST
    fffff8a00abb5030 000007a8 000000000000103c:000000000000050c 0000000000000000 fffffa800725b580 LPC_REQUEST
    fffff8a00239cab0 000000b8 0000000000000480:00000000000015f8 0000000000000000 fffffa80077f0b60 LPC_REQUEST
    fffff8a00ac81a90 00000a18 00000000000028ac:0000000000001e54 0000000000000000 fffffa8007fba060 LPC_CANCELED
    fffff8a005879140 00000f80 0000000000001260:0000000000000730 fffffa8006432060 fffffa8006b18060 LPC_REQUEST
    fffff8a013720d00 00000c6c 0000000000003764:00000000000032a8 0000000000000000 fffffa8006b00a60 LPC_CANCELED
    fffff8a00ac82660 00000810 0000000000003af4:0000000000002a98 0000000000000000 fffffa80068c0b60 LPC_CANCELED
    fffff8a00bdeca50 00000ec8 000000000000233c:00000000000013f8 0000000000000000 fffffa80079455b0 LPC_CANCELED
    fffff8a00b662830 000005cc 00000000000005e4:0000000000000e0c fffffa800791a7a0 fffffa8007376580 LPC_REQUEST
    fffff8a003d57150 00000f08 0000000000002678:0000000000003e0c 0000000000000000 fffffa8007e4a870 LPC_CANCELED
    fffff8a00cd08830 00000750 0000000000003408:0000000000003adc 0000000000000000 fffffa8008631b60 LPC_CANCELED
    fffff8a01855b2f0 000004f4 0000000000002c74:0000000000002d00 0000000000000000 fffffa800746b890 LPC_CANCELED
    fffff8a00da0d0b0 00000db0 0000000000001a34:0000000000002d80 0000000000000000 fffffa800aff4b60 LPC_CANCELED
    fffff8a00eddb030 0000059c 0000000000003f34:0000000000003c8c 0000000000000000 fffffa8008f96060 LPC_CANCELED
    fffff8a017a14d00 00000920 0000000000003850:0000000000002588 0000000000000000 fffffa8009f66060 LPC_CANCELED
    fffff8a01792d030 000007f8 0000000000003844:00000000000028d0 0000000000000000 fffffa800ad56260 LPC_CANCELED
    fffff8a00f8d6ae0 00000f30 000000000000239c:0000000000001694 0000000000000000 fffffa8008b86060 LPC_CANCELED
    fffff8a01395ab80 00000cdc 0000000000003630:00000000000018f8 0000000000000000 fffffa8005bc0770 LPC_CANCELED
    fffff8a0166ff800 00000984 00000000000005e4:00000000000025f4 fffffa8009718910 fffffa8008cbfb60 LPC_REQUEST
    fffff8a012b9f5a0 00000ac8 0000000000002d34:0000000000001b24 0000000000000000 fffffa8009cd8410 LPC_CANCELED
    fffff8a014313830 00000afc 00000000000005e4:00000000000023bc fffffa80073f0230 fffffa80054d7060 LPC_REQUEST
    fffff8a00a34a6b0 00000ca8 0000000000002534:0000000000002dd0 0000000000000000 fffffa80064c3980 LPC_CANCELED
[...]
    fffff8a00ad8f610 00000e64 0000000000003714:00000000000030b8 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a015720710 00001594 0000000000003638:00000000000029b8 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a009bac560 00001508 0000000000003994:0000000000001aac 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a00b6e78f0 00001574 0000000000002938:0000000000001998 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a00b5716b0 00001570 0000000000002938:0000000000001698 0000000000000000 fffffa800a3b8620 LPC_REQUEST
    fffff8a018531d00 00000db8 00000000000016d8:00000000000031c4 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a01112f410 000014b0 0000000000001b6c:0000000000001618 0000000000000000 fffffa800a3b8620 LPC_CANCELED

  Canceled queue is empty.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the most common patterns is Crash Signature. It consists of a set of attributes derivable from saved execution context for exceptions, faults and traps. For example, on x64 Windows it is usually RIP and RSP addresses. For x86 it is usually EIP, ESP and EBP. It can also include the application module name.

0:009> !analyze -v

[...]

FAILURE_BUCKET_ID:  SOFTWARE_NX_FAULT_c0000005_ApplicationA.exe!Unknown

BUCKET_ID:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_STACK_CORRUPTION_BAD_IP_ApplicationA+2d560

[...]

0:009> kL
ChildEBP RetAddr 
0354f270 75bc0962 ntdll!NtWaitForMultipleObjects+0x15
0354f30c 7651162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0354f354 76511921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0354f370 76539b0d kernel32!WaitForMultipleObjects+0x18
0354f3dc 76539baa kernel32!WerpReportFaultInternal+0x186
0354f3f0 765398d8 kernel32!WerpReportFault+0x70
0354f400 76539855 kernel32!BasepReportFault+0x20
0354f48c 77750727 kernel32!UnhandledExceptionFilter+0x1af
0354f494 77750604 ntdll!__RtlUserThreadStart+0x62
0354f4a8 777504a9 ntdll!_EH4_CallFilterFunc+0x12
0354f4d0 777387b9 ntdll!_except_handler4+0x8e
0354f4f4 7773878b ntdll!ExecuteHandler2+0x26
0354f5a4 776f010f ntdll!ExecuteHandler+0x24
0354f5a4 0354f958 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 0×354f958
00000000 00000000 0×2ff0340

0:009> kv
ChildEBP RetAddr  Args to Child             
[...]
0354f5a4 0354f958 0154f5bc 0354f60c 0354f5bc ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0354f60c)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 00000000 00000000 00000000 0×354f958
00000000 00000000 00000000 00000000 00000000 0×2ff0340

0:009> .cxr 0354f60c
eax=80010105 ebx=0354f924 ecx=00000003 edx=0000ffff esi=00d7dce0 edi=00d7e0c8
eip=0354f958 esp=0354f8f4 ebp=0354f908 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
0354f958 64f9            stc

0:009> !address 0354f958
 TEB 7efdd000 in range 7efdb000 7efde000
 TEB 7efda000 in range 7efd8000 7efdb000
 TEB 7efd7000 in range 7efd5000 7efd8000
 TEB 7efaf000 in range 7efad000 7efb0000
 TEB 7efac000 in range 7efaa000 7efad000
 TEB 7efa9000 in range 7efa7000 7efaa000
 TEB 7efa6000 in range 7efa4000 7efa7000
 TEB 7efa3000 in range 7efa1000 7efa4000
 TEB 7ef9f000 in range 7ef9d000 7efa0000
 TEB 7ef9c000 in range 7ef9a000 7ef9d000
 TEB 7ef99000 in range 7ef97000 7ef9a000
 ProcessParametrs 007714b0 in range 00770000 00870000
 Environment 007707f0 in range 00770000 00870000
    03450000 : 0354d000 - 00003000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageStack
                    Pid.Tid  1ea0.12dc

0:009> !address 02ff0340
 TEB 7efdd000 in range 7efdb000 7efde000
 TEB 7efda000 in range 7efd8000 7efdb000
 TEB 7efd7000 in range 7efd5000 7efd8000
 TEB 7efaf000 in range 7efad000 7efb0000
 TEB 7efac000 in range 7efaa000 7efad000
 TEB 7efa9000 in range 7efa7000 7efaa000
 TEB 7efa6000 in range 7efa4000 7efa7000
 TEB 7efa3000 in range 7efa1000 7efa4000
 TEB 7ef9f000 in range 7ef9d000 7efa0000
 TEB 7ef9c000 in range 7ef9a000 7ef9d000
 TEB 7ef99000 in range 7ef97000 7ef9a000
 ProcessParametrs 007714b0 in range 00770000 00870000
 Environment 007707f0 in range 00770000 00870000
    02fc0000 : 02fc0000 - 00043000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   00d70000

Stack trace may or may not be included here and it might be incorrect, heuristic and not fully discernible automatically (requires raw stack semantic analysis) like in the example above. In some cases exception information might not be valid though, for example, in the case of laterally damaged or truncated memory dump files.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 4 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4

It features:

- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art

Its table of contents is available here:

http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I have discovered rules that make it possible to devise a memory dump and software trace analysis equivalent of the Periodic Table of Elements in Chemistry. It allows prediction of abnormal software behaviour and structural defects and what patterns to look for after deploying software and collecting its artifacts. More on this is in the next part of these series.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a kernel mode counterpart of Divide by Zero pattern in user mode. It manifests under different bugchecks, for example:

1: kd> !analyze -v

[...]

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind that the kernel isn't allowed to have/catch (bound trap) or that is always instant death (double fault).  The first number in the bugcheck params is the number of the trap (8 = double fault, etc) Consult an Intel x86 family manual to learn more about what these traps are. Here is a *portion* of those codes:
If kv shows a taskGate
        use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
        use .trap on that value
Else
        .trap on the appropriate frame will show where the trap was taken
        (on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000000, EXCEPTION_DIVIDED_BY_ZERO
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

[...]

TRAP_FRAME:  a8954c8c -- (.trap 0xffffffffa8954c8c)
ErrCode = 00000000
eax=ffffffff ebx=00000000 ecx=00000005 edx=00000000 esi=00000000 edi=00000000
eip=975c42cd esp=a8954d00 ebp=a8954d4c iopl=0 nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000  efl=00010246
win32k!NtGdiEnumObjects+0xc6:
975c42cd f7f6            div     eax,esi
Resetting default scope

PROCESS_NAME:  Application.EXE

[...]

STACK_TEXT: 
a8954c2c 81ac2b76 0000007f 5317512a 975c42cd nt!KeBugCheck+0x14
a8954c80 81899808 a8954c8c a8954d4c 975c42cd nt!Ki386CheckDivideByZeroTrap+0×44
a8954c80 975c42cd a8954c8c a8954d4c 975c42cd nt!KiTrap00+0×88
a8954d4c 81898a7a 062102ce 00000001 00000000 Driver!EnumObjects+0xc6
a8954d4c 77a59a94 062102ce 00000001 00000000 nt!KiFastCallEntry+0×12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012ca70 00000000 00000000 00000000 00000000 0×77a59a94

0: kd> !analyze -v

[...]

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000094, Exception code that caused the bugcheck
Arg2: fffff9600025ba6d, Address of the exception record for the exception that caused the bugcheck
Arg3: fffff8800ac361d0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

[...]

EXCEPTION_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION}  Integer division by zero.

FAULTING_IP:
Driver!EnumObjects+e9
fffff960`0025ba6d f7f6            div     eax,esi

CONTEXT:  fffff8800ac361d0 -- (.cxr 0xfffff8800ac361d0)
rax=00000000ffffffff rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff9600025ba6d rsp=fffff8800ac36ba0 rbp=fffff8800ac36ca0
 r8=0000000000000000  r9=0000000000000000 r10=0000000005892f18
r11=fffff900c28379e0 r12=0000000000000000 r13=0000000000000002
r14=0000000000000001 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b  efl=00010286
Driver!EnumObjects+0xe9:
fffff960`0025ba6d f7f6            div     eax,esi
Resetting default scope

[...]

STACK_TEXT:
fffff880`0ac36ba0 fffff800`01682993 Driver!EnumObjects+0xe9
fffff880`0ac36c20 00000000`748a1b3a nt!KiSystemServiceCopyEnd+0x13
00000000`001cdf08 00000000`00000000 0x748a1b3a

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -