Software Diagnostics Library

In addition to hooked functions pattern we should also pay attention to Hooking Level. The latter is the number of patched functions. Often value-added hooksware has configuration options that fine-tune hooking behavior. For example, an application with the less number of patched functions behaved incorrectly and two process user dumps were saved from the working and non-working environment:  

0:000> * problem behavior

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\user32.dll\49E0380E9d000\user32.dll
No range specified

Scanning section:    .text
Size: 422527
Range to scan: 76e31000-76e9827f
    76e3d6f8-76e3d6fc  5 bytes - user32!NtUserSetThreadDesktop
 [ b8 30 12 00 00:e9 03 29 13 09 ]
    76e3dc2a-76e3dc2e  5 bytes - user32!CreateWindowExA (+0x532)
 [ 8b ff 55 8b ec:e9 d1 23 15 09 ]
    76e3f8f8-76e3f8fc  5 bytes - user32!PostMessageA (+0x1cce)
 [ 8b ff 55 8b ec:e9 03 07 fa 08 ]
    76e41305-76e41309  5 bytes - user32!CreateWindowExW (+0x1a0d)
 [ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
    76e435e3-76e435e7  5 bytes - user32!NtUserSetWindowPos (+0x22de)
 [ b8 38 12 00 00:e9 18 ca 11 09 ]
    76e48343-76e48347  5 bytes - user32!PeekMessageA (+0x4d60)
 [ 8b ff 55 8b ec:e9 b8 7c fb 08 ]
    76e48ab3-76e48ab7  5 bytes - user32!GetMessageA (+0x770)
 [ 8b ff 55 8b ec:e9 48 75 fd 08 ]
    76e4a175-76e4a179  5 bytes - user32!PostMessageW (+0x16c2)
 [ 8b ff 55 8b ec:e9 86 5e f8 08 ]
    76e4fef7-76e4fefb  5 bytes - user32!GetMessageW (+0x5d82)
 [ 8b ff 55 8b ec:e9 04 01 fc 08 ]
    76e5045a-76e5045e  5 bytes - user32!PeekMessageW (+0x563)
 [ 8b ff 55 8b ec:e9 a1 fb f9 08 ]
    76e8d37d-76e8d381  5 bytes - user32!MessageBoxTimeoutW (+0x3cf23)
 [ 8b ff 55 8b ec:e9 7e 2c fd 08 ]
    76e8d4d9-76e8d4dd  5 bytes - user32!MessageBoxIndirectA (+0x15c)
 [ 8b ff 55 8b ec:e9 22 2b ff 08 ]
    76e8d5d3-76e8d5d7  5 bytes - user32!MessageBoxIndirectW (+0xfa)
 [ 8b ff 55 8b ec:e9 28 2a fe 08 ]
    76e8d65d-76e8d661  5 bytes - user32!MessageBoxExW (+0x8a)
 [ 8b ff 55 8b ec:e9 9e 29 00 09 ]
Total bytes compared: 422527(100%)
Number of errors: 70
70 errors : !user32 (76e3d6f8-76e8d661)

0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b 8bff            mov     edi,edi
76e3ba5d 55              push    ebp
76e3ba5e 8bec            mov     ebp,esp
76e3ba60 81ec54030000    sub     esp,354h
76e3ba66 a1c090e976      mov     eax,dword ptr [user32!__security_cookie (76e990c0)]
76e3ba6b 33c5            xor     eax,ebp
76e3ba6d 8945fc          mov     dword ptr [ebp-4],eax
76e3ba70 53              push    ebx

0:000> * expected behavior 

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\user32.dll\49E0380E9d000\user32.dll
No range specified

Scanning section:    .text
Size: 422527
Range to scan: 76e31000-76e9827f
    76e39c11-76e39c15  5 bytes - user32!MonitorFromPoint
 [ 6a 08 68 50 9c:e9 ea 63 10 09 ]
    76e3b8ea-76e3b8ee  5 bytes - user32!GetMonitorInfoA (+0x1cd9)
 [ 8b ff 55 8b ec:e9 11 47 12 09 ]
    76e3ba5b-76e3ba5f  5 bytes - user32!EnumDisplayDevicesW (+0×171)
 [ 8b ff 55 8b ec:e9 a0 45 0b 09 ]
    76e3d6f8-76e3d6fa  3 bytes - user32!NtUserSetThreadDesktop (+0×1c9d)
 [ b8 30 12:e9 03 29 ]
    76e3d6fc - user32!NtUserSetThreadDesktop+4 (+0×04)
 [ 00:09 ]
    76e3dc2a-76e3dc2e  5 bytes - user32!CreateWindowExA (+0×52e)
 [ 8b ff 55 8b ec:e9 d1 23 15 09 ]
    76e3e7cd-76e3e7d1  5 bytes - user32!SetWindowLongA (+0xba3)
 [ 8b ff 55 8b ec:e9 2e 18 03 09 ]
    76e3f8f8-76e3f8fc  5 bytes - user32!PostMessageA (+0×112b)
 [ 8b ff 55 8b ec:e9 03 07 e7 08 ]
    76e41305-76e41309  5 bytes - user32!CreateWindowExW (+0×1a0d)
 [ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
    76e413b4-76e413b8  5 bytes - user32!SetWindowLongW (+0xaf)
 [ 8b ff 55 8b ec:e9 47 ec 03 09 ]
    76e41709-76e4170d  5 bytes - user32!MonitorFromRect (+0×355)
 [ 6a 08 68 48 17:e9 f2 e8 0e 09 ]
    76e435e3-76e435e7  5 bytes - user32!NtUserSetWindowPos (+0×1eda)
 [ b8 38 12 00 00:e9 18 ca fe 08 ]
    76e440c5-76e440c9  5 bytes - user32!EnumDisplaySettingsExW (+0xae2)
 [ 8b ff 55 8b ec:e9 36 bf 06 09 ]
    76e441a1-76e441a5  5 bytes - user32!EnumDisplaySettingsW (+0xdc)
 [ 8b ff 55 8b ec:e9 5a be 08 09 ]
    76e46d4a-76e46d4e  5 bytes - user32!EnumDisplayDevicesA (+0×2ba9)
 [ 8b ff 55 8b ec:e9 b1 92 0b 09 ]
    76e46fe6-76e46fea  5 bytes - user32!EnumDisplaySettingsA (+0×29c)
 [ 8b ff 55 8b ec:e9 15 90 09 09 ]
    76e47010-76e47014  5 bytes - user32!EnumDisplaySettingsExA (+0×2a)
 [ 8b ff 55 8b ec:e9 eb 8f 07 09 ]
    76e47d12-76e47d16  5 bytes - user32!GetMonitorInfoW (+0xd02)
 [ 8b ff 55 8b ec:e9 e9 82 10 09 ]
    76e48343-76e48347  5 bytes - user32!PeekMessageA (+0×631)
 [ 8b ff 55 8b ec:e9 b8 7c e8 08 ]
    76e4844c-76e48450  5 bytes - user32!NtUserEnumDisplayMonitors (+0×109)
 [ b8 81 11 00 00:e9 af 7b 0c 09 ]
    76e488d4-76e488d8  5 bytes - user32!MonitorFromWindow (+0×488)
 [ 6a 08 68 28 89:e9 27 77 0d 09 ]
    76e48ab3-76e48ab7  5 bytes - user32!GetMessageA (+0×1df)
 [ 8b ff 55 8b ec:e9 48 75 ea 08 ]
    76e49994-76e49998  5 bytes - user32!GetWindowLongA (+0xee1)
 [ 6a 08 68 d0 99:e9 67 66 00 09 ]
    76e49af1-76e49af5  5 bytes - user32!GetSystemMetrics (+0×15d)
 [ 6a 0c 68 58 9b:e9 0a 65 12 09 ]
    76e4a175-76e4a179  5 bytes - user32!PostMessageW (+0×684)
 [ 8b ff 55 8b ec:e9 86 5e e5 08 ]
    76e4f8bf-76e4f8c3  5 bytes - user32!GetWindowLongW (+0×574a)
 [ 6a 08 68 00 f9:e9 3c 07 01 09 ]
    76e4fef7-76e4fefb  5 bytes - user32!GetMessageW (+0×638)
 [ 8b ff 55 8b ec:e9 04 01 e9 08 ]
    76e5045a-76e5045e  5 bytes - user32!PeekMessageW (+0×563)
 [ 8b ff 55 8b ec:e9 a1 fb e6 08 ]
    76e8d37d-76e8d381  5 bytes - user32!MessageBoxTimeoutW (+0×3cf23)
 [ 8b ff 55 8b ec:e9 7e 2c ea 08 ]
    76e8d4d9-76e8d4dd  5 bytes - user32!MessageBoxIndirectA (+0×15c)
 [ 8b ff 55 8b ec:e9 22 2b ec 08 ]
    76e8d5d3-76e8d5d7  5 bytes - user32!MessageBoxIndirectW (+0xfa)
 [ 8b ff 55 8b ec:e9 28 2a eb 08 ]
    76e8d65d-76e8d661  5 bytes - user32!MessageBoxExW (+0×8a)
 [ 8b ff 55 8b ec:e9 9e 29 ed 08 ]
Total bytes compared: 422527(100%)
Number of errors: 154
154 errors : !user32 (76e39c11-76e8d661)

0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b e9a0450b09      jmp     7fef0000
76e3ba60 81ec54030000    sub     esp,354h
76e3ba66 a1c090e976      mov     eax,dword ptr [user32!__security_cookie (76e990c0)]
76e3ba6b 33c5            xor     eax,ebp
76e3ba6d 8945fc          mov     dword ptr [ebp-4],eax
76e3ba70 53              push    ebx
76e3ba71 56              push    esi
76e3ba72 8b7510          mov     esi,dword ptr [ebp+10h]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Embedded Comments in dump files are useful to record external information like the reason for saving a memory dump, a tool used to do that, and some pre-analysis and monitoring data that might help or guide in the future analysis. Comments are not widely used but some examples include Manual Process Dump, False Positive Dump patterns, and process and thread CPU consumption comments in dump files saved by Sysinternals ProcDump tool. Such comments may not be necessarily saved by IDebugClient2 :: WriteDumpFile2 function but any buffer saved in memory that is accessible later from a dump file will do as can be easily demonstrated by the old Citrix SystemDump tool.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If we look at any non-trivial trace we would see different Implementation Discourses. Components are written in different languages and adhere to different runtime environments, binary models and interface frameworks. All these implementation variations influence the structure, syntax and semantics of trace messages. For example, .NET debugging traces differ from file system driver or COM debugging messages. Therefore we establish the new field of Software Trace Linguistics as a science of software trace languages. Some parallels can be drawn here towards software linguistics (the science of software languages) although we came to that conclusion independently while thinking about applying “ethnography of speaking” to software trace narration. More on this in the following posts.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

By analogy with Well-Tested Function we introduce another pattern called Well-Tested Module. This is a module we usually skip when analyzing a stack trace because we suspect it the least. WinDbg can also be customized to skip such modules for the default analysis command as shown in the following example: Minidump Analysis (Part 2)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We now start unifying software behavior analysis patterns with debugging architecture, design, implementation and usage. This is analogous to software construction where a problem analysis leads to various software engineering phases. The important difference here is the addition of debugging usage patterns. Let’s look at an example (we discuss suggested patterns later):

- Analysis Patterns

Shared Buffer Overwrite

- Architectural Patterns

Debug Event Subscription / Notification

- Design Patterns

Punctuated Execution

- Implementation Patterns

Breakpoint (software and hardware)

- Usage Patterns

Kernel vs. user space breakpoints

To differentiate this systematic approach from the various published ad hoc debugging patterns we call it Unified Debugging Pattern Language. ADI parts can also correspond to various DebugWare patterns where we provide a mapping later.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This morning it was -2 with lost of snow left from the yesterday spike. Here is Dublin Citrix Office in Eastpoint Business Park as seen from outside:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first big one was in January this year: System Freeze in Nature. It is December now and we have the second snowfall in just one year which is the longest and coldest as well. The pictures below were taken 5 days ago when it started with temperatures down to -7 in the morning so we were able to practice only small scale architecture while building a snowman:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One frequently useful analysis pattern is the presence of String Parameter on a function call stack. The trivial case is when a function parameter is a pointer to an ASCII or a Unicode string (da and du WinDbg commands). More interesting case is when we have a function that takes pointers to a structure that has string fields (dpa and dpu commands), for example:

0:018> kv 100
ChildEBP RetAddr  Args to Child             
00de8c7c 7739bf53 7739610a 07750056 00000000 ntdll!KiFastSystemCallRet
00de8cb4 7738965e 00080126 07750056 00000001 user32!NtUserWaitMessage+0xc
00de8cdc 7739f762 77380000 0012b238 07750056 user32!InternalDialogBox+0xd0
00de8f9c 7739f047 00de90f8 00000000 ffffffff user32!SoftModalMessageBox+0x94b
00de90ec 7739eec9 00de90f8 00000028 07750056 user32!MessageBoxWorker+0x2ba
00de9144 773d7d0d 07750056 0015cd68 00132a60 user32!MessageBoxTimeoutW+0x7a
00de9178 773c42c8 07750056 00de923f 00de91ec user32!MessageBoxTimeoutA+0x9c
00de9198 773c42a4 07750056 00de923f 00de91ec user32!MessageBoxExA+0x1b
00de91b4 6dfcf8c2 07750056 00de923f 00de91ec user32!MessageBoxA+0×45
00de99f0 6dfcfad2 00de9285 00de9a1c 77bc6cd5 compstui!FilterException+0×174
00dead94 7739b6e3 0038010e 00000110 00000000 compstui!CPSUIPageDlgProc+0xf3
00deadc0 77395f82 6dfcf9df 0038010e 00000110 user32!InternalCallWinProc+0×28
00deae3c 77395e22 0015d384 6dfcf9df 0038010e user32!UserCallDlgProcCheckWow+0×147
00deae84 7738aaa4 00000000 00000110 00000000 user32!DefDlgProcWorker+0xa8
00deaeb4 77388c01 004673d0 00461130 00000000 user32!SendMessageWorker+0×43e
00deaf6c 77387910 6dfc0000 004673d0 00000404 user32!InternalCreateDialog+0×9cf
00deaf90 7739fb5b 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamAorW+0×33
00deafb0 774279a5 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamW+0×1b
00deb000 77427abc 02192c78 000ddd08 07750056 comctl32!_CreatePageDialog+0×79
00deb028 77429d12 02192c78 6dff5c30 07750056 comctl32!_CreatePage+0xb1
00deb244 7742b8b6 02192c78 00000001 00290110 comctl32!PageChange+0xcc
00deb604 7742c446 07750056 02192c78 00deb6ec comctl32!InitPropSheetDlg+0xbb8
00deb674 7739b6e3 07750056 00000110 00290110 comctl32!PropSheetDlgProc+0×4cb
00deb6a0 77395f82 7742bf7b 07750056 00000110 user32!InternalCallWinProc+0×28
00deb71c 77395e22 0008c33c 7742bf7b 07750056 user32!UserCallDlgProcCheckWow+0×147
00deb764 7738aaa4 00000000 00000110 00290110 user32!DefDlgProcWorker+0xa8
00deb794 77388c01 004652e0 00461130 00290110 user32!SendMessageWorker+0×43e
00deb84c 77387910 77420000 004652e0 00000100 user32!InternalCreateDialog+0×9cf
00deb870 7739fb5b 77420000 02184be8 00000000 user32!CreateDialogIndirectParamAorW+0×33
00deb890 774ab1c5 77420000 02184be8 00000000 user32!CreateDialogIndirectParamW+0×1b
00deb8d8 7742ca78 77420000 02184be8 00000000 comctl32!SHFusionCreateDialogIndirectParam+0×36
00deb93c 7742ccea 00000000 000000a0 00000000 comctl32!_RealPropertySheet+0×242
00deb954 7742cd05 00deb9b4 00000000 00deb99c comctl32!_PropertySheet+0×146
00deb964 6dfd1178 00deb9b4 000000a0 00deba30 comctl32!PropertySheetW+0xf
00deb99c 6dfcf49b 00deb9b4 0256b3f8 0013fbe0 compstui!PropertySheetW+0×4b
00deba14 6dfd0718 00000000 00134da4 00debae8 compstui!DoComPropSheet+0×2ef
00deba44 6dfd0799 00000000 7307c8da 00debad0 compstui!DoCommonPropertySheetUI+0xe9
00deba5c 730801c5 00000000 7307c8da 00debad0 compstui!CommonPropertySheetUIW+0×17
00debaa4 73080f5d 00000000 7307c8da 00debad0 winspool!CallCommonPropertySheetUI+0×43
00debeec 4f49cdfe 00000000 0218bd84 02277fe8 winspool!PrinterPropertiesNative+0×10c
WARNING: Stack unwind information not available. Following frames may be wrong.
00debf2c 4f4950a5 00deea08 00000002 02277fe8 PrintDriverA!DllGetClassObject+0xdb7e
00deee18 4f4904fb 00ca6ee0 00000003 00000001 PrintDriverA!DllGetClassObject+0×5e25
00deee30 18f60282 02277fe8 00ca6ee0 00000003 PrintDriverA!DllGetClassObject+0×127b
00deee58 18f5abce 001042e4 00ca6ee0 00000003 ps5ui!HComOEMPrinterEvent+0×33
00deee9c 7308218c 00ca6ee0 00000003 00000001 ps5ui!DrvPrinterEvent+0×22e
00deeee8 761543c8 00ca6ee0 00000003 00000001 winspool!SpoolerPrinterEventNative+0×57
00deef04 761560d2 00ca6ee0 00000003 00000000 localspl!SplDriverEvent+0×21
00deef28 761447f9 00cb2160 00000003 00000000 localspl!PrinterDriverEvent+0×46
00def3f0 76144b12 00000000 00000002 00d12020 localspl!SplAddPrinter+0×5f3
00def41c 74070193 00000000 00000002 00d12020 localspl!LocalAddPrinterEx+0×2e
00def86c 7407025c 00000000 00000002 00d12020 spoolss!AddPrinterExW+0×151
00def888 01007a93 00000000 00000002 00d12020 spoolss!AddPrinterW+0×17
00def8a4 01006772 00000000 00ce74b0 021b6278 spoolsv!YAddPrinter+0×75
00def8c8 77c80355 00000000 00ce74b0 021b6278 spoolsv!RpcAddPrinter+0×37
00def8f0 77ce43e1 0100673b 00defae0 00000005 rpcrt4!Invoke+0×30
00defcf8 77ce45c4 00000000 00000000 000e8584 rpcrt4!NdrStubCall2+0×299
00defd14 77c8013a 000e8584 000d63d8 000e8584 rpcrt4!NdrServerCall2+0×19
00defd48 77c805ef 01002c57 000e8584 00defdec rpcrt4!DispatchToStubInCNoAvrf+0×38
00defd9c 77c80515 00000005 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
00defdc0 77c8139e 000e8584 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
00defdfc 77c814b2 000e1c48 000d85b8 02154180 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×42c
00defe20 77c88848 000d85f0 00defe38 000e1c48 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
00deff84 77c88962 00deffac 77c888fd 000d85b8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
00deff8c 77c888fd 000d85b8 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
00deffac 77c7b293 0008b038 00deffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0×9d
00deffb8 77e6482f 000bdba8 00000000 00000000 rpcrt4!ThreadStartRoutine+0×1b
00deffec 00000000 77c7b278 000bdba8 00000000 kernel32!BaseThreadStart+0×34

0:018> da 00de923f
00de923f  “Function address 0×77481456 caus”
00de925f  “ed a protection fault. (exceptio”
00de927f  “n code 0xc0000005).The applicati”
00de929f  “on property sheet page(s) may no”
00de92bf  “t function properly.”

0:018> dpu 00d12020
00d12020  00000000
00d12024  021b6088 “Printer A User B Server C”
00d12028  00000000
00d1202c  021b6124 “Remote Printer Address for User C”
00d12030  021b6190 “Printer Name and Family”
00d12034  021b61c4 “Printer Client Name”
00d12038  021b6228 “Printer Location”
00d1203c  00000000
00d12040  00000000
00d12044  021b6264 “Printer Module Name”
00d12048  00000000
00d1204c  00000000
00d12050  021b628c
00d12054  00008841
00d12058  00000000
00d1205c  00000000
00d12060  00000000
00d12064  00000000
00d12068  00000000
00d1206c  00000000
00d12070  00000000
00d12074  00000000
00d12078  00000000
00d1207c  00000000
00d12080  00000000
00d12084  00000000
00d12088  00000000
00d1208c  00000000
00d12090  00000000
00d12094  00000000
00d12098  00000000
00d1209c  00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This specially designed logo explores the concept of Twitter message stream as a software (t)race:

Follow DumpAnalysis @ Twitter: http://twitter.com/DumpAnalysis

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -