Software Diagnostics Library

Tool Objects name was motivated by the title of the book Tool and Object: A History and Philosophy of Category Theory that I’m reading:

I realized that troubleshooting can be represented as a category of memory states (or collections of proximate states) as objects and troubleshooting tools as arrows.

We can also consider tools as categories with arrows as troubleshooting actions. In the latter case we can model a collection of tools and transformations (morphisms) between them as a functor.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It’s time to update my old Resume and CV: As a Book that was on sale for almost 2 years to reflect exponential changes in objectives, skills, achievements and deliverables since summer, 2003. Actually the number of pages has doubled. The 2nd edition is planned to be 111th book scheduled for publication in December:

• Title: Resume and CV: As a Book (2nd Edition)
• Author: Dmitry Vostokov  
• Paperback: 32 pages
• Publisher: OpenTask (01 Dec 2010)
• ISBN-13: 978-1-908043-00-9

The front cover color follows the proverbs about “green fields” (I also fixed a bug that was on the first edition front cover)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Suspended Thread pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The previously published script to dump raw stack of all threads dumps only 64-bit raw stack from 64-bit WOW64 process memory dumps (a 32-bit process saved in a 64-bit dump). In order to dump WOW64 32-bit raw stack from such 64-bit dumps we need another script. After I found a location of 32-bit TEB pointers (WOW64 TEB32) inside a 64-bit TEB structure I was able to create such a script:

0:000> .load wow64exts

0:000> !teb
Wow64 TEB32 at 000000007efdd000

Wow64 TEB at 000000007efdb000
    ExceptionList:        000000007efdd000
    StackBase:            000000000008fd20
    StackLimit:           0000000000082000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efdb000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000f34 . 0000000000000290
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0x7efdd000
TEB64: 0x7efdb000

32 bit, StackBase   : 0×1a0000
        StackLimit  : 0×190000
        Deallocation: 0xa0000

64 bit, StackBase   : 0x8fd20
        StackLimit  : 0x82000
        Deallocation: 0x50000

[...]

0:000:x86> dd 000000007efdd000 L4
7efdd000  0019fa84 001a0000 00190000 00000000

So the script obviously should be this:

~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->ExceptionList; !wow64exts.info; dds poi(@$t1+8) poi(@$t1+4)

Before running it against a freshly opened user dump we need to execute the following commands first after setting symbols right:

.load wow64exts;
.effmach x86

I’ve also created a page to put all such scripts together:

Raw Stack Analysis Scripts 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I aim to release a beta version of that unified troubleshooting system together with EasyDbg (as WinDbg add-on) in a few months after finishing porting it to x64 platform. The system is very customizable and similar buttons can be created to launch troubleshooting tools and execute troubleshooting units of work for any client and server applications. It will also be covered in the forthcoming Software Troubleshooting and Debugging Tools presentation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This pattern is called Invalid Exception Information and shows how to recognize it and get a stack trace right when a debugger is not able to locate a crash point. For example, WinDbg default analysis command is not able to locate the exception context for a crash and provides a heuristic stack trace:

0:000> !analyze -v

[...]

EXCEPTION_RECORD:  001150fc -- (.exr 0x1150fc)
ExceptionAddress: 7c7e2afb (kernel32!RaiseException+0x00000053)
   ExceptionCode: 0eedfade
  ExceptionFlags: 00000001
NumberParameters: 7
   Parameter[0]: 0098fa49
   Parameter[1]: 0374c200
   Parameter[2]: 00000000
   Parameter[3]: 005919b4
   Parameter[4]: 01d80010
   Parameter[5]: 00115704
   Parameter[6]: 001154a4

[...]

CONTEXT:  0012ffb4 — (.cxr 0×12ffb4)
eax=00000000 ebx=00000000 ecx=0000019c edx=00000214 esi=00000000 edi=00000000
eip=000003b0 esp=000002d8 ebp=2d59495b iopl=0 nv up ei ng zr na pe nc
cs=0032 ss=0010 ds=0002 es=0000 fs=0000 gs=0000 efl=000003e4
0032:000003b0 ??              ???

[...]

STACK_TEXT: 
7c910328 ntdll!`string'+0x4
7c7db7d0 kernel32!ConsoleApp+0xe
7c7db7a4 kernel32!ConDllInitialize+0x20f
7c7db7b9 kernel32!ConDllInitialize+0x224
7c915239 ntdll!bsearch+0x42
7c91542b ntdll!RtlpLocateActivationContextSection+0x15a
7c915474 ntdll!RtlpCompareActivationContextDataTOCEntryById+0x0
7c916104 ntdll!RtlpFindUnicodeStringInSection+0x23d
7c91534a ntdll!RtlpFindNextActivationContextSection+0x61
7c915742 ntdll!RtlFindNextActivationContextSection+0x46
7c9155ed ntdll!RtlFindActivationContextSectionString+0xde
7c915ce9 ntdll!RtlDecodeSystemPointer+0x9e7
7c915d47 ntdll!RtlDecodeSystemPointer+0xb0b
7c9158ff ntdll!RtlDecodeSystemPointer+0x45b
7c915bf8 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x346
7c915c5d ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3de
7c97e214 ntdll!DllExtension+0xc
00800000 ApplicationA+0x400000
7c910000 ntdll!RtlFreeHeap+0x1a4
7c914a53 ntdll!LdrLockLoaderLock+0x146
7c912d04 ntdll!LdrLockLoaderLock+0x1d2
7c912d71 ntdll!LdrUnlockLoaderLock+0x88
7c916768 ntdll!LdrGetDllHandleEx+0xc9
7c912d80 ntdll!`string'+0x84
7c91690e ntdll!LdrGetDllHandleEx+0x2f1
7c912d78 ntdll!LdrUnlockLoaderLock+0xb1
7c97ecc0 ntdll!LdrpHotpatchCount+0x8
7c9167e8 ntdll!`string'+0xc4
7c9168d6 ntdll!LdrGetDllHandleEx+0x2de
7c9166b8 ntdll!LdrGetDllHandle+0x18
7c7de534 kernel32!GetModuleHandleForUnicodeString+0x1d
7c7de544 kernel32!GetModuleHandleForUnicodeString+0xa0
7c7de64b kernel32!BasepGetModuleHandleExW+0x18e
7c7de6cb kernel32!BasepGetModuleHandleExW+0x250
79000000 mscoree!_imp__EnterCriticalSection <PERF> +0x0
7c809ad8 kernel32!_except_handler3+0x0
7c7de548 kernel32!`string'+0x28
79002280 mscoree!`string'+0x0
02080000 xpsp2res+0xc0000
7c7db6d4 kernel32!_BaseDllInitialize+0x7a
7c7db6e9 kernel32!_BaseDllInitialize+0x488
7c917ef3 ntdll!LdrpSnapThunk+0xbd
7c9048b8 ntdll!$$VProc_ImageExportDirectory+0x14b8
7c9000d0 ntdll!RtlDosPathSeperatorsString <PERF> +0x0
7c905d48 ntdll!$$VProc_ImageExportDirectory+0x2948
7c910228 ntdll!RtlpRunTable+0x448
7c910222 ntdll!RtlpAllocateFromHeapLookaside+0x42
7c911086 ntdll!RtlAllocateHeap+0x43d
7c903400 ntdll!$$VProc_ImageExportDirectory+0x0
7c7d9036 kernel32!$$VProc_ImageExportDirectory+0x6a0a
791c6f2d mscorwks!DllMain+0x117
7c917e10 ntdll!`string'+0xc
7c918047 ntdll!LdrpSnapThunk+0x317
7c7d00f0 kernel32!_imp___wcsnicmp <PERF> +0x0
7c7d903c kernel32!$$VProc_ImageExportDirectory+0x6a10
7c917dba ntdll!LdrpGetProcedureAddress+0x186
7c900000 ntdll!RtlDosPathSeperatorsString <PERF> +0x0
7c917e5f ntdll!LdrpGetProcedureAddress+0x29b
7c7d262c kernel32!$$VProc_ImageExportDirectory+0x0
7c7d0000 kernel32!_imp___wcsnicmp <PERF> +0x0
79513870 mscorsn!DllMain+0x119
7c913425 ntdll!RtlDecodePointer+0x0
00726574 ApplicationA+0x326574
7c917e09 ntdll!LdrpGetProcedureAddress+0xa6
7c917ec0 ntdll!LdrGetProcedureAddress+0x18
7c9101e0 ntdll!CheckHeapFillPattern+0x54
7c9101db ntdll!RtlAllocateHeap+0xeac
40ae17ea msxml6!calloc+0xa9
40ae181f msxml6!calloc+0xde
40a30000 msxml6!_imp__OpenThreadToken <PERF> +0x0
7c910323 ntdll!RtlpImageNtHeader+0x56
7c910385 ntdll!RtlImageDirectoryEntryToData+0x57
00400100 ApplicationA+0x100
7c928595 ntdll!LdrpCallTlsInitializers+0x1d
00400000 ApplicationA+0x0
7c9285c7 ntdll!LdrpCallTlsInitializers+0xd8
7c90118a ntdll!LdrpCallInitRoutine+0x14
00a23010 ApplicationA+0x623010
7c9285d0 ntdll!`string'+0x18
7c935e24 ntdll!LdrpInitializeThread+0x147
7c91b1b7 ntdll!LdrpInitializeThread+0x13b
778e159a SETUPAPI!_DllMainCRTStartup+0x0
7c91b100 ntdll!`string'+0x88
7c91b0a4 ntdll!_LdrpInitialize+0x25b
7c90de9a ntdll!NtTestAlert+0xc
7c91b030 ntdll!`string'+0xc8
7c91b02a ntdll!_LdrpInitialize+0x246
7c90d06a ntdll!NtContinue+0xc
7c90e45f ntdll!KiUserApcDispatcher+0xf
00780010 ApplicationA+0x380010
7c951e13 ntdll!DbgUiRemoteBreakin+0x0
7c97e178 ntdll!LdrpLoaderLock+0x0
00d10000 ApplicationA+0x910000
7c951e40 ntdll!DbgUiRemoteBreakin+0x2d
7c90e920 ntdll!_except_handler3+0x0
7c951e60 ntdll!`string'+0x7c

Compare our invalid context data with the normal one having good efl and segment register values:

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206

We look at our stack trace after resetting context and using kv command. We see that KiUserExceptionDispatcher has the valid exception context but exception pointers for UnhandledExceptionFilter are not valid:

0:000> .ecxr

0:000> kv
ChildEBP RetAddr  Args to Child             
001132d0 7c90df4a 7c7d9590 00000002 001132fc ntdll!KiFastSystemCallRet
001132d4 7c7d9590 00000002 001132fc 00000001 ntdll!ZwWaitForMultipleObjects+0xc
00113370 7c7da115 00000002 001134a0 00000000 kernel32!WaitForMultipleObjectsEx+0x12c
0011338c 6993763c 00000002 001134a0 00000000 kernel32!WaitForMultipleObjects+0x18
00113d20 699382b1 00115018 00000001 00198312 faultrep!StartDWException+0x5df
00114d94 7c834526 00115018 00000001 00000000 faultrep!ReportFault+0x533
00115008 0040550c 00115018 7c9032a8 001150fc kernel32!UnhandledExceptionFilter+0×55b
WARNING: Stack unwind information not available. Following frames may be wrong.
00115034 7c90327a 001150fc 0012ffb4 0011512c ApplicationA+0×550c
001150e4 7c90e48a 00000000 0011512c 001150fc ntdll!ExecuteHandler+0×24
001150e4 7c7e2afb 00000000 0011512c 001150fc ntdll!KiUserExceptionDispatcher+0xe (CONTEXT @ 0011512c)
0011544c 0057ac37 0eedfade 00000001 00000007 kernel32!RaiseException+0×53
00115470 0098fa49 0eedfade 00000001 00000007 ApplicationA+0×17ac37
[…]
0012268c 7e398816 017d0f87 000607e8 0000001a USER32!InternalCallWinProc+0×28
001226f4 7e3a8ea0 00000000 017d0f87 000607e8 USER32!UserCallWinProcCheckWow+0×150

0:000> dd 00115018 L4
00115018  001150fc 0012ffb4 0011512c 001150d0

So we use the valid context pointer now:

0:000> .cxr 0011512c
eax=001153fc ebx=0eedfade ecx=00000000 edx=001537a8 esi=001154a4 edi=00000007
eip=7c7e2afb esp=001153f8 ebp=0011544c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00200202
kernel32!RaiseException+0×53:
7c7e2afb 5e              pop     esi

0:000> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
0011544c 0057ac37 0eedfade 00000001 00000007 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
00115470 0098fa49 0eedfade 00000001 00000007 ApplicationA+0x17ac37
[...]
0012268c 7e398816 017d0f87 000607e8 0000001a USER32!InternalCallWinProc+0x28
001226f4 7e3a8ea0 00000000 017d0f87 000607e8 USER32!UserCallWinProcCheckWow+0x150
00122748 7e3aacd1 00fd2ad0 0000001a 00000000 USER32!DispatchClientMessage+0xa3
00122778 7c90e473 00122788 00000030 00000030 USER32!__fnINSTRING+0x37
001227b4 7e3993e9 7e3993a8 00122840 00000000 ntdll!KiUserCallbackDispatcher+0x13
001227e0 7e3aa43b 00122840 00000000 00000000 USER32!NtUserPeekMessage+0xc
0012280c 004794d9 00122840 00000000 00000000 USER32!PeekMessageA+0xeb
001228bc 00461667 0012ff7c 00461680 001228e0 ApplicationA+0x794d9
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is time now to introduce a syntactical notation for memory (dump) and software trace analysis pattern languages (in addition to graphical notation proposed earlier). It should be simple and concise: allow easy grammar with plain syntax and obvious reading semantics. We propose to use capitalized letters for major pattern categories, for example, W for wait chains and D for deadlocks. Then use subscripts (or small letters) for pattern subcategories, for example, Wcs and Dlpc. Several categories and subcategories can be combined by using slash (/), for example, Wcs/Dcs/lpc. Slash notation is better viewed using subscripts:

W_cs/D_cs/lpc

Next part will introduce more categories and propose notational adornments for pattern succession, space differentiation and the inclusion of details in notational sentences.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for No System Dumps pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The choice of webinars below mostly reflects my personal preferences and long time desire to speak on topics like systems thinking, troubleshooting tool design and development. The other topic about BSOD minidump analysis was requested by participants in an ongoing survey. There will be more topics in 2011. All forthcoming webinars will be hosted by Memory Dump Analysis Services. The planning list includes:

Systems Thinking in Memory Dump and Software Trace Analysis

Software Troubleshooting and Debugging Tools: Objects, Components, Patterns and Frameworks with UML

• UML basics
• DebugWare patterns
• Unified Troubleshooting Framework
• RADII software development process
• Hands-on exercise: designing and building a tool

Blue Screen Of Death Analysis Done Right: Minidump Investigation for System Administrators

• Making sense of !analyze –v output
• Get extra troubleshooting information with additional WinDbg commands
• Guessing culprits with raw stack analysis
• Who’s responsible: hardware or software?
• Checklist and patterns
• Including hands-on exercises: send your own minidumps

More detailed information will be available soon. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Don’t name your driver a “Missile” blog post dealt with funny names seen in crash dumps. However, even innocuous driver names may occasionally provoke a laughter from people in the know. For example, SGUB32.SYS can be read 23BUGS in reverse. My recent encounter is a print driver SGNUD64.dll where we read 46DUNGS in reverse. Don’t rush to Google the name to find ISV, it was modified to avoid an engineering embarrassment, although a dung was really there

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is an image fragment from the front cover of the forthcoming Debugged! MZ/PE December issue:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is a common technique in driver development to copy/paste an entire driver sample from WDK and modify it for a specific device or a filter value-adding functionality in a specific driver stack or framework. The problem here is that developers sometimes forget to change module resources and certain amount of detective work is required to find out the module vendor. I call this pattern Template Module. Here is an example. In a spooler service there were many threads blocked in displaying a dialog box:

0:000> ~34kL 100
Child-SP          RetAddr           Call Site
00000000`08a49368 00000000`774c5118 user32!NtUserWaitMessage+0xa
00000000`08a49370 00000000`774c5770 user32!DialogBox2+0x261
00000000`08a493f0 00000000`774c57e6 user32!InternalDialogBox+0x134
00000000`08a49450 00000000`774c3e36 user32!DialogBoxIndirectParamAorW+0x58
00000000`08a49490 000007fe`fa27cc97 user32!DialogBoxParamW+0x66
00000000`08a494d0 000007fe`fa28832b unidrvui!ICheckConstraintsDlg+0xbf
00000000`08a49950 000007fe`fa29423d unidrvui!BUpdateUISettingForOEM+0x2f
00000000`08a49980 00000000`50036d2c unidrvui!CPrintOemDriverUI::DrvUpdateUISetting+0x1d
00000000`08a499b0 00000000`50038a1d ModuleZ!DllGetClassObject+0×1fe74
[…]
00000000`08a4b250 000007fe`f759546b unidrvui!OEMDevicePropertySheets+0×56
00000000`08a4b280 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4b330 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4b5f0 000007fe`fa2880e9 compstui!CPSUICallBack+0×3ec
00000000`08a4b6a0 000007fe`fa2836c4 unidrvui!BAddOemPluginPages+0×12d
00000000`08a4b6d0 000007fe`f759546b unidrvui!DrvDevicePropertySheets+0×2c8
00000000`08a4bb60 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4bc10 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4bed0 000007fe`fb452838 compstui!CPSUICallBack+0×3ec
00000000`08a4bf80 000007fe`f759546b winspool!DevicePropertySheets+0×108
00000000`08a4bfb0 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4c060 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4c320 000007fe`f759758e compstui!CPSUICallBack+0×3ec
00000000`08a4c3d0 000007fe`f75976b2 compstui!DoCommonPropertySheetUI+0xbe
00000000`08a4c430 000007fe`fb446339 compstui!CommonPropertySheetUIW+0xe
00000000`08a4c470 000007fe`fb44b425 winspool!CallCommonPropertySheetUI+0×65
00000000`08a4c4c0 00000000`5003623c winspool!PrinterPropertiesNative+0×121
00000000`08a4c950 00000000`50035d16 ModuleZ!DllGetClassObject+0×1f384
[…]
00000000`08a4dd70 000007fe`fb4472d8 unidrvui!DrvPrinterEvent+0×419
00000000`08a4de00 000007fe`fb44737f winspool!SpoolerPrinterEventNative+0×84
00000000`08a4de60 000007fe`faedc957 winspool!SpoolerPrinterEvent+0×13
00000000`08a4dea0 000007fe`faedc8c7 localspl!SplDriverEvent+0×4f
00000000`08a4def0 000007fe`faec3d74 localspl!PrinterDriverEvent+0xcf
00000000`08a4df30 000007fe`fa771f20 localspl!SplAddPrinter+0xae0
00000000`08a4e4e0 000007fe`fa7491d8 win32spl!NCSRCommon::TLocalPrinter::AddPrinterW+0xb4
00000000`08a4e5b0 000007fe`fa747511 win32spl!TPrintOpen::AddLocalPrinter+0xb8
00000000`08a4e6b0 000007fe`fa746dfb win32spl!TPrintOpen::AddAndInstallLocalPrinter+0×34d
00000000`08a4e830 000007fe`fa746bb0 win32spl!TPrintOpen::ReEstablishCacheConnectionNoGuidPrinter+0×157
00000000`08a4e900 000007fe`fa7467d1 win32spl!TPrintOpen::ReEstablishCacheConnection+0×178
00000000`08a4e980 000007fe`fa7465c1 win32spl!TPrintOpen::ReEstablishPrinterConnection+0×16d
00000000`08a4ea30 000007fe`fa73e5ad win32spl!TPrintOpen::ReEstablishConnectionFromKey+0×1fd
00000000`08a4eb30 000007fe`fa733492 win32spl!TPrintOpen::RediscoverPrinterConnections+0xd7
00000000`08a4ebe0 000007fe`fb3f2332 win32spl!TPrintProviderTable::forwardEnumPrinters+0×47
00000000`08a4ec70 00000000`ff3414c8 spoolss!EnumPrintersW+0×176
00000000`08a4ed20 00000000`ff3413cc spoolsv!YEnumPrinters+0×112
00000000`08a4eda0 000007fe`fe225ec5 spoolsv!RpcEnumPrinters+0×30
00000000`08a4edf0 000007fe`fe2bebed rpcrt4!Invoke+0×65
00000000`08a4ee70 000007fe`fe1f5df0 rpcrt4!Ndr64StubWorker+0×5a9
00000000`08a4f440 000007fe`fe2268d4 rpcrt4!NdrServerCallAll+0×40
00000000`08a4f490 000007fe`fe2269f0 rpcrt4!DispatchToStubInCNoAvrf+0×14
00000000`08a4f4c0 000007fe`fe227402 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×100
00000000`08a4f5b0 000007fe`fe227080 rpcrt4!LRPC_SCALL::DispatchRequest+0×1c2
00000000`08a4f620 000007fe`fe2262bb rpcrt4!LRPC_SCALL::HandleRequest+0×200
00000000`08a4f740 000007fe`fe225e1a rpcrt4!LRPC_ADDRESS::ProcessIO+0×44a
00000000`08a4f860 000007fe`fe207769 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0×24a
00000000`08a4f910 000007fe`fe207714 rpcrt4!ProcessIOEventsWrapper+0×9
00000000`08a4f940 000007fe`fe2077a4 rpcrt4!BaseCachedThreadRoutine+0×94
00000000`08a4f980 00000000`7758be3d rpcrt4!ThreadStartRoutine+0×24
00000000`08a4f9b0 00000000`776c6a51 kernel32!BaseThreadInitThunk+0xd
00000000`08a4f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We suspect ModuleZ but its module information points to Microsoft: 

0:000> lmv m ModuleZ
start             end                 module name
00000000`50000000 00000000`500a4000   ModuleZ   (export symbols)       ModuleZ.DLL
    Loaded symbol image file: ModuleZ.DLL
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleZ.DLL
    Image name: ModuleZ.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     2.5.0.0
    Product version:  2.5.0.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0407.04b0
    CompanyName:      Microsoft Corp.
    ProductName:      Microsoft PS UI Replacement Sample
    InternalName:     PSUIREP
    OriginalFilename: PSUIREP.dll
    ProductVersion:   2.5
    FileVersion:      2.5.0.0
    FileDescription:  PS UI Replacement Sample
    LegalCopyright:   Copyright © 1998 - 2009 Microsoft Corp.
    LegalTrademarks:  Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
    Comments:         Written by Windows Printing & Imaging Team

Having never seen ModuleZ in Microsoft module lists and suspecting the word “Sample” in a file and product description we did Internet search and found the module name on various “DLL fixing” websites but still pointing to Microsoft in module description. However, in a full module list (lmt WinDbg command) we found more modules having Module* name structure:

0:000> lmv m ModuleC
start             end                 module name
00000000`10000000 00000000`100b7000   ModuleC   (deferred)            
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleC.DLL
    Image name: ModuleC.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     20.1.0.0
    Product version:  20.1.0.0
    File flags:       0 (Mask 17)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      
    ProductName:      Printer Driver
    InternalName:     MC.dll
    OriginalFilename: MC.dll
    ProductVersion:   20.1.0.0
    FileVersion:      20.1.0.0
    FileDescription:  Printer Driver
    LegalCopyright:   

0:000> lmv m ModuleO
start             end                 module name
00000000`6f280000 00000000`6f2e2000   ModuleO   (deferred)            
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleO.DLL
    Image name: ModuleO.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     2.4.0.0
    Product version:  2.4.0.0
    File flags:       8 (Mask 3F) Private
    File OS:          40004 NT Win32
    File type:        3.1 Driver
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      CompanyA
    ProductName:      CompanyA Printer driver
    InternalName:     ModuleO.dll
    OriginalFilename: ModuleO.dll
    ProductVersion:   2.4
    FileVersion:      2.4.0.0
    FileDescription:  CompanyA Printer driver
    LegalCopyright:   Copyright © CompanyA
    Comments:        

We see that both module names and time stamps follow the same pattern so our “Microsoft” ModuleZ is definitely from CompanyA instead. We also check more detailed information:

0:000> !lmi 00000000`50000000
Loaded Module Info: [00000000`50000000]
         Module: ModuleZ
[...]
               Pdb: N:\ServerQ\[…]
[…]

0:000> !lmi 00000000`10000000
Loaded Module Info: [00000000`10000000]
         Module: ModuleC
[...]
               Pdb: N:\ServerQ\[…]
[…]

0:000> !lmi 00000000`6f280000
Loaded Module Info: [00000000`6f280000]
         Module: ModuleO
[...]
               Pdb: N:\ServerQ\[…]
[…]

All three modules have the same build server in their PDB file name path. We advised to contact CompanyA for updates.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After firing 3 bugtations in a row noticed that I lost all followers

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A Momentary Lapse of Computation.

Pink Floyd, A Momentary Lapse of Reason

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first bugtation where the source book title and the chapter number and name were bugtated too:

The engineer who has no tincture for memory dump analysis goes through life cycle imprisoned in the prejudices derived from coding… (The Problems of Computation. Chapter 0x5: The Value of Memory Dump Analysis)

Bertrand Russell, The Problems of Philosophy, Chapter XV: The Value of Philosophy

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Yet another variation on Bugtation No.98 theme:

I am because we memory dump and because we memory dump therefore I am.

African proverb

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was thinking for some time about applying crash dump analysis patterns (later including software trace analysis patterns and more recently structural memory patterns) to History (one of my favourite study subjects) using metaphorical bijectionism as I tried before with the analysis of project failures. Yesterday I found this book that applies the perspective of patterns in natural sciences to History (according to its description):

Pattern and Repertoire in History

I plan to review the book and highlight the differences and similarities between the authors’ and mine patternist approaches to History.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The next pattern is called Region Boundary. It is an inaccessible range of memory that surrounds Closed Memory Region. For example, the closed region of a kernel stack for the following thread has a one page boundary region next to its Base:

1: kd> !thread
THREAD fffffa8004544b60  Cid 0a6c.0acc  Teb: 000007fffffde000 Win32Thread: fffff900c1eb4010 RUNNING on processor 1
IRP List:
    fffffa8004d7e010: (0006,0118) Flags: 00060000  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a001e84c00
Owning Process            fffffa8004f68370       Image:         NotMyfault.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      40290          Ticks: 0
Context Switch Count      408                 LargeStack
UserTime                  00:00:00.015
KernelTime                00:00:00.015
Win32 Start Address NotMyfault (0x0000000140002708)
Stack Init fffff8800587cdb0 Current fffff8800587c6f0
Base fffff8800587d000 Limit fffff88005874000 Call 0
[…]

1: kd> !pte fffff880`0587d000
                                           VA fffff8800587d000
PXE at FFFFF6FB7DBEDF88    PPE at FFFFF6FB7DBF1000    PDE at FFFFF6FB7E200160    PTE at FFFFF6FC4002C3E8
contains 0000000133D84863  contains 0000000133D83863  contains 00000000AA9AE863  contains 0000000000000000
pfn 133d84    ---DA--KWEV  pfn 133d83    ---DA--KWEV  pfn aa9ae     ---DA--KWEV  not valid

1: kd> !pte fffff880`0587d000+0n4096-1
  VA fffff8800587dfff
PXE at FFFFF6FB7DBEDF88 PPE at FFFFF6FB7DBF1000 PDE at FFFFF6FB7E200160 PTE at FFFFF6FC4002C3E8
contains 0000000133D84863 contains 0000000133D83863 contains 00000000AA9AE863 contains 0000000000000000
pfn 133d84 —DA–KWEV pfn 133d83 —DA–KWEV pfn aa9ae —DA–KWEV not valid

1: kd> !pte fffff880`0587d000+0n4096
                                           VA fffff8800587e000
PXE at FFFFF6FB7DBEDF88    PPE at FFFFF6FB7DBF1000    PDE at FFFFF6FB7E200160    PTE at FFFFF6FC4002C3F0
contains 0000000133D84863  contains 0000000133D83863  contains 00000000AA9AE863  contains 8000000028230963
pfn 133d84    —DA–KWEV  pfn 133d83    —DA–KWEV  pfn aa9ae     —DA–KWEV  pfn 28230     -G-DA–KW-V

The region after boundary belongs to another process thread kernel stack (I use CodeMachine WinDbg extension here):

1: kd> !cmkd.kvas  fffff8800587e000
kvas : Show region containing fffff8800587e000
### Start            End                                  Length Type   
006 fffff88000000000 fffff89fffffffff       2000000000 ( 128 GB) SystemPTEs
Thread FFFFFA80053D0B60 [0798.0a5c] Stack fffff88005884000 - fffff8800587e000

1: kd> !thread FFFFFA80053D0B60
THREAD fffffa80053d0b60  Cid 0798.0a5c  Teb: 000007fffffdd000 Win32Thread: 0000000000000000 WAIT: (DelayExecution) UserMode Non-Alertable
    fffffa80053d0f20  Semaphore Limit 0x1
Not impersonating
DeviceMap                 fffff8a001e84c00
Owning Process            fffffa80053ce630       Image:         ApplicationB.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      34911          Ticks: 5379 (0:00:01:23.912)
Context Switch Count      22            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x000000013fe01538
Stack Init fffff88005883db0 Current fffff88005883970
Base fffff88005884000 Limit fffff8800587e000 Call 0
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Now we propose the next group of general patterns related to memory regions (the terminology was partially influenced from topology). The first one we call Memory Region, for example:

1: kd> dp fffff88005875000 fffff88005875010
fffff880`05875000  039b5000`6e696268 00000000`00002000
fffff880`05875010  00000000`00000000

There are Open and Closed memory regions. We can extend the former ones in one or both directions:

1: kd> dp fffff88005875000-10 fffff88005875010+10
fffff880`05874ff0  6ab90c4f`039ba1b8 039b8050`fffffff8
fffff880`05875000  039b5000`6e696268 00000000`00002000
fffff880`05875010  00000000`00000000 00000000`00000000
fffff880`05875020  00000294`ffffffd0

The closed regions cannot be read past its boundary, like this kernel stack region [fffff880`05874000 fffff8800587d000):

1: kd> dp fffff88005874000-30
fffff880`05873fd0  ????????`???????? ????????`????????
fffff880`05873fe0  ????????`???????? ????????`????????
fffff880`05873ff0  ????????`???????? ????????`????????
fffff880`05874000  039ba000`6e696268 00000000`00001000
fffff880`05874010  00000000`00000000 00000000`00000000
fffff880`05874020  00206b6e`ffffffa8 01cae7bd`b8aca323
fffff880`05874030  039b6698`00000000 00000000`00000001
fffff880`05874040  ffffffff`039bafe8 039b6710`00000004

1: kd> dp fffff8800587d000-30
fffff880`0587cfd0  00000000`00000000 00000000`00000000
fffff880`0587cfe0  00000000`00000000 00000000`00000000
fffff880`0587cff0  00000000`00000000 00000000`00000000
fffff880`0587d000  ????????`???????? ????????`????????
fffff880`0587d010  ????????`???????? ????????`????????
fffff880`0587d020  ????????`???????? ????????`????????
fffff880`0587d030  ????????`???????? ????????`????????
fffff880`0587d040  ????????`???????? ????????`????????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Another tool that supports adjoint threading in addition to Citrix CDFAnalyzer (see also Debugging Experts magazine article for pictorial description of this concept) is Process Monitor. We can view adjoint threads having common attributes like TID (ordinary threads), PID, operation (function), process name, etc. by using this right click context menu:

For example, this adjoint thread having RegOpenKey as its ATID (Adjoint Thread ID) where we excluded Path, Result and Detail fields for viewing clarity (together these fields can constitute an analogous Message field in TMF traces):

Time of Day      Process Name PID  TID  Operation
[…]
09:33:25.9545410 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9548650 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9550234 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9551656 Explorer.EXE 1292 1032 RegOpenKey
09:33:25.9692456 WFICA32.EXE  3588 3496 RegOpenKey
09:33:25.9761325 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9761912 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9762295 wfcrun32.exe 852  1148 RegOpenKey
09:33:25.9984547 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0023831 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0074675 wfcrun32.exe 852  1148 RegOpenKey
09:33:26.0087191 Explorer.EXE 1292 1032 RegOpenKey
09:33:26.1618595 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1625697 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1632745 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1633924 iexplore.exe 1348 2228 RegOpenKey
09:33:26.1639209 iexplore.exe 1348 2228 RegOpenKey
[…]

So if someone writes a converter from TMF to PML format…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -