Crash Dump Analysis Patterns (Part 108)

It is a common technique in driver development to copy/paste an entire driver sample from WDK and modify it for a specific device or a filter value-adding functionality in a specific driver stack or framework. The problem here is that developers sometimes forget to change module resources and certain amount of detective work is required to find out the module vendor. I call this pattern Template Module. Here is an example. In a spooler service there were many threads blocked in displaying a dialog box:

0:000> ~34kL 100
Child-SP          RetAddr           Call Site
00000000`08a49368 00000000`774c5118 user32!NtUserWaitMessage+0xa
00000000`08a49370 00000000`774c5770 user32!DialogBox2+0x261
00000000`08a493f0 00000000`774c57e6 user32!InternalDialogBox+0x134
00000000`08a49450 00000000`774c3e36 user32!DialogBoxIndirectParamAorW+0x58
00000000`08a49490 000007fe`fa27cc97 user32!DialogBoxParamW+0x66
00000000`08a494d0 000007fe`fa28832b unidrvui!ICheckConstraintsDlg+0xbf
00000000`08a49950 000007fe`fa29423d unidrvui!BUpdateUISettingForOEM+0x2f
00000000`08a49980 00000000`50036d2c unidrvui!CPrintOemDriverUI::DrvUpdateUISetting+0x1d
00000000`08a499b0 00000000`50038a1d ModuleZ!DllGetClassObject+0×1fe74
[…]
00000000`08a4b250 000007fe`f759546b unidrvui!OEMDevicePropertySheets+0×56
00000000`08a4b280 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4b330 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4b5f0 000007fe`fa2880e9 compstui!CPSUICallBack+0×3ec
00000000`08a4b6a0 000007fe`fa2836c4 unidrvui!BAddOemPluginPages+0×12d
00000000`08a4b6d0 000007fe`f759546b unidrvui!DrvDevicePropertySheets+0×2c8
00000000`08a4bb60 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4bc10 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4bed0 000007fe`fb452838 compstui!CPSUICallBack+0×3ec
00000000`08a4bf80 000007fe`f759546b winspool!DevicePropertySheets+0×108
00000000`08a4bfb0 000007fe`f759653e compstui!CallpfnPSUI+0×137
00000000`08a4c060 000007fe`f7596b84 compstui!InsertPSUIPage+0×24a
00000000`08a4c320 000007fe`f759758e compstui!CPSUICallBack+0×3ec
00000000`08a4c3d0 000007fe`f75976b2 compstui!DoCommonPropertySheetUI+0xbe
00000000`08a4c430 000007fe`fb446339 compstui!CommonPropertySheetUIW+0xe
00000000`08a4c470 000007fe`fb44b425 winspool!CallCommonPropertySheetUI+0×65
00000000`08a4c4c0 00000000`5003623c winspool!PrinterPropertiesNative+0×121
00000000`08a4c950 00000000`50035d16 ModuleZ!DllGetClassObject+0×1f384
[…]
00000000`08a4dd70 000007fe`fb4472d8 unidrvui!DrvPrinterEvent+0×419
00000000`08a4de00 000007fe`fb44737f winspool!SpoolerPrinterEventNative+0×84
00000000`08a4de60 000007fe`faedc957 winspool!SpoolerPrinterEvent+0×13
00000000`08a4dea0 000007fe`faedc8c7 localspl!SplDriverEvent+0×4f
00000000`08a4def0 000007fe`faec3d74 localspl!PrinterDriverEvent+0xcf
00000000`08a4df30 000007fe`fa771f20 localspl!SplAddPrinter+0xae0
00000000`08a4e4e0 000007fe`fa7491d8 win32spl!NCSRCommon::TLocalPrinter::AddPrinterW+0xb4
00000000`08a4e5b0 000007fe`fa747511 win32spl!TPrintOpen::AddLocalPrinter+0xb8
00000000`08a4e6b0 000007fe`fa746dfb win32spl!TPrintOpen::AddAndInstallLocalPrinter+0×34d
00000000`08a4e830 000007fe`fa746bb0 win32spl!TPrintOpen::ReEstablishCacheConnectionNoGuidPrinter+0×157
00000000`08a4e900 000007fe`fa7467d1 win32spl!TPrintOpen::ReEstablishCacheConnection+0×178
00000000`08a4e980 000007fe`fa7465c1 win32spl!TPrintOpen::ReEstablishPrinterConnection+0×16d
00000000`08a4ea30 000007fe`fa73e5ad win32spl!TPrintOpen::ReEstablishConnectionFromKey+0×1fd
00000000`08a4eb30 000007fe`fa733492 win32spl!TPrintOpen::RediscoverPrinterConnections+0xd7
00000000`08a4ebe0 000007fe`fb3f2332 win32spl!TPrintProviderTable::forwardEnumPrinters+0×47
00000000`08a4ec70 00000000`ff3414c8 spoolss!EnumPrintersW+0×176
00000000`08a4ed20 00000000`ff3413cc spoolsv!YEnumPrinters+0×112
00000000`08a4eda0 000007fe`fe225ec5 spoolsv!RpcEnumPrinters+0×30
00000000`08a4edf0 000007fe`fe2bebed rpcrt4!Invoke+0×65
00000000`08a4ee70 000007fe`fe1f5df0 rpcrt4!Ndr64StubWorker+0×5a9
00000000`08a4f440 000007fe`fe2268d4 rpcrt4!NdrServerCallAll+0×40
00000000`08a4f490 000007fe`fe2269f0 rpcrt4!DispatchToStubInCNoAvrf+0×14
00000000`08a4f4c0 000007fe`fe227402 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×100
00000000`08a4f5b0 000007fe`fe227080 rpcrt4!LRPC_SCALL::DispatchRequest+0×1c2
00000000`08a4f620 000007fe`fe2262bb rpcrt4!LRPC_SCALL::HandleRequest+0×200
00000000`08a4f740 000007fe`fe225e1a rpcrt4!LRPC_ADDRESS::ProcessIO+0×44a
00000000`08a4f860 000007fe`fe207769 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0×24a
00000000`08a4f910 000007fe`fe207714 rpcrt4!ProcessIOEventsWrapper+0×9
00000000`08a4f940 000007fe`fe2077a4 rpcrt4!BaseCachedThreadRoutine+0×94
00000000`08a4f980 00000000`7758be3d rpcrt4!ThreadStartRoutine+0×24
00000000`08a4f9b0 00000000`776c6a51 kernel32!BaseThreadInitThunk+0xd
00000000`08a4f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We suspect ModuleZ but its module information points to Microsoft: 

0:000> lmv m ModuleZ
start             end                 module name
00000000`50000000 00000000`500a4000   ModuleZ   (export symbols)       ModuleZ.DLL
    Loaded symbol image file: ModuleZ.DLL
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleZ.DLL
    Image name: ModuleZ.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     2.5.0.0
    Product version:  2.5.0.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0407.04b0
    CompanyName:      Microsoft Corp.
    ProductName:      Microsoft PS UI Replacement Sample

    InternalName:     PSUIREP
    OriginalFilename: PSUIREP.dll
    ProductVersion:   2.5
    FileVersion:      2.5.0.0
    FileDescription:  PS UI Replacement Sample
    LegalCopyright:   Copyright © 1998 - 2009 Microsoft Corp.
    LegalTrademarks:  Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation

    Comments:         Written by Windows Printing & Imaging Team

Having never seen ModuleZ in Microsoft module lists and suspecting the word “Sample” in a file and product description we did Internet search and found the module name on various “DLL fixing” websites but still pointing to Microsoft in module description. However, in a full module list (lmt WinDbg command) we found more modules having Module* name structure:

0:000> lmv m ModuleC
start             end                 module name
00000000`10000000 00000000`100b7000   ModuleC   (deferred)            
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleC.DLL
    Image name: ModuleC.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     20.1.0.0
    Product version:  20.1.0.0
    File flags:       0 (Mask 17)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      
    ProductName:      Printer Driver
    InternalName:     MC.dll
    OriginalFilename: MC.dll
    ProductVersion:   20.1.0.0
    FileVersion:      20.1.0.0
    FileDescription:  Printer Driver
    LegalCopyright:
   

0:000> lmv m ModuleO
start             end                 module name
00000000`6f280000 00000000`6f2e2000   ModuleO   (deferred)            
    Image path: C:\Windows\System32\spool\drivers\x64\3\ModuleO.DLL
    Image name: ModuleO.DLL
    Timestamp:        Feb […] 2010
    […]
    File version:     2.4.0.0
    Product version:  2.4.0.0
    File flags:       8 (Mask 3F) Private
    File OS:          40004 NT Win32
    File type:        3.1 Driver
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      CompanyA
    ProductName:      CompanyA Printer driver

    InternalName:     ModuleO.dll
    OriginalFilename: ModuleO.dll
    ProductVersion:   2.4
    FileVersion:      2.4.0.0
    FileDescription:  CompanyA Printer driver
    LegalCopyright:   Copyright © CompanyA

    Comments:
        

We see that both module names and time stamps follow the same pattern so our “Microsoft” ModuleZ is definitely from CompanyA instead. We also check more detailed information:

0:000> !lmi 00000000`50000000
Loaded Module Info: [00000000`50000000]
         Module: ModuleZ
[...]
               Pdb: N:\ServerQ\[…]
[…]

0:000> !lmi 00000000`10000000
Loaded Module Info: [00000000`10000000]
         Module: ModuleC
[...]
               Pdb: N:\ServerQ\[…]
[…]

0:000> !lmi 00000000`6f280000
Loaded Module Info: [00000000`6f280000]
         Module: ModuleO
[...]
               Pdb: N:\ServerQ\[…]
[…]

All three modules have the same build server in their PDB file name path. We advised to contact CompanyA for updates.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One Response to “Crash Dump Analysis Patterns (Part 108)”

  1. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 113) Says:

    […] also check application or service vendor and copyright in the output of lmv command. Similar to Template Module Same Vendor pattern can be useful to relate such different incidents. Usually in the same company […]

Leave a Reply