Software Diagnostics Library » Crash Dump Patterns

Crash Dump Analysis Patterns (Part 199)

Tuesday, May 28th, 2013

Processes with one thread like Notepad are rare. Such a process is always suspicious especially if it is a service or belongs to a complex product. We call such a pattern One-Thread Process. Usually this happens when all other threads terminated and the remaining thread is blocked in some wait chain. For example, this process has a thread which is blocked in an ALPC request to itself (the same process):

0: kd> !process fffffa8013ed9b30 ff PROCESS fffffa8013ed9b30 SessionId: 0 Cid: 44b4 Peb: 7fffffd8000 ParentCid: 0114 DirBase: 2da448000 ObjectTable: fffff8a01948c670 HandleCount: 660. Image: ServiceA.exe VadRoot fffffa801356dd10 Vads 398 Clone 0 Private 5795. Modified 204253. Locked 0. DeviceMap fffff8a000008340 Token fffff8a01b546060 ElapsedTime 01:32:37.622 UserTime 00:00:01.421 KernelTime 00:00:01.578 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (1525, 50, 345) (6100KB, 200KB, 1380KB) PeakWorkingSetSize 7607 VirtualSize 178 Mb PeakVirtualSize 182 Mb PageFaultCount 752709 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 8043

THREAD fffffa8012caab50 Cid 44b4.4f70 Teb: 000007fffff5a000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) KernelMode Non-Alertable fffffa8012caaf18 Semaphore Limit 0x1 Waiting for reply to ALPC Message fffff8a0194d4780 : queued at port fffffa8012911c80 : owned by process fffffa8013ed9b30 IRP List: fffffa8013923300: (0006,0118) Flags: 00060000 Mdl: 00000000 Not impersonating DeviceMap fffff8a000008340 Owning Process fffffa8013ed9b30 Image: ServiceA.exe Attached Process N/A Image: N/A Wait Start TickCount 139828 Ticks: 347372 (0:01:30:27.687) Context Switch Count 7380 UserTime 00:00:00.031 KernelTime 00:00:04.890 Win32 Start Address ServiceA (0×00000001401156e0) Stack Init fffff88014c9ddb0 Current fffff88014c9c6b0 Base fffff88014c9e000 Limit fffff88014c98000 Call 0 Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`14c9c6f0 fffff800`01873652 nt!KiSwapContext+0×7a fffff880`14c9c830 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2 fffff880`14c9c8c0 fffff800`0189f04f nt!KeWaitForSingleObject+0×19f fffff880`14c9c960 fffff800`01b919f6 nt!AlpcpSignalAndWait+0×8f fffff880`14c9ca10 fffff800`01b910f0 nt!AlpcpReceiveSynchronousReply+0×46 fffff880`14c9ca70 fffff800`01b9519d nt!AlpcpProcessSynchronousRequest+0×33d fffff880`14c9cbb0 fffff800`01b95276 nt!LpcpRequestWaitReplyPort+0×9c fffff880`14c9cc10 fffff800`0187ced3 nt!NtRequestWaitReplyPort+0×76 fffff880`14c9cc60 fffff800`01879490 nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9cc60) fffff880`14c9cdf8 fffff880`05c31050 nt!KiServiceLinkage fffff880`14c9ce70 fffff880`045ce005 ModuleA+0×12468 […] fffff880`14c9da10 fffff800`01b9d3b6 nt!IopXxxControlFile+0×607 fffff880`14c9db40 fffff800`0187ced3 nt!NtDeviceIoControlFile+0×56 fffff880`14c9dbb0 00000000`76d8138a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9dc20) 00000000`082af028 000007fe`fd366cf6 ntdll!NtDeviceIoControlFile+0xa 00000000`082af030 00000000`76c2683f KERNELBASE!TlsGetValue+0×1a36 00000000`082af0a0 00000001`4019d38c kernel32!DeviceIoControlImplementation+0×7f […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 198)

Sunday, May 26th, 2013

All previous wait chain patterns were about single wait chains. However, it is often a case when there are many different wait chains in a memory dump especially in terminal services environments. There can be ALPC and critical section wait chains at the same time. The can be related or completely disjoint. Here we call a special case of several wait chains having the same structure (and possibly pointing to one direction) Distributed Wait Chain. One such example we put below. In a stack trace collection from a complete memory dump from a hanging system we found several explorer.exe processes with critical section wait chains having the same structure and endpoint of a top and blocking ModuleA:

THREAD fffffa80137cf060 Cid 4884.4f9c Teb: 000007fffffaa000 Win32Thread: fffff900c0fb98b0 WAIT: (UserRequest) UserMode Non-Alertable fffffa8013570dc0 SynchronizationEvent Not impersonating DeviceMap fffff8a014e21d90 Owning Process fffffa80131a75d0 Image: explorer.exe Attached Process N/A Image: N/A Wait Start TickCount 274752 Ticks: 212448 (0:00:55:19.500) Context Switch Count 9889 LargeStack UserTime 00:00:00.093 KernelTime 00:00:00.171 Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefdafc608) Stack Init fffff88013c25db0 Current fffff88013c25900 Base fffff88013c26000 Limit fffff88013c1b000 Call 0 Priority 11 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Kernel stack not resident. Child-SP RetAddr Call Site fffff880`13c25940 fffff800`01873652 nt!KiSwapContext+0×7a fffff880`13c25a80 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2 fffff880`13c25b10 fffff800`01b7768e nt!KeWaitForSingleObject+0×19f fffff880`13c25bb0 fffff800`0187ced3 nt!NtWaitForSingleObject+0xde fffff880`13c25c20 00000000`76d8135a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`13c25c20) 00000000`0489e518 00000000`76d7e4e8 ntdll!ZwWaitForSingleObject+0xa 00000000`0489e520 00000000`76d7e3db ntdll!RtlpWaitOnCriticalSection+0xe8 00000000`0489e5d0 000007fe`fdf8ff50 ntdll!RtlEnterCriticalSection+0xd1 00000000`0489e600 000007fe`fdf8fbd3 SHELL32!CFSFolder::GetIconOf+0×24b 00000000`0489f3a0 000007fe`fdf903d3 SHELL32!SHGetIconIndexFromPIDL+0×3f 00000000`0489f3d0 00000000`ff900328 SHELL32!SHMapIDListToSystemImageListIndexAsync+0×73 00000000`0489f470 00000000`ff8fff4b Explorer!SFTBarHost::AddImageForItem+0×9c 00000000`0489f4d0 00000000`ff8fd2f1 Explorer!SFTBarHost::_InternalRepopulateList+0×4ad 00000000`0489f5d0 00000000`ff8fd0b4 Explorer!SFTBarHost::_RepopulateList+0×1f3 00000000`0489f600 00000000`ff8fcccd Explorer!SFTBarHost::_OnBackgroundEnumDone+0xc1 00000000`0489f630 00000000`ff8fc9e2 Explorer!SFTBarHost::_WndProc+0×451 00000000`0489f680 00000000`76669bd1 Explorer!SFTBarHost::_WndProc_ProgramsMFU+0×1b 00000000`0489f6b0 00000000`766698da USER32!UserCallWinProcCheckWow+0×1ad 00000000`0489f770 00000000`ff8f1177 USER32!DispatchMessageWorker+0×3b5 00000000`0489f7f0 00000000`ff9130e9 Explorer!CTray::_MessageLoop+0×446 00000000`0489f880 000007fe`fdafc71e Explorer!CTray::MainThreadProc+0×8a 00000000`0489f8b0 00000000`76c2652d SHLWAPI!WrapperThreadProc+0×19b 00000000`0489f9b0 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd 00000000`0489f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .process /r /p fffffa80131a75d0 Implicit process is now fffffa80`131a75d0 Loading User Symbols

0: kd> !cs -l -o -s ----------------------------------------- DebugInfo = 0x0000000000499d90 Critical section = 0x000007fefe3d5900 (SHELL32!g_csIconCache+0x0) LOCKED LockCount = 0×2 WaiterWoken = No OwningThread = 0×0000000000002b34 RecursionCount = 0×1 LockSemaphore = 0×7F8 SpinCount = 0×0000000000000000 OwningThread = .thread fffffa8013dc3b00

THREAD fffffa8013dc3b00 Cid 4884.2b34 Teb: 000007fffffac000 Win32Thread: fffff900c2bc1010 WAIT: (Executive) KernelMode Non-Alertable fffff88011c03600 SynchronizationEvent IRP List: fffffa800f8fc790: (0006,0430) Flags: 00000404 Mdl: 00000000 Not impersonating DeviceMap fffff8a014e21d90 Owning Process fffffa80131a75d0 Image: explorer.exe Attached Process N/A Image: N/A Wait Start TickCount 170052 Ticks: 317148 (0:01:22:35.437) Context Switch Count 2 LargeStack UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address SHELL32!ShutdownThreadProc (0x000007fefe13ef54) Stack Init fffff88011c03db0 Current fffff88011c03320 Base fffff88011c04000 Limit fffff88011bfd000 Call 0 Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`11c03360 fffff800`01873652 nt!KiSwapContext+0x7a fffff880`11c034a0 fffff800`01884a9f nt!KiCommitThreadWait+0x1d2 fffff880`11c03530 fffff880`05c12383 nt!KeWaitForSingleObject+0x19f fffff880`11c035d0 fffff880`012b9288 ModuleA+0×12468 fffff880`11c03750 fffff880`012b7d1b fltmgr!FltpPerformPostCallbacks+0×368 fffff880`11c03820 fffff880`012b66df fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×39b fffff880`11c038b0 fffff880`01b895ff fltmgr!FltpDispatch+0xcf fffff880`11c03a30 fffff800`01b783b4 nt!IopCloseFile+0×11f fffff880`11c03ac0 fffff800`01b78171 nt!ObpDecrementHandleCount+0xb4 fffff880`11c03b40 fffff800`01b78734 nt!ObpCloseHandleTableEntry+0xb1 fffff880`11c03bd0 fffff800`0187ced3 nt!ObpCloseHandle+0×94 fffff880`11c03c20 00000000`76d8140a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`11c03c20) 00000000`0754f348 000007fe`fd341873 ntdll!NtClose+0xa 00000000`0754f350 00000000`76c32f51 KERNELBASE!CloseHandle+0×13 00000000`0754f380 000007fe`fdaf9690 kernel32!CloseHandleImplementation+0×3d 00000000`0754f490 000007fe`fe191d7f SHLWAPI!CFileStream::Release+0×84 00000000`0754f4c0 000007fe`fe13ed57 SHELL32!IconCacheSave+0×2b7 00000000`0754f780 000007fe`fe13f0c6 SHELL32!CommonRestart+0×2f 00000000`0754f7f0 00000000`76c2652d SHELL32!ShutdownThreadProc+0×172 00000000`0754f820 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd 00000000`0754f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Memory Dump Analysis Best Practices (Part 3)

Wednesday, May 8th, 2013

Another best practice that is directly related to productivity is a parallel processing of the same memory dump especially in the case of complete memory dumps. Here an analysis might start with running time consuming scripts that dump all process and threads in the variety of formats such as x64 and x86 thread stack traces. However, if the nature of the problem is such that it is possible to start with some pattern and continue unfolding its analysis then we can do that in parallel. One of examples may be a discovered Incomplete Session with an ALPC Wait Chain. Here we can follow such a wait chain while another WinDbg instance dumps all threads for further pattern search later.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Best Practices, Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology | No Comments »

Memory Dump Analysis Best Practices (Part 2)

Tuesday, May 7th, 2013

We continue with best practices, the previous was SCP. The second best practice is to check the system for additional patterns after the main pattern was found (similar to avoiding Common Mistake 8). For example, in the case of a bug check resulted from NULL pointer dereference or any other exception in some 3rd-party driver code don’t stop but look at all CPUs, processes and threads to find any other patterns such as Spiking Threads, Busy System, and Contention. Inspection of associated thread stack traces might reveal the same module and/or give additional clues to system behaviour prior to the fault.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Best Practices, Crash Dump Analysis, Crash Dump Patterns, Debugging, Debugging Methodology | No Comments »

Generalized Software Narrative and Trace

Monday, March 25th, 2013

In the past we viewed software traces and logs as temporarily ordered event sequences. Since events are just memory data we have a map

T -> M

as can be seen in the definition of a software trace. Here we generalize the domain to any arbitrary set, for example, it can be a list of indexes or pointers or even memory itself. The latter map can give us narrative chains such as

M -> M -> M -> M

and even give us a grand unification of memory and log analysis and the possibility to apply software narratology to memory dump analysis as well. We talk about it soon and provide some generalized software narrative examples.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Software Narrative Science, Software Narratology, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 197)

Wednesday, February 27th, 2013

Injected Symbols pattern can be used to add missing symbols when we have Reduced Symbol Information like it was done previously in this old case study. For example, TestWER module was compiled with static MFC and CRT libraries and its private PDB file contains all necessary symbols including MSG structure. We can load that module into notepad.exe process space and apply symbols:

0:000:x86> lm start end module name 00fc0000 00ff0000 notepad (pdb symbols) c:\mss\notepad.pdb\E325F5195AE94FAEB58D25C9DF8C0CFD2\notepad.pdb 10000000 10039000 WinCRT (deferred) 727f0000 7298e000 comctl32 (deferred) 72aa0000 72af1000 winspool (deferred) 72b10000 72b19000 version (deferred) 72e40000 72e48000 wow64cpu (deferred) 72e50000 72eac000 wow64win (pdb symbols) c:\mss\wow64win.pdb\B2D08CC152D64E71B79167DC0A0A53E91\wow64win.pdb 72eb0000 72eef000 wow64 (deferred) 733d0000 733e3000 dwmapi (deferred) 735b0000 73606000 uxtheme (deferred) 746f0000 746fc000 CRYPTBASE (deferred) 74700000 74760000 sspicli (deferred) 747c0000 74817000 shlwapi (deferred) 74830000 7547a000 shell32 (deferred) 755d0000 7564b000 comdlg32 (deferred) 75650000 7567e000 imm32 (deferred) 75770000 75810000 advapi32 (deferred) 75810000 75920000 kernel32 (pdb symbols) c:\mss\wkernel32.pdb\1C690A8592304467BB15A09CEA7180FA2\wkernel32.pdb 75920000 759b0000 gdi32 (deferred) 759b0000 759f7000 KERNELBASE (deferred) 75a00000 75b00000 user32 (pdb symbols) c:\mss\wuser32.pdb\0FCE9CC301ED4567A819705B2718E1D62\wuser32.pdb 75b00000 75b8f000 oleaut32 (deferred) 75be0000 75c7d000 usp10 (deferred) 75ff0000 76009000 sechost (deferred) 76010000 76100000 rpcrt4 (deferred) 76230000 762dc000 msvcrt (deferred) 76470000 7647a000 lpk (deferred) 76480000 7654c000 msctf (deferred) 76550000 766ac000 ole32 (deferred) 766d0000 76753000 clbcatq (deferred) 76e40000 76fe9000 ntdll (deferred) 77020000 771a0000 ntdll_77020000 (pdb symbols) c:\mss\wntdll.pdb\D74F79EB1F8D4A45ABCD2F476CCABACC2\wntdll.pdb

0:000:x86> .sympath+ C:\DebuggingTV\TestWER\x86 Symbol search path is: srv*;C:\DebuggingTV\TestWER\x86 Expanded Symbol search path is: SRV*c:\mss*http://msdl.microsoft.com/download/symbols;c:\debuggingtv\testwer\x86

0:000:x86> .reload /f /i C:\DebuggingTV\TestWER\x86\TestWER.exe=10000000

0:000:x86> lm start end module name 00fc0000 00ff0000 notepad (pdb symbols) c:\mss\notepad.pdb\E325F5195AE94FAEB58D25C9DF8C0CFD2\notepad.pdb 10000000 10039000 TestWER (private pdb symbols) c:\debuggingtv\testwer\x86\TestWER.pdb 727f0000 7298e000 comctl32 (deferred) 72aa0000 72af1000 winspool (deferred) 72b10000 72b19000 version (deferred) 72e40000 72e48000 wow64cpu (deferred) 72e50000 72eac000 wow64win (pdb symbols) c:\mss\wow64win.pdb\B2D08CC152D64E71B79167DC0A0A53E91\wow64win.pdb 72eb0000 72eef000 wow64 (deferred) 733d0000 733e3000 dwmapi (deferred) 735b0000 73606000 uxtheme (deferred) 746f0000 746fc000 CRYPTBASE (deferred) 74700000 74760000 sspicli (deferred) 747c0000 74817000 shlwapi (deferred) 74830000 7547a000 shell32 (deferred) 755d0000 7564b000 comdlg32 (deferred) 75650000 7567e000 imm32 (deferred) 75770000 75810000 advapi32 (deferred) 75810000 75920000 kernel32 (pdb symbols) c:\mss\wkernel32.pdb\1C690A8592304467BB15A09CEA7180FA2\wkernel32.pdb 75920000 759b0000 gdi32 (deferred) 759b0000 759f7000 KERNELBASE (deferred) 75a00000 75b00000 user32 (pdb symbols) c:\mss\wuser32.pdb\0FCE9CC301ED4567A819705B2718E1D62\wuser32.pdb 75b00000 75b8f000 oleaut32 (deferred) 75be0000 75c7d000 usp10 (deferred) 75ff0000 76009000 sechost (deferred) 76010000 76100000 rpcrt4 (deferred) 76230000 762dc000 msvcrt (deferred) 76470000 7647a000 lpk (deferred) 76480000 7654c000 msctf (deferred) 76550000 766ac000 ole32 (deferred) 766d0000 76753000 clbcatq (deferred) 76e40000 76fe9000 ntdll (deferred) 77020000 771a0000 ntdll_77020000 (pdb symbols) c:\mss\wntdll.pdb\D74F79EB1F8D4A45ABCD2F476CCABACC2\wntdll.pdb

0:000:x86> kv ChildEBP RetAddr Args to Child 0013fe34 75a1790d 0013fe74 00000000 00000000 user32!NtUserGetMessage+0x15 0013fe50 00fc148a 0013fe74 00000000 00000000 user32!GetMessageW+0×33 0013fe90 00fc16ec 00fc0000 00000000 00354082 notepad!WinMain+0xe6 0013ff20 758233aa 7efde000 0013ff6c 77059ef2 notepad!_initterm_e+0×1a1 0013ff2c 77059ef2 7efde000 57785ae5 00000000 kernel32!BaseThreadInitThunk+0xe 0013ff6c 77059ec5 00fc3689 7efde000 00000000 ntdll_77020000!__RtlUserThreadStart+0×70 0013ff84 00000000 00fc3689 7efde000 00000000 ntdll_77020000!_RtlUserThreadStart+0×1b

0:000:x86> dt -r MSG 0013fe74
TestWER!MSG
  +0x000 hwnd             : 0x0007149c HWND__
    +0x000 unused           : ??
  +0×004 message          : 0×113
  +0×008 wParam           : 0×38a508
  +0×00c lParam           : 0n1921500630
  +0×010 time             : 0×2079a177
  +0×014 pt               : tagPOINT
    +0×000 x                : 0n1337
    +0×004 y                : 0n448

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 196)

Tuesday, February 26th, 2013

Sometimes we have Reduced Symbolic Information for modules which can range from stripped or public symbol files to exported only function names. In such cases we can use API function prototypes, structure definitions and possible String Parameters to make sense of function arguments:

0:000:x86> kv ChildEBP RetAddr Args to Child 0013fe34 75a1790d 0013fe74 00000000 00000000 user32!NtUserGetMessage+0x15 0013fe50 00fc148a 0013fe74 00000000 00000000 user32!GetMessageW+0×33 0013fe90 00fc16ec 00fc0000 00000000 00354082 notepad!WinMain+0xe6 0013ff20 758233aa 7efde000 0013ff6c 77059ef2 notepad!_initterm_e+0×1a1 0013ff2c 77059ef2 7efde000 57785ae5 00000000 kernel32!BaseThreadInitThunk+0xe 0013ff6c 77059ec5 00fc3689 7efde000 00000000 ntdll_77020000!__RtlUserThreadStart+0×70 0013ff84 00000000 00fc3689 7efde000 00000000 ntdll_77020000!_RtlUserThreadStart+0×1b

The first parameter of GetMessage API is a pointer to MSG structure:

0:000:x86> dt MSG 0013fe74 Symbol MSG not found.

From MSDN we find this structure definition:

typedef struct tagMSG {
    HWND   hwnd;
    UINT   message;
    WPARAM wParam;
    LPARAM lParam;
    DWORD  time;
    POINT  pt;
} MSG, *PMSG, *LPMSG;

0:000:x86> dc 0013fe74 L7 0013fe74 0007149c 00000113 0038a508 7287c5d6 ……….8….r 0013fe84 2079a177 00000539 000001c0 w.y 9…….

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 195)

Tuesday, February 26th, 2013

Sometimes we have a Truncated Stack Trace and need to perform manual stack trace reconstruction of the missing part to get approximate full stack trace. Often we are only able to reconstruct some parts and glue them together perhaps with some missing intermediate frames:

For example, we have this truncated stack trace due to the lack of symbols:

1: kd> k ChildEBP RetAddr 97543b6c 85adf579 nt!KiTrap0E+0x2ac WARNING: Stack unwind information not available. Following frames may be wrong. 97543be8 85adf770 myfault+0x579 97543bf4 85adf7fc myfault+0x770 97543c2c 81827ecf myfault+0x7fc 97543c44 81988f65 nt!IofCallDriver+0x63 97543c64 81989f25 nt!IopSynchronousServiceTail+0x1e0 97543d00 8198ee8d nt!IopXxxControlFile+0x6b7 97543d34 8188c96a nt!NtDeviceIoControlFile+0x2a 97543d34 77510f34 nt!KiFastCallEntry+0x12a 0012f9a0 7750f850 ntdll!KiFastSystemCallRet 0012f9a4 77417c92 ntdll!NtDeviceIoControlFile+0xc 0012fa04 00401a5b kernel32!DeviceIoControl+0x14a 0012fa94 7700becf NotMyfault+0x1a5b 0012facc 00000000 USER32!xxxDrawButton+0xc1

Manual stack reconstruction brings this fragment:

1: kd> k L=0012fb94 0012fb94 0012fb94 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 0012fb94 77001ae8 0x12fb94 0012fc0c 7700286a USER32!UserCallWinProcCheckWow+0x14b 0012fc4c 77002bba USER32!SendMessageWorker+0x4b7 0012fc6c 7700c6b4 USER32!SendMessageW+0x7c 0012fc84 7700c7c9 USER32!xxxButtonNotifyParent+0x41 0012fca0 7700c7e8 USER32!xxxBNReleaseCapture+0xf7 0012fd24 7701632e USER32!ButtonWndProcWorker+0x910 0012fd44 77001a10 USER32!ButtonWndProcA+0x4c 0012fd70 77001ae8 USER32!InternalCallWinProc+0x23 0012fde8 77002a47 USER32!UserCallWinProcCheckWow+0x14b 0012fe4c 77002a98 USER32!DispatchMessageWorker+0x322 0012fe5c 76ff11fc USER32!DispatchMessageW+0xf 0012fe80 76fe98d2 USER32!IsDialogMessageW+0x586 0012fea0 00401cc9 USER32!IsDialogMessageA+0xff 0012ff10 004022ec NotMyfault+0x1cc9 00000000 00000000 NotMyfault+0x22ec

And finally we get the 3rd usual thread start fragment:

1: kd> k L=0012ffa0 0012ffa0 0012ffa0 ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 0012ffa0 77413833 0x12ffa0 0012ffac 774ea9bd kernel32!BaseThreadInitThunk+0xe 0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

Gluing them together we get this approx. stack trace:

97543b6c 85adf579 nt!KiTrap0E+0x2ac WARNING: Stack unwind information not available. Following frames may be wrong. 97543be8 85adf770 myfault+0x579 97543bf4 85adf7fc myfault+0x770 97543c2c 81827ecf myfault+0x7fc 97543c44 81988f65 nt!IofCallDriver+0x63 97543c64 81989f25 nt!IopSynchronousServiceTail+0x1e0 97543d00 8198ee8d nt!IopXxxControlFile+0x6b7 97543d34 8188c96a nt!NtDeviceIoControlFile+0x2a 97543d34 77510f34 nt!KiFastCallEntry+0x12a 0012f9a0 7750f850 ntdll!KiFastSystemCallRet 0012f9a4 77417c92 ntdll!NtDeviceIoControlFile+0xc 0012fa04 00401a5b kernel32!DeviceIoControl+0x14a 0012fa94 7700becf NotMyfault+0x1a5b 0012fc0c 7700286a USER32!UserCallWinProcCheckWow+0x14b 0012fc4c 77002bba USER32!SendMessageWorker+0x4b7 0012fc6c 7700c6b4 USER32!SendMessageW+0x7c 0012fc84 7700c7c9 USER32!xxxButtonNotifyParent+0x41 0012fca0 7700c7e8 USER32!xxxBNReleaseCapture+0xf7 0012fd24 7701632e USER32!ButtonWndProcWorker+0x910 0012fd44 77001a10 USER32!ButtonWndProcA+0x4c 0012fd70 77001ae8 USER32!InternalCallWinProc+0x23 0012fde8 77002a47 USER32!UserCallWinProcCheckWow+0x14b 0012fe4c 77002a98 USER32!DispatchMessageWorker+0x322 0012fe5c 76ff11fc USER32!DispatchMessageW+0xf 0012fe80 76fe98d2 USER32!IsDialogMessageW+0x586 0012fea0 00401cc9 USER32!IsDialogMessageA+0xff 0012ff10 004022ec NotMyfault+0x1cc9 0012ffac 774ea9bd kernel32!BaseThreadInitThunk+0xe 0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

We call this pattern Glued Stack Trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 194)

Friday, January 18th, 2013

Whereas some false positives can be considered soft debugger bugs false negatives can have more severe impact on software behavior analysis especially in malware analysis. We name this pattern Debugger Omission. Typical example here is current .imgscan command which according to documentation should by default scan virtual process space for MZ/PE signatures. Unfortunately it doesn’t detect such signatures in resource pages (we haven’t checked stack regions yet):

0000000000fd0000 image base

SECTION HEADER #4 .rsrc name 6430 virtual size 4000 virtual address 6600 size of raw data 1600 file pointer to raw data 0 file pointer to relocation table 0 file pointer to line numbers 0 number of relocations 0 number of line numbers 40000040 flags Initialized Data (no align specified) Read Only

0:000> .imgscan /r 00000000`00fd4000 L200

0:000> s -[l2]sa 00000000`00fd4000 l200 00000000`00fd40b0 "MZ" 00000000`00fd40fd "!This program cannot be run in D" 00000000`00fd411d "OS mode." 00000000`00fd4188 "Rich" 00000000`00fd4198 "PE"

0:000> !dh 00000000`00fd40b0

File Type: DLL FILE HEADER VALUES 14C machine (i386) 3 number of sections time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table 0 number of symbols E0 size of optional header 2102 characteristics Executable 32 bit word machine DLL [...]

Another other analysis scenarios found will be added to this pattern. Milder version of it includes !analyze -v that shows us a breakpoint instead of an exception violation from a parallel thread.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging, Malware Analysis, x64 Windows | No Comments »

Static Code Analysis Patterns (Part 1)

Friday, January 11th, 2013

Static program analysis is used to eliminate certain coding errors that may lead to abnormal software behaviour. So it is naturally a part of software diagnostics but at source code level. Our goal here is to identify certain patterns directly linkable to patterns we see in memory dumps and software logs and collect them into a catalog. One such pattern candidate is called Loop Construct. It covers conditional and unconditional loops, for example, in one of modern languages:

extern bool soonToBeTrue;

int mediumValue = ...;

while (true)

{

  TRACE("Waiting");

  sleep(mediumValue);

  if (soonToBeTrue)

  {

    break;

  }

  doHeavyWork();

}

while (--pControl->aFewPasses)

{

  TRACE("Waiting");

  sleep(mediumValue);

  doHeavyWork();

}

Such loops may potentially lead to Spiking Thread memory dump analysis and High Message Current and Density trace analysis patterns. Of course, we shouldn’t suspect every loop but only some that have potential to be altered by Local Buffer Overflow (for mediumValue) or Shared Buffer Overwrite (for Control.aFewPasses) or by a race condition (soonToBeTrue).

We expect things to get more interesting when we start associating source code that uses certain API with patterns of abnormal behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in C and C++, Code Reading, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Security, Software Behavior Patterns, Software Diagnostics, Software Trace Analysis, Static Code Analysis Patterns, Trace Analysis Patterns, Victimware Analysis | No Comments »

Crash Dump Analysis Patterns (Part 193)

Wednesday, January 9th, 2013

Sometimes we have a Broken Link for some reason, either from memory corruption, Lateral Damage or Truncated Dump. For example, an active process list enumeration stops after showing some processes (!for_each_thread and !vm also don’t work):

0: kd> !process 0 ff

[...]

TYPE mismatch for process object at fffffa80041da5c0

0: kd> !validatelist nt!PsActiveProcessHead Blink at address fffffa80041da748 does not point back to previous at fffffa8005bc8cb8

Here we can either try to repair or navigate links manually or use other means such as dumping pool allocations for process structures with Proc pool tag:

0: kd> !poolfind Proc

Searching NonPaged pool (fffffa80032fc000 : ffffffe000000000) for Tag: Proc

*fffffa80033879a0 size: 510 previous size: a0 (Allocated) Proc (Protected) *fffffa80033ffad0 size: 530 previous size: 280 (Allocated) Proc (Protected) *fffffa80041a2af0 size: 510 previous size: 90 (Allocated) Proc (Protected) *fffffa800439c5c0 size: 530 previous size: 80 (Allocated) Proc (Protected) [...] *fffffa8007475ad0 size: 530 previous size: 30 (Allocated) Proc (Protected) *fffffa80074e8490 size: 530 previous size: 100 (Allocated) Proc (Protected) *fffffa80075ee0b0 size: 530 previous size: b0 (Free) Pro. *fffffa800761d000 size: 530 previous size: 0 (Free) Pro. *fffffa8007645ad0 size: 530 previous size: b0 (Allocated) Proc (Protected)

0: kd> dc fffffa8007645ad0 fffffa80`07645ad0 0253000b e36f7250 07644030 fffffa80 ..S.Pro.0.d..... fffffa80`07645ae0 00001000 00000528 00000068 fffff800 ....(...h....... fffffa80`07645af0 01a1a940 fffff800 00080090 00490024 @...........$.I. fffffa80`07645b00 000000c4 00000000 00000008 00000000 ................ fffffa80`07645b10 00000000 00000000 00080007 00300033 ............3.0. fffffa80`07645b20 01a1a940 fffff800 013cfeae fffff8a0 @.........<..... fffffa80`07645b30 00580003 00000000 05ba19a0 fffffa80 ..X…………. fffffa80`07645b40 05ba19a0 fffffa80 07645b48 fffffa80 ……..H[d…..

0: kd> !process fffffa80`07645b30 ff PROCESS fffffa8007645b30 SessionId: 0 Cid: 14c4 Peb: 7fffffd4000 ParentCid: 02c4 DirBase: 7233e000 ObjectTable: fffff8a0014d4220 HandleCount: 399. Image: AppA.exe VadRoot fffffa80072bc5b0 Vads 239 Clone 0 Private 24675. Modified 23838. Locked 0. DeviceMap fffff8a0000088f0 Token fffff8a000f28060 ElapsedTime 00:00:53.066 UserTime 00:00:00.000 KernelTime 00:00:00.000 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (11960, 50, 345) (47840KB, 200KB, 1380KB) PeakWorkingSetSize 74346 VirtualSize 331 Mb PeakVirtualSize 478 Mb PageFaultCount 92214 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 25905

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, WinDbg Tips and Tricks, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 42k)

Tuesday, January 8th, 2013

Here we provide examples of threads waiting for pushlocks as they are not normally seen in crash dumps:

THREAD fffffa80033b5b50 Cid 0004.0030 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable fffff880021d9750 SynchronizationEvent Not impersonating DeviceMap fffff8a0000088f0 Owning Process fffffa80033879e0 Image: System Attached Process fffffa800439c620 Image: AppA.exe Wait Start TickCount 30819 Ticks: 14746574 (2:15:54:08.028) Context Switch Count 2800 UserTime 00:00:00.000 KernelTime 00:00:00.374 Win32 Start Address nt!ExpWorkerThread (0xfffff8000189e530) Stack Init fffff880021d9db0 Current fffff880021d9470 Base fffff880021da000 Limit fffff880021d4000 Call 0 Priority 12 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`021d94b0 fffff800`0188aa32 nt!KiSwapContext+0×7a fffff880`021d95f0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2 fffff880`021d9680 fffff800`018c4bf8 nt!KeWaitForSingleObject+0×19f fffff880`021d9720 fffff800`01c2915d nt!ExfAcquirePushLockShared+0×138 fffff880`021d97a0 fffff800`01c6da31 nt!MmEnumerateAndReferenceImages+0×6d […] fffff880`021d9cb0 fffff800`01b2be5a nt!ExpWorkerThread+0×111 fffff880`021d9d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a fffff880`021d9d80 00000000`00000000 nt!KxStartSystemThread+0×16

THREAD fffffa8003c9d600 Cid 0004.00ac Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable fffff880023d1b30 SynchronizationEvent Not impersonating DeviceMap fffff8a0000088f0 Owning Process fffffa80033879e0 Image: System Attached Process N/A Image: N/A Wait Start TickCount 177686 Ticks: 14599707 (2:15:15:56.888) Context Switch Count 1590 UserTime 00:00:00.000 KernelTime 00:00:00.124 Win32 Start Address 0xfffff80001bac754 Stack Init fffff880023d1db0 Current fffff880023d1850 Base fffff880023d2000 Limit fffff880023cc000 Call 0 Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`023d1890 fffff800`0188aa32 nt!KiSwapContext+0×7a fffff880`023d19d0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2 fffff880`023d1a60 fffff800`01886183 nt!KeWaitForSingleObject+0×19f fffff880`023d1b00 fffff800`01cd9982 nt!ExfAcquirePushLockExclusive+0×188 […] fffff880`023d1d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a fffff880`023d1d80 00000000`00000000 nt!KxStartSystemThread+0×16

Instead of explaining what a pushlock is we provide a link to ntdebugging blog article.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Process Patterns

Saturday, January 5th, 2013

A page to reference all different kinds of process related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Thread Patterns

Saturday, January 5th, 2013

A page to reference all different kinds of thread related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 13i)

Saturday, January 5th, 2013

Although we briefly mentioned session pool in Insufficient Memory (kernel pool) pattern we decided to factor it into a separate (sub)pattern and provide WinDbg commands to analyze possible leaks. The following output shows the sequence of commands that gives you an idea although the example itself was taken from a healthy dump so no red coloring (from my memory leaks in session pool happened mostly in 32-bit past):

1: kd> !vm 4

Terminal Server Memory Usage By Session:

Session ID 0 @ fffff8800324d000: Paged Pool Usage: 4128K Commit Usage: 7488K

Session ID 1 @ fffff88002f65000: Paged Pool Usage: 32852K Commit Usage: 36488K

1: kd> !session Sessions on machine: 2 Valid Sessions: 0 1 Error in reading current session

1: kd> !session -s 1 Sessions on machine: 2 Implicit process is now fffffa80`07d79730 Using session 1

1: kd> !poolused 8 Sorting by Session Tag

Pool Used: NonPaged Paged Tag Allocs Used Allocs Used TOTAL 4 4208 9500 33475120 [...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, WinDbg Tips and Tricks, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 192)

Friday, January 4th, 2013

It is common to get dozens of process memory dumps saved sequentially, for example, after each second. Then we can first analyze memory dumps corresponding to changes in their file sizes ignoring plateaus to save analysis time. We call such pattern Step Dumps by an analogy with step functions. For example, we have this dump set (it was reported that an application was freezing for some time until its disappearance from a user screen) with comments from WinDbg analysis sessions:

C:\MemoryDumps>dir [...] 12/30/2012 8:33 PM 218,252,862 AppA-1.dmp // normal 12/30/2012 8:34 PM 218,541,762 AppA-2.dmp // slightly increased CPU consumption for thread #11 12/30/2012 8:37 PM 218,735,848 AppA-3.dmp // spiking thread #11 12/30/2012 8:38 PM 218,735,848 AppA-4.dmp 12/30/2012 8:38 PM 218,735,848 AppA-5.dmp 12/30/2012 8:39 PM 218,735,848 AppA-6.dmp 12/30/2012 8:39 PM 218,735,848 AppA-7.dmp 12/30/2012 8:39 PM 218,735,848 AppA-8.dmp 12/30/2012 8:40 PM 218,735,848 AppA-9.dmp 12/30/2012 8:40 PM 218,735,848 AppA-10.dmp 12/30/2012 8:41 PM 218,735,848 AppA-11.dmp 12/30/2012 8:41 PM 218,735,848 AppA-12.dmp // spiking thread #11 12/30/2012 8:42 PM 219,749,040 AppA-13.dmp // spiking thread #11, another thread blocked in ALPC 12/30/2012 8:42 PM 219,048,842 AppA-14.dmp // only one thread left […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 191)

Monday, December 31st, 2012

Sometimes we need to check under what security principal or group we run a process or what privileges it has or whether it has impersonating threads. We call this pattern Deviant Token when we find, for example, an unexpected token with a different security identifier, for example, Network Service instead of Local System (SID: S-1-5-18):

PROCESS 8f218d88 SessionId: 0 Cid: 09c4 Peb: 7ffdf000 ParentCid: 0240 DirBase: bffd4260 ObjectTable: e10eae90 HandleCount: 93. Image: ServiceA.exe VadRoot 8f1f70e8 Vads 141 Clone 0 Private 477. Modified 2. Locked 0. DeviceMap e10038d8 Token e10ff5d8 […]

0: kd> !token e10ff5d8 _TOKEN e10ff5d8 TS Session ID: 0 User: S-1-5-20 […]

Well-known SIDs can be found in this MS article: http://support.microsoft.com/kb/243330

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Security | No Comments »

Malware Analysis Patterns (Part 2)

Saturday, December 29th, 2012

As was announced earlier we start cataloguing elemental malware detection and analysis patterns. We skip Part 1 because we assign Deviant Module to it. Part 2 deals with Fake Module pattern where one of loaded modules masquerades as a legitimate system DLL or a widely known value adding DLL from some popular 3rd party product. To illustrate this pattern we modeled it as Victimware: a process crashed after loading a malware module:

0:000> k *** Stack trace for last set context - .thread/.cxr resets it Child-SP RetAddr Call Site 00000000`0026f978 00000001`3f89103a 0x0 00000000`0026f980 00000001`3f8911c4 FakeModule!wmain+0x3a 00000000`0026f9c0 00000000`76e3652d FakeModule!__tmainCRTStartup+0x144 00000000`0026fa00 00000000`7752c521 kernel32!BaseThreadInitThunk+0xd 00000000`0026fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

When we inspect loaded modules we don’t find anything suspicious:

0:000> lmp start end module name 00000000`76e20000 00000000`76f3f000 kernel32 <none> 00000000`77500000 00000000`776a9000 ntdll <none> 00000001`3f890000 00000001`3f8a6000 FakeModule <none> 000007fe`f8cb0000 000007fe`f8cc7000 winspool <none> 000007fe`fdb30000 000007fe`fdb9c000 KERNELBASE <none>

However, when checking modules images for any modifications we find that winspool was not compared with existing binary from Microsoft symbol server:

0:000> !for_each_module "!chkimg -v -d @#ModuleName" Searching for module with expression: kernel32 Will apply relocation fixups to file used for comparison Will ignore NOP/LOCK errors Will ignore patched instructions Image specific ignores will be applied Comparison image path: C:\WSDK8\Debuggers\x64\sym\kernel32.dll\503285C111f000\kernel32.dll No range specified

Scanning section: .text Size: 633485 Range to scan: 76e21000-76ebba8d Total bytes compared: 633485(100%) Number of errors: 0 0 errors : kernel32 Searching for module with expression: ntdll Will apply relocation fixups to file used for comparison Will ignore NOP/LOCK errors Will ignore patched instructions Image specific ignores will be applied Comparison image path: C:\WSDK8\Debuggers\x64\sym\ntdll.dll\4EC4AA8E1a9000\ntdll.dll No range specified

Scanning section: .text Size: 1049210 Range to scan: 77501000-7760127a Total bytes compared: 1049210(100%) Number of errors: 0

Scanning section: RT Size: 474 Range to scan: 77602000-776021da Total bytes compared: 474(100%) Number of errors: 0 0 errors : ntdll Searching for module with expression: FakeModule Error for FakeModule: Could not find image file for the module. Make sure binaries are included in the symbol path. Searching for module with expression: winspool Error for winspool: Could not find image file for the module. Make sure binaries are included in the symbol path. Searching for module with expression: KERNELBASE Will apply relocation fixups to file used for comparison Will ignore NOP/LOCK errors Will ignore patched instructions Image specific ignores will be applied Comparison image path: C:\WSDK8\Debuggers\x64\sym\KERNELBASE.dll\503285C26c000\KERNELBASE.dll No range specified

Scanning section: .text Size: 302047 Range to scan: 7fefdb31000-7fefdb7abdf Total bytes compared: 302047(100%) Number of errors: 0 0 errors : KERNELBASE

Checking module data reveals that it was loaded not from System32 folder and doesn’t have any version information:

0:000> lmv m winspool start end module name 000007fe`f8cb0000 000007fe`f8cc7000 winspool (deferred) Image path: C:\Work\AWMA\FakeModule\x64\Release\winspool.drv Image name: winspool.drv Timestamp: Fri Dec 28 22:22:42 2012 (50DE1BB2) CheckSum: 00000000 ImageSize: 00017000 File version: 0.0.0.0 Product version: 0.0.0.0 File flags: 0 (Mask 0) File OS: 0 Unknown Base File type: 0.0 Unknown File date: 00000000.00000000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

We could see that path from running this command as well :

0:000> !for_each_module 00: 0000000076e20000 0000000076f3f000 kernel32 C:\Windows\System32\kernel32.dll kernel32.dll 01: 0000000077500000 00000000776a9000 ntdll C:\Windows\System32\ntdll.dll ntdll.dll 02: 000000013f890000 000000013f8a6000 FakeModule C:\Work\AWMA\FakeModule\x64\Release\FakeModule.exe FakeModule.exe 03: 000007fef8cb0000 000007fef8cc7000 winspool C:\Work\AWMA\FakeModule\x64\Release\winspool.drv 04: 000007fefdb30000 000007fefdb9c000 KERNELBASE C:\Windows\System32\KERNELBASE.dll KERNELBASE.dll

or from PEB:

0:000> !peb PEB at 000007fffffdf000 [...] 7fef8cb0000 50de1bb2 Dec 28 22:22:42 2012 C:\Work\AWMA\FakeModule\x64\Release\winspool.drv […]

Another sign is module size in memory which is much smaller than real winspool.drv:

0:000> ? 000007fe`f8cc7000 - 000007fe`f8cb0000 Evaluate expression: 94208 = 00000000`0001700

Module size can help if legitimate module from well-known folder was replaced. Module debug directory and the size of export and import directories are also different with the former revealing the development folder:

0:000> !dh 000007fe`f8cb0000 [...] 0 [ 0] address [size] of Export Directory […] 9000 [ 208] address [size] of Import Address Table Directory […] Debug Directories(2) Type Size Address Pointer cv 49 e2c0 cac0 Format: RSDS, guid, 1, C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb

This can also be seen from the output of !lmi command:

0:000> !lmi 7fef8cb0000 Loaded Module Info: [7fef8cb0000] Module: winspool Base Address: 000007fef8cb0000 Image Name: winspool.drv Machine Type: 34404 (X64) Time Stamp: 50de1bb2 Fri Dec 28 22:22:42 2012 Size: 17000 CheckSum: 0 Characteristics: 2022 Debug Data Dirs: Type Size VA Pointer CODEVIEW 49, e2c0, cac0 RSDS - GUID: {29D85193-1C9D-4997-95BA-DD190FA3C1BF} Age: 1, Pdb: C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb ?? 10, e30c, cb0c [Data not mapped] Symbol Type: DEFERRED - No error - symbol load deferred Load Report: no symbols loaded

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Malware Analysis, Malware Patterns, Software Diagnostics, Victimware, Victimware Analysis, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 190)

Monday, December 24th, 2012

In addition to stack trace collection we often are interested in Module Collection (we called this pattern initially Vendor Collection), especially if we would like to check if some vendor DLL is present in some process address space in a complete memory dump (kernel module list or module list from a process memory dump is trivial). Or we need to check for some vendor information from problem description (lmv command). If we have a complete memory dump from x64 system then listing modules for each process is not enough. For example, we might have this:

0: kd> lmu start end module name 00000000`00ab0000 00000000`00ae8000 AppA (deferred) 00000000`74fe0000 00000000`7502e000 wow64win (deferred) 00000000`75030000 00000000`75075000 wow64 (deferred) 00000000`750c0000 00000000`750c9000 wow64cpu (deferred) 00000000`77b70000 00000000`77cf7000 ntdll (pdb symbols)

AppA is a 32-bit process and has an additional 32-bit module list that is more useful. We can set x86 context for a thread from that process and get the list of 32-bit modules:

0: kd> .load wow64exts

0: kd> .thread /w fffffa800e372060 Implicit thread is now fffffa80`0e372060 x86 context set

0: kd:x86> .reload Loading Kernel Symbols Loading User Symbols Loading unloaded module list Loading Wow64 Symbols

0: kd:x86> lmu start end module name 00000000`00ab0000 00000000`00ae8000 AppA (deferred) 00000000`73490000 00000000`73515000 COMCTL32 (deferred) 00000000`73520000 00000000`735c3000 MSVCR90 (deferred) 00000000`735d0000 00000000`7365e000 MSVCP90 (deferred) 00000000`74920000 00000000`7493e000 USERENV (deferred) 00000000`74940000 00000000`74ade000 comctl32_74940000 (deferred) 00000000`74af0000 00000000`74b02000 MSASN1 (deferred) 00000000`74b10000 00000000`74c03000 CRYPT32 (deferred) 00000000`74dc0000 00000000`74e5b000 MSVCR80 (deferred) 00000000`74f60000 00000000`74fd6000 NETAPI32 (deferred) 00000000`74fe0000 00000000`7502e000 wow64win (deferred) 00000000`75030000 00000000`75075000 wow64 (deferred) 00000000`750b0000 00000000`750ba000 WTSAPI32 (deferred) 00000000`750c0000 00000000`750c9000 wow64cpu (deferred) 00000000`75cf0000 00000000`75d50000 Secur32 (deferred) 00000000`75d50000 00000000`76861000 SHELL32 (deferred) 00000000`76a10000 00000000`76aa0000 GDI32 (deferred) 00000000`76b30000 00000000`76b90000 IMM32 (deferred) 00000000`76be0000 00000000`76cf0000 kernel32 (deferred) 00000000`76e30000 00000000`76f75000 ole32 (deferred) 00000000`76f80000 00000000`7702a000 msvcrt (deferred) 00000000`77030000 00000000`77037000 PSAPI (deferred) 00000000`77040000 00000000`77110000 USER32 (deferred) 00000000`77110000 00000000`77169000 SHLWAPI (deferred) 00000000`77170000 00000000`771ed000 USP10 (deferred) 00000000`77380000 00000000`7740d000 OLEAUT32 (deferred) 00000000`77640000 00000000`77649000 LPK (deferred) 00000000`776e0000 00000000`777d0000 RPCRT4 (deferred) 00000000`777d0000 00000000`77898000 MSCTF (deferred) 00000000`778a0000 00000000`77966000 ADVAPI32 (deferred) 00000000`77b70000 00000000`77cf7000 ntdll (pdb symbols) 00000000`77d30000 00000000`77e90000 ntdll_77d30000 # (pdb symbols)

So it looks like we need to dump modules for each thread. However, the output would be enormous unless we skip threads having the same PID. After some tinkering I wrote this WinDbg script with moderate output volume:

.load wow64exts !for_each_thread ".thread @#Thread; .if (@$t0 != @@c++(@$thread->Cid.UniqueProcess)) {.reload /user;lmvu;.thread /w @#Thread;.reload /user;lmvu;r $t0 = @@c++(@$thread->Cid.UniqueProcess);.effmach AMD64; }"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, WinDbg Scripts, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 189)

Sunday, December 23rd, 2012

Although a handle leak may lead to Insufficient Memory it is not always the case especially if pool structures are small such as events. So we describe another pattern called Handle Leak that covers high memory usage (including fat structures), high handle counts and also abnormal differences in allocations and deallocations. As an example for the latter here is a nonpaged pool leak of Event objects and correlated pooltag ABCD. Although memory usage footprint is small compared with other nonleaking pooltags we see the difference between Allocs and Frees is surely abnormal correlating with high handle counts:

0: kd> !poolused 3 Sorting by NonPaged Pool Consumed

Pool Used: NonPaged Paged Tag Allocs Frees Diff Used Allocs Frees Diff Used […] ABCD 1778517 1704538 73979 4734656 0 0 0 0 UNKNOWN pooltag ‘ABCD’, please update pooltag.txt Even 6129633 6063728 65905 4224528 0 0 0 0 Event objects […]

0: kd> !process 0 0

[...]

PROCESS d2b85360 SessionId: 2 Cid: 1bf4 Peb: 7ffdf000 ParentCid: 1688 DirBase: 7d778dc0 ObjectTable: e53dda08 HandleCount: 18539. Image: AppA.exe

PROCESS b2fcd670 SessionId: 2 Cid: 0818 Peb: 7ffd4000 ParentCid: 1688 DirBase: 7d778400 ObjectTable: b3ffd8c0 HandleCount: 36252. Image: AppB.exe

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | 1 Comment »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘Crash Dump Patterns’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta