Crash Dump Analysis

Crash Dump Analysis Patterns (Part 145)

Forthcoming Webinar on Pattern-Driven Software Diagnostics

2012 - The Year of Software Trace Analysis

Accelerated Memory Dump Analysis Training

Sponsored link: Memory Dump Analysis Services

In addition to Blocked Thread, endpoint threads of Wait Chain patterns, and Blocking Module we would like to add Blocking File pattern that often happen (but not limited to) in roaming profile scenarios. For example, an application was reported hanging and in a complete memory dump we can see a thread in stack trace collection:

THREAD fffffa8005eca060 Cid 14b0.1fec Teb: 000000007ef84000 Win32Thread: fffff900c26c2c30 WAIT: (Executive) KernelMode Non-Alertable fffffa80048e6758 NotificationEvent IRP List: fffffa8005a6c160: (0006,03e8) Flags: 00060000 Mdl: 00000000 Not impersonating DeviceMap fffff8a0055b6620 Owning Process fffffa80063dd970 Image: Application.exe Attached Process N/A Image: N/A Wait Start TickCount 171988390 Ticks: 26963639 (4:21:01:46.859) Context Switch Count 226 LargeStack UserTime 00:00:00.015 KernelTime 00:00:00.015 Win32 Start Address 0×000000006d851f62 Stack Init fffff880075a9db0 Current fffff880075a9770 Base fffff880075aa000 Limit fffff880075a4000 Call 0 Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`075a97b0 fffff800`0167f752 nt!KiSwapContext+0×7a fffff880`075a98f0 fffff800`016818af nt!KiCommitThreadWait+0×1d2 fffff880`075a9980 fffff800`019b612a nt!KeWaitForSingleObject+0×19f fffff880`075a9a20 fffff800`0198feaa nt! ?? ::NNGAKEGL::`string’+0×1d61a fffff880`075a9a60 fffff800`018ed0e3 nt!IopSynchronousServiceTail+0×35a fffff880`075a9ad0 fffff800`01677853 nt!NtLockFile+0×514 fffff880`075a9bb0 00000000`77840cea nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`075a9c20) 00000000`0798e488 00000000`7543293b ntdll!ZwLockFile+0xa 00000000`0798e490 00000000`7541cf87 wow64!whNtLockFile+0×7f 00000000`0798e510 00000000`7536276d wow64!Wow64SystemServiceEx+0xd7 00000000`0798edd0 00000000`7541d07e wow64cpu!TurboDispatchJumpAddressEnd+0×24 00000000`0798ee90 00000000`7541c549 wow64!RunCpuSimulation+0xa 00000000`0798eee0 00000000`7786d177 wow64!Wow64LdrpInitialize+0×429 00000000`0798f430 00000000`7782308e ntdll! ?? ::FNODOBFM::`string’+0×2bfe4 00000000`0798f4a0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

We immediately spot the anomaly of a lock file attempt and look at its IRP:

0: kd> !irp fffffa8005a6c160 Irp is active with 7 stacks 7 is current (= 0xfffffa8005a6c3e0) No Mdl: No System Buffer: Thread fffffa8005eca060: Irp stack trace. cmd flg cl Device File Completion-Context [ 0, 0] 0 2 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 ffffffffc000020c [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000 [ 11, 0] 0 2 fffffa8004da0620 00000000 fffff8800177d9cc-fffffa800710e580 \FileSystem\mrxsmb mup!MupiUncProviderCompletion Args: 00000000 00000000 00000000 00000000 >[ 11, 1] 0 0 fffffa8004066400 fffffa80048e66c0 00000000-00000000 \FileSystem\Mup Args: fffffa8004a98120 00000001 00000000 00000000

From that IRP we see a file name:

0: kd> !fileobj fffffa80048e66c0

[...]\AppData\Roaming\Vendor\Product\Recent\index.dat

LockOperation Set Device Object: 0xfffffa8004066400 \FileSystem\Mup Vpb is NULL Access: Read SharedRead SharedWrite SharedDelete

Flags: 0x40002 Synchronous IO Handle Created

File Object is currently busy and has 0 waiters.

FsContext: 0xfffff8a00e8d9010 FsContext2: 0xfffff8a012e4d688 CurrentByteOffset: 0 Cache Data: Section Object Pointers: fffffa8006086928 Shared Cache Map: 00000000 File object extension is at fffffa8005c8cbe0:

Alternatively we get a 32-bit stack trace from the virtualized process:

0: kd> .process /r /p fffffa80063dd970 Implicit process is now fffffa80`063dd970 Loading User Symbols

0: kd> .thread /w fffffa8005eca060 Implicit thread is now fffffa80`05eca060 The context is partially valid. Only x86 user-mode context is available. x86 context set

0: kd:x86> .reload Loading Kernel Symbols Loading User Symbols Loading unloaded module list Loading Wow64 Symbols

0: kd:x86> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child 07ac8510 774f033f 00000390 00000000 00000000 ntdll_779d0000!ZwLockFile+0×12 07ac8590 774f00d3 061b2b68 ada9964d c0000016 kernel32!BaseDllOpenIniFileOnDisk+0×246 07ac85d0 774efae9 061b2b68 00001000 6d352f20 kernel32!BaseDllReadWriteIniFileOnDisk+0×2d 07ac85e8 775001bf 00000001 00000000 061b2b68 kernel32!BaseDllReadWriteIniFile+0xed 07ac861c 6d928401 07aca71c 00000000 00001000 kernel32!GetPrivateProfileStringW+0×35 WARNING: Stack unwind information not available. Following frames may be wrong. 07ac8640 6d9282f5 07aca71c 00000000 00000000 DLL+0×618401 […] 07acfb14 774e3677 06757d20 07acfb60 77a09d72 DLL+0×541f6d 07acfb20 77a09d72 06757d20 eca51e43 00000000 kernel32!BaseThreadInitThunk+0xe 07acfb60 77a09d45 6d851f62 06757d20 ffffffff ntdll_779d0000!__RtlUserThreadStart+0×70 07acfb78 00000000 6d851f62 06757d20 00000000 ntdll_779d0000!_RtlUserThreadStart+0×1b

We get the same file name from a file handle:

0: kd> !handle 00000390 processor number 0, process fffffa80063dd970 PROCESS fffffa80063dd970 SessionId: 5 Cid: 14b0 Peb: 7efdf000 ParentCid: 1fac DirBase: 48293000 ObjectTable: fffff8a010515f90 HandleCount: 342. Image: Application.exe

Handle table at fffff8a0083e9000 with 444 Entries in use 0390: Object: fffffa80048e66c0 GrantedAccess: 00120089 Entry: fffff8a00866fe40 Object: fffffa80048e66c0 Type: (fffffa8003cf0b40) File ObjectHeader: fffffa80048e6690 (new version) HandleCount: 1 PointerCount: 3 Directory Object: 00000000 Name: […]\AppData\Roaming\Vendor\Product\Recent\index.dat {Mup}

We also c0000016 error code on raw stack and examine it too:

0: kd> !error c0000016 Error code: (NTSTATUS) 0xc0000016 (3221225494) - {Still Busy} The specified I/O request packet (IRP) cannot be disposed of because the I/O operation is not complete.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sponsored link: Professional Software Debugging Services

/* Malware and Software Defects -> Victimware.org */

Citrix and Microsoft Customer Forum

Museum of Debugging and Memory Dumps

7/7/2011 - 8/8/2011 Annual Competition: Tell Your Windows Debugging Story

Crash and Hang Analysis Audit Service

CARE: Crash Analysis Report Environment

Crash Dump and Software Trace Analysis Training and Seminars

Access OpenTask Titles on Safari Books Online

DATA (Dump Analysis + Trace Analysis) Facebook group
Please join the community of memory (dump) and trace analysis engineers. This group promotes scientific methods and memory dump-based worldview.

Twitter @ DumpAnalysis
You can now follow portal and blog news at DumpAnalysis on Twitter

LinkedIn Group Dr. Watson Enthusiasts
All about Dr. Watson errors and more. Get news, excerpts and progress reports about the forthcoming book The Science of Dr. Watson: An Illustrated History of Debugging (ISBN 978-1906717070)

2010 (0x7DA) - The Year of Dump Analysis
2011 (0x7DB) - 2020 (0x7E4) The Debugging Decade

International Memory Analysts and Debuggers Day:
07.07 and/or 08.08 starting from The Year of Dump Analysis, 2010, 7DA

Announcements

Coming Soon:

Resume and CV: As a Book

Fundamentals of Complete Crash and Hang Memory Dump Analysis

Management Bits: An Anthology from Reductionist Manager

Crash Dump Analysis: Practical Foundations (Windows Edition, Systematic Software Fault Analysis Series)

Crash Dump Analysis for System Administrators and Support Engineers

New Magazines:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers

New Books:

Advanced Windows Memory Dump Analysis with Data Structures: Training Course Transcript and WinDbg Practice Exercises with Notes

Accelerated .NET Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes

Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes

Introduction to Pattern-Driven Software Problem Solving

Memory Dump Analysis Anthology: Color Supplement for Volumes 4-5

Windows Debugging Notebook: Essential User Space WinDbg Commands

Memory Dump Analysis Anthology, Volume 5

Memory Dump Analysis Anthology, Volume 4

Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3

Memory Dump Analysis Anthology, Volume 3

First Fault Software Problem Solving: A Guide for Engineers, Managers and Users

x64 Windows Debugging: Practical Foundations

Also available:

Windows Debugging: Practical Foundations

DLL List Landscape: The Art from Computer Memory Space

Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

WinDbg: A Reference Poster and Learning Cards

Memory Dump Analysis Anthology, Volume 2

Memory Dump Analysis Anthology, Volume 1

New Children's Book:

Baby Turing

Memory Dump It

This entry was posted on Saturday, June 25th, 2011 at 11:00 pm and is filed under Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, x64 Windows. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Crash Dump Analysis Patterns (Part 145)

Leave a Reply