Archive for February, 2007

Crash Dump Analysis AntiPatterns (Part 3)

Wednesday, February 28th, 2007

I have heard engineers saying, “I didn’t know about this debugging command, let’s use it!” after training session or reading other people’s analysis of crash dumps. A year later I hear the same phrase from them about another debugging command. In the mean time they continue to use the same set of commands they know about until they hear the old new one.

This is a manifestation of Word of Mouth anti-pattern.

General solution: Know your tools. Study them proactively. RTFM.

Example solution: periodically read and re-read WinDbg help.

More refined solution: debugger.chm on Windows Mobile PC.

- Dmitry Vostokov -

Crash Dump Analysis AntiPatterns (Part 2)

Wednesday, February 28th, 2007

Let’s define Zippocricy - common sin in software support environments worldwide: someone gets something from a customer in archived form and without checking the contents forwards it further to another person in support chain. By the time the evidence gets unzipped somewhere, checked and found corrupt or irrelevant the customer suffers not hours but days.

Happens not only with crash dumps but with any type of problem evidence. 

- Dmitry Vostokov -

The Elements of Crash Dump Analysis Style

Monday, February 26th, 2007

After looking at multitude of crash dump analysis reports from different companies and engineers I would like to highlight several rules for good analysis reports:

  • Format your debugger output in fixed size font (Courier New or Lucida Console). This is very important for readability
  • Bold and highlight (using different colors) important addresses or data
  • Keep the same color for the same address or data consistently
  • Use red color for bug manifestation points
  • If you refer to some dump put a link to it

What is considered bad crash dump analysis style? These are:

  • Variable size font (you copy your debugger output to Outlook e-mail as is and it is using the default font)
  • Highlight the whole data set (for example, stack trace) in red
  • Too much irrelevant information

As an example of the good style I advocate (albeit not perfect) please look at the previous post Crash Dump Analysis Case Study

These are my first thoughts about crash and memory dump analysis style and I continue to elaborate it and present more examples later.

- Dmitry Vostokov -

Heap stack traces from W2K3/XP user dump

Saturday, February 24th, 2007

If you have user mode stack trace DB enabled on Windows 2003 for some service or application (here is an example for Citrix IMA service) and if you get a dump and try to get saved stack traces using !heap extension command you get these errors:

0:000> !heap -k -h 000a0000
    Heap entries for Segment00 in Heap 000a0000
        000a0c50: 00c50 . 00040 [01] - busy (40)
        000a0c90: 00040 . 01818 [07] - busy (1800), tail fill - unable to read heap entry extra at 000a24a0
        000a24a8: 01818 . 00030 [07] - busy (18), tail fill - unable to read heap entry extra at 000a24d0
        000a24d8: 00030 . 005a0 [07] - busy (588), tail fill - unable to read heap entry extra at 000a2a70

The solution is to use Windows 2000 extension ntsdexts.dll:

0:000> !.\w2kfre\ntsdexts.heap -k -h 000a0000
Stack trace (12) at 1021bfc:
   7c85fc22: ntdll!RtlAllocateHeapSlowly+0×00000041
   7c81d4df: ntdll!RtlAllocateHeap+0×00000E9F
   7c83467a: ntdll!LdrpAllocateUnicodeString+0×00000035
   7c8354f4: ntdll!LdrpCopyUnicodeString+0×00000031
   7c83517b: ntdll!LdrpResolveDllName+0×00000195
   7c834b2a: ntdll!LdrpMapDll+0×0000014F
   7c837474: ntdll!LdrpLoadImportModule+0×0000017C
   7c837368: ntdll!LdrpHandleOneNewFormatImportDescriptor+0×0000004D
   7c837317: ntdll!LdrpHandleNewFormatImportDescriptors+0×0000001D
   7c837441: ntdll!LdrpWalkImportDescriptor+0×00000195
   7c80f560: ntdll!LdrpInitializeProcess+0×00000E3E
   7c80ea0b: ntdll!_LdrpInitialize+0×000000D0
   7c82ec2d: ntdll!KiUserApcDispatcher+0×00000025

- Dmitry Vostokov @ DumpAnalysis.org -

Crash dump analysis case study (1)

Wednesday, February 21st, 2007

Consider the following legacy C++/Win32 code fragment highlighted in WinDbg after opening a crash dump:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
5:    DWORD dwRead = 0;
6:    CHAR *bufferA = new CHAR[dwSize+2];
7:    memset(bufferA, 0, dwSize+2);
8:    if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:    {
10:      DWORD i = 0, j = 0;
11:      for (; i < dwSize+2-7; ++i)
12:      {
13:         if (bufferA[i] == 0xD && bufferA[i+1] != 0xA)

At the first glance the code seems to be right: we open a file, get its size, allocate a buffer to read, etc. All loop indexes are within array bounds too. Let’s look at disassembly and crash point:

0:000> uf component!CMyDlg::OnTimer



004021bc push    0
004021be push    esi
004021bf call    dword ptr [component!_imp__GetFileSize (0042e26c)]
004021c5 mov     edi,eax ; dwSize
004021c7 lea     ebx,[edi+2] ; dwSize+2
004021ca push    ebx
004021cb mov     dword ptr [esp+34h],0
004021d3 call    component!operator new[] (00408e35)
004021d8 push    ebx
004021d9 mov     ebp,eax ; bufferA
004021db push    0
004021dd push    ebp
004021de call    component!memset (00418500)
004021e3 add     esp,10h
004021e6 push    0
004021e8 lea     edx,[esp+34h]
004021ec push    edx
004021ed push    edi
004021ee push    ebp
004021ef push    esi
004021f0 call    dword ptr [component!_imp__ReadFile (0042e264)]
004021f6 test    eax,eax
004021f8 jne     component!CMyDlg::OnTimer+0×3b1 (00402331)



00402331 xor     esi,esi ; i
00402333 add     edi,0FFFFFFFBh ; +2-7 (edi contains dwSize)
00402336 cmp     edi,esi ; loop condition
00402338 mov     dword ptr [esp+24h],esi
0040233c jbe     component!CMyDlg::OnTimer+0×43e (004023be)
00402342 mov     al,byte ptr [esi+ebp] ; bufferA[i]

0:000> r
eax=00002b00 ebx=00000002 ecx=00431000 edx=00000000 esi=00002b28 edi=fffffffb
eip=00402342 esp=0012efd4 ebp=0095b4d8 iopl=0 nv up ei pl nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000217
component!CMyDlg::OnTimer+0×3c2:
00402342 8a042e mov al,byte ptr [esi+ebp] ds:0023:0095e000=??

If we look at ebx (dwSize+2) and edi registers (array upper bound, dwSize+2-7) we can easily see that dwSize was zero. Clearly we had buffer overrun because upper array bound was calculated as 0+2-7 = FFFFFFFB (the loop index was unsigned integer, DWORD). Were the index signed integer variable (int) we wouldn’t have had any problem because the condition 0 < 0+2-7 is always false and the loop body would have never been executed.

Based on that the following fix was proposed:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
5:    DWORD dwRead = 0;
6:    CHAR *bufferA = new CHAR[dwSize+2];
7:    memset(bufferA, 0, dwSize+2);
8:    if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:    {
10:      DWORD i = 0, j = 0;
10:      int i = 0, j = 0;
11:      for (; i < dwSize+2-7; ++i)
11:      for (; i < (int)dwSize+2-7; ++i)
12:      {

GetFileSize can return INVALID_FILE_SIZE (0xFFFFFFFF) and operator new can fail theoretically (if the size is too big) so we can correct the code even further:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
4a:   if (dwSize != INVALID_FILE_SIZE)
4b:   {
5:       DWORD dwRead = 0;
6:       CHAR *bufferA = new CHAR[dwSize+2];
6a:      if (bufferA)
6b:      {
7:          memset(bufferA, 0, dwSize+2);
8:          if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:          {
10:            int i = 0, j = 0;
11:            for (; i < (int)dwSize+2-7; ++i)
12:            {

- Dmitry Vostokov @ DumpAnalysis.org -

Suspending threads (live kernel debugging)

Tuesday, February 20th, 2007

I couldn’t find any WinDbg command to suspend threads during live kernel debugging session even if you debug a process. This can be useful for debugging or reproducing race condition issues. ~n (suspend) and ~f (freeze) are for user mode live debugging only.

For example, you have one thread that depends on another thread finishing its work earlier. Sometimes, very rarely the latter thread finishes after the moment  the first thread would expect it. In order to model this race condition you can simply patch the prologue code of the second thread worker function with ret instruction. This has the same effect as suspending the thread so it cannot produce required data. 

- Dmitry Vostokov -

InstantDump (JIT Process Dumper)

Monday, February 19th, 2007

Techniques utilizing user mode process dumpers and debuggers like Microsoft userdump.exe, NTSD or WinDbg and CDB from Debugging Tools for Windows are too slow to pick up a process and dump it. You need either to attach a debugger manually, run the command line prompt or switch to Task Manager. This deficiency was the primary motivation for me to use JIT (just-in-time) technology for process dumpers. The new tool, InstantDump, will dump a process instantly and non-invasively in a moment when you need it. How does it work? You point to any window and press hot key.

InstantDump could be useful to study hang GUI processes or to get several dumps of the same process during some period of time (CPU spiking case or memory leak, for example) or just dump the process for the sake of dumping it (for curiosity). The tool uses the same tooltip technology introduced in WindowHistory 4.0 to dynamically display window information.

Short user guide:

1. The program will run only on XP/W2K3/Vista (in fact it will not load on W2K).

2. Run InstantDump.exe on 32-bit system or InstantDump64.exe on x64 Windows. If you attempt to run InstantDump.exe on x64 Windows it will show this message box and quit:

 

3. InstantDump puts itself into task bar icon notification area:

4. By default when you move the mouse pointer over windows the tooltip follows the cursor describing the process and thread id and process image path (you can disable tips in Options dialog box):

5. If you hold Ctrl-RightShift-Break for less than a second then the process (which window is under the cursor) will be dumped according to the settings for external process dumper in options dialog (accessible via task bar icon right mouse click):

 

The saved dump name will be (in our Calculator window case): calc.exe_9f8(2552)_22-17-56_18-Feb-2007.dmp

Looks like there is no NTSD in Vista so you have to use another user mode dumper, for example, install MS userdump.exe and specify the following command line in Options dialog:

userdump.exe %d %s

or resort to WinDbg or CDB command line.

The tool can be downloaded from here.

The new version of this tool is under development that will automatically pick up a process name from Task Manager, Process Explorer or Process Monitor (in fact, from any tool that displays the list of processes) and then instantly dump it.

- Dmitry Vostokov -

WindowHistory 4.0

Thursday, February 15th, 2007

I’ve added tool tips showing window information when pointing at any window. Here are some screenshots:

This works inside Citrix seamless ICA sessions too. Additionally the new version tracks client window rectangle and its changes (this was missing in the previous versions).

It works on Vista (doesn’t require elevation):

32-bit version can be downloaded from Citrix support web site.

64-bit version, which tracks changes for both 32-bit and 64-bit windows on x64 Windows platform, can also be downloaded from Citrix support web site.

- Dmitry Vostokov @ DumpAnalysis.org -

savedump.exe and pagefile

Wednesday, February 14th, 2007

Today I was curious about what savedump.exe does. And after some research I found the following:

On Windows 2000 it is a part of the logon process where it copies the crash dump section from the pagefile to a memory dump and also creates a mini dump file.

http://support.microsoft.com/kb/257299

http://support.microsoft.com/kb/262077

On Windows 2003 and XP savedump.exe doesn’t touch the pagefile. Session manager smss.exe truncates the pagefile and renames it to dumpxxx.dmp file (copying it to the boot volume if necessary). Then savedump.exe copies that file to the correct location and then creates the mini dump file.

http://support.microsoft.com/kb/886429

Also I couldn’t find savedump.exe on Vista. It looks like it was finally removed.

- Dmitry Vostokov -

Easy list traversing (dt vs. !list)

Sunday, February 11th, 2007

I recently discovered in WinDbg help that dt command can be used for traversing linked lists. Most structures I work with have LIST_ENTRY as their first member and it is much easier to use dt command than !list (less typing) For example:

0:000> dt _MYBIGSTRUCTURE
   +0x000 Links : _LIST_ENTRY
    ...
   +0x080 SomeName : [33] Uint2B

0:000> dd component!MyBigStructureListHead l1
01022cd0  0007fe58

0:000> .enable_unicode 1

The following command outputs the whole list of structures:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink 0007fe58

And the following command outputs the list of SomeName members:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink -y SomeName 0007fe58
Links.Flink at 0×7fe58
   +0×000 Links :  [ 0×8e090 - 0×1022cd0 ]
   +0×080 SomeName : [33]  “Foo”
Links.Flink at 0×8e090
   +0×000 Links :  [ 0×913f8 - 0×7fe58 ]
   +0×080 SomeName : [33]  “Bar”

If you don’t remember exact member name you can specify the partial name and any member that matches will be shown:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink -y S 0007fe58

However it your structure doesn’t have LIST_ENTRY as its first member then you need to subtract its offset, for example:

kd> dd nt!PsActiveProcessHead l1
808af068  85fa48b0

kd> dt _EPROCESS
   +0x000 Pcb              : _KPROCESS
   +0x078 ProcessLock      : _EX_PUSH_LOCK
   +0x080 CreateTime       : _LARGE_INTEGER
   +0x088 ExitTime         : _LARGE_INTEGER
   +0x090 RundownProtect   : _EX_RUNDOWN_REF
   +0x094 UniqueProcessId  : Ptr32 Void
   +0×098 ActiveProcessLinks : _LIST_ENTRY

kd> dt _EPROCESS -l ActiveProcessLinks.Flink -y ImageFileName 85fa48b0-0×98
ActiveProcessLinks.Flink at 0×85fa4818
   +0×098 ActiveProcessLinks :  [ 0×85d1ce20 - 0×808af068 ]
   +0×164 ImageFileName : [16]  “System”
ActiveProcessLinks.Flink at 0×85d1cd88
   +0×098 ActiveProcessLinks :  [ 0×85dba6b8 - 0×85fa48b0 ]
   +0×164 ImageFileName : [16]  “smss.exe”
ActiveProcessLinks.Flink at 0×85dba620
   +0×098 ActiveProcessLinks :  [ 0×858d20b8 - 0×85d1ce20 ]
   +0×164 ImageFileName : [16]  “csrss.exe”
ActiveProcessLinks.Flink at 0×858d2020
   +0×098 ActiveProcessLinks :  [ 0×858c20b8 - 0×85dba6b8 ]
   +0×164 ImageFileName : [16]  “winlogon.exe”
ActiveProcessLinks.Flink at 0×858c2020
   +0×098 ActiveProcessLinks :  [ 0×8589f0b8 - 0×858d20b8 ]
   +0×164 ImageFileName : [16]  “services.exe”

Here is another example, not involving LIST_ENTRY but rather a classic single list forward pointer: 

0:000> !teb
TEB at 7FFDE000
    ExceptionList:    6fc54
    Stack Base:       70000
    Stack Limit:      6d000
    SubSystemTib:     0
    FiberData:        1e00
    ArbitraryUser:    0
    Self:             7ffde000
    EnvironmentPtr:   0
    ClientId:         22c.228
    Real ClientId:    22c.228
    RpcHandle:        0
    Tls Storage:      742b8
    PEB Address:      7ffdf000
    LastErrorValue:   997
    LastStatusValue:  103
    Count Owned Locks:0
    HardErrorsMode:   0

0:000> dt -r _TEB
   +0x000 NtTib : _NT_TIB
      +0x000 ExceptionList : Ptr32 _EXCEPTION_REGISTRATION_RECORD
         +0×000 Next : Ptr32 _EXCEPTION_REGISTRATION_RECORD
         +0×004 Handler : Ptr32
      +0×004 StackBase : Ptr32 Void
      +0×008 StackLimit : Ptr32 Void
      +0×00c SubSystemTib : Ptr32 Void
      +0×010 FiberData : Ptr32 Void
      +0×010 Version : Uint4B
      +0×014 ArbitraryUserPointer : Ptr32 Void
      +0×018 Self : Ptr32 _NT_TIB

0:000> dt _EXCEPTION_REGISTRATION_RECORD -l Next 7FFDE000
Next at 0x7ffde000
   +0x000 Next : 0x0006fc54 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x00070000 +70000
Next at 0x6fc54
   +0x000 Next : 0x0006fcfc _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c5c1f44 KERNEL32!_except_handler3+0
Next at 0x6fcfc
   +0x000 Next : 0x0006ff5c _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c2e5649 ADVAPI32!_except_handler3+0
Next at 0x6ff5c
   +0x000 Next : 0x0006ffb0 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c2e5649 ADVAPI32!_except_handler3+0
Next at 0x6ffb0
   +0x000 Next : 0x0006ffe0 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x01015878 component!_except_handler3+0
Next at 0x6ffe0
   +0x000 Next : 0xffffffff _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c5c1f44 KERNEL32!_except_handler3+0

- Dmitry Vostokov -

Crash Dump Analysis in Visual Studio 2005

Saturday, February 10th, 2007

If you open a user crash dump as a solution/project, not a as file then you can do crash dump analysis by using Visual Studio debug windows, for example, powerful Watch window. As you can see from the picture I loaded a crash dump from my test application saved by NTSD and it shows assembly code and source code nicely interleaved.

If you need to specify additional symbol paths or symbol server settings you can do it in Tools \ Options \ Debugging \ Symbols dialog.

This might be good for you if you are used to do live debugging in Visual Studio and have no experience in using WinDbg and you need to look at your application crash dumps sent by customers.

- Dmitry Vostokov -

Exported NTDLL and kernel structures

Saturday, February 10th, 2007

It happens sometimes that during crash dump analysis or debugging session I forget exact structure name when I want to use it in dt WinDbg command. In this case wildcards help me: dt module!*, for example,

0:000> dt ntdll!*
          ntdll!LIST_ENTRY64
          ntdll!LIST_ENTRY32
          ntdll!_ULARGE_INTEGER
          ntdll!_LIST_ENTRY
          ntdll!_IMAGE_NT_HEADERS
          ntdll!_IMAGE_FILE_HEADER
          ntdll!_IMAGE_OPTIONAL_HEADER
          ntdll!_IMAGE_NT_HEADERS
          ntdll!_LARGE_INTEGER
          ntdll!_LUID
          ntdll!_KPRCB
          ntdll!_KTHREAD
          ntdll!_KPROCESSOR_STATE
          ntdll!_KSPIN_LOCK_QUEUE
          ntdll!_KNODE
          ntdll!_PP_LOOKASIDE_LIST
          ntdll!_KPRCB
          ntdll!_KDPC_DATA
          ntdll!_KEVENT
          ntdll!_KDPC
          ntdll!_SINGLE_LIST_ENTRY
          ntdll!_FX_SAVE_AREA
          ntdll!_PROCESSOR_POWER_STATE
          ntdll!_KPRCB
          ntdll!_KPCR
          ntdll!_NT_TIB
          ntdll!_EXCEPTION_REGISTRATION_RECORD
          ntdll!_KIDTENTRY
          ntdll!_KGDTENTRY
          ntdll!_KTSS
          ntdll!_KPCR
          ntdll!_KAPC
          ntdll!_SINGLE_LIST_ENTRY
          ntdll!_KDPC_IMPORTANCE
          ntdll!_KDPC
          ntdll!_DISPATCHER_HEADER
          ntdll!_KAPC_STATE
          ntdll!_KWAIT_BLOCK
          ntdll!_KGATE
          ntdll!_KQUEUE
          ntdll!_KTIMER
          ntdll!_KTRAP_FRAME
          ntdll!_KPROCESS
          ntdll!_KSEMAPHORE
          ntdll!_KTHREAD
          ntdll!_KSPIN_LOCK_QUEUE_NUMBER
          ntdll!_FAST_MUTEX
          ntdll!_SLIST_HEADER
          ntdll!_NPAGED_LOOKASIDE_LIST
          ntdll!_GENERAL_LOOKASIDE
          ntdll!_NPAGED_LOOKASIDE_LIST
          ntdll!_PAGED_LOOKASIDE_LIST
          ntdll!_PP_NPAGED_LOOKASIDE_NUMBER
          ntdll!_POOL_TYPE
          ntdll!_GENERAL_LOOKASIDE
          ntdll!_EX_RUNDOWN_REF
          ntdll!_EX_FAST_REF
          ntdll!_EX_PUSH_LOCK
          ntdll!_EX_PUSH_LOCK_WAIT_BLOCK
          ntdll!_EX_PUSH_LOCK_CACHE_AWARE
          ntdll!_ETHREAD
          ntdll!_TERMINATION_PORT
          ntdll!_CLIENT_ID
          ntdll!_PS_IMPERSONATION_INFORMATION
          ntdll!_DEVICE_OBJECT
          ntdll!_EPROCESS
          ntdll!_ETHREAD
          ntdll!_HANDLE_TABLE
          ntdll!_KGUARDED_MUTEX
          ntdll!_MM_AVL_TABLE
          ntdll!_EJOB
          ntdll!_EPROCESS_QUOTA_BLOCK
          ntdll!_PAGEFAULT_HISTORY
          ntdll!_HARDWARE_PTE_X86
          ntdll!_PEB
          ntdll!_SE_AUDIT_PROCESS_CREATION_INFO
          ntdll!_MMSUPPORT
          ntdll!_EPROCESS
          ntdll!_OBJECT_HEADER
          ntdll!_OBJECT_TYPE
          ntdll!_OBJECT_CREATE_INFORMATION
          ntdll!_QUAD
          ntdll!_OBJECT_HEADER
          ntdll!_OBJECT_HEADER_QUOTA_INFO
          ntdll!_OBJECT_HEADER_HANDLE_INFO
          ntdll!_OBJECT_HANDLE_COUNT_DATABASE
          ntdll!_OBJECT_HANDLE_COUNT_ENTRY
          ntdll!_OBJECT_HEADER_HANDLE_INFO
          ntdll!_OBJECT_HEADER_NAME_INFO
          ntdll!_OBJECT_DIRECTORY
          ntdll!_UNICODE_STRING
          ntdll!_OBJECT_HEADER_NAME_INFO
          ntdll!_OBJECT_HEADER_CREATOR_INFO
          ntdll!_OBJECT_ATTRIBUTES
          ntdll!_ERESOURCE
          ntdll!_OBJECT_TYPE_INITIALIZER
          ntdll!_OBJECT_TYPE
          ntdll!_OBJECT_HANDLE_INFORMATION
          ntdll!_PERFINFO_GROUPMASK
          ntdll!_KGUARDED_MUTEX
          ntdll!_DISPATCHER_HEADER
          ntdll!_PF_SCENARIO_TYPE
          ntdll!_HANDLE_TRACE_DEBUG_INFO
          ntdll!_HANDLE_TABLE
          ntdll!_KWAIT_BLOCK
          ntdll!_MMSUPPORT_FLAGS
          ntdll!_MMWSL
          ntdll!_MMSUPPORT
          ntdll!_EPROCESS_QUOTA_ENTRY
          ntdll!_EPROCESS_QUOTA_BLOCK
          ntdll!_UNICODE_STRING
          ntdll!_NT_TIB
          ntdll!_PS_JOB_TOKEN_FILTER
          ntdll!_IO_COUNTERS
          ntdll!_EJOB
          ntdll!_PEB_LDR_DATA
          ntdll!_RTL_USER_PROCESS_PARAMETERS
          ntdll!_RTL_CRITICAL_SECTION
          ntdll!_PEB_FREE_BLOCK
          ntdll!_ACTIVATION_CONTEXT_DATA
          ntdll!_ASSEMBLY_STORAGE_MAP
          ntdll!_PEB
          ntdll!_KGATE
          ntdll!_IMAGE_FILE_HEADER
          ntdll!_RTL_STACK_TRACE_ENTRY
          ntdll!_PEB_FREE_BLOCK
          ntdll!_KSPIN_LOCK_QUEUE
          ntdll!_PP_LOOKASIDE_LIST
          ntdll!_KEXECUTE_OPTIONS
          ntdll!_KPROCESS
          ntdll!_PEB_LDR_DATA
          ntdll!_DPH_BLOCK_INFORMATION
          ntdll!_SECURITY_IMPERSONATION_LEVEL
          ntdll!_PS_IMPERSONATION_INFORMATION
          ntdll!_EPROCESS_QUOTA_ENTRY
          ntdll!_FNSAVE_FORMAT
          ntdll!_FX_SAVE_AREA
          ntdll!PROCESSOR_IDLE_TIMES
          ntdll!PROCESSOR_PERF_STATE
          ntdll!_PROCESSOR_POWER_STATE
          ntdll!_IO_COUNTERS
          ntdll!_KiIoAccessMap
          ntdll!_KTSS
          ntdll!_KIDTENTRY
          ntdll!_MMSUPPORT_FLAGS
          ntdll!_HEAP
          ntdll!_HEAP_ENTRY
          ntdll!_HEAP_TAG_ENTRY
          ntdll!_HEAP_UCR_SEGMENT
          ntdll!_HEAP_UNCOMMMTTED_RANGE
          ntdll!_HEAP_SEGMENT
          ntdll!_HEAP_PSEUDO_TAG_ENTRY
          ntdll!_HEAP_LOCK
          ntdll!_HEAP
          ntdll!_TERMINATION_PORT
          ntdll!LSA_FOREST_TRUST_RECORD_TYPE
          ntdll!_HEAP_UNCOMMMTTED_RANGE
          ntdll!_OBJECT_HANDLE_COUNT_DATABASE
          ntdll!_FNSAVE_FORMAT
          ntdll!PROCESSOR_PERF_STATE
          ntdll!PROCESSOR_IDLE_TIMES
          ntdll!_HANDLE_TRACE_DB_ENTRY
          ntdll!_HANDLE_TRACE_DEBUG_INFO
          ntdll!_PROCESS_WS_WATCH_INFORMATION
          ntdll!_PAGEFAULT_HISTORY
          ntdll!_SECURITY_QUALITY_OF_SERVICE
          ntdll!_OBJECT_CREATE_INFORMATION
          ntdll!_MMADDRESS_NODE
          ntdll!_MM_AVL_TABLE
          ntdll!_HARDWARE_PTE_X86
          ntdll!_HEAP_ENTRY
          ntdll!_GENERIC_MAPPING
          ntdll!_OBJECT_DUMP_CONTROL
          ntdll!_OB_OPEN_REASON
          ntdll!_ACCESS_STATE
          ntdll!_SECURITY_OPERATION_CODE
          ntdll!_OBJECT_NAME_INFORMATION
          ntdll!_OBJECT_TYPE_INITIALIZER
          ntdll!_LARGE_INTEGER
          ntdll!_RTL_TRACE_BLOCK
          ntdll!_HEAP_UCR_SEGMENT
          ntdll!_KEXECUTE_OPTIONS
          ntdll!_OWNER_ENTRY
          ntdll!_ERESOURCE
          ntdll!_GENERIC_MAPPING
          ntdll!_SID_AND_ATTRIBUTES
          ntdll!_LUID_AND_ATTRIBUTES
          ntdll!_PS_JOB_TOKEN_FILTER
          ntdll!_MEMORY_CACHING_TYPE_ORIG
          ntdll!_KiIoAccessMap
          ntdll!_EXCEPTION_DISPOSITION
          ntdll!_EXCEPTION_RECORD
          ntdll!_CONTEXT
          ntdll!_EXCEPTION_REGISTRATION_RECORD
          ntdll!_DRIVER_OBJECT
          ntdll!_IRP
          ntdll!_IO_TIMER
          ntdll!_VPB
          ntdll!_WAIT_CONTEXT_BLOCK
          ntdll!_KDEVICE_QUEUE
          ntdll!_DEVOBJ_EXTENSION
          ntdll!_DEVICE_OBJECT
          ntdll!_PROCESS_WS_WATCH_INFORMATION
          ntdll!_SECURITY_QUALITY_OF_SERVICE
          ntdll!_FLOATING_SAVE_AREA
          ntdll!_CONTEXT
          ntdll!_IMAGE_DATA_DIRECTORY
          ntdll!_IMAGE_OPTIONAL_HEADER
          ntdll!_KUSER_SHARED_DATA
          ntdll!_KSYSTEM_TIME
          ntdll!_NT_PRODUCT_TYPE
          ntdll!_ALTERNATIVE_ARCHITECTURE_TYPE
          ntdll!_KUSER_SHARED_DATA
          ntdll!_QUAD
          ntdll!_KAPC_STATE
          ntdll!_MODE
          ntdll!_HEAP_PSEUDO_TAG_ENTRY
          ntdll!_RTL_CRITICAL_SECTION_DEBUG
          ntdll!_RTL_CRITICAL_SECTION
          ntdll!_HEAP_SEGMENT
          ntdll!_KTRAP_FRAME
          ntdll!_KGDTENTRY
          ntdll!_KDEVICE_QUEUE_ENTRY
          ntdll!_IO_ALLOCATION_ACTION
          ntdll!_WAIT_CONTEXT_BLOCK
          ntdll!_KTIMER
          ntdll!_MDL
          ntdll!_IO_STATUS_BLOCK
          ntdll!_IO_STACK_LOCATION
          ntdll!_FILE_OBJECT
          ntdll!_IRP
          ntdll!_VPB
          ntdll!_KOBJECTS
          ntdll!_KSEMAPHORE
          ntdll!_MMADDRESS_NODE
          ntdll!_CURDIR
          ntdll!_RTL_DRIVE_LETTER_CURDIR
          ntdll!_RTL_USER_PROCESS_PARAMETERS
          ntdll!_OWNER_ENTRY
          ntdll!_SE_AUDIT_PROCESS_CREATION_INFO
          ntdll!_OBJECT_HANDLE_COUNT_ENTRY
          ntdll!_CLIENT_ID
          ntdll!_RTL_TRACE_DATABASE
          ntdll!_RTL_TRACE_SEGMENT
          ntdll!_RTL_TRACE_DATABASE
          ntdll!_HEAP_LOCK
          ntdll!_HANDLE_TRACE_DB_ENTRY
          ntdll!ReplacesCorHdrNumericDefines
          ntdll!_MEMORY_TYPE
          ntdll!_IO_TIMER
          ntdll!_FXSAVE_FORMAT
          ntdll!_OBJECT_DIRECTORY_ENTRY
          ntdll!_DEVICE_MAP
          ntdll!_OBJECT_DIRECTORY
          ntdll!_STACK_TRACE_DATABASE
          ntdll!_KDPC_DATA
          ntdll!_STRING
          ntdll!_RTL_DRIVE_LETTER_CURDIR
          ntdll!_SID_AND_ATTRIBUTES
          ntdll!_DPH_HEAP_ROOT
          ntdll!_DPH_HEAP_BLOCK
          ntdll!_RTL_AVL_TABLE
          ntdll!_DPH_HEAP_ROOT
          ntdll!_DEVICE_OBJECT_POWER_EXTENSION
          ntdll!_DEVOBJ_EXTENSION
          ntdll!_FLOATING_SAVE_AREA
          ntdll!_KSYSTEM_TIME
          ntdll!_KQUEUE
          ntdll!_RTL_BALANCED_LINKS
          ntdll!_RTL_GENERIC_COMPARE_RESULTS
          ntdll!_RTL_AVL_TABLE
          ntdll!_HEAP_TAG_ENTRY
          ntdll!_RTL_CRITICAL_SECTION_DEBUG
          ntdll!_MDL
          ntdll!_DPH_HEAP_BLOCK
          ntdll!_PS_QUOTA_TYPE
          ntdll!_flags
          ntdll!_KNODE
          ntdll!_LDR_DATA_TABLE_ENTRY
          ntdll!_ACTIVATION_CONTEXT
          ntdll!_LDR_DATA_TABLE_ENTRY
          ntdll!_TEB
          ntdll!_ACTIVATION_CONTEXT_STACK
          ntdll!_GDI_TEB_BATCH
          ntdll!_TEB_ACTIVE_FRAME
          ntdll!_TEB
          ntdll!_KEVENT
          ntdll!_IO_STATUS_BLOCK
          ntdll!_RTL_TRACE_SEGMENT
          ntdll!_SECURITY_SUBJECT_CONTEXT
          ntdll!_INITIAL_PRIVILEGE_SET
          ntdll!_PRIVILEGE_SET
          ntdll!_ACCESS_STATE
          ntdll!_KSPECIAL_REGISTERS
          ntdll!_KPROCESSOR_STATE
          ntdll!_STRING
          ntdll!_flags
          ntdll!_REG_NOTIFY_CLASS
          ntdll!_OBJECT_DUMP_CONTROL
          ntdll!_SECURITY_SUBJECT_CONTEXT
          ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
          ntdll!_ACTIVATION_CONTEXT_STACK
          ntdll!_MMSYSTEM_PTE_POOL_TYPE
          ntdll!_KDEVICE_QUEUE
          ntdll!_LUID_AND_ATTRIBUTES
          ntdll!_EXCEPTION_RECORD
          ntdll!_INITIAL_PRIVILEGE_SET
          ntdll!_TEB_ACTIVE_FRAME_CONTEXT
          ntdll!_TEB_ACTIVE_FRAME
          ntdll!_OBJECT_NAME_INFORMATION
          ntdll!_SECTION_OBJECT_POINTERS
          ntdll!_IO_COMPLETION_CONTEXT
          ntdll!_FILE_OBJECT
          ntdll!_IO_COMPLETION_CONTEXT
          ntdll!_DRIVER_EXTENSION
          ntdll!_FAST_IO_DISPATCH
          ntdll!_DRIVER_OBJECT
          ntdll!_IO_CLIENT_EXTENSION
          ntdll!_FS_FILTER_CALLBACKS
          ntdll!_DRIVER_EXTENSION
          ntdll!_TEB_ACTIVE_FRAME_CONTEXT
          ntdll!_IMAGE_DATA_DIRECTORY
          ntdll!_CURDIR
          ntdll!_GDI_TEB_BATCH
          ntdll!_RTL_BALANCED_LINKS
          ntdll!_KDEVICE_QUEUE_ENTRY
          ntdll!_SECTION_OBJECT_POINTERS
          ntdll!_IO_CLIENT_EXTENSION
          ntdll!_IO_SECURITY_CONTEXT
          ntdll!_NAMED_PIPE_CREATE_PARAMETERS
          ntdll!_MAILSLOT_CREATE_PARAMETERS
          ntdll!_FILE_INFORMATION_CLASS
          ntdll!_FSINFOCLASS
          ntdll!_SCSI_REQUEST_BLOCK
          ntdll!_FILE_GET_QUOTA_INFORMATION
          ntdll!_DEVICE_RELATION_TYPE
          ntdll!_GUID
          ntdll!_INTERFACE
          ntdll!_DEVICE_CAPABILITIES
          ntdll!_IO_RESOURCE_REQUIREMENTS_LIST
          ntdll!BUS_QUERY_ID_TYPE
          ntdll!DEVICE_TEXT_TYPE
          ntdll!_DEVICE_USAGE_NOTIFICATION_TYPE
          ntdll!_SYSTEM_POWER_STATE
          ntdll!_POWER_SEQUENCE
          ntdll!_POWER_STATE_TYPE
          ntdll!_POWER_STATE
          ntdll!POWER_ACTION
          ntdll!_CM_RESOURCE_LIST
          ntdll!_IO_STACK_LOCATION
          ntdll!_INTERFACE
          ntdll!_DEVICE_POWER_STATE
          ntdll!_POWER_STATE
          ntdll!_FS_FILTER_CALLBACK_DATA
          ntdll!_FS_FILTER_CALLBACKS
          ntdll!_DEVICE_MAP
          ntdll!_INTERFACE_TYPE
          ntdll!_IO_RESOURCE_LIST
          ntdll!_IO_RESOURCE_REQUIREMENTS_LIST
          ntdll!_SID
          ntdll!_FILE_GET_QUOTA_INFORMATION
          ntdll!_FS_FILTER_PARAMETERS
          ntdll!_FS_FILTER_CALLBACK_DATA
          ntdll!_FILE_BASIC_INFORMATION
          ntdll!_FILE_STANDARD_INFORMATION
          ntdll!_FILE_NETWORK_OPEN_INFORMATION
          ntdll!_COMPRESSED_DATA_INFO
          ntdll!_FAST_IO_DISPATCH
          ntdll!_OBJECT_DIRECTORY_ENTRY
          ntdll!_FILE_BASIC_INFORMATION
          ntdll!_PRIVILEGE_SET
          ntdll!_IO_SECURITY_CONTEXT
          ntdll!_DESCRIPTOR
          ntdll!_KSPECIAL_REGISTERS
          ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
          ntdll!_MAILSLOT_CREATE_PARAMETERS
          ntdll!_NAMED_PIPE_CREATE_PARAMETERS
          ntdll!_IO_RESOURCE_DESCRIPTOR
          ntdll!_IO_RESOURCE_LIST
          ntdll!_FILE_NETWORK_OPEN_INFORMATION
          ntdll!_CM_FULL_RESOURCE_DESCRIPTOR
          ntdll!_CM_RESOURCE_LIST
          ntdll!_POWER_SEQUENCE
          ntdll!_IO_RESOURCE_DESCRIPTOR
          ntdll!_FS_FILTER_SECTION_SYNC_TYPE
          ntdll!_FS_FILTER_PARAMETERS
          ntdll!_COMPRESSED_DATA_INFO
          ntdll!_FILE_STANDARD_INFORMATION
          ntdll!_DESCRIPTOR
          ntdll!_GUID
          ntdll!_SID_IDENTIFIER_AUTHORITY
          ntdll!_SID
          ntdll!_SID_IDENTIFIER_AUTHORITY
          ntdll!_CM_PARTIAL_RESOURCE_LIST
          ntdll!_CM_FULL_RESOURCE_DESCRIPTOR
          ntdll!_DEVICE_CAPABILITIES
          ntdll!_CM_PARTIAL_RESOURCE_DESCRIPTOR
          ntdll!_CM_PARTIAL_RESOURCE_LIST
          ntdll!_CM_PARTIAL_RESOURCE_DESCRIPTOR
          ntdll!__unnamed

You might have noticed that many structures are listed twice in the output. Actually all of them appear twice and there are many __unnamed (I edited the output before posting to save space). I was wondering why they are listed twice and after some research I found that Visual Studio contains DIA SDK (Debug Interface Access SDK) and you can build DIA2Dump sample to dump PDB files. Unfortunately this tool displays them twice too without any hints:

UDT            : LIST_ENTRY32
Data           :   this+0×0, Member, Type: unsigned long, Flink
Data           :   this+0×4, Member, Type: unsigned long, Blink
UDT            : LIST_ENTRY32
Data           :   this+0×0, Member, Type: unsigned long, Flink
Data           :   this+0×4, Member, Type: unsigned long, Blink

__unnamed datatype is for unions, for example:

0:000> dt -r _ULARGE_INTEGER
   +0x000 LowPart          : Uint4B
   +0x004 HighPart         : Uint4B
   +0x000 u                : __unnamed
      +0×000 LowPart          : Uint4B
      +0×004 HighPart         : Uint4B
   +0×000 QuadPart         : Uint8B

Here’s the definition taken from winnt.h:

typedef union _ULARGE_INTEGER
{
   struct
   {
      DWORD LowPart;
      DWORD HighPart;
   };
   struct
   {
      DWORD LowPart;
      DWORD HighPart;
   } u;
   ULONGLONG QuadPart;
} ULARGE_INTEGER, *PULARGE_INTEGER;

- Dmitry Vostokov -

Crash Dump Analysis Patterns (Part 9a)

Friday, February 9th, 2007

Next pattern is Deadlock. If you don’t know what “deadlock” is read Dumps for Dummes (Part 4). Deadlocks do not only happen with synchronization primitives like mutexes, events or more complex objects (built upon primitives) like critical sections or executive resources (ERESOURCE). They can happen from high level or systems perspective in inter-process or inter-component communication, for example, mutually waiting on messages: GUI window messages, LPC messages, RPC calls. This is a big pattern and I’m going to split it into several parts.

How can we see deadlocks in dumps? Let’s start with user dumps and critical sections.

First I would recommend to read the following excellent MSDN article to understand various members of CRITICAL_SECTION structure:

Break Free of Code Deadlocks in Critical Sections Under Windows

WinDbg !locks command will examine process critical section list and display all locked critical sections, lock count and thread id of current critical section owner. This is the output from a dump of hanging Windows print spooler process (spoolsv.exe):

0:000> !locks
CritSec NTDLL!LoaderLock+0 at 784B0348
LockCount          4
RecursionCount     1
OwningThread       624
EntryCount         6c3
ContentionCount    6c3
*** Locked

CritSec LOCALSPL!SpoolerSection+0 at 76AB8070
LockCount          3
RecursionCount     1
OwningThread       1c48
EntryCount         646
ContentionCount    646
*** Locked

If we look at threads #624 and #1c48 we could see them mutually waiting for each other:

  • TID#624 owns CritSec 784B0348 and is waiting for CritSec 76AB8070

  • TID#1c48 owns CritSec 76AB8070 and is waiting for CritSec 784B0348

0:000>~*kv

. 12 Id: bc0.624 Suspend: 1 Teb: 7ffd3000 Unfrozen
0000024c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
76ab8000 76a815ef 76ab8070 NTDLL!RtlpWaitForCriticalSection+0×9e
76ab8070 76a844f8 00cd1f38 NTDLL!RtlEnterCriticalSection+0×46
00cd1f38 76a8a1d7 00000000 LOCALSPL!EnterSplSem+0xb
00000000 00000000 00cd1f38 LOCALSPL!FindSpoolerByNameIncRef+0×1f
00000000 777f19bc 00000001 LOCALSPL!LocalGetPrinterDriverDirectory+0xe
00000000 777f19bc 00000001 spoolss!GetPrinterDriverDirectoryW+0×59
00000000 777f19bc 00000001 spoolsv!YGetPrinterDriverDirectory+0×27
00000000 777f19bc 00000001 WINSPOOL!GetPrinterDriverDirectoryW+0×7b
50000000 00000001 00000000 BRHLUI04+0×14ea
50002ea0 50000000 00000001 BRHLUI04!DllGetClassObject+0×1705
00000000 00000000 000cb570 NTDLL!LdrpRunInitializeRoutines+0×1df
000cc8f8 0288ea30 0288ea38 NTDLL!LdrpLoadDll+0×2e6
000cc8f8 0288ea30 0288ea38 NTDLL!LdrLoadDll+0×17)
000c1258 00000000 00000008 KERNEL32!LoadLibraryExW+0×231
000c150c 0288efd8 00000000 UNIDRVUI!PLoadCommonInfo+0×17e
000c150c 0288efd8 00000007 UNIDRVUI!DwDeviceCapabilities+0×1a
00070000 00071378 00000045 UNIDRVUI!DrvDeviceCapabilities+0×19

. 13 Id: bc0.1c48 Suspend: 1 Teb: 7ffd2000 Unfrozen
0000010c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
784b0301 78468d38 784b0348 NTDLL!RtlpWaitForCriticalSection+0×9e
784b0348 74fb4344 00000000 NTDLL!RtlEnterCriticalSection+0×46
74fb0000 02c0f2a8 00000000 NTDLL!LdrpGetProcedureAddress+0×122
74fb0000 02c0f2a8 00000000 NTDLL!LdrGetProcedureAddress+0×17
74fb0000 74fb4344 02c0f449 KERNEL32!GetProcAddress+0×41
017924b0 00000000 00000001 ws2_32!CheckForHookersOrChainers+0×1f
00000101 02c0f344 017924b0 ws2_32!WSAStartup+0×10f
00cdf79c 02c0f4f4 76a8c9bc LOCALSPL!GetDNSMachineName+0×1e
00000000 76a8c9bc 780276a2 LOCALSPL!GetPrinterUrl+0×2c
0176f570 ffffffff 01000000 LOCALSPL!UpdateDsSpoolerKey+0×322
0176f570 76a8c9bc 01792b90 LOCALSPL!RecreateDsKey+0×50
00000000 00000002 01792b90 LOCALSPL!SplAddPrinter+0×521
01791faa 0176a684 76a5cd34 WIN32SPL!InternalAddPrinterConnection+0×1b4
01791faa 02c0fa00 02c0fabc WIN32SPL!AddPrinterConnectionW+0×15
00076f1c 02c0fabc 01006873 spoolss!AddPrinterConnectionW+0×49
00076f1c 00000001 77107fb0 spoolsv!YAddPrinterConnection+0×17
00076f1c 02020202 00000001 spoolsv!RpcAddPrinterConnection+0xb
01006868 02c0fac0 00000001 rpcrt4!Invoke+0×30
00000000 00000000 000d22c8 rpcrt4!NdrStubCall2+0×655
000d22c8 00076fe0 000d22c8 rpcrt4!NdrServerCall2+0×17
010045fc 000d22c8 02c0fe0c rpcrt4!DispatchToStubInC+0×32
0000002b 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×100
000d22c8 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStub+0×5e
000d3210 00076608 813b0013 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×1dd
000d21d0 02c0fe50 000d3210 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×10c
770c9ad0 00076608 770cb6d8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×229
00076608 770cb6d8 0288f9a8 rpcrt4!RecvLotsaCallsWrapper+0×9
00074a50 02c0ffec 77e7438b rpcrt4!BaseCachedThreadRoutine+0×11f
00076e68 770cb6d8 0288f9a8 rpcrt4!ThreadStartRoutine+0×18
770d1c54 00076e68 00000000 KERNEL32!BaseThreadStart+0×52

This analysis looks pretty simple and easy. What about kernel and complete memory dumps? Of course we cannot see user space critical sections in kernel memory dumps but we can see them in complete memory dumps after switching to appropriate process context and using !ntsdexts.locks. This can be done via simple script adapted from debugger.chm: Deadlocks and Critical Sections

Why it is so easy to see deadlocks when critical sections are involved? Because their structures have a member that records their owner. So it is very easy to map them to corresponding threads. The same is with kernel ERESOURCE synchronization objects (we will see them in the next part). Other objects do not have an owner, for example, in case of events it is not so easy to find an owner just by looking at an event object. You need to examine thread call stacks, other structures or have access to source code.

- Dmitry Vostokov @ DumpAnalysis.org -

Asmpedia.org update (2007, Week 6)

Tuesday, February 6th, 2007

Added EFLAGS/RFLAGS template:

http://www.asmpedia.org/index.php?title=EFLAGS/RFLAGS

AAA, MOV, NOP instructions have been updated to include EFLAGS (all other instructions will have it automatically):

Added 16-bit addressing ModRegRM table (for the sake of completeness):

http://www.asmpedia.org/index.php?title=ModRegRM_byte_%2816-bit_addressing%29

Added SIB byte translation table:

http://www.asmpedia.org/index.php?title=SIB_byte_%2832/64-bit_addressing%29

- Dmitry Vostokov -

Dumping Vista

Sunday, February 4th, 2007

32-bit Vista

If you need to dump a running 32-bit Vista system you can do it with Citrix SystemDump tool. You just need to run it with elevated administrator rights:

  • right click SystemDump.exe in appropriate Computer explorer folder and choose “Run as administrator”

  • if you use command line options run SystemDump.exe from elevated command prompt (Start -> All Programs -> Accessories, right click Command Prompt, and then select “Run as administrator”)

Here is a screenshot before dumping my Vista and WinDbg output from saved kernel dump:

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Windows Vista Kernel Version 6000 UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 6000.16386.x86fre.vista_rtm.061101-2205
Kernel base = 0x81800000 PsLoadedModuleList = 0x81908ab0
Debug session time: Sat Jan 27 20:13:10.917 2007 (GMT+0)
System Uptime: 0 days 1:33:13.589
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
BugCheck E2, {cccccccc, 83286f08, 1a, 0}
Probably caused by : SystemDump.sys

64-bit Vista

Currently in order to use 64-bit SystemDump you have to disable “Driver Signature Enforcement” by:

  • F8 Advanced Boot Option
  • command line tool BCDedit
  • attaching an active kernel debugger

Then you need to run SystemDump64.exe as administrator.

- Dmitry Vostokov -

Crash Dump Analysis Patterns (Part 8)

Friday, February 2nd, 2007

Today I will talk about another pattern occurring frequently and I call it Hidden Exception. You run !analyze -v command and you don’t see an exception or you see only a breakpoint hit. In this case manual analysis is required. This happens sometimes because of another pattern: Multiple Exceptions. In other cases an exception happens and it is handled by an exception handler dismissing it and a process continues execution slowly accumulating corruption inside its data leading to a new crash or hang. Sometimes you see a process hanging during its termination like the case I present here.

We have a process dump with only one thread:

0:000> kv
ChildEBP RetAddr
0096fcdc 7c822124 ntdll!KiFastSystemCallRet
0096fce0 77e6baa8 ntdll!NtWaitForSingleObject+0xc
0096fd50 77e6ba12 kernel32!WaitForSingleObjectEx+0xac
0096fd64 67f016ce kernel32!WaitForSingleObject+0x12
0096fd78 7c82257a component!DllInitialize+0xc2
0096fd98 7c8118b0 ntdll!LdrpCallInitRoutine+0x14
0096fe34 77e52fea ntdll!LdrShutdownProcess+0x130
0096ff20 77e5304d kernel32!_ExitProcess+0x43
0096ff34 77bcade4 kernel32!ExitProcess+0x14
0096ff40 77bcaefb msvcrt!__crtExitProcess+0x32
0096ff70 77bcaf6d msvcrt!_cinit+0xd2
0096ff84 77bcb555 msvcrt!_exit+0x11
0096ffb8 77e66063 msvcrt!_endthreadex+0xc8
0096ffec 00000000 kernel32!BaseThreadStart+0x34

We can look at its raw stack and try to find the following address:

KiUserExceptionDispatcher

This function calls RtlDispatchException:

0:000> !teb
TEB at 7ffdc000
    ExceptionList:        0096fd40
    StackBase:            00970000
    StackLimit:           0096a000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdc000
    EnvironmentPointer:   00000000
    ClientId:             00000858 . 000008c0
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffdd000
    LastErrorValue:       0
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000>dds 0096a000 00970000
...
...
...
0096c770  7c8140cc ntdll!RtlDispatchException+0x91
0096c774  0096c808
0096c778  0096ffa8
0096c77c  0096c824
0096c780  0096c7e4
0096c784  77bc6c74 msvcrt!_except_handler3
0096c788  00000000
0096c78c  0096c808
0096c790  01030064
0096c794  00000000
0096c798  00000000
0096c79c  00000000
0096c7a0  00000000
0096c7a4  00000000
0096c7a8  00000000
0096c7ac  00000000
0096c7b0  00000000
0096c7b4  00000000
0096c7b8  00000000
0096c7bc  00000000
0096c7c0  00000000
0096c7c4  00000000
0096c7c8  00000000
0096c7cc  00000000
0096c7d0  00000000
0096c7d4  00000000
0096c7d8  00000000
0096c7dc  00000000
0096c7e0  00000000
0096c7e4  00000000
0096c7e8  00970000
0096c7ec  00000000
0096c7f0  0096caf0
0096c7f4  7c82ecc6 ntdll!KiUserExceptionDispatcher+0xe
0096c7f8  0096c000
0096c7fc  0096c824 ; a pointer to an exception context
0096c800  0096c808
0096c804  0096c824
0096c808  c0000005
0096c80c  00000000
0096c810  00000000
0096c814  77bd8df3 msvcrt!wcschr+0×15
0096c818  00000002
0096c81c  00000000
0096c820  01031000
0096c824  0001003f
0096c828  00000000
0096c82c  00000000
0096c830  00000000
0096c834  00000000
0096c838  00000000
0096c83c  00000000

A second parameter to both functions is a pointer to a so called exception context (processor state when an exception occurred). We can use .cxr command to change thread execution context to what it was at the time of exception:

After changing the context we can see thread stack prior to that exception:

0:000> kL
ChildEBP RetAddr
0096caf0 67b11808 msvcrt!wcschr+0×15
0096cb10 67b1194d component2!function1+0×50
0096cb24 67b11afb component2!function2+0×1a
0096eb5c 67b11e10 component2!function3+0×39
0096ed94 67b14426 component2!function4+0×155
0096fdc0 67b164b7 component2!function5+0×3b
0096fdcc 00402831 component2!function6+0×5b
0096feec 0096ff14 program!function+0×1d1
0096ffec 00000000 kernel32!BaseThreadStart+0×34

We see that the exception happened when component2 was searching a Unicode string for a character (wcschr). Most likely the string was not zero terminated:

To summarize and show you the common exception handling path in user space here is another thread stack taken from a different dump:

ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+0xc
kernel32!UnhandledExceptionFilter+0×746
kernel32!_except_handler3+0×61
ntdll!ExecuteHandler2+0×26
ntdll!ExecuteHandler+0×24
ntdll!RtlDispatchException+0×91
ntdll!KiUserExceptionDispatcher+0xe
ntdll!RtlpCoalesceFreeBlocks+0×36e ; crash is here
ntdll!RtlFreeHeap+0×38e
msvcrt!free+0xc3
msvcrt!_freefls+0×124
msvcrt!_freeptd+0×27
msvcrt!__CRTDLL_INIT+0×1da
ntdll!LdrpCallInitRoutine+0×14
ntdll!LdrShutdownThread+0xd2
kernel32!ExitThread+0×2f
kernel32!BaseThreadStart+0×39

When RtlpCoalesceFreeBlocks (this function compacts heap and it is called from RtlFreeHeap) does an illegal memory access then this exception is first processed in kernel and because it happened in user space and mode the execution is transferred to RtlDispatchException which searches for exception handler and in this case there is a default one installed: UnhandledExceptionFilter.

If you see this function on call stack you can also manually get an exception context and a thread stack leading to it like in this example below taken from other dump:

The most likely reason of this crash is an instance of Dynamic Memory Corruption pattern - heap corruption.

- Dmitry Vostokov -