Software Diagnostics Library

If you have user mode stack trace DB enabled on Windows 2003 for some service or application (here is an example for Citrix IMA service) and if you get a dump and try to get saved stack traces using !heap extension command you get these errors:

0:000> !heap -k -h 000a0000
    Heap entries for Segment00 in Heap 000a0000
        000a0c50: 00c50 . 00040 [01] - busy (40)
        000a0c90: 00040 . 01818 [07] - busy (1800), tail fill - unable to read heap entry extra at 000a24a0
        000a24a8: 01818 . 00030 [07] - busy (18), tail fill - unable to read heap entry extra at 000a24d0
        000a24d8: 00030 . 005a0 [07] - busy (588), tail fill - unable to read heap entry extra at 000a2a70

The solution is to use Windows 2000 extension ntsdexts.dll:

0:000> !.\w2kfre\ntsdexts.heap -k -h 000a0000
Stack trace (12) at 1021bfc:
   7c85fc22: ntdll!RtlAllocateHeapSlowly+0×00000041
   7c81d4df: ntdll!RtlAllocateHeap+0×00000E9F
   7c83467a: ntdll!LdrpAllocateUnicodeString+0×00000035
   7c8354f4: ntdll!LdrpCopyUnicodeString+0×00000031
   7c83517b: ntdll!LdrpResolveDllName+0×00000195
   7c834b2a: ntdll!LdrpMapDll+0×0000014F
   7c837474: ntdll!LdrpLoadImportModule+0×0000017C
   7c837368: ntdll!LdrpHandleOneNewFormatImportDescriptor+0×0000004D
   7c837317: ntdll!LdrpHandleNewFormatImportDescriptors+0×0000001D
   7c837441: ntdll!LdrpWalkImportDescriptor+0×00000195
   7c80f560: ntdll!LdrpInitializeProcess+0×00000E3E
   7c80ea0b: ntdll!_LdrpInitialize+0×000000D0
   7c82ec2d: ntdll!KiUserApcDispatcher+0×00000025

- Dmitry Vostokov @ DumpAnalysis.org -

Consider the following legacy C++/Win32 code fragment highlighted in WinDbg after opening a crash dump:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
5:    DWORD dwRead = 0;
6:    CHAR *bufferA = new CHAR[dwSize+2];
7:    memset(bufferA, 0, dwSize+2);
8:    if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:    {
10:      DWORD i = 0, j = 0;
11:      for (; i < dwSize+2-7; ++i)
12:      {
13:         if (bufferA[i] == 0xD && bufferA[i+1] != 0xA)

At the first glance the code seems to be right: we open a file, get its size, allocate a buffer to read, etc. All loop indexes are within array bounds too. Let’s look at disassembly and crash point:

0:000> uf component!CMyDlg::OnTimer
…
…
…
004021bc push    0
004021be push    esi
004021bf call    dword ptr [component!_imp__GetFileSize (0042e26c)]
004021c5 mov     edi,eax ; dwSize
004021c7 lea     ebx,[edi+2] ; dwSize+2
004021ca push    ebx
004021cb mov     dword ptr [esp+34h],0
004021d3 call    component!operator new[] (00408e35)
004021d8 push    ebx
004021d9 mov     ebp,eax ; bufferA
004021db push    0
004021dd push    ebp
004021de call    component!memset (00418500)
004021e3 add     esp,10h
004021e6 push    0
004021e8 lea     edx,[esp+34h]
004021ec push    edx
004021ed push    edi
004021ee push    ebp
004021ef push    esi
004021f0 call    dword ptr [component!_imp__ReadFile (0042e264)]
004021f6 test    eax,eax
004021f8 jne     component!CMyDlg::OnTimer+0×3b1 (00402331)
…
…
…
00402331 xor     esi,esi ; i
00402333 add     edi,0FFFFFFFBh ; +2-7 (edi contains dwSize)
00402336 cmp     edi,esi ; loop condition
00402338 mov     dword ptr [esp+24h],esi
0040233c jbe     component!CMyDlg::OnTimer+0×43e (004023be)
00402342 mov     al,byte ptr [esi+ebp] ; bufferA[i]

0:000> r
eax=00002b00 ebx=00000002 ecx=00431000 edx=00000000 esi=00002b28 edi=fffffffb
eip=00402342 esp=0012efd4 ebp=0095b4d8 iopl=0 nv up ei pl nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000217
component!CMyDlg::OnTimer+0×3c2:
00402342 8a042e mov al,byte ptr [esi+ebp] ds:0023:0095e000=??

If we look at ebx (dwSize+2) and edi registers (array upper bound, dwSize+2-7) we can easily see that dwSize was zero. Clearly we had buffer overrun because upper array bound was calculated as 0+2-7 = FFFFFFFB (the loop index was unsigned integer, DWORD). Were the index signed integer variable (int) we wouldn’t have had any problem because the condition 0 < 0+2-7 is always false and the loop body would have never been executed.

Based on that the following fix was proposed:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
5:    DWORD dwRead = 0;
6:    CHAR *bufferA = new CHAR[dwSize+2];
7:    memset(bufferA, 0, dwSize+2);
8:    if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:    {
10:      DWORD i = 0, j = 0;
10:      int i = 0, j = 0;
11:      for (; i < dwSize+2-7; ++i)
11:      for (; i < (int)dwSize+2-7; ++i)
12:      {


GetFileSize can return INVALID_FILE_SIZE (0xFFFFFFFF) and operator new can fail theoretically (if the size is too big) so we can correct the code even further:

1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
2: if (hFile != INVALID_HANDLE_VALUE)
3: {
4:    DWORD dwSize = GetFileSize(hFile, NULL);
4a:   if (dwSize != INVALID_FILE_SIZE)
4b:   {
5:       DWORD dwRead = 0;
6:       CHAR *bufferA = new CHAR[dwSize+2];
6a:      if (bufferA)
6b:      {
7:          memset(bufferA, 0, dwSize+2);
8:          if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL))
9:          {
10:            int i = 0, j = 0;
11:            for (; i < (int)dwSize+2-7; ++i)
12:            {


- Dmitry Vostokov @ DumpAnalysis.org -

Techniques utilizing user mode process dumpers and debuggers like Microsoft userdump.exe, NTSD or WinDbg and CDB from Debugging Tools for Windows are too slow to pick up a process and dump it. You need either to attach a debugger manually, run the command line prompt or switch to Task Manager. This deficiency was the primary motivation for me to use JIT (just-in-time) technology for process dumpers. The new tool, InstantDump, will dump a process instantly and non-invasively in a moment when you need it. How does it work? You point to any window and press hot key.

InstantDump could be useful to study hang GUI processes or to get several dumps of the same process during some period of time (CPU spiking case or memory leak, for example) or just dump the process for the sake of dumping it (for curiosity). The tool uses the same tooltip technology introduced in WindowHistory 4.0 to dynamically display window information.

Short user guide:

1. The program will run only on XP/W2K3/Vista (in fact it will not load on W2K).

2. Run InstantDump.exe on 32-bit system or InstantDump64.exe on x64 Windows. If you attempt to run InstantDump.exe on x64 Windows it will show this message box and quit:

3. InstantDump puts itself into task bar icon notification area:

4. By default when you move the mouse pointer over windows the tooltip follows the cursor describing the process and thread id and process image path (you can disable tips in Options dialog box):

5. If you hold Ctrl-RightShift-Break for less than a second then the process (which window is under the cursor) will be dumped according to the settings for external process dumper in options dialog (accessible via task bar icon right mouse click):

The saved dump name will be (in our Calculator window case): calc.exe_9f8(2552)_22-17-56_18-Feb-2007.dmp

Looks like there is no NTSD in Vista so you have to use another user mode dumper, for example, install MS userdump.exe and specify the following command line in Options dialog:

userdump.exe %d %s

or resort to WinDbg or CDB command line.

The tool can be downloaded from here.

The new version of this tool is under development that will automatically pick up a process name from Task Manager, Process Explorer or Process Monitor (in fact, from any tool that displays the list of processes) and then instantly dump it.

- Dmitry Vostokov -

Today I was curious about what savedump.exe does. And after some research I found the following:

On Windows 2000 it is a part of the logon process where it copies the crash dump section from the pagefile to a memory dump and also creates a mini dump file.

http://support.microsoft.com/kb/257299

http://support.microsoft.com/kb/262077

On Windows 2003 and XP savedump.exe doesn’t touch the pagefile. Session manager smss.exe truncates the pagefile and renames it to dumpxxx.dmp file (copying it to the boot volume if necessary). Then savedump.exe copies that file to the correct location and then creates the mini dump file.

http://support.microsoft.com/kb/886429

Also I couldn’t find savedump.exe on Vista. It looks like it was finally removed.

- Dmitry Vostokov -

I recently discovered in WinDbg help that dt command can be used for traversing linked lists. Most structures I work with have LIST_ENTRY as their first member and it is much easier to use dt command than !list (less typing).  For example:

0:000> dt _MYBIGSTRUCTURE
   +0x000 Links : _LIST_ENTRY
    ...
   +0x080 SomeName : [33] Uint2B

0:000> dd component!MyBigStructureListHead l1
01022cd0  0007fe58

0:000> .enable_unicode 1

The following command outputs the whole list of structures:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink 0007fe58

And the following command outputs the list of SomeName members:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink -y SomeName 0007fe58
Links.Flink at 0×7fe58
   +0×000 Links :  [ 0×8e090 - 0×1022cd0 ]
   +0×080 SomeName : [33]  “Foo”
Links.Flink at 0×8e090
   +0×000 Links :  [ 0×913f8 - 0×7fe58 ]
   +0×080 SomeName : [33]  “Bar”

If you don’t remember exact member name you can specify the partial name and any member that matches will be shown:

0:000> dt _MYBIGSTRUCTURE -l Links.Flink -y S 0007fe58

However it your structure doesn’t have LIST_ENTRY as its first member then you need to subtract its offset, for example:

kd> dd nt!PsActiveProcessHead l1
808af068  85fa48b0

kd> dt _EPROCESS
   +0x000 Pcb              : _KPROCESS
   +0x078 ProcessLock      : _EX_PUSH_LOCK
   +0x080 CreateTime       : _LARGE_INTEGER
   +0x088 ExitTime         : _LARGE_INTEGER
   +0x090 RundownProtect   : _EX_RUNDOWN_REF
   +0x094 UniqueProcessId  : Ptr32 Void
   +0×098 ActiveProcessLinks : _LIST_ENTRY

kd> dt _EPROCESS -l ActiveProcessLinks.Flink -y ImageFileName 85fa48b0-0×98
ActiveProcessLinks.Flink at 0×85fa4818
   +0×098 ActiveProcessLinks :  [ 0×85d1ce20 - 0×808af068 ]
   +0×164 ImageFileName : [16]  “System”
ActiveProcessLinks.Flink at 0×85d1cd88
   +0×098 ActiveProcessLinks :  [ 0×85dba6b8 - 0×85fa48b0 ]
   +0×164 ImageFileName : [16]  “smss.exe”
ActiveProcessLinks.Flink at 0×85dba620
   +0×098 ActiveProcessLinks :  [ 0×858d20b8 - 0×85d1ce20 ]
   +0×164 ImageFileName : [16]  “csrss.exe”
ActiveProcessLinks.Flink at 0×858d2020
   +0×098 ActiveProcessLinks :  [ 0×858c20b8 - 0×85dba6b8 ]
   +0×164 ImageFileName : [16]  “winlogon.exe”
ActiveProcessLinks.Flink at 0×858c2020
   +0×098 ActiveProcessLinks :  [ 0×8589f0b8 - 0×858d20b8 ]
   +0×164 ImageFileName : [16]  “services.exe”

Here is another example, not involving LIST_ENTRY but rather a classic single list forward pointer: 

0:000> !teb
TEB at 7FFDE000
    ExceptionList:    6fc54
    Stack Base:       70000
    Stack Limit:      6d000
    SubSystemTib:     0
    FiberData:        1e00
    ArbitraryUser:    0
    Self:             7ffde000
    EnvironmentPtr:   0
    ClientId:         22c.228
    Real ClientId:    22c.228
    RpcHandle:        0
    Tls Storage:      742b8
    PEB Address:      7ffdf000
    LastErrorValue:   997
    LastStatusValue:  103
    Count Owned Locks:0
    HardErrorsMode:   0

0:000> dt -r _TEB
   +0x000 NtTib : _NT_TIB
      +0x000 ExceptionList : Ptr32 _EXCEPTION_REGISTRATION_RECORD
         +0×000 Next : Ptr32 _EXCEPTION_REGISTRATION_RECORD
         +0×004 Handler : Ptr32
      +0×004 StackBase : Ptr32 Void
      +0×008 StackLimit : Ptr32 Void
      +0×00c SubSystemTib : Ptr32 Void
      +0×010 FiberData : Ptr32 Void
      +0×010 Version : Uint4B
      +0×014 ArbitraryUserPointer : Ptr32 Void
      +0×018 Self : Ptr32 _NT_TIB

0:000> dt _EXCEPTION_REGISTRATION_RECORD -l Next 7FFDE000
Next at 0x7ffde000
   +0x000 Next : 0x0006fc54 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x00070000 +70000
Next at 0x6fc54
   +0x000 Next : 0x0006fcfc _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c5c1f44 KERNEL32!_except_handler3+0
Next at 0x6fcfc
   +0x000 Next : 0x0006ff5c _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c2e5649 ADVAPI32!_except_handler3+0
Next at 0x6ff5c
   +0x000 Next : 0x0006ffb0 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c2e5649 ADVAPI32!_except_handler3+0
Next at 0x6ffb0
   +0x000 Next : 0x0006ffe0 _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x01015878 component!_except_handler3+0
Next at 0x6ffe0
   +0x000 Next : 0xffffffff _EXCEPTION_REGISTRATION_RECORD
   +0x004 Handler : 0x7c5c1f44 KERNEL32!_except_handler3+0

- Dmitry Vostokov -

If you open a user crash dump as a solution/project, not a as file then you can do crash dump analysis by using Visual Studio debug windows, for example, powerful Watch window. As you can see from the picture I loaded a crash dump from my test application saved by NTSD and it shows assembly code and source code nicely interleaved.

If you need to specify additional symbol paths or symbol server settings you can do it in Tools \ Options \ Debugging \ Symbols dialog.

This might be good for you if you are used to do live debugging in Visual Studio and have no experience in using WinDbg and you need to look at your application crash dumps sent by customers.

- Dmitry Vostokov -

It happens sometimes that during crash dump analysis or debugging session I forget exact structure name when I want to use it in dt WinDbg command. In this case wildcards help me: dt module!*, for example,

0:000> dt ntdll!*
          ntdll!LIST_ENTRY64
          ntdll!LIST_ENTRY32
          ntdll!_ULARGE_INTEGER
          ntdll!_LIST_ENTRY
          ntdll!_IMAGE_NT_HEADERS
          ntdll!_IMAGE_FILE_HEADER
          ntdll!_IMAGE_OPTIONAL_HEADER
          ntdll!_IMAGE_NT_HEADERS
          ntdll!_LARGE_INTEGER
          ntdll!_LUID
          ntdll!_KPRCB
          ntdll!_KTHREAD
          ntdll!_KPROCESSOR_STATE
          ntdll!_KSPIN_LOCK_QUEUE
          ntdll!_KNODE
          ntdll!_PP_LOOKASIDE_LIST
          ntdll!_KPRCB
          ntdll!_KDPC_DATA
          ntdll!_KEVENT
          ntdll!_KDPC
          ntdll!_SINGLE_LIST_ENTRY
          ntdll!_FX_SAVE_AREA
          ntdll!_PROCESSOR_POWER_STATE
          ntdll!_KPRCB
          ntdll!_KPCR
          ntdll!_NT_TIB
          ntdll!_EXCEPTION_REGISTRATION_RECORD
          ntdll!_KIDTENTRY
          ntdll!_KGDTENTRY
          ntdll!_KTSS
          ntdll!_KPCR
          ntdll!_KAPC
          ntdll!_SINGLE_LIST_ENTRY
          ntdll!_KDPC_IMPORTANCE
          ntdll!_KDPC
          ntdll!_DISPATCHER_HEADER
          ntdll!_KAPC_STATE
          ntdll!_KWAIT_BLOCK
          ntdll!_KGATE
          ntdll!_KQUEUE
          ntdll!_KTIMER
          ntdll!_KTRAP_FRAME
          ntdll!_KPROCESS
          ntdll!_KSEMAPHORE
          ntdll!_KTHREAD
          ntdll!_KSPIN_LOCK_QUEUE_NUMBER
          ntdll!_FAST_MUTEX
          ntdll!_SLIST_HEADER
          ntdll!_NPAGED_LOOKASIDE_LIST
          ntdll!_GENERAL_LOOKASIDE
          ntdll!_NPAGED_LOOKASIDE_LIST
          ntdll!_PAGED_LOOKASIDE_LIST
          ntdll!_PP_NPAGED_LOOKASIDE_NUMBER
          ntdll!_POOL_TYPE
          ntdll!_GENERAL_LOOKASIDE
          ntdll!_EX_RUNDOWN_REF
          ntdll!_EX_FAST_REF
          ntdll!_EX_PUSH_LOCK
          ntdll!_EX_PUSH_LOCK_WAIT_BLOCK
          ntdll!_EX_PUSH_LOCK_CACHE_AWARE
          ntdll!_ETHREAD
          ntdll!_TERMINATION_PORT
          ntdll!_CLIENT_ID
          ntdll!_PS_IMPERSONATION_INFORMATION
          ntdll!_DEVICE_OBJECT
          ntdll!_EPROCESS
          ntdll!_ETHREAD
          ntdll!_HANDLE_TABLE
          ntdll!_KGUARDED_MUTEX
          ntdll!_MM_AVL_TABLE
          ntdll!_EJOB
          ntdll!_EPROCESS_QUOTA_BLOCK
          ntdll!_PAGEFAULT_HISTORY
          ntdll!_HARDWARE_PTE_X86
          ntdll!_PEB
          ntdll!_SE_AUDIT_PROCESS_CREATION_INFO
          ntdll!_MMSUPPORT
          ntdll!_EPROCESS
          ntdll!_OBJECT_HEADER
          ntdll!_OBJECT_TYPE
          ntdll!_OBJECT_CREATE_INFORMATION
          ntdll!_QUAD
          ntdll!_OBJECT_HEADER
          ntdll!_OBJECT_HEADER_QUOTA_INFO
          ntdll!_OBJECT_HEADER_HANDLE_INFO
          ntdll!_OBJECT_HANDLE_COUNT_DATABASE
          ntdll!_OBJECT_HANDLE_COUNT_ENTRY
          ntdll!_OBJECT_HEADER_HANDLE_INFO
          ntdll!_OBJECT_HEADER_NAME_INFO
          ntdll!_OBJECT_DIRECTORY
          ntdll!_UNICODE_STRING
          ntdll!_OBJECT_HEADER_NAME_INFO
          ntdll!_OBJECT_HEADER_CREATOR_INFO
          ntdll!_OBJECT_ATTRIBUTES
          ntdll!_ERESOURCE
          ntdll!_OBJECT_TYPE_INITIALIZER
          ntdll!_OBJECT_TYPE
          ntdll!_OBJECT_HANDLE_INFORMATION
          ntdll!_PERFINFO_GROUPMASK
          ntdll!_KGUARDED_MUTEX
          ntdll!_DISPATCHER_HEADER
          ntdll!_PF_SCENARIO_TYPE
          ntdll!_HANDLE_TRACE_DEBUG_INFO
          ntdll!_HANDLE_TABLE
          ntdll!_KWAIT_BLOCK
          ntdll!_MMSUPPORT_FLAGS
          ntdll!_MMWSL
          ntdll!_MMSUPPORT
          ntdll!_EPROCESS_QUOTA_ENTRY
          ntdll!_EPROCESS_QUOTA_BLOCK
          ntdll!_UNICODE_STRING
          ntdll!_NT_TIB
          ntdll!_PS_JOB_TOKEN_FILTER
          ntdll!_IO_COUNTERS
          ntdll!_EJOB
          ntdll!_PEB_LDR_DATA
          ntdll!_RTL_USER_PROCESS_PARAMETERS
          ntdll!_RTL_CRITICAL_SECTION
          ntdll!_PEB_FREE_BLOCK
          ntdll!_ACTIVATION_CONTEXT_DATA
          ntdll!_ASSEMBLY_STORAGE_MAP
          ntdll!_PEB
          ntdll!_KGATE
          ntdll!_IMAGE_FILE_HEADER
          ntdll!_RTL_STACK_TRACE_ENTRY
          ntdll!_PEB_FREE_BLOCK
          ntdll!_KSPIN_LOCK_QUEUE
          ntdll!_PP_LOOKASIDE_LIST
          ntdll!_KEXECUTE_OPTIONS
          ntdll!_KPROCESS
          ntdll!_PEB_LDR_DATA
          ntdll!_DPH_BLOCK_INFORMATION
          ntdll!_SECURITY_IMPERSONATION_LEVEL
          ntdll!_PS_IMPERSONATION_INFORMATION
          ntdll!_EPROCESS_QUOTA_ENTRY
          ntdll!_FNSAVE_FORMAT
          ntdll!_FX_SAVE_AREA
          ntdll!PROCESSOR_IDLE_TIMES
          ntdll!PROCESSOR_PERF_STATE
          ntdll!_PROCESSOR_POWER_STATE
          ntdll!_IO_COUNTERS
          ntdll!_KiIoAccessMap
          ntdll!_KTSS
          ntdll!_KIDTENTRY
          ntdll!_MMSUPPORT_FLAGS
          ntdll!_HEAP
          ntdll!_HEAP_ENTRY
          ntdll!_HEAP_TAG_ENTRY
          ntdll!_HEAP_UCR_SEGMENT
          ntdll!_HEAP_UNCOMMMTTED_RANGE
          ntdll!_HEAP_SEGMENT
          ntdll!_HEAP_PSEUDO_TAG_ENTRY
          ntdll!_HEAP_LOCK
          ntdll!_HEAP
          ntdll!_TERMINATION_PORT
          ntdll!LSA_FOREST_TRUST_RECORD_TYPE
          ntdll!_HEAP_UNCOMMMTTED_RANGE
          ntdll!_OBJECT_HANDLE_COUNT_DATABASE
          ntdll!_FNSAVE_FORMAT
          ntdll!PROCESSOR_PERF_STATE
          ntdll!PROCESSOR_IDLE_TIMES
          ntdll!_HANDLE_TRACE_DB_ENTRY
          ntdll!_HANDLE_TRACE_DEBUG_INFO
          ntdll!_PROCESS_WS_WATCH_INFORMATION
          ntdll!_PAGEFAULT_HISTORY
          ntdll!_SECURITY_QUALITY_OF_SERVICE
          ntdll!_OBJECT_CREATE_INFORMATION
          ntdll!_MMADDRESS_NODE
          ntdll!_MM_AVL_TABLE
          ntdll!_HARDWARE_PTE_X86
          ntdll!_HEAP_ENTRY
          ntdll!_GENERIC_MAPPING
          ntdll!_OBJECT_DUMP_CONTROL
          ntdll!_OB_OPEN_REASON
          ntdll!_ACCESS_STATE
          ntdll!_SECURITY_OPERATION_CODE
          ntdll!_OBJECT_NAME_INFORMATION
          ntdll!_OBJECT_TYPE_INITIALIZER
          ntdll!_LARGE_INTEGER
          ntdll!_RTL_TRACE_BLOCK
          ntdll!_HEAP_UCR_SEGMENT
          ntdll!_KEXECUTE_OPTIONS
          ntdll!_OWNER_ENTRY
          ntdll!_ERESOURCE
          ntdll!_GENERIC_MAPPING
          ntdll!_SID_AND_ATTRIBUTES
          ntdll!_LUID_AND_ATTRIBUTES
          ntdll!_PS_JOB_TOKEN_FILTER
          ntdll!_MEMORY_CACHING_TYPE_ORIG
          ntdll!_KiIoAccessMap
          ntdll!_EXCEPTION_DISPOSITION
          ntdll!_EXCEPTION_RECORD
          ntdll!_CONTEXT
          ntdll!_EXCEPTION_REGISTRATION_RECORD
          ntdll!_DRIVER_OBJECT
          ntdll!_IRP
          ntdll!_IO_TIMER
          ntdll!_VPB
          ntdll!_WAIT_CONTEXT_BLOCK
          ntdll!_KDEVICE_QUEUE
          ntdll!_DEVOBJ_EXTENSION
          ntdll!_DEVICE_OBJECT
          ntdll!_PROCESS_WS_WATCH_INFORMATION
          ntdll!_SECURITY_QUALITY_OF_SERVICE
          ntdll!_FLOATING_SAVE_AREA
          ntdll!_CONTEXT
          ntdll!_IMAGE_DATA_DIRECTORY
          ntdll!_IMAGE_OPTIONAL_HEADER
          ntdll!_KUSER_SHARED_DATA
          ntdll!_KSYSTEM_TIME
          ntdll!_NT_PRODUCT_TYPE
          ntdll!_ALTERNATIVE_ARCHITECTURE_TYPE
          ntdll!_KUSER_SHARED_DATA
          ntdll!_QUAD
          ntdll!_KAPC_STATE
          ntdll!_MODE
          ntdll!_HEAP_PSEUDO_TAG_ENTRY
          ntdll!_RTL_CRITICAL_SECTION_DEBUG
          ntdll!_RTL_CRITICAL_SECTION
          ntdll!_HEAP_SEGMENT
          ntdll!_KTRAP_FRAME
          ntdll!_KGDTENTRY
          ntdll!_KDEVICE_QUEUE_ENTRY
          ntdll!_IO_ALLOCATION_ACTION
          ntdll!_WAIT_CONTEXT_BLOCK
          ntdll!_KTIMER
          ntdll!_MDL
          ntdll!_IO_STATUS_BLOCK
          ntdll!_IO_STACK_LOCATION
          ntdll!_FILE_OBJECT
          ntdll!_IRP
          ntdll!_VPB
          ntdll!_KOBJECTS
          ntdll!_KSEMAPHORE
          ntdll!_MMADDRESS_NODE
          ntdll!_CURDIR
          ntdll!_RTL_DRIVE_LETTER_CURDIR
          ntdll!_RTL_USER_PROCESS_PARAMETERS
          ntdll!_OWNER_ENTRY
          ntdll!_SE_AUDIT_PROCESS_CREATION_INFO
          ntdll!_OBJECT_HANDLE_COUNT_ENTRY
          ntdll!_CLIENT_ID
          ntdll!_RTL_TRACE_DATABASE
          ntdll!_RTL_TRACE_SEGMENT
          ntdll!_RTL_TRACE_DATABASE
          ntdll!_HEAP_LOCK
          ntdll!_HANDLE_TRACE_DB_ENTRY
          ntdll!ReplacesCorHdrNumericDefines
          ntdll!_MEMORY_TYPE
          ntdll!_IO_TIMER
          ntdll!_FXSAVE_FORMAT
          ntdll!_OBJECT_DIRECTORY_ENTRY
          ntdll!_DEVICE_MAP
          ntdll!_OBJECT_DIRECTORY
          ntdll!_STACK_TRACE_DATABASE
          ntdll!_KDPC_DATA
          ntdll!_STRING
          ntdll!_RTL_DRIVE_LETTER_CURDIR
          ntdll!_SID_AND_ATTRIBUTES
          ntdll!_DPH_HEAP_ROOT
          ntdll!_DPH_HEAP_BLOCK
          ntdll!_RTL_AVL_TABLE
          ntdll!_DPH_HEAP_ROOT
          ntdll!_DEVICE_OBJECT_POWER_EXTENSION
          ntdll!_DEVOBJ_EXTENSION
          ntdll!_FLOATING_SAVE_AREA
          ntdll!_KSYSTEM_TIME
          ntdll!_KQUEUE
          ntdll!_RTL_BALANCED_LINKS
          ntdll!_RTL_GENERIC_COMPARE_RESULTS
          ntdll!_RTL_AVL_TABLE
          ntdll!_HEAP_TAG_ENTRY
          ntdll!_RTL_CRITICAL_SECTION_DEBUG
          ntdll!_MDL
          ntdll!_DPH_HEAP_BLOCK
          ntdll!_PS_QUOTA_TYPE
          ntdll!_flags
          ntdll!_KNODE
          ntdll!_LDR_DATA_TABLE_ENTRY
          ntdll!_ACTIVATION_CONTEXT
          ntdll!_LDR_DATA_TABLE_ENTRY
          ntdll!_TEB
          ntdll!_ACTIVATION_CONTEXT_STACK
          ntdll!_GDI_TEB_BATCH
          ntdll!_TEB_ACTIVE_FRAME
          ntdll!_TEB
          ntdll!_KEVENT
          ntdll!_IO_STATUS_BLOCK
          ntdll!_RTL_TRACE_SEGMENT
          ntdll!_SECURITY_SUBJECT_CONTEXT
          ntdll!_INITIAL_PRIVILEGE_SET
          ntdll!_PRIVILEGE_SET
          ntdll!_ACCESS_STATE
          ntdll!_KSPECIAL_REGISTERS
          ntdll!_KPROCESSOR_STATE
          ntdll!_STRING
          ntdll!_flags
          ntdll!_REG_NOTIFY_CLASS
          ntdll!_OBJECT_DUMP_CONTROL
          ntdll!_SECURITY_SUBJECT_CONTEXT
          ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
          ntdll!_ACTIVATION_CONTEXT_STACK
          ntdll!_MMSYSTEM_PTE_POOL_TYPE
          ntdll!_KDEVICE_QUEUE
          ntdll!_LUID_AND_ATTRIBUTES
          ntdll!_EXCEPTION_RECORD
          ntdll!_INITIAL_PRIVILEGE_SET
          ntdll!_TEB_ACTIVE_FRAME_CONTEXT
          ntdll!_TEB_ACTIVE_FRAME
          ntdll!_OBJECT_NAME_INFORMATION
          ntdll!_SECTION_OBJECT_POINTERS
          ntdll!_IO_COMPLETION_CONTEXT
          ntdll!_FILE_OBJECT
          ntdll!_IO_COMPLETION_CONTEXT
          ntdll!_DRIVER_EXTENSION
          ntdll!_FAST_IO_DISPATCH
          ntdll!_DRIVER_OBJECT
          ntdll!_IO_CLIENT_EXTENSION
          ntdll!_FS_FILTER_CALLBACKS
          ntdll!_DRIVER_EXTENSION
          ntdll!_TEB_ACTIVE_FRAME_CONTEXT
          ntdll!_IMAGE_DATA_DIRECTORY
          ntdll!_CURDIR
          ntdll!_GDI_TEB_BATCH
          ntdll!_RTL_BALANCED_LINKS
          ntdll!_KDEVICE_QUEUE_ENTRY
          ntdll!_SECTION_OBJECT_POINTERS
          ntdll!_IO_CLIENT_EXTENSION
          ntdll!_IO_SECURITY_CONTEXT
          ntdll!_NAMED_PIPE_CREATE_PARAMETERS
          ntdll!_MAILSLOT_CREATE_PARAMETERS
          ntdll!_FILE_INFORMATION_CLASS
          ntdll!_FSINFOCLASS
          ntdll!_SCSI_REQUEST_BLOCK
          ntdll!_FILE_GET_QUOTA_INFORMATION
          ntdll!_DEVICE_RELATION_TYPE
          ntdll!_GUID
          ntdll!_INTERFACE
          ntdll!_DEVICE_CAPABILITIES
          ntdll!_IO_RESOURCE_REQUIREMENTS_LIST
          ntdll!BUS_QUERY_ID_TYPE
          ntdll!DEVICE_TEXT_TYPE
          ntdll!_DEVICE_USAGE_NOTIFICATION_TYPE
          ntdll!_SYSTEM_POWER_STATE
          ntdll!_POWER_SEQUENCE
          ntdll!_POWER_STATE_TYPE
          ntdll!_POWER_STATE
          ntdll!POWER_ACTION
          ntdll!_CM_RESOURCE_LIST
          ntdll!_IO_STACK_LOCATION
          ntdll!_INTERFACE
          ntdll!_DEVICE_POWER_STATE
          ntdll!_POWER_STATE
          ntdll!_FS_FILTER_CALLBACK_DATA
          ntdll!_FS_FILTER_CALLBACKS
          ntdll!_DEVICE_MAP
          ntdll!_INTERFACE_TYPE
          ntdll!_IO_RESOURCE_LIST
          ntdll!_IO_RESOURCE_REQUIREMENTS_LIST
          ntdll!_SID
          ntdll!_FILE_GET_QUOTA_INFORMATION
          ntdll!_FS_FILTER_PARAMETERS
          ntdll!_FS_FILTER_CALLBACK_DATA
          ntdll!_FILE_BASIC_INFORMATION
          ntdll!_FILE_STANDARD_INFORMATION
          ntdll!_FILE_NETWORK_OPEN_INFORMATION
          ntdll!_COMPRESSED_DATA_INFO
          ntdll!_FAST_IO_DISPATCH
          ntdll!_OBJECT_DIRECTORY_ENTRY
          ntdll!_FILE_BASIC_INFORMATION
          ntdll!_PRIVILEGE_SET
          ntdll!_IO_SECURITY_CONTEXT
          ntdll!_DESCRIPTOR
          ntdll!_KSPECIAL_REGISTERS
          ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
          ntdll!_MAILSLOT_CREATE_PARAMETERS
          ntdll!_NAMED_PIPE_CREATE_PARAMETERS
          ntdll!_IO_RESOURCE_DESCRIPTOR
          ntdll!_IO_RESOURCE_LIST
          ntdll!_FILE_NETWORK_OPEN_INFORMATION
          ntdll!_CM_FULL_RESOURCE_DESCRIPTOR
          ntdll!_CM_RESOURCE_LIST
          ntdll!_POWER_SEQUENCE
          ntdll!_IO_RESOURCE_DESCRIPTOR
          ntdll!_FS_FILTER_SECTION_SYNC_TYPE
          ntdll!_FS_FILTER_PARAMETERS
          ntdll!_COMPRESSED_DATA_INFO
          ntdll!_FILE_STANDARD_INFORMATION
          ntdll!_DESCRIPTOR
          ntdll!_GUID
          ntdll!_SID_IDENTIFIER_AUTHORITY
          ntdll!_SID
          ntdll!_SID_IDENTIFIER_AUTHORITY
          ntdll!_CM_PARTIAL_RESOURCE_LIST
          ntdll!_CM_FULL_RESOURCE_DESCRIPTOR
          ntdll!_DEVICE_CAPABILITIES
          ntdll!_CM_PARTIAL_RESOURCE_DESCRIPTOR
          ntdll!_CM_PARTIAL_RESOURCE_LIST
          ntdll!_CM_PARTIAL_RESOURCE_DESCRIPTOR
          ntdll!__unnamed

You might have noticed that many structures are listed twice in the output. Actually all of them appear twice and there are many __unnamed (I edited the output before posting to save space). I was wondering why they are listed twice and after some research I found that Visual Studio contains DIA SDK (Debug Interface Access SDK) and you can build DIA2Dump sample to dump PDB files. Unfortunately this tool displays them twice too without any hints:

UDT            : LIST_ENTRY32
Data           :   this+0×0, Member, Type: unsigned long, Flink
Data           :   this+0×4, Member, Type: unsigned long, Blink
UDT            : LIST_ENTRY32
Data           :   this+0×0, Member, Type: unsigned long, Flink
Data           :   this+0×4, Member, Type: unsigned long, Blink

__unnamed datatype is for unions, for example:

0:000> dt -r _ULARGE_INTEGER
   +0x000 LowPart          : Uint4B
   +0x004 HighPart         : Uint4B
   +0x000 u                : __unnamed
      +0×000 LowPart          : Uint4B
      +0×004 HighPart         : Uint4B
   +0×000 QuadPart         : Uint8B

Here’s the definition taken from winnt.h:

- Dmitry Vostokov -

Next pattern is Deadlock. If you don’t know what “deadlock” is read Dumps for Dummes (Part 4). Deadlocks do not only happen with synchronization primitives like mutexes, events or more complex objects (built upon primitives) like critical sections or executive resources (ERESOURCE). They can happen from high level or systems perspective in inter-process or inter-component communication, for example, mutually waiting on messages: GUI window messages, LPC messages, RPC calls. This is a big pattern and I’m going to split it into several parts.

How can we see deadlocks in dumps? Let’s start with user dumps and critical sections.

First I would recommend to read the following excellent MSDN article to understand various members of CRITICAL_SECTION structure:

Break Free of Code Deadlocks in Critical Sections Under Windows

WinDbg !locks command will examine process critical section list and display all locked critical sections, lock count and thread id of current critical section owner. This is the output from a dump of hanging Windows print spooler process (spoolsv.exe):

0:000> !locks
CritSec NTDLL!LoaderLock+0 at 784B0348
LockCount          4
RecursionCount     1
OwningThread       624
EntryCount         6c3
ContentionCount    6c3
*** Locked

CritSec LOCALSPL!SpoolerSection+0 at 76AB8070
LockCount          3
RecursionCount     1
OwningThread       1c48
EntryCount         646
ContentionCount    646
*** Locked

If we look at threads #624 and #1c48 we could see them mutually waiting for each other:

• TID#624 owns CritSec 784B0348 and is waiting for CritSec 76AB8070

• TID#1c48 owns CritSec 76AB8070 and is waiting for CritSec 784B0348

0:000>~*kv

. 12 Id: bc0.624 Suspend: 1 Teb: 7ffd3000 Unfrozen
0000024c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
76ab8000 76a815ef 76ab8070 NTDLL!RtlpWaitForCriticalSection+0×9e
76ab8070 76a844f8 00cd1f38 NTDLL!RtlEnterCriticalSection+0×46
00cd1f38 76a8a1d7 00000000 LOCALSPL!EnterSplSem+0xb
00000000 00000000 00cd1f38 LOCALSPL!FindSpoolerByNameIncRef+0×1f
00000000 777f19bc 00000001 LOCALSPL!LocalGetPrinterDriverDirectory+0xe
00000000 777f19bc 00000001 spoolss!GetPrinterDriverDirectoryW+0×59
00000000 777f19bc 00000001 spoolsv!YGetPrinterDriverDirectory+0×27
00000000 777f19bc 00000001 WINSPOOL!GetPrinterDriverDirectoryW+0×7b
50000000 00000001 00000000 BRHLUI04+0×14ea
50002ea0 50000000 00000001 BRHLUI04!DllGetClassObject+0×1705
00000000 00000000 000cb570 NTDLL!LdrpRunInitializeRoutines+0×1df
000cc8f8 0288ea30 0288ea38 NTDLL!LdrpLoadDll+0×2e6
000cc8f8 0288ea30 0288ea38 NTDLL!LdrLoadDll+0×17)
000c1258 00000000 00000008 KERNEL32!LoadLibraryExW+0×231
000c150c 0288efd8 00000000 UNIDRVUI!PLoadCommonInfo+0×17e
000c150c 0288efd8 00000007 UNIDRVUI!DwDeviceCapabilities+0×1a
00070000 00071378 00000045 UNIDRVUI!DrvDeviceCapabilities+0×19

. 13 Id: bc0.1c48 Suspend: 1 Teb: 7ffd2000 Unfrozen
0000010c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb
784b0301 78468d38 784b0348 NTDLL!RtlpWaitForCriticalSection+0×9e
784b0348 74fb4344 00000000 NTDLL!RtlEnterCriticalSection+0×46
74fb0000 02c0f2a8 00000000 NTDLL!LdrpGetProcedureAddress+0×122
74fb0000 02c0f2a8 00000000 NTDLL!LdrGetProcedureAddress+0×17
74fb0000 74fb4344 02c0f449 KERNEL32!GetProcAddress+0×41
017924b0 00000000 00000001 ws2_32!CheckForHookersOrChainers+0×1f
00000101 02c0f344 017924b0 ws2_32!WSAStartup+0×10f
00cdf79c 02c0f4f4 76a8c9bc LOCALSPL!GetDNSMachineName+0×1e
00000000 76a8c9bc 780276a2 LOCALSPL!GetPrinterUrl+0×2c
0176f570 ffffffff 01000000 LOCALSPL!UpdateDsSpoolerKey+0×322
0176f570 76a8c9bc 01792b90 LOCALSPL!RecreateDsKey+0×50
00000000 00000002 01792b90 LOCALSPL!SplAddPrinter+0×521
01791faa 0176a684 76a5cd34 WIN32SPL!InternalAddPrinterConnection+0×1b4
01791faa 02c0fa00 02c0fabc WIN32SPL!AddPrinterConnectionW+0×15
00076f1c 02c0fabc 01006873 spoolss!AddPrinterConnectionW+0×49
00076f1c 00000001 77107fb0 spoolsv!YAddPrinterConnection+0×17
00076f1c 02020202 00000001 spoolsv!RpcAddPrinterConnection+0xb
01006868 02c0fac0 00000001 rpcrt4!Invoke+0×30
00000000 00000000 000d22c8 rpcrt4!NdrStubCall2+0×655
000d22c8 00076fe0 000d22c8 rpcrt4!NdrServerCall2+0×17
010045fc 000d22c8 02c0fe0c rpcrt4!DispatchToStubInC+0×32
0000002b 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×100
000d22c8 00000000 02c0fe0c rpcrt4!RPC_INTERFACE::DispatchToStub+0×5e
000d3210 00076608 813b0013 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×1dd
000d21d0 02c0fe50 000d3210 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×10c
770c9ad0 00076608 770cb6d8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×229
00076608 770cb6d8 0288f9a8 rpcrt4!RecvLotsaCallsWrapper+0×9
00074a50 02c0ffec 77e7438b rpcrt4!BaseCachedThreadRoutine+0×11f
00076e68 770cb6d8 0288f9a8 rpcrt4!ThreadStartRoutine+0×18
770d1c54 00076e68 00000000 KERNEL32!BaseThreadStart+0×52

This analysis looks pretty simple and easy. What about kernel and complete memory dumps? Of course we cannot see user space critical sections in kernel memory dumps but we can see them in complete memory dumps after switching to appropriate process context and using !ntsdexts.locks. This can be done via simple script adapted from debugger.chm: Deadlocks and Critical Sections

Why it is so easy to see deadlocks when critical sections are involved? Because their structures have a member that records their owner. So it is very easy to map them to corresponding threads. The same is with kernel ERESOURCE synchronization objects (we will see them in the next part). Other objects do not have an owner, for example, in case of events it is not so easy to find an owner just by looking at an event object. You need to examine thread call stacks, other structures or have access to source code.

- Dmitry Vostokov @ DumpAnalysis.org -

Added EFLAGS/RFLAGS template:

http://www.asmpedia.org/index.php?title=EFLAGS/RFLAGS

AAA, MOV, NOP instructions have been updated to include EFLAGS (all other instructions will have it automatically):

• http://www.asmpedia.org/index.php?title=AAA
• http://www.asmpedia.org/index.php?title=MOV
• http://www.asmpedia.org/index.php?title=NOP

Added 16-bit addressing ModRegRM table (for the sake of completeness):

http://www.asmpedia.org/index.php?title=ModRegRM_byte_%2816-bit_addressing%29

Added SIB byte translation table:

http://www.asmpedia.org/index.php?title=SIB_byte_%2832/64-bit_addressing%29

- Dmitry Vostokov -

32-bit Vista

If you need to dump a running 32-bit Vista system you can do it with Citrix SystemDump tool. You just need to run it with elevated administrator rights:

• right click SystemDump.exe in appropriate Computer explorer folder and choose “Run as administrator”

• if you use command line options run SystemDump.exe from elevated command prompt (Start -> All Programs -> Accessories, right click Command Prompt, and then select “Run as administrator”)

Here is a screenshot before dumping my Vista and WinDbg output from saved kernel dump:

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Windows Vista Kernel Version 6000 UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 6000.16386.x86fre.vista_rtm.061101-2205
Kernel base = 0x81800000 PsLoadedModuleList = 0x81908ab0
Debug session time: Sat Jan 27 20:13:10.917 2007 (GMT+0)
System Uptime: 0 days 1:33:13.589
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
BugCheck E2, {cccccccc, 83286f08, 1a, 0}
Probably caused by : SystemDump.sys

64-bit Vista

Currently in order to use 64-bit SystemDump you have to disable “Driver Signature Enforcement” by:

• F8 Advanced Boot Option
• command line tool BCDedit
• attaching an active kernel debugger

Then you need to run SystemDump64.exe as administrator.

- Dmitry Vostokov -

Today I will talk about another pattern occurring frequently and I call it Hidden Exception. You run !analyze -v command and you don’t see an exception or you see only a breakpoint hit. In this case manual analysis is required. This happens sometimes because of another pattern: Multiple Exceptions. In other cases an exception happens and it is handled by an exception handler dismissing it and a process continues execution slowly accumulating corruption inside its data leading to a new crash or hang. Sometimes you see a process hanging during its termination like the case I present here.

We have a process dump with only one thread:

0:000> kv
ChildEBP RetAddr
0096fcdc 7c822124 ntdll!KiFastSystemCallRet
0096fce0 77e6baa8 ntdll!NtWaitForSingleObject+0xc
0096fd50 77e6ba12 kernel32!WaitForSingleObjectEx+0xac
0096fd64 67f016ce kernel32!WaitForSingleObject+0x12
0096fd78 7c82257a component!DllInitialize+0xc2
0096fd98 7c8118b0 ntdll!LdrpCallInitRoutine+0x14
0096fe34 77e52fea ntdll!LdrShutdownProcess+0x130
0096ff20 77e5304d kernel32!_ExitProcess+0x43
0096ff34 77bcade4 kernel32!ExitProcess+0x14
0096ff40 77bcaefb msvcrt!__crtExitProcess+0x32
0096ff70 77bcaf6d msvcrt!_cinit+0xd2
0096ff84 77bcb555 msvcrt!_exit+0x11
0096ffb8 77e66063 msvcrt!_endthreadex+0xc8
0096ffec 00000000 kernel32!BaseThreadStart+0x34


We can look at its raw stack and try to find the following address:

KiUserExceptionDispatcher

This function calls RtlDispatchException:

0:000> !teb
TEB at 7ffdc000
    ExceptionList:        0096fd40
    StackBase:            00970000
    StackLimit:           0096a000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdc000
    EnvironmentPointer:   00000000
    ClientId:             00000858 . 000008c0
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffdd000
    LastErrorValue:       0
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000>dds 0096a000 00970000
...
...
...
0096c770  7c8140cc ntdll!RtlDispatchException+0x91
0096c774  0096c808
0096c778  0096ffa8
0096c77c  0096c824
0096c780  0096c7e4
0096c784  77bc6c74 msvcrt!_except_handler3
0096c788  00000000
0096c78c  0096c808
0096c790  01030064
0096c794  00000000
0096c798  00000000
0096c79c  00000000
0096c7a0  00000000
0096c7a4  00000000
0096c7a8  00000000
0096c7ac  00000000
0096c7b0  00000000
0096c7b4  00000000
0096c7b8  00000000
0096c7bc  00000000
0096c7c0  00000000
0096c7c4  00000000
0096c7c8  00000000
0096c7cc  00000000
0096c7d0  00000000
0096c7d4  00000000
0096c7d8  00000000
0096c7dc  00000000
0096c7e0  00000000
0096c7e4  00000000
0096c7e8  00970000
0096c7ec  00000000
0096c7f0  0096caf0
0096c7f4  7c82ecc6 ntdll!KiUserExceptionDispatcher+0xe
0096c7f8  0096c000
0096c7fc  0096c824 ; a pointer to an exception context
0096c800  0096c808
0096c804  0096c824
0096c808  c0000005
0096c80c  00000000
0096c810  00000000
0096c814  77bd8df3 msvcrt!wcschr+0×15
0096c818  00000002
0096c81c  00000000
0096c820  01031000
0096c824  0001003f
0096c828  00000000
0096c82c  00000000
0096c830  00000000
0096c834  00000000
0096c838  00000000
0096c83c  00000000


A second parameter to both functions is a pointer to a so called exception context (processor state when an exception occurred). We can use .cxr command to change thread execution context to what it was at the time of exception:

After changing the context we can see thread stack prior to that exception:

0:000> kL
ChildEBP RetAddr
0096caf0 67b11808 msvcrt!wcschr+0×15
0096cb10 67b1194d component2!function1+0×50
0096cb24 67b11afb component2!function2+0×1a
0096eb5c 67b11e10 component2!function3+0×39
0096ed94 67b14426 component2!function4+0×155
0096fdc0 67b164b7 component2!function5+0×3b
0096fdcc 00402831 component2!function6+0×5b
0096feec 0096ff14 program!function+0×1d1
0096ffec 00000000 kernel32!BaseThreadStart+0×34

We see that the exception happened when component2 was searching a Unicode string for a character (wcschr). Most likely the string was not zero terminated:

To summarize and show you the common exception handling path in user space here is another thread stack taken from a different dump:

ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+0xc
kernel32!UnhandledExceptionFilter+0×746
kernel32!_except_handler3+0×61
ntdll!ExecuteHandler2+0×26
ntdll!ExecuteHandler+0×24
ntdll!RtlDispatchException+0×91
ntdll!KiUserExceptionDispatcher+0xe
ntdll!RtlpCoalesceFreeBlocks+0×36e ; crash is here
ntdll!RtlFreeHeap+0×38e
msvcrt!free+0xc3
msvcrt!_freefls+0×124
msvcrt!_freeptd+0×27
msvcrt!__CRTDLL_INIT+0×1da
ntdll!LdrpCallInitRoutine+0×14
ntdll!LdrShutdownThread+0xd2
kernel32!ExitThread+0×2f
kernel32!BaseThreadStart+0×39

When RtlpCoalesceFreeBlocks (this function compacts heap and it is called from RtlFreeHeap) does an illegal memory access then this exception is first processed in kernel and because it happened in user space and mode the execution is transferred to RtlDispatchException which searches for exception handler and in this case there is a default one installed: UnhandledExceptionFilter.

If you see this function on call stack you can also manually get an exception context and a thread stack leading to it like in this example below taken from other dump:

The most likely reason of this crash is an instance of Dynamic Memory Corruption pattern - heap corruption.

- Dmitry Vostokov -

Just a short note that might be useful if you get a 64-bit dump of a 32-bit process from x64 Windows. As a reminder I wrote already about separate dump types and tools for 32-bit and 64-bit processes.

Since then I’ve been looking for a simple solution in case we get a 64-bit dump so we could look at 32-bit thread stacks, etc. Yesterday I accidentally found wow64exts.dll extension in 64-bit WinDbg. I should have been looked at \winxp folder before :-) 

The whole discussion and solution can be found on Crash Dump Analysis forum:

http://www.dumpanalysis.org/forum/viewtopic.php?t=39

Now if by accident a customer sends a 64-bit dump of a 32-bit process (some Citrix services like IMA service are still 32-bit in x64 CPS4) then we can look at threads without asking for a new dump and therefore avoid roundtrips and shorten time to problem resolution.

- Dmitry Vostokov -

We have to live with tools that produce inconsistent dumps. For example, LiveKd.exe from sysinternals.com which is widely used by Microsoft and Citrix technical support to save complete memory dumps without server reboot. I even wrote an article for Citrix customers:

Using LiveKD to Save a Complete Memory Dump for Session or System Hangs

If you read it you will find an important note which is reproduced here:

LiveKd.exe-generated dumps are always inconsistent and cannot be a reliable source for certain types of dump analysis, for example, looking at resource contention. This is because it takes a considerable amount of time to save a dump on a live system and the system is being changed during that process. The instantaneous traditional CrashOnCtrlScroll method or SystemDump tool always save a reliable and consistent dump because the system is frozen first (any process or kernel activity is disabled), then a dump is saved to a page file.

If you look at such inconsistent dump you will find that many useful kernel structures such as ERESOURCE list (!locks) are broken and even circular referenced and therefore WinDbg commands display “strange” output.

Easy and painless (for customers) dump generation using such “Live” tools means that it is widely used and we have to analyze dumps saved by these tools and sent from customers. This brings us to the next crash dump analysis pattern called “Inconsistent Dump”.

If you have such dump you should look at it in order to extract maximum useful information that helps in identifying the root cause or give you further directions. Not all information is inconsistent in such dumps. For example, drivers, processes, thread stacks and IRP lists can give you some clues about activities. Even some information not visible in consistent dump can surface in inconsistent dump (subject to commands used).

For example, I had a LiveKd dump where I looked at process stacks by running the script I created earlier:

Yet another WinDbg script

and I found that for some processes in addition to their own threads the script lists additional terminated threads that belong to a completely different process (have never seen it in consistent dump):

Process 89d97d88 is not visible in the active process list (script mentioned above or !process 0 0 command). However, if we feed this memory address to !process command (or explore it as _EPROCESS structure, dt command) we get its contents:

What might have happened there: terminated process 89d97d88 was excluded from active processes list but its structure was left in memory and due to inconsistency thread lists were also broken and therefore terminated threads surfaced when listing other processes and their threads. 

I suspected here that winlogon.exe died in session 2 and left empty desktop window which a customer saw and complained about. The only left and visible process from session 2 was csrss.exe. The conclusion was to enable NTSD as a default postmortem debugger to catch winlogon.exe crash when it happens next time.

- Dmitry Vostokov @ DumpAnalysis.org -

New version of SystemDump for 32-bit and 64-bit platforms has been released. What’s new in this version:

• Fixed the bug that prevented SystemDump to bugcheck second time after the first bugcheck and reboot happened (the known workaround was to run it twice second time)
• Easter egg (hold key and click on About button)

The tool can be downloaded from Citrix support web site.

- Dmitry Vostokov -

Suppose you have 100 - 200 user dumps from various user processes in the system and you want to quickly check their thread stacks, locks, etc. to see something suspicious related to your product or its environment your customers complaining about. It is much easier to collect such information into text files and browse them quickly than open every dump in WinDbg. I used shell script (VBScript) to automate loading dumps into WinDbg and used WinDbg scripts to run complex commands against loaded user dumps. For example, I used the following shell script:

'
' UDumps2Txt.vbs
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set Folder = fso.GetFolder(".")
Set Files = Folder.Files
Set WshShell = CreateObject("WScript.Shell")
For Each File In Files
  Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\mss*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\UDmp2Txt.txt;q"" -Q -QS -QY –QSY")
  Do While oExec.Status = 0
     WScript.Sleep 1000
  Loop
Next
'
' UDumps2Txt.vbs: End of File
'

and the following WinDbg script:

$$
$$ UDmp2Txt: Dump information from user dump into log
$$
.logopen /d
!analyze -v
!locks
~*kv
lmv
.logclose
$$
$$ UDmp2Txt: End of File
$$

The following command launches multiple Dmp2Txt conversions:

C:\UserDumps>cscript /nologo c:\scripts\UDumps2Txt.vbs

You can also use CDB from Debugging Tools for Windows (console debugger) instead of WinDbg. I just use WinDbg uniformly instead of using separately CDB for user process dumps and KD for kernel and complete memory dumps. 

Now when you have text files you can search for patterns using regular expressions. I will write more about applying them later. There is a very good book about them from practical point of view I read 6 years ago when I needed to understand them beyond wildcards and question marks. Since that time the book has undergone another two editions:

Mastering Regular Expressions, 3rd edition

Or you can process text files further and feed them into your database - part of automated crash dump analysis system.

- Dmitry Vostokov -

I’ve been doing some research in this direction and found so many patents filed, to name a few:

Method and expert system for analysis of crash dumps

System for performing dump analysis

Some companies have their own systems. For example, Microsoft has its own Online Crash Analysis system (OCA) and even promotes its Corporate Error Reporting (CER) tool. CER architecture is described in the following document:

CER_Implementation_Plan

In the next parts I will try to outline different implementation choices for building automated crash dump analysis system and discuss their advantages and disadvantages from expert systems perspective.

- Dmitry Vostokov -

Microsoft describes the reason behind these changes: silent process death if thread stack is corrupt. In Vista such crashes will be reported to MS via Windows Error Reporting mechanism.

Presentation, Reliability and Recovery, slide 42

- Dmitry Vostokov -

I have been thinking for a while what kind of a marketing card www.dumpanalysis.org should have (which should be useful to its users) and finally came up with the following design which is being printed now:

Front

Backside

I put most used commands (at least by me) and hope the backside of this card will be useful. If you see me in person you have a chance to get this card in hardcopy If after reading this post you got an idea that we need a crash dump analysis and debugging poster (WinDbg related or a general one) then don’t worry and this is being designed now and details will be announced shortly… All suggestions are welcome anyway and if they are genuine and original then full credit will be given.

- Dmitry Vostokov -

WinDbg help states that NMI_HARDWARE_FAILURE (0×80) bugcheck 80 indicates a hardware fault. This description can easily lead to a conclusion that a kernel or complete crash dump you just got from your customer doesn’t worth examining. But hardware malfunction is not always the case especially if your customer mentions that their system was hanging and they forced a manual dump. Here I would advise to check whether they have a special hardware for debugging purposes, for example, a card or an integrated iLO chip (Integrated Lights-Out) for remote server administration. Both can generate NMI (Non Maskable Interrupt) on demand and therefore bugcheck the system. If this is the case then it is worth examining their dump to see why the system was hanging.

- Dmitry Vostokov -

Welcome to the new blog location at dumpanalysis.org/blog/ 

Its feed address is

http://feeds.feedburner.com/CrashDumpAnalysis

The blog has been moved from its original location at

citrite.org/blogs/dmitryv/ 

in order to bring all crash dump analysis and debugging information to one place including www.dumpanalysis.org/forum and the forthcoming online encyclopedia about assembly languages:

www.asmpedia.org

Thank you and sorry for any inconvenience this might have caused.

Merry Christmas and Happy Debugging in New Year!

- Dmitry Vostokov -