Archive for April, 2008

Command Autocompletion in WinDbg

Wednesday, April 30th, 2008

To my shame I’ve just discovered this feature in WinDbg by reading WinDbg Help :-) For example, just type !a<TAB>. No longer I need to type command !analyze fully. As by product, I also discovered the existence of !analyzeuexception command which seems identical to !analyze command at least in user dumps.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 60)

Tuesday, April 29th, 2008

In the pattern about NULL code pointer I created a simple program that crashes when we pass a NULL thread procedure pointer to CreateThread function. We might expect to see little in the raw stack data because there was no user-supplied thread code. In reality, if we dump it we would see lots of symbolic information for code and data including ASCII and UNICODE fragments that I call Execution Residue patterns and one of them is Exception Handling Residue we can use to check for hidden exceptions and differentiate between 1st and 2nd chance exceptions. Code residues are very powerful in reconstructing stack traces manually or looking for partial stack traces and historical information.

To show typical execution residues I created the small program with two additionally created threads based on Visual Studio Win32 project. After we dismiss About box we create the first thread and then we crash the process when creating the second thread because of the NULL thread procedure:

typedef DWORD (WINAPI *THREADPROC)(PVOID);

DWORD WINAPI ThreadProc(PVOID pvParam)
{
   for (unsigned int i = 0xFFFFFFFF; i; --i);
   return 0;
}

// Message handler for about box.
INT_PTR CALLBACK About(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
   UNREFERENCED_PARAMETER(lParam);
   switch (message)
   {
   case WM_INITDIALOG:
      return (INT_PTR)TRUE;

   case WM_COMMAND:
      if (LOWORD(wParam) == IDOK || LOWORD(wParam) == IDCANCEL)
      {
         EndDialog(hDlg, LOWORD(wParam));
         THREADPROC thProc = ThreadProc;
         HANDLE hThread = CreateThread(NULL, 0, ThreadProc, 0, 0, NULL);
         CloseHandle(hThread);
         Sleep(1000);
         hThread = CreateThread(NULL, 0, NULL, 0, 0, NULL);
         CloseHandle(hThread);
         return (INT_PTR)TRUE;
      }
      break;
   }
   return (INT_PTR)FALSE;
}

When we open the crash dump we see these threads:

0:002> ~*kL

   0  Id: cb0.9ac Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr 
0012fdf4 00411554 user32!NtUserGetMessage+0x15
0012ff08 00412329 NullThread!wWinMain+0xa4
0012ffb8 0041208d NullThread!__tmainCRTStartup+0x289
0012ffc0 7d4e7d2a NullThread!wWinMainCRTStartup+0xd
0012fff0 00000000 kernel32!BaseProcessStart+0x28

   1  Id: cb0.8b4 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr 
01eafea4 7d63f501 ntdll!NtWaitForMultipleObjects+0x15
01eaff48 7d63f988 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
01eaffb8 7d4dfe21 ntdll!EtwpEventPump+0x27f
01eaffec 00000000 kernel32!BaseThreadStart+0x34

   2  Id: cb0.ca8 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr 
0222ffb8 7d4dfe21 NullThread!ThreadProc+0×34
0222ffec 00000000 kernel32!BaseThreadStart+0×34

#  3  Id: cb0.5bc Suspend: 1 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0236ffb8 7d4dfe21 0×0
0236ffec 00000000 kernel32!BaseThreadStart+0×34

   4  Id: cb0.468 Suspend: -1 Teb: 7efac000 Unfrozen
ChildEBP RetAddr 
01f7ffb4 7d674807 ntdll!NtTerminateThread+0x12
01f7ffc4 7d66509f ntdll!RtlExitUserThread+0x26
01f7fff4 00000000 ntdll!DbgUiRemoteBreakin+0x41

We see our first created thread looping:

0:003> ~2s
eax=cbcf04b5 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=0222ffb8
eip=00411aa4 esp=0222fee0 ebp=0222ffb8 iopl=0 nv up ei ng nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000282
NullThread!ThreadProc+0x34:
00411aa4 7402            je      NullThread!ThreadProc+0x38 (00411aa8)   [br=0]

0:002> u
NullThread!ThreadProc+0x34:
00411aa4 je      NullThread!ThreadProc+0x38 (00411aa8)
00411aa6 jmp     NullThread!ThreadProc+0x27 (00411a97)
00411aa8 xor     eax,eax
00411aaa pop     edi
00411aab pop     esi
00411aac pop     ebx
00411aad mov     esp,ebp
00411aaf pop     ebp

We might expect it having very little in its raw stack data but what we see when we dump stack range from !teb command is Thread Startup Residue where some symbolic information might be coincidental:

0:002> dds 0222f000  02230000
0222f000  00000000
0222f004  00000000
0222f008  00000000
[...]
0222f104  00000000
0222f108  00000000
0222f10c  00000000
0222f110  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f114  7efde000
0222f118  00000000
0222f11c  00000001
0222f120  000000e8
0222f124  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0222f128  00000000
0222f12c  0222f114
0222f130  00000000
0222f134  0222fca0
0222f138  7d61f1f8 ntdll!_except_handler3
0222f13c  7d621958 ntdll!RtlpRunTable+0x4a0
0222f140  ffffffff
0222f144  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f148  7d6218ab ntdll!RtlImageNtHeader+0x1b
0222f14c  00000001
0222f150  00400000 NullThread!_enc$textbss$begin <PERF> (NullThread+0x0)
0222f154  00000000
0222f158  00000000
0222f15c  0222f160
0222f160  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0222f164  0222f7bc
0222f168  7d4dfea3 kernel32!ConsoleApp+0xe
0222f16c  00400000 NullThread!_enc$textbss$begin <PERF> (NullThread+0x0)
0222f170  7d4dfe77 kernel32!ConDllInitialize+0x1f5
0222f174  00000000
0222f178  7d4dfe8c kernel32!ConDllInitialize+0x20a
0222f17c  00000000
0222f180  00000000
[...]
0222f290  00000000
0222f294  0222f2b0
0222f298  7d6256e8 ntdll!bsearch+0x42
0222f29c  00180144
0222f2a0  0222f2b4
0222f2a4  7d625992 ntdll!ARRAY_FITS+0x29
0222f2a8  00000a8c
0222f2ac  00001f1c
0222f2b0  0222f2c0
0222f2b4  0222f2f4
0222f2b8  7d625944 ntdll!RtlpLocateActivationContextSection+0x1da
0222f2bc  00001f1c
0222f2c0  000029a8
[...]
0222f2e0  536cd652
0222f2e4  0222f334
0222f2e8  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0222f2ec  0222f418
0222f2f0  00000000
0222f2f4  0222f324
0222f2f8  7d6257f1 ntdll!RtlpFindNextActivationContextSection+0x64
0222f2fc  00181f1c
0222f300  c0150008
[...]
0222f320  7efd7000
0222f324  0222f344
0222f328  7d625cd2 ntdll!RtlFindNextActivationContextSection+0x46
0222f32c  0222f368
0222f330  0222f3a0
0222f334  0222f38c
0222f338  0222f340
0222f33c  00181f1c
0222f340  00000000
0222f344  0222f390
0222f348  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0222f34c  0222f368
0222f350  0222f3a0
[...]
0222f38c  00000a8c
0222f390  0222f454
0222f394  7d626381 ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0xa57
0222f398  00000003
0222f39c  00000000
0222f3a0  00181f1c
0222f3a4  0222f418
0222f3a8  0222f3b4
0222f3ac  7d6a0340 ntdll!LdrApiDefaultExtension
0222f3b0  7d6263df ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0xb73
0222f3b4  00000040
0222f3b8  00000000
[...]
0222f420  00000000
0222f424  0222f458
0222f428  7d625f9a ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0x4c1
0222f42c  00020000
0222f430  0222f44c
0222f434  0222f44c
0222f438  0222f44c
0222f43c  00000002
0222f440  00000002
0222f444  7d625f9a ntdll!CsrCaptureMessageMultiUnicodeStringsInPlace+0x4c1
0222f448  00020000
0222f44c  00000000
0222f450  00003cfb
0222f454  0222f5bc
0222f458  0222f4f4
0222f45c  0222f5bc
0222f460  7d626290 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x346
0222f464  0222f490
0222f468  00000000
0222f46c  0222f69c
0222f470  7d6262f5 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3de
0222f474  0222f510
0222f478  7d6a0340 ntdll!LdrApiDefaultExtension
0222f47c  7d626290 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x346
0222f480  00000000
0222f484  00800000
[...]
0222f544  00000000
0222f548  00000001
0222f54c  7d6a0290 ntdll!LdrpHashTable+0x50
0222f550  00000000
0222f554  00500000
[...]
0222f59c  00000000
0222f5a0  0222f5d4
0222f5a4  7d6251d0 ntdll!LdrUnlockLoaderLock+0x84
0222f5a8  7d6251d7 ntdll!LdrUnlockLoaderLock+0xad
0222f5ac  00000000
0222f5b0  0222f69c
0222f5b4  00000000
0222f5b8  00003cfb
0222f5bc  0222f5ac
0222f5c0  7d626de0 ntdll!LdrGetDllHandleEx+0xbe
0222f5c4  0222f640
0222f5c8  7d61f1f8 ntdll!_except_handler3
0222f5cc  7d6251e0 ntdll!`string'+0x74
0222f5d0  ffffffff
0222f5d4  7d6251d7 ntdll!LdrUnlockLoaderLock+0xad
0222f5d8  7d626fb3 ntdll!LdrGetDllHandleEx+0x368
0222f5dc  00000001
0222f5e0  0ca80042
0222f5e4  7d626f76 ntdll!LdrGetDllHandleEx+0x329
0222f5e8  00000000
0222f5ec  7d626d0b ntdll!LdrGetDllHandle
0222f5f0  00000002
0222f5f4  001a0018
[...]
0222f640  0222f6a8
0222f644  7d61f1f8 ntdll!_except_handler3
0222f648  7d626e60 ntdll!`string'+0xb4
0222f64c  ffffffff
0222f650  7d626f76 ntdll!LdrGetDllHandleEx+0x329
0222f654  7d626d23 ntdll!LdrGetDllHandle+0x18
0222f658  00000001
[...]
0222f66c  0222f6b8
0222f670  7d4dff0e kernel32!GetModuleHandleForUnicodeString+0x20
0222f674  00000001
0222f678  00000000
0222f67c  0222f6d4
0222f680  7d4dff1e kernel32!GetModuleHandleForUnicodeString+0x97
0222f684  00000000
0222f688  7efd7c00
0222f68c  00000002
0222f690  00000001
0222f694  00000000
0222f698  0222f6f0
0222f69c  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f6a0  0222f684
0222f6a4  7efd7c00
0222f6a8  0222fb20
0222f6ac  7d4d89c4 kernel32!_except_handler3
0222f6b0  7d4dff28 kernel32!`string'+0x18
0222f6b4  ffffffff
0222f6b8  7d4dff1e kernel32!GetModuleHandleForUnicodeString+0x97
0222f6bc  7d4e001f kernel32!BasepGetModuleHandleExW+0x17f
0222f6c0  7d4e009f kernel32!BasepGetModuleHandleExW+0x23c
0222f6c4  00000000
0222f6c8  0222fc08
0222f6cc  00000001
0222f6d0  ffffffff
0222f6d4  001a0018
0222f6d8  7efd7c00
0222f6dc  0222fb50
0222f6e0  00000000
0222f6e4  00000000
0222f6e8  00000000
0222f6ec  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f6f0  0222f90c
0222f6f4  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f6f8  0222f704
0222f6fc  00000000
0222f700  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f704  00000000
0222f708  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f70c  0222f928
0222f710  02080000 oleaut32!_PictSaveEnhMetaFile+0x76
0222f714  0222f720
0222f718  00000000
0222f71c  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f720  00000000
0222f724  00000000
[...]
0222f7b8  0000f949
0222f7bc  0222fbf4
0222f7c0  7d4dfdd0 kernel32!_BaseDllInitialize+0x6b
0222f7c4  00000002
0222f7c8  00000000
0222f7cc  00000000
0222f7d0  7d4dfde4 kernel32!_BaseDllInitialize+0x495
0222f7d4  00000000
0222f7d8  7efde000
0222f7dc  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222f7e0  00000000
0222f7e4  00000000
[...]
0222f894  01c58ae0
0222f898  0222fac0
0222f89c  7d62155b ntdll!RtlAllocateHeap+0x460
0222f8a0  7d61f78c ntdll!RtlAllocateHeap+0xee7
0222f8a4  00000000
0222f8a8  0222fc08
[...]
0222f8d8  00000000
0222f8dc  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0222f8e0  0222f9a4
0222f8e4  7d614c88 ntdll!$$VProc_ImageExportDirectory+0x2c48
0222f8e8  0222f9a6
0222f8ec  7d612040 ntdll!$$VProc_ImageExportDirectory
0222f8f0  00000221
0222f8f4  0222f944
0222f8f8  7d627405 ntdll!LdrpSnapThunk+0xc0
0222f8fc  0222f9a6
0222f900  00000584
0222f904  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222f908  7d613678 ntdll!$$VProc_ImageExportDirectory+0x1638
0222f90c  7d614c88 ntdll!$$VProc_ImageExportDirectory+0x2c48
0222f910  0222f9a4
0222f914  00000001
0222f918  0222f9a4
0222f91c  00000000
0222f920  0222f990
0222f924  7d6000f0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xf0)
0222f928  0222f968
0222f92c  00000001
0222f930  0222f9a4
0222f934  7d6000f0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xf0)
0222f938  0222f954
0222f93c  00000000
0222f940  00000000
0222f944  0222fa00
0222f948  7d62757a ntdll!LdrpGetProcedureAddress+0x189
0222f94c  0222f95c
0222f950  00000098
0222f954  00000005
0222f958  01c44f48
0222f95c  0222fb84
0222f960  7d62155b ntdll!RtlAllocateHeap+0x460
0222f964  7d61f78c ntdll!RtlAllocateHeap+0xee7
0222f968  00000000
0222f96c  0000008c
0222f970  00000000
0222f974  7d4d8472 kernel32!$$VProc_ImageExportDirectory+0x6d4e
0222f978  0222fa1c
0222f97c  7d627607 ntdll!LdrpGetProcedureAddress+0x274
0222f980  7d612040 ntdll!$$VProc_ImageExportDirectory
0222f984  002324f8
0222f988  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222f98c  0222faa8
0222f990  0000a7bb
0222f994  00221f08
0222f998  0222f9a4
0222f99c  7d627c2e ntdll!RtlDecodePointer
0222f9a0  00000000
0222f9a4  74520000
0222f9a8  6365446c
0222f9ac  5065646f
0222f9b0  746e696f
0222f9b4  00007265
0222f9b8  7d627c2e ntdll!RtlDecodePointer
0222f9bc  00000000
[...]
0222f9f8  01c40640
0222f9fc  00000000
0222fa00  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fa04  7d627772 ntdll!LdrpSnapThunk+0x31c
0222fa08  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222fa0c  0222fa44
0222fa10  00000000
0222fa14  0222faa8
0222fa18  00000000
0222fa1c  0222fab0
0222fa20  00000001
0222fa24  00000001
0222fa28  00000000
0222fa2c  0222fa9c
0222fa30  7d4c00e8 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0xe8)
0222fa34  01c44fe0
0222fa38  00000001
0222fa3c  01c401a0
0222fa40  7d4c00e8 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0xe8)
0222fa44  00110010
0222fa48  7d4d8478 kernel32!$$VProc_ImageExportDirectory+0x6d54
0222fa4c  00000000
0222fa50  0222fb0c
0222fa54  7d62757a ntdll!LdrpGetProcedureAddress+0x189
0222fa58  7d600000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
0222fa5c  00000000
0222fa60  0022faa8
0222fa64  0222fab0
0222fa68  0222fb0c
0222fa6c  7d627607 ntdll!LdrpGetProcedureAddress+0x274
0222fa70  7d6a0180 ntdll!LdrpLoaderLock
0222fa74  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fa78  102ce1ac msvcr80d!`string'
0222fa7c  0222fc08
0222fa80  0000ffff
0222fa84  0022f8b0
0222fa88  0022f8a0
0222fa8c  00000003
0222fa90  0222fbd4
0222fa94  020215fc oleaut32!DllMain+0x2c
0222fa98  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fa9c  00000002
0222faa0  00000000
0222faa4  00000000
0222faa8  00000002
0222faac  0202162d oleaut32!DllMain+0x203
0222fab0  65440000
0222fab4  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fab8  00000001
0222fabc  00726574
0222fac0  0222facc
0222fac4  7d627c2e ntdll!RtlDecodePointer
0222fac8  00000000
0222facc  65440000
0222fad0  00000000
0222fad4  00000000
0222fad8  00726574
0222fadc  00000005
0222fae0  00000000
0222fae4  1021af95 msvcr80d!_heap_alloc_dbg+0x375
0222fae8  002322f0
0222faec  00000000
0222faf0  01c40238
0222faf4  0222fa78
0222faf8  7efd7bf8
0222fafc  00000020
0222fb00  7d61f1f8 ntdll!_except_handler3
0222fb04  7d6275b8 ntdll!`string'+0xc
0222fb08  ffffffff
0222fb0c  7d6275b2 ntdll!LdrpGetProcedureAddress+0xb3
0222fb10  00000000
0222fb14  00000000
0222fb18  0222fb48
0222fb1c  00000000
0222fb20  01000000
0222fb24  00000001
0222fb28  0222fb50
0222fb2c  7d4dac3a kernel32!GetProcAddress+0x44
0222fb30  0222fb50
0222fb34  7d4dac4c kernel32!GetProcAddress+0x5c
0222fb38  0222fc08
0222fb3c  00000013
0222fb40  00000000
0222fb44  01c44f40
0222fb48  01c4015c
0222fb4c  00000098
0222fb50  01c44f40
0222fb54  01c44f48
0222fb58  01c40238
0222fb5c  10204f9f msvcr80d!_initptd+0x10f
0222fb60  00000098
0222fb64  00000000
0222fb68  01c40000
0222fb6c  0222f968
0222fb70  7d4c0000 kernel32!_imp__NtFsControlFile <PERF> (kernel32+0x0)
0222fb74  00000ca8
0222fb78  4b405064 msctf!g_timlist
0222fb7c  0222fbb8
0222fb80  4b3c384f msctf!CTimList::Leave+0x6
0222fb84  4b3c14d7 msctf!CTimList::IsThreadId+0x5a
0222fb88  00000ca8
0222fb8c  4b405064 msctf!g_timlist
0222fb90  4b3c0000 msctf!_imp__CheckTokenMembership <PERF> (msctf+0x0)
0222fb94  01c70000
0222fb98  00000000
0222fb9c  4b405064 msctf!g_timlist
0222fba0  0222fb88
0222fba4  7d4dfd40 kernel32!FlsSetValue+0xc7
0222fba8  0222fca0
0222fbac  4b401dbd msctf!_except_handler3
0222fbb0  4b3c14e0 msctf!`string'+0x78
0222fbb4  0222fbd4
0222fbb8  0022f8a0
0222fbbc  00000001
0222fbc0  00000000
0222fbc4  00000000
0222fbc8  0222fc80
0222fbcc  0022f8a0
0222fbd0  0000156f
0222fbd4  0222fbf4
0222fbd8  020215a4 oleaut32!_DllMainCRTStartup+0x52
0222fbdc  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fbe0  00000002
0222fbe4  00000000
0222fbe8  00000000
0222fbec  0222fc08
0222fbf0  00000001
0222fbf4  0222fc14
0222fbf8  7d610024 ntdll!LdrpCallInitRoutine+0x14
0222fbfc  02020000 oleaut32!_imp__RegFlushKey <PERF> (oleaut32+0x0)
0222fc00  00000001
0222fc04  00000000
0222fc08  00000001
0222fc0c  00000000
0222fc10  0022f8a0
0222fc14  00000001
0222fc18  00000000
0222fc1c  0222fcb0
0222fc20  7d62822e ntdll!LdrpInitializeThread+0x1a5
0222fc24  7d6a0180 ntdll!LdrpLoaderLock
0222fc28  7d62821c ntdll!LdrpInitializeThread+0x18f
0222fc2c  00000000
0222fc30  7efde000
0222fc34  00000000
[...]
0222fc6c  00000070
0222fc70  ffffffff
0222fc74  ffffffff
0222fc78  7d6281c7 ntdll!LdrpInitializeThread+0xd8
0222fc7c  7d6280d6 ntdll!LdrpInitializeThread+0x12c
0222fc80  00000000
0222fc84  00000000
0222fc88  0022f8a0
0222fc8c  0202155c oleaut32!_DllMainCRTStartup
0222fc90  7efde000
0222fc94  7d6a01f4 ntdll!PebLdr+0x14
0222fc98  0222fc2c
0222fc9c  00000000
0222fca0  0222fcfc
0222fca4  7d61f1f8 ntdll!_except_handler3
0222fca8  7d628148 ntdll!`string'+0xac
0222fcac  ffffffff
0222fcb0  7d62821c ntdll!LdrpInitializeThread+0x18f
0222fcb4  7d61e299 ntdll!ZwTestAlert+0x15
0222fcb8  7d628088 ntdll!_LdrpInitialize+0x1de
0222fcbc  0222fd20
0222fcc0  00000000
[...]
0222fcfc  0222ffec
0222fd00  7d61f1f8 ntdll!_except_handler3
0222fd04  7d628090 ntdll!`string'+0xfc
0222fd08  ffffffff
0222fd0c  7d628088 ntdll!_LdrpInitialize+0x1de
0222fd10  7d61ce0d ntdll!NtContinue+0x12
0222fd14  7d61e9b2 ntdll!KiUserApcDispatcher+0x3a
0222fd18  0222fd20
0222fd1c  00000001
0222fd20  0001002f
[...]
0222fdc8  00000000
0222fdcc  00000000
0222fdd0  00411032 NullThread!ILT+45(?ThreadProcYGKPAXZ)
0222fdd4  00000000
0222fdd8  7d4d1504 kernel32!BaseThreadStartThunk
0222fddc  00000023
0222fde0  00000202
[...]
0222ffb4  cccccccc
0222ffb8  0222ffec
0222ffbc  7d4dfe21 kernel32!BaseThreadStart+0x34
0222ffc0  00000000
0222ffc4  00000000
0222ffc8  00000000
0222ffcc  00000000
0222ffd0  00000000
0222ffd4  0222ffc4
0222ffd8  00000000
0222ffdc  ffffffff
0222ffe0  7d4d89c4 kernel32!_except_handler3
0222ffe4  7d4dfe28 kernel32!`string'+0x18
0222ffe8  00000000
0222ffec  00000000
0222fff0  00000000
0222fff4  00411032 NullThread!ILT+45(?ThreadProcYGKPAXZ)
0222fff8  00000000
0222fffc  00000000
02230000  ????????

The second crashed thread has much more symbolic information in it overwriting previous thread startup residue. It is mostly exception handling residue because exception handling consumes stack space as explained in the post Who calls the postmortem debugger?:

0:003> dds 0236a000 02370000
0236a000  00000000
[...]
0236a060  00000000
0236a064  0236a074
0236a068  00220000
0236a06c  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236a070  00221378
0236a074  0236a29c
0236a078  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236a07c  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236a080  0236a5f4
0236a084  00000000
[...]
0236a1b4  0236a300
0236a1b8  0236a1dc
0236a1bc  7d624267 ntdll!RtlIsDosDeviceName_Ustr+0x2f
0236a1c0  0236a21c
0236a1c4  7d624274 ntdll!RtlpDosSlashCONDevice
0236a1c8  00000001
0236a1cc  0236a317
0236a1d0  00000000
0236a1d4  0236a324
0236a1d8  0236a290
0236a1dc  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236a1e0  7d6a00e0 ntdll!FastPebLock
0236a1e4  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236a1e8  0236a5f4
0236a1ec  00000208
[...]
0236a224  00000000
0236a228  00000038
0236a22c  02080038 oleaut32!_PictSaveMetaFile+0x33
0236a230  00000000
[...]
0236a27c  00000000
0236a280  0236a53c
0236a284  7d61f1f8 ntdll!_except_handler3
0236a288  7d6245f0 ntdll!`string'+0x5c
0236a28c  ffffffff
0236a290  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236a294  0236a5c8
0236a298  00000008
0236a29c  00000000
0236a2a0  0236a54c
0236a2a4  7d624bcf ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3d8
0236a2a8  7d6a00e0 ntdll!FastPebLock
0236a2ac  7d624ba1 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3cb
0236a2b0  00000000
0236a2b4  0236e6d0
[...]
0236a2e0  000a0008
0236a2e4  7d624be8 ntdll!`string'
0236a2e8  00000000
0236a2ec  003a0038
[...]
0236a330  00650070
0236a334  0050005c
0236a338  00480043 advapi32!LsaGetQuotasForAccount+0x25
0236a33c  00610046
0236a340  006c0075
0236a344  00520074
0236a348  00700065
0236a34c  00780045
0236a350  00630065
0236a354  00690050
0236a358  00650070
0236a35c  00000000
0236a360  00000000
[..]
0236a4a0  0236a4b0
0236a4a4  00000001
0236a4a8  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236a4ac  00230b98
0236a4b0  0236a590
0236a4b4  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236a4b8  00221378
0236a4bc  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236a4c0  00000000
0236a4c4  7d61f4ab ntdll!RtlFreeHeap
0236a4c8  00000000
0236a4cc  00000000
[...]
0236a538  00000000
0236a53c  0236a678
0236a540  7d61f1f8 ntdll!_except_handler3
0236a544  7d624ba8 ntdll!`string'+0x1c
0236a548  ffffffff
0236a54c  7d624ba1 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3cb
0236a550  7d624c43 ntdll!RtlpDosPathNameToRelativeNtPathName_U+0x55
0236a554  00000001
0236a558  0236a56c
[...]
0236a590  0236a5c0
0236a594  7d620304 ntdll!RtlNtStatusToDosError+0x38
0236a598  7d620309 ntdll!RtlNtStatusToDosError+0x3d
0236a59c  7d61c828 ntdll!ZwWaitForSingleObject+0x15
0236a5a0  7d4d8c82 kernel32!WaitForSingleObjectEx+0xac
0236a5a4  00000124
0236a5a8  00000000
0236a5ac  7d4d8ca7 kernel32!WaitForSingleObjectEx+0xdc
0236a5b0  00000124
0236a5b4  7d61f49c ntdll!RtlGetLastWin32Error
0236a5b8  80070000
0236a5bc  00000024
[...]
0236a5f8  00000000
0236a5fc  0236a678
0236a600  7d4d89c4 kernel32!_except_handler3
0236a604  7d4d8cb0 kernel32!`string'+0x68
0236a608  ffffffff
0236a60c  7d4d8ca7 kernel32!WaitForSingleObjectEx+0xdc
0236a610  7d4d8bf1 kernel32!WaitForSingleObject+0x12
0236a614  7d61f49c ntdll!RtlGetLastWin32Error
0236a618  7d61c92d ntdll!NtClose+0x12
0236a61c  7d4d8e4f kernel32!CloseHandle+0x59
0236a620  00000124
0236a624  0236a688
0236a628  69511753 <Unloaded_faultrep.dll>+0x11753
0236a62c  6951175b <Unloaded_faultrep.dll>+0x1175b
0236a630  0236c6d0
[...]
0236a668  00000120
0236a66c  00000000
0236a670  0236a630
0236a674  7d94a2e9 user32!GetSystemMetrics+0x62
0236a678  0236f920
0236a67c  69510078 <Unloaded_faultrep.dll>+0x10078
0236a680  69503d10 <Unloaded_faultrep.dll>+0x3d10
0236a684  ffffffff
0236a688  6951175b <Unloaded_faultrep.dll>+0x1175b
0236a68c  69506136 <Unloaded_faultrep.dll>+0x6136
0236a690  0236e6d0
0236a694  0236c6d0
0236a698  0000009c
0236a69c  0236a6d0
0236a6a0  00002000
0236a6a4  0236eae4
0236a6a8  695061ff <Unloaded_faultrep.dll>+0x61ff
0236a6ac  00000000
0236a6b0  00000001
0236a6b4  0236f742
0236a6b8  69506210 <Unloaded_faultrep.dll>+0x6210
0236a6bc  00000028
0236a6c0  0236c76c
[...]
0236e6e0  0050005c
0236e6e4  00480043 advapi32!LsaGetQuotasForAccount+0x25
0236e6e8  00610046
[...]
0236e718  002204d8
0236e71c  0236e890
0236e720  77b940bb <Unloaded_VERSION.dll>+0x40bb
0236e724  77b91798 <Unloaded_VERSION.dll>+0x1798
0236e728  ffffffff
0236e72c  77b9178e <Unloaded_VERSION.dll>+0x178e
0236e730  69512587 <Unloaded_faultrep.dll>+0x12587
0236e734  0236e744
0236e738  00220000
0236e73c  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236e740  00221378
0236e744  0236e96c
0236e748  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236e74c  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e750  0236eca4
0236e754  00000000
0236e758  0236ec94
0236e75c  7d620309 ntdll!RtlNtStatusToDosError+0x3d
0236e760  0236e7c8
0236e764  7d61c9db ntdll!NtQueryValueKey
0236e768  0236e888
0236e76c  0236e760
0236e770  7d61c9ed ntdll!NtQueryValueKey+0x12
0236e774  0236f920
0236e778  7d61f1f8 ntdll!_except_handler3
0236e77c  7d620310 ntdll!RtlpRunTable+0x490
0236e780  0236e790
0236e784  00220000
0236e788  7d61f7b4 ntdll!RtlpAllocateFromHeapLookaside+0x13
0236e78c  00221378
0236e790  0236e9b8
0236e794  7d61f748 ntdll!RtlAllocateHeap+0x1dd
0236e798  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e79c  0236ef18
0236e7a0  00000000
0236e7a4  00000000
0236e7a8  00220000
0236e7ac  0236e89c
0236e7b0  00000000
0236e7b4  00000128
0236e7b8  00000000
0236e7bc  0236e8c8
0236e7c0  0236e7c8
0236e7c4  c0000034
0236e7c8  0236e814
0236e7cc  7d61f1f8 ntdll!_except_handler3
0236e7d0  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236e7d4  ffffffff
0236e7d8  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236e7dc  7d4ded95 kernel32!FindClose+0x9b
0236e7e0  00220000
0236e7e4  00000000
0236e7e8  00220000
0236e7ec  00000000
0236e7f0  002314b4
0236e7f4  7d61ca1d ntdll!NtQueryInformationProcess+0x12
0236e7f8  7d4da465 kernel32!GetErrorMode+0x18
0236e7fc  ffffffff
0236e800  0000000c
0236e804  7d61ca65 ntdll!ZwSetInformationProcess+0x12
0236e808  7d4da441 kernel32!SetErrorMode+0x37
0236e80c  ffffffff
0236e810  0000000c
0236e814  0236e820
0236e818  00000004
0236e81c  00000000
0236e820  00000005
0236e824  0236eae8
0236e828  7d4e445f kernel32!GetLongPathNameW+0x38f
0236e82c  7d4e4472 kernel32!GetLongPathNameW+0x3a2
0236e830  00000001
0236e834  00000103
0236e838  00000000
0236e83c  0236f712
0236e840  7efaf000
0236e844  002316f0
0236e848  0000005c
0236e84c  7efaf000
0236e850  00000004
0236e854  002314b4
0236e858  0000ea13
0236e85c  0236e894
0236e860  00456b0d advapi32!RegQueryValueExW+0x96
0236e864  00000128
0236e868  0236e888
0236e86c  0236e8ac
0236e870  0236e8c8
0236e874  0236e8a4
0236e878  0236e89c
0236e87c  0236e88c
0236e880  7d635dc4 ntdll!iswdigit+0xf
0236e884  00000064
0236e888  00000004
0236e88c  7d624d81 ntdll!RtlpValidateCurrentDirectory+0xf6
0236e890  7d635d4e ntdll!RtlIsDosDeviceName_Ustr+0x1c0
0236e894  00000064
0236e898  0236e9d0
0236e89c  0236e9e7
0236e8a0  00000000
0236e8a4  0236e9f4
0236e8a8  0236e960
0236e8ac  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236e8b0  7d6a00e0 ntdll!FastPebLock
0236e8b4  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236e8b8  0236eca4
0236e8bc  00000208
0236e8c0  0236ec94
0236e8c4  00000000
0236e8c8  00220178
0236e8cc  00000004
0236e8d0  0236eb3c
0236e8d4  0236e8c8
0236e8d8  7d624d81 ntdll!RtlpValidateCurrentDirectory+0xf6
0236e8dc  0236e8f8
0236e8e0  7d6246c1 ntdll!RtlIsDosDeviceName_Ustr+0x14
0236e8e4  0236ea1c
0236e8e8  0236ea33
0236e8ec  00000000
0236e8f0  0236ea40
0236e8f4  0236e9ac
0236e8f8  7d6248af ntdll!RtlGetFullPathName_Ustr+0x80b
0236e8fc  7d6a00e0 ntdll!FastPebLock
0236e900  7d62489d ntdll!RtlGetFullPathName_Ustr+0x15b
0236e904  0236ef18
0236e908  00000208
[...]
0236e934  00000022
0236e938  00460044 advapi32!GetPerflibKeyValue+0x19e
0236e93c  0236ecd0
0236e940  00000000
0236e944  00000044
0236e948  02080044 oleaut32!_PictSaveMetaFile+0x3f
0236e94c  00000000
0236e950  4336ec0c
[...]
0236e9a8  0236ebd0
0236e9ac  7d62155b ntdll!RtlAllocateHeap+0x460
0236e9b0  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e9b4  00000000
0236e9b8  000003ee
0236e9bc  0236ed2c
0236e9c0  7d624bcf ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x3d8
0236e9c4  7d6a00e0 ntdll!FastPebLock
0236e9c8  00000ab0
0236e9cc  00000381
0236e9d0  00233950
0236e9d4  0236ebfc
0236e9d8  7d62155b ntdll!RtlAllocateHeap+0x460
0236e9dc  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236e9e0  00000003
0236e9e4  fffffffc
0236e9e8  00000aa4
0236e9ec  00230ba0
0236e9f0  00000004
0236e9f4  003a0043
0236e9f8  00000000
0236e9fc  000a0008
0236ea00  7d624be8 ntdll!`string'
0236ea04  00000000
0236ea08  00460044 advapi32!GetPerflibKeyValue+0x19e
0236ea0c  0236ecd0
0236ea10  00233948
[...]
0236ea44  00220640
0236ea48  7d62273d ntdll!RtlIntegerToUnicode+0x126
0236ea4c  0000000c
[...]
0236eab4  0236f79c
0236eab8  7d61f1f8 ntdll!_except_handler3
0236eabc  7d622758 ntdll!RtlpIntegerWChars+0x54
0236eac0  00220178
0236eac4  0236ed3c
0236eac8  00000005
0236eacc  0236ed00
0236ead0  7d622660 ntdll!RtlConvertSidToUnicodeString+0x1cb
0236ead4  00220178
0236ead8  0236eaf0
0236eadc  0236eaec
0236eae0  00000001
0236eae4  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236eae8  00223620
0236eaec  00220178
0236eaf0  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236eaf4  002217f8
0236eaf8  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eafc  00000000
0236eb00  00220178
[...]
0236eb48  0236eb58
0236eb4c  7d635dc4 ntdll!iswdigit+0xf
0236eb50  00220178
0236eb54  00000381
0236eb58  002343f8
0236eb5c  0236eb78
0236eb60  7d620deb ntdll!RtlpCoalesceFreeBlocks+0x383
0236eb64  00000381
0236eb68  002343f8
0236eb6c  00220000
0236eb70  00233948
0236eb74  00220000
0236eb78  00000000
0236eb7c  00220000
0236eb80  0236ec60
0236eb84  7d620fbe ntdll!RtlFreeHeap+0x6b0
0236eb88  00220608
0236eb8c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eb90  000000e8
0236eb94  7d61cd23 ntdll!ZwWriteVirtualMemory
0236eb98  7efde000
0236eb9c  000000e8
0236eba0  00233948
0236eba4  7efde000
0236eba8  000002e8
0236ebac  0000005d
0236ebb0  00220178
0236ebb4  00000156
0236ebb8  0236e9b4
0236ebbc  00233948
0236ebc0  7d61f1f8 ntdll!_except_handler3
0236ebc4  00000ab0
0236ebc8  00233948
0236ebcc  00233950
0236ebd0  00220178
0236ebd4  00220000
0236ebd8  00000ab0
0236ebdc  00220178
0236ebe0  00000000
0236ebe4  00233950
0236ebe8  7d4ddea8 kernel32!`string'+0x50
0236ebec  00000000
0236ebf0  00233950
0236ebf4  00220178
0236ebf8  00000aa4
0236ebfc  00000000
0236ec00  0236ec54
0236ec04  7d63668a ntdll!RtlCreateProcessParameters+0x375
0236ec08  7d63668f ntdll!RtlCreateProcessParameters+0x37a
0236ec0c  7d6369e9 ntdll!RtlCreateProcessParameters+0x35f
0236ec10  00000000
[...]
0236ec4c  0000007f
0236ec50  0236ef4c
0236ec54  7d61f1f8 ntdll!_except_handler3
0236ec58  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236ec5c  ffffffff
0236ec60  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ec64  7d6365e2 ntdll!RtlDestroyProcessParameters+0x1b
0236ec68  00220000
0236ec6c  00000000
0236ec70  00233950
0236ec74  0236ef5c
0236ec78  7d4ec4bc kernel32!BasePushProcessParameters+0x806
0236ec7c  00233950
0236ec80  7d4ec478 kernel32!BasePushProcessParameters+0x7c5
0236ec84  7efde000
0236ec88  0236f748
0236ec8c  00000000
0236ec90  0236ed92
0236ec94  00000000
0236ec98  00000000
0236ec9c  01060104
0236eca0  0236f814
0236eca4  0020001e
0236eca8  7d535b50 kernel32!`string'
0236ecac  00780076
0236ecb0  002314e0
0236ecb4  00780076
0236ecb8  0236ed2c
0236ecbc  00020000
0236ecc0  7d4ddee4 kernel32!`string'
0236ecc4  0236efec
[...]
0236ed3c  006d0061
0236ed40  00460020 advapi32!GetPerflibKeyValue+0x17a
0236ed44  006c0069
0236ed48  00730065
0236ed4c  00280020
0236ed50  00380078
0236ed54  00290036
0236ed58  0044005c advapi32!CryptDuplicateHash+0x3
0236ed5c  00620065
0236ed60  00670075
[...]
0236ee7c  0236ee8c
0236ee80  00000001
0236ee84  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236ee88  00230dc0
0236ee8c  0236ef6c
0236ee90  0236eea0
0236ee94  00000001
0236ee98  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236ee9c  00223908
0236eea0  0236ef80
0236eea4  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236eea8  00221d38
0236eeac  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236eeb0  7d61f4ab ntdll!RtlFreeHeap
0236eeb4  7d61c91b ntdll!NtClose
0236eeb8  00000000
[...]
0236ef08  00000000
0236ef0c  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0236ef10  7efde000
0236ef14  00001000
0236ef18  00000000
0236ef1c  000000e8
0236ef20  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0236ef24  00000000
0236ef28  0236ef10
0236ef2c  00000000
0236ef30  0236f79c
0236ef34  7d61f1f8 ntdll!_except_handler3
0236ef38  7d621954 ntdll!RtlImageNtHeaderEx+0xee
0236ef3c  00220000
[...]
0236ef68  0236eeb0
0236ef6c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ef70  0236f79c
0236ef74  7d61f1f8 ntdll!_except_handler3
0236ef78  7d61f5f0 ntdll!CheckHeapFillPattern+0x64
0236ef7c  ffffffff
0236ef80  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236ef84  7d4ea183 kernel32!CreateProcessInternalW+0x21f5
0236ef88  00220000
0236ef8c  00000000
0236ef90  00223910
0236ef94  7d4ebc0b kernel32!CreateProcessInternalW+0x1f26
0236ef98  00000000
0236ef9c  00000096
0236efa0  0236f814
0236efa4  00000103
0236efa8  7efde000
0236efac  00000001
0236efb0  0236effc
0236efb4  00000200
0236efb8  00000cb0
0236efbc  0236f00c
0236efc0  0236efdc
0236efc4  7d6256e8 ntdll!bsearch+0x42
0236efc8  00180144
0236efcc  0236efe0
0236efd0  7d625992 ntdll!ARRAY_FITS+0x29
0236efd4  00000a8c
0236efd8  00000000
0236efdc  00000000
0236efe0  00080000
0236efe4  00070000
0236efe8  00040000
0236efec  00000044
0236eff0  00000000
0236eff4  7d535b50 kernel32!`string'
0236eff8  00000000
0236effc  00000000
[...]
0236f070  00000001
0236f074  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0236f078  004000e8 NullThread!_enc$textbss$begin <PERF> (NullThread+0xe8)
0236f07c  0236f0cc
0236f080  00000000
0236f084  7d6256e8 ntdll!bsearch+0x42
0236f088  00180144
0236f08c  0236f0a0
0236f090  7d625992 ntdll!ARRAY_FITS+0x29
0236f094  00000a8c
[...]
0236f0d0  0236f120
0236f0d4  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f0d8  0236f204
0236f0dc  00000020
[...]
0236f190  000002a8
0236f194  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f198  00000001
0236f19c  00000000
0236f1a0  0236f1d0
0236f1a4  7d6257f1 ntdll!RtlpFindNextActivationContextSection+0x64
0236f1a8  00181f1c
[...]
0236f1f0  7efaf000
0236f1f4  7d625ad8 ntdll!RtlFindActivationContextSectionString+0xe1
0236f1f8  0236f214
0236f1fc  0236f24c
0236f200  00000000
0236f204  7d6256e8 ntdll!bsearch+0x42
0236f208  00180144
[...]
0236f24c  00000200
0236f250  00000734
0236f254  7d625b62 ntdll!RtlpFindUnicodeStringInSection+0x7b
0236f258  0236f384
[...]
0236f3f0  00000000
0236f3f4  00000000
0236f3f8  01034236
0236f3fc  00000000
0236f400  7d4d1510 kernel32!BaseProcessStartThunk
0236f404  00000018
0236f408  00003000
[...]
0236f62c  0236f63c
0236f630  00000001
0236f634  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236f638  00231088
0236f63c  0236f71c
[...]
0236f70c  002333b8
0236f710  0236f720
0236f714  00000001
0236f718  7d61f645 ntdll!RtlpFreeToHeapLookaside+0x22
0236f71c  00228fb0
0236f720  0236f800
0236f724  7d61f5d1 ntdll!RtlFreeHeap+0x20e
0236f728  00221318
0236f72c  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236f730  00000000
0236f734  00000096
0236f738  0236f814
0236f73c  00220608
0236f740  7d61f5ed ntdll!RtlFreeHeap+0x70f
0236f744  0236f904
0236f748  008e0000
0236f74c  002334c2
[...]
0236f784  0236f7bc
0236f788  7d63d275 ntdll!_vsnwprintf+0x30
0236f78c  0236f79c
0236f790  0000f949
0236f794  0236ef98
0236f798  00000095
0236f79c  0236fb7c
0236f7a0  7d4d89c4 kernel32!_except_handler3
0236f7a4  7d4ed1d0 kernel32!`string'+0xc
0236f7a8  ffffffff
0236f7ac  7d4ebc0b kernel32!CreateProcessInternalW+0x1f26
0236f7b0  7d4d14a2 kernel32!CreateProcessW+0x2c
0236f7b4  00000000
[...]
0236f7f0  0236fb7c
0236f7f4  7d61f1f8 ntdll!_except_handler3
0236f7f8  7d61d051 ntdll!NtWaitForMultipleObjects+0x15
0236f7fc  7d61c92d ntdll!NtClose+0x12
0236f800  7d4d8e4f kernel32!CloseHandle+0x59
0236f804  00000108
0236f808  0236fb8c
0236f80c  7d535b07 kernel32!UnhandledExceptionFilter+0x815
0236f810  00000108
0236f814  00430022 advapi32!_imp__OutputDebugStringW <PERF> (advapi32+0x22)
0236f818  005c003a
0236f81c  00720050
[...]
0236f8ec  0055005c
0236f8f0  00650073
0236f8f4  00440072 advapi32!CryptDuplicateHash+0x19
0236f8f8  006d0075
0236f8fc  00730070
0236f900  006e005c
0236f904  00770065
0236f908  0064002e
0236f90c  0070006d
0236f910  0020003b
0236f914  00220071
0236f918  00000000
0236f91c  00000096
0236f920  7d4dda47 kernel32!DuplicateHandle+0xd0
0236f924  7d4dda47 kernel32!DuplicateHandle+0xd0
0236f928  0236fb8c
0236f92c  7d5358cb kernel32!UnhandledExceptionFilter+0x5f1
0236f930  0236f9f0
0236f934  00000001
0236f938  00000000
0236f93c  7d535b43 kernel32!UnhandledExceptionFilter+0x851
0236f940  00000000
0236f944  00000000
0236f948  00000000
0236f94c  0236f95c
0236f950  00000098
0236f954  000001a2
0236f958  01c423b0
0236f95c  0236fb84
0236f960  7d62155b ntdll!RtlAllocateHeap+0x460
0236f964  7d61f78c ntdll!RtlAllocateHeap+0xee7
0236f968  00000000
0236f96c  0000008c
0236f970  00000000
0236f974  7d4d8472 kernel32!$$VProc_ImageExportDirectory+0x6d4e
0236f978  0236fa1c
0236f97c  00000044
0236f980  00000000
0236f984  7d535b50 kernel32!`string'
0236f988  00000000
0236f98c  00000000
0236f990  00000000
0236f994  00000000
0236f998  00000000
0236f99c  00000000
0236f9a0  00000000
0236f9a4  00000000
0236f9a8  00000000
0236f9ac  00000000
0236f9b0  00000000
0236f9b4  00000000
0236f9b8  00000000
0236f9bc  00000000
0236f9c0  0010000e
0236f9c4  7ffe0030 SharedUserData+0x30
0236f9c8  000000e8
0236f9cc  00000108
0236f9d0  00000200
0236f9d4  00000734
0236f9d8  00000018
0236f9dc  00000000
0236f9e0  7d5621d0 kernel32!ProgramFilesEnvironment+0x74
0236f9e4  00000040
0236f9e8  00000000
0236f9ec  00000000
0236f9f0  0000000c
0236f9f4  00000000
0236f9f8  00000001
0236f9fc  00000118
0236fa00  000000e8
0236fa04  c0000005
0236fa08  00000000
0236fa0c  00000008
0236fa10  00000000
0236fa14  00000110
0236fa18  0236f814
0236fa1c  6950878a <Unloaded_faultrep.dll>+0x878a
0236fa20  00120010
0236fa24  7d51c5e4 kernel32!`string'
0236fa28  00000003
0236fa2c  05bc0047
[...]
0236fa74  0057005c
0236fa78  004b0032 advapi32!szPerflibSectionName <PERF> (advapi32+0x80032)
0236fa7c  005c0033
0236fa80  00790073
[...]
0236fac8  0000002b
0236facc  00000000
0236fad0  7d61e3e6 ntdll!ZwWow64CsrNewThread+0x12
0236fad4  00000000
[...]
0236fb44  00000000
0236fb48  00000000
0236fb4c  7d61cb0d ntdll!ZwQueryVirtualMemory+0x12
0236fb50  7d54eeb8 kernel32!_ValidateEH3RN+0xb6
0236fb54  ffffffff
0236fb58  7d4dfe28 kernel32!`string'+0x18
0236fb5c  00000000
0236fb60  0236fb78
0236fb64  0000001c
0236fb68  0000000f
0236fb6c  7d4dfe28 kernel32!`string'+0x18
0236fb70  0000f949
0236fb74  0236f814
0236fb78  7d4df000 kernel32!CheckForSameCurdir+0x39
0236fb7c  0236fbd4
0236fb80  7d4d89c4 kernel32!_except_handler3
0236fb84  7d535be0 kernel32!`string'+0xc
0236fb88  ffffffff
0236fb8c  7d535b43 kernel32!UnhandledExceptionFilter+0x851
0236fb90  7d508f4e kernel32!BaseThreadStart+0x4a
0236fb94  0236fbb4
0236fb98  7d4d8a25 kernel32!_except_handler3+0x61
0236fb9c  0236fbbc
0236fba0  00000000
0236fba4  0236fbbc
0236fba8  00000000
0236fbac  00000000
0236fbb0  00000000
0236fbb4  0236fca0
0236fbb8  0236fcf0
0236fbbc  0236fbe0
0236fbc0  7d61ec2a ntdll!ExecuteHandler2+0x26
0236fbc4  0236fca0
0236fbc8  0236ffdc
0236fbcc  0236fcf0
0236fbd0  0236fc7c
0236fbd4  0236ffdc
0236fbd8  7d61ec3e ntdll!ExecuteHandler2+0x3a
0236fbdc  0236ffdc
0236fbe0  0236fc88
0236fbe4  7d61ebfb ntdll!ExecuteHandler+0x24
0236fbe8  0236fca0
0236fbec  0236ffdc
0236fbf0  00000000
0236fbf4  0236fc7c
0236fbf8  7d4d89c4 kernel32!_except_handler3
0236fbfc  00000000
0236fc00  0036fca0
0236fc04  0236fc18
0236fc08  7d640ca6 ntdll!RtlCallVectoredContinueHandlers+0x15
0236fc0c  0236fca0
0236fc10  0236fcf0
0236fc14  7d6a0608 ntdll!RtlpCallbackEntryList
0236fc18  0236fc88
0236fc1c  7d6354c9 ntdll!RtlDispatchException+0x11f
0236fc20  0236fca0
0236fc24  0236fcf0
0236fc28  00000000
0236fc2c  00000000
[...]
0236fc88  0236ffec
0236fc8c  7d61dd26 ntdll!NtRaiseException+0x12
0236fc90  7d61ea51 ntdll!KiUserExceptionDispatcher+0x29
0236fc94  0236fca0
0236fc98  0236fcf0
0236fc9c  00000000
0236fca0  c0000005
0236fca4  00000000
0236fca8  00000000
0236fcac  00000000
0236fcb0  00000002
0236fcb4  00000008
0236fcb8  00000000
0236fcbc  00000000
0236fcc0  00000000
0236fcc4  6b021fa0
0236fcc8  78b83980
0236fccc  00000000
0236fcd0  00000000
0236fcd4  00000000
0236fcd8  7efad000
0236fcdc  023afd00
0236fce0  023af110
0236fce4  78b83980
0236fce8  010402e1
0236fcec  00000000
0236fcf0  0001003f
0236fcf4  00000000
0236fcf8  00000000
0236fcfc  00000000
0236fd00  00000000
0236fd04  00000000
0236fd08  00000000
0236fd0c  0000027f
0236fd10  00000000
0236fd14  0000ffff
0236fd18  00000000
0236fd1c  00000000
0236fd20  00000000
0236fd24  00000000
0236fd28  00000000
0236fd2c  00000000
0236fd30  00000000
0236fd34  00000000
0236fd38  00000000
0236fd3c  00000000
0236fd40  00000000
0236fd44  00000000
0236fd48  00000000
0236fd4c  00000000
0236fd50  00000000
0236fd54  00000000
0236fd58  00000000
0236fd5c  00000000
0236fd60  00000000
0236fd64  00000000
0236fd68  00000000
0236fd6c  00000000
0236fd70  00000000
0236fd74  00000000
0236fd78  00000000
0236fd7c  0000002b
0236fd80  00000053
0236fd84  0000002b
0236fd88  0000002b
0236fd8c  00000000
0236fd90  00000000
0236fd94  00000000
0236fd98  00000000
0236fd9c  47f30000
0236fda0  00000000
0236fda4  0236ffec
0236fda8  00000000
0236fdac  00000023
0236fdb0  00010246
0236fdb4  0236ffbc
0236fdb8  0000002b
0236fdbc  0000027f
0236fdc0  00000000
0236fdc4  00000000
0236fdc8  00000000
0236fdcc  00000000
0236fdd0  00000000
0236fdd4  00001f80
0236fdd8  00000000
0236fddc  00000000
[...]
0236ffb4  00000000
0236ffb8  00000000
0236ffbc  7d4dfe21 kernel32!BaseThreadStart+0x34
0236ffc0  00000000
0236ffc4  00000000
0236ffc8  00000000
0236ffcc  00000000
0236ffd0  c0000005
0236ffd4  0236ffc4
0236ffd8  0236fbb4
0236ffdc  ffffffff
0236ffe0  7d4d89c4 kernel32!_except_handler3
0236ffe4  7d4dfe28 kernel32!`string'+0x18
0236ffe8  00000000
0236ffec  00000000
0236fff0  00000000
0236fff4  00000000
0236fff8  00000000
0236fffc  00000000
02370000  ????????

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 6a)

Monday, April 28th, 2008

This is a specialization of Invalid Pointer pattern called NULL Pointer and it is the most easily recognized pattern with a straightforward fix most of the time according to my experience. Checking the pointer value to be non-NULL might not work if the pointer value is random (Wild Pointer pattern) but at least it eliminates this class of problems. NULL pointers can be NULL data pointers or NULL code pointers. The latter happens when we have a pointer to some function and we try to call it. Consider this example:

0:002> r
eax=00000000 ebx=00000000 ecx=93630000 edx=00000000 esi=00000000 edi=00000000
eip=00000000 esp=0222ffbc ebp=0222ffec iopl=0  nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
00000000 ??              ???

0:002> kv
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
0222ffb8 7d4dfe21 00000000 00000000 00000000 0×0
0222ffec 00000000 00000000 00000000 00000000 kernel32!BaseThreadStart+0×34

Clearly we have a NULL code pointer here and if we disassemble backwards the return address 7d4dfe21 or BaseThreadStart+0×34 we would suspect that BaseThreadStart function tried to call a thread start procedure:

0:002> ub 7d4dfe21
kernel32!BaseThreadStart+0x10:
7d4dfdfd mov     eax,dword ptr fs:[00000018h]
7d4dfe03 cmp     dword ptr [eax+10h],1E00h
7d4dfe0a jne     kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe0c cmp     byte ptr [kernel32!BaseRunningInServerProcess (7d560008)],0
7d4dfe13 jne     kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe15 call    dword ptr [kernel32!_imp__CsrNewThread (7d4d0310)]
7d4dfe1b push    dword ptr [ebp+0Ch]
7d4dfe1e call    dword ptr [ebp+8]

0:002> dp ebp+8 l1
0222fff4  00000000

To confirm this suspicion we can write a code that calls CreateThread function similar to this one:

typedef DWORD (WINAPI *THREADPROC)(PVOID);

DWORD WINAPI ThreadProc(PVOID pvParam)
{
  // Does some work
  return 0;
}

void foo()
{
  //..
  THREADPROC thProc = ThreadProc;
  //..
  // thProc becomes NULL because of a bug
  //..
  HANDLE Thread = CreateThread(NULL, 0, thProc, 0, 0, NULL);
  CloseHandle(hThread);
}

- Dmitry Vostokov @ DumpAnalysis.org -

Debugging MDAA Volume One (Errata)

Monday, April 28th, 2008

I have created Book Errata page at this address:

http://www.dumpanalysis.org/MDAA-Volume1-Errata

The link to it has been also added to the main book ad page:

Memory Dump Analysis Anthology, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -

Windows® Debugging Notebook

Friday, April 25th, 2008

This is the next scheduled book from Crash Dump Analysis Publishing Roadmap:

  • Title: Windows® Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
  • Authors: Roberto Alexis Farah, Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • ISBN-13: 978-1-906717-00-1
  • Publisher: Opentask (1 December 2009)
  • Paperback: 256 pages
  • ISBN-13: 978-0-9558328-5-7
  • Publisher: Opentask (1 February 2010)
  • Hardcover (Cloth): 256 pages

Draft Table of Contents will be published next month together with a sample chapter.

- Dmitry Vostokov @ DumpAnalysis.org -

MDAA Volume One Goes Digital

Friday, April 25th, 2008

Due to demand from people that prefer ebooks I published Memory Dump Analysis Anthology, Volume 1 in a digital format that can be purchased in Crash Dump Analysis Store. This format has color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

Search Inside for MDAA Volume 1

Thursday, April 24th, 2008

I’ve made available Search Inside for MDAA V1 book on Amazon:

Amazon Search Inside

It is still not available for purchase there but will be in a few weeks because I use a different POD company for WW distribution and I made my submission too late. In the mean time you can buy it directly from Dump Analysis Store. Google Book Search will also be available soon too.

Note: The book is shown as color book inside but it is B/W in printed form. I apologize for any confusion that might have arisen from this fact. I’m working on a digital version and it will have color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

Google Finance and Crash Dumps

Wednesday, April 23rd, 2008

In the past two hours I noticed visitors from this URL:

http://finance.google.com/finance?q=AAPL

Checked it and found that my previous post with secondary bugcheck callback data from my Mac Mini running Windows Vista hit complex RSS feed triggers. I saved the picture of this historic moment. Click on it to enlarge:

Has anyone studied the influence of crash dumps on stock market volatility and crashes? Perhaps there is some similarity between OS thread exceptions and share price trends and it is possible to metaphorically map crash dump analysis patterns to finance domain like I’m doing for Project Failure Analysis.

:-)

- Dmitry Vostokov @ DumpAnalysis.org -

Bugcheck Callbacks

Wednesday, April 23rd, 2008

There are some improvements in Vista and Windows Server 2008 regarding various WER callbacks to write user-defined data in the case of application crashes and hangs. See MSDN documentation:

What’s New in WER

However I have found that many engineers are not aware that the similar mechanism exists in kernel for many years:

Writing a Bug Check Callback Routine

You can check this data using !bugdump and .enumtag WinDbg commands:

0: kd> !bugdump
**** Dump of Bug Check Data ****
8526ba7c: Bug check callback record could not be read

We get “could not be read” message probably because for systems newer than Windows XP SP1 !bugdump command shows callback data written to memory after the crash dump was saved. So it is useful for live debugging only. However we can see that bugcheck callbacks form a linked list:

0: kd> dps 8526ba7c
8526ba7c  849eca7c
8526ba80  81b36ce0 nt!KeBugCheckCallbackListHead
8526ba84  858a7dea ndis!ndisBugcheckHandler
8526ba88  8526b438
8526ba8c  00000b28
8526ba90  8594dd76 ndis! ?? ::LNCPHCLB::`string’
8526ba94  90461ac0
8526ba98  00000001
8526ba9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa4  85969274 ndis!ethFilterDprIndicateReceivePacket
8526baa8  8de66c5c bthpan!MpReturnPacket
8526baac  8526ea80
8526bab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
8526bab4  8526b438
8526bab8  00000000

0: kd> !list -x "dps @$extret l10" 81b36ce0
81b36ce0  8526ba7c
81b36ce4  81ddbe40 hal!HalpCallbackRecord
81b36ce8  00000000
81b36cec  00000001
81b36cf0  00000000
81b36cf4  00000000
81b36cf8  00000101
81b36cfc  00000001
81b36d00  00000000
81b36d04  00000000
81b36d08  00000000
81b36d0c  00000000
81b36d10  00000000
81b36d14  00000000
81b36d18  00000000
81b36d1c  00000000

8526ba7c  849eca7c
8526ba80  81b36ce0 nt!KeBugCheckCallbackListHead
8526ba84  858a7dea ndis!ndisBugcheckHandler
8526ba88  8526b438
8526ba8c  00000b28
8526ba90  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8526ba94  90461ac0
8526ba98  00000001
8526ba9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa4  85969274 ndis!ethFilterDprIndicateReceivePacket
8526baa8  8de66c5c bthpan!MpReturnPacket
8526baac  8526ea80
8526bab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
8526bab4  8526b438
8526bab8  00000000

849eca7c  849ea72c
849eca80  8526ba7c
849eca84  858a7dea ndis!ndisBugcheckHandler
849eca88  849ec438
849eca8c  00000b28
849eca90  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849eca94  8fbe2ac0
849eca98  00000001
849eca9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ecaa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ecaa4  859432ca ndis!ndisMIndicatePacket
849ecaa8  00000000
849ecaac  00000000
849ecab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
849ecab4  849ec438
849ecab8  00000000

849ea72c  849c272c
849ea730  849eca7c
849ea734  858a7dea ndis!ndisBugcheckHandler
849ea738  849ea0e8
849ea73c  00000b28
849ea740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849ea744  8fbe0770
849ea748  00000001
849ea74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ea750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ea754  85969274 ndis!ethFilterDprIndicateReceivePacket
849ea758  00000000
849ea75c  00000000
849ea760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849ea764  849ea0e8
849ea768  00000000

849c272c  849c172c
849c2730  849ea72c
849c2734  858a7dea ndis!ndisBugcheckHandler
849c2738  849c20e8
849c273c  00000b28
849c2740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849c2744  8fbb8770
849c2748  00000001
849c274c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c2750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c2754  85969274 ndis!ethFilterDprIndicateReceivePacket
849c2758  85df579a tunmp!TunMpReturnPacket
849c275c  84a45538
849c2760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849c2764  849c20e8
849c2768  00000000

849c172c  849a072c
849c1730  849c272c
849c1734  858a7dea ndis!ndisBugcheckHandler
849c1738  849c10e8
849c173c  00000b28
849c1740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849c1744  8fbb7770
849c1748  00000001
849c174c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c1750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c1754  859432ca ndis!ndisMIndicatePacket
849c1758  00000000
849c175c  00000000
849c1760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849c1764  849c10e8
849c1768  00000000

849a072c  8499d72c
849a0730  849c172c
849a0734  858a7dea ndis!ndisBugcheckHandler
849a0738  849a00e8
849a073c  00000b28
849a0740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849a0744  8fb96770
849a0748  00000001
849a074c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849a0750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849a0754  859432ca ndis!ndisMIndicatePacket
849a0758  00000000
849a075c  00000000
849a0760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849a0764  849a00e8
849a0768  00000000

8499d72c  8499f72c
8499d730  849a072c
8499d734  858a7dea ndis!ndisBugcheckHandler
8499d738  8499d0e8
8499d73c  00000b28
8499d740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8499d744  8fb93770
8499d748  00000001
8499d74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499d750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499d754  859432ca ndis!ndisMIndicatePacket
8499d758  00000000
8499d75c  00000000
8499d760  859495ef ndis!ndisSynchReturnPacketsForTranslation
8499d764  8499d0e8
8499d768  00000000

8499f72c  81ddbe40 hal!HalpCallbackRecord
8499f730  8499d72c
8499f734  858a7dea ndis!ndisBugcheckHandler
8499f738  8499f0e8
8499f73c  00000b28
8499f740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8499f744  8fb95770
8499f748  00000001
8499f74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499f750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499f754  859432ca ndis!ndisMIndicatePacket
8499f758  00000000
8499f75c  00000000
8499f760  859495ef ndis!ndisSynchReturnPacketsForTranslation
8499f764  8499f0e8
8499f768  00000000

81ddbe40  81b36ce0 nt!KeBugCheckCallbackListHead
81ddbe44  8499f72c
81ddbe48  81dcebdc hal!HalpBugCheckCallback
81ddbe4c  00000000
81ddbe50  00000000
81ddbe54  81dc2550 hal!HalName
81ddbe58  03b9112c
81ddbe5c  00000001
81ddbe60  00000000
81ddbe64  00000000
81ddbe68  00000000
81ddbe6c  00000000
81ddbe70  6d46da80
81ddbe74  00000000
81ddbe78  00000000
81ddbe7c  00000000

Another WinDbg command .enumtag shows data written before saving a crash dump and therefore useful for postmortem crash dump analysis (binary output is removed for visual clarity):

0: kd> .enumtag
{BC5C008F-1E3A-44D7-988D86F6884C6758} - 0x5cd bytes
  ...$............
  ................
  Apple Inc..    M
  M21.88Z.009A.B00
  .0706281359.06/2
  8/07............
  ................
  .Apple Inc..Macm
  ini2,1.1.0.    
        .System SK
  UNumber.Napa Mac
  ................
  ..Apple Inc..Mac
  -F4208EAA.PVT. .
  .Part Compon
  ent.............
  ..........Apple
  Inc..Mac-F4208EA
  A.           . 
  ............J6H1
  :1-X CMOS CLEAR(
  default); J8H1:1
  -X BIOS RECOVERY
  ...........None.
  Ethernet........
  ...None.DVI.....
  ......None.USB0.
  ..........None.U
  SB1...........No
  ne.USB2.........
  ..None.USB3.....
  ....!.None.FireW
  ire0...........N
  one.Audio Line I
  n...........None
  .Audio Line Out.
  ..............Ai
  rPort........Int
  egrated Graphics
  Controller ....
  ....Yukon Ethern
  et Controller...
  .....Azalia Audi
  o Codec........S
  ATA........PATA.
  ..........#.....
  .............&.&
  .A..........Inte
  l(R) Core(TM)2 C
  PU         T.Int
  el(R) Corporatio
  n.U2E1.       ..
[...]
  .......Intel(R)
  Core(TM)2 CPU  
       T.Intel(R)
  Corporation.U2E
  1.       .......
[...]
  ...........DIMM0
  .BANK 0.0x2C0000
  0000000000.    
      .       .0x
  3848544636343634
  4844592D36363744
  3320....!.......
  .. .$........"..
  ...@.@..........
  ......DIMM1.BANK
  1.0x2C000000000
  00000.         
  .       .0x38485
  4463634363448445
  92D363637443320.
[...]
{6C7AC389-4313-47DC-9F34A8800A0FB56C} - 0x266 bytes
  ....~.M.H.z.....
  ......)...,...C.
  o.m.p.o.n.e.n.t.
  .I.n.f.o.r.m.a.
  t.i.o.n.........
  ..&...C.o.n.f.i.
  g.u.r.a.t.i.o.n.
  .D.a.t.a.......
  ........I.d.e.n.
  t.i.f.i.e.r.....
  ..B...x.8.6. .F.
  a.m.i.l.y. .6. .
  M.o.d.e.l. .1.5.
  .S.t.e.p.p.i.n.
  g. .2...(...P.r.
  o.c.e.s.s.o.r.N.
  a.m.e.S.t.r.i.n.
  g.......`...I.n.
  t.e.l.(.R.). .C.
  o.r.e.(.T.M.).2.
  .C.P.U. . . . .
  . . . . .T.5.6.
  0.0. . .@. .1...
  8.3.G.H.z..."...
  U.p.d.a.t.e. .S.
  i.g.n.a.t.u.r.e.
  ..............W.
  ......U.p.d.a.t.
  e. .S.t.a.t.u.s.
  ..............".
  ..V.e.n.d.o.r.I.
  d.e.n.t.i.f.i.e.
  r...........G.e.
  n.u.i.n.e.I.n.t.
  e.l.......M.S.R.
[...]
{D03DC06F-D88E-44C5-BA2AFAE035172D19} - 0x438 bytes
  ............Genu
  ntelineI....Genu
  ntelineI........
[...]
  ........Intel(R)
  Core(TMIntel(R)
  Core(TM........
  )2 CPU         T
  )2 CPU         T
  ........5600  @
  1.83GHz.5600  @
  1.83GHz.........
[...]
{E83B40D2-B0A0-4842-ABEA71C9E3463DD1} - 0x184 bytes
  APICh.....APPLE
  Apple00.....Loki
  _.......FACP....
  .aAPPLE Apple00.
  ....Loki_......>
  HPET8.....APPLE
  Apple00.....Loki
  _.......MCFG<...
  ..APPLE Apple00.
  ....Loki_.......
  ASF!.... .APPLE
  Apple00.....Loki
  _.......SBST0...
  ..APPLE Apple00.
  ....Loki_.......
  ECDTS....9APPLE
  Apple00.....Loki
  _.......SSDTO...
  .>APPLE SataPri.
  ....INTL... SSDT
  O....>APPLE Sata
  Pri.....INTL...
  SSDTO....>APPLE
  SataPri.....INTL
{270A33FD-3DA6-460D-BA893C1BAE21E39B} - 0xfc8 bytes
  ........H.......
  H.......H.......
[...]

Of course, this is much more useful if your drivers save additional data for troubleshooting and you have written a WinDbg extension to interpret it.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 59)

Tuesday, April 22nd, 2008

David V provided an idea and a user dump for the next pattern which I call Missing Component. Sometimes the code raises an exception when certain DLL is missing. We need to guess that component name if we don’t have symbols and source code. This can be done by inspecting raw stack data in the close proximity of the exception ESP/RSP.

Consider the crash dump of Zune.exe with the following incomplete unmanaged stack trace:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 76f442eb (kernel32!RaiseException+0x00000058)
   ExceptionCode: c06d007f
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0024f21c

0:000> kL
ChildEBP RetAddr 
0024f1f8 6eb1081e kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
0024f260 6eac62fb ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f2ac 6ea9e269 ZuneNativeLib!ZuneLibraryExports::Phase2Initialization+0x24c9
0024f32c 79e74d79 ZuneNativeLib!ZuneLibraryExports::QueryDatabase+0x99da
0024f3d4 664bd6af mscorwks!MethodTable::IsValueType+0x35
0024f3e8 319cec9e ZuneShell_ni+0x2d6af
0024f3f4 31a15d19 UIX_ni+0x1ec9e
0024f3f8 00000000 UIX_ni+0x65d19

We can try to interpret the crash as Managed Code Exception but let’s first to check the exception code. Google search shows that the error code c06d007f means “DelayLoad Export Missing” and this definitely has to do with some missing DLL. It is not possible to tell which one was missing from the stack trace output. Additional digging is required.

Let’s look at the raw stack. First, we can try to see whether there are any calls to LoadLibrary on thread raw stack data:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0024f8c4
    StackBase:            00250000
    StackLimit:           00249000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             000012f4 . 00001080
    RpcHandle:            00000000
    Tls Storage:          004e8a18
    PEB Address:          7ffde000
    LastErrorValue:       126
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00249000 00250000
00249000 00000000
00249004 00000000
00249008 00000000
0024900c 00000000
00249010 00000000
00249014 00000000
00249018 00000000
[...]
0024f1a0 00000000
0024f1a4 00000000
0024f1a8 c06d007f
0024f1ac 00000000
0024f1b0 00000000
0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0x11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0x15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
[...]

There are no such calls in our crash dump. Then we can try to interpret raw stack data as a byte stream to see “.dll” strings:

0:000> db 00249000 00250000
00249000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249030 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
[...]

There are no such strings except “user32.dll”.

Now we can try to interpret every double word as a pointer to a Unicode string:

0:000> dpu 00249000 00250000
[...]

There are no strings with “.dll” inside. Finally, if we try to interpret every double word as a pointer to an ASCII string we get a few references to “ZuneService.dll”:

0:000> dpa 00249000 00250000
[...]
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 "MZ."
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8 ""
0024f1dc 76fb8e8f "..t-.E."
0024f1e0 6ecb9b40 “ZuneService.dll”
0024f1e4 0024f21c “$”
0024f1e8 0024f200 “.”
0024f1ec 6ec74e2a “..^.._]..”
0024f1f0 6ecb9b40 “ZuneService.dll”
0024f1f4 6ecb9ff0 “CreateServiceInstance”
0024f1f8 0024f260 “..$”
0024f1fc 6eb1081e “.]…….e.”
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268 “..$”
0024f210 00000000
0024f214 0024f2c8 “…n ..n<.$”
0024f218 6ecbe220 “”
0024f21c 00000024
0024f220 6ecb9960 “.”
0024f224 6ecbe05c “.c.n.2.n”
0024f228 6ecb9b40 “ZuneService.dll”
0024f22c 00000001
0024f230 6ecb9ff0 “CreateServiceInstance”
0024f234 ffffffff
0024f238 00000000

If we search for 0024f1e0 pointer in dps WinDbg command output we would see that it is in a close proximity to RaiseException call and it seems that all our pointers to “ZuneService.dll” string fall into ZuneNativeLib address range:

0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0×11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0×6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268
0024f210 00000000
0024f214 0024f2c8
0024f218 6ecbe220 ZuneNativeLib!ShutdownSingletonMgr+0×15f704
0024f21c 00000024
0024f220 6ecb9960 ZuneNativeLib!ShutdownSingletonMgr+0×15ae44
0024f224 6ecbe05c ZuneNativeLib!ShutdownSingletonMgr+0×15f540
0024f228 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f22c 00000001
0024f230 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f234 ffffffff
0024f238 00000000

When examining the system it was found that ZuneService.dll was missing there indeed.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugchecks: SYSTEM_SERVICE_EXCEPTION

Tuesday, April 22nd, 2008

Bugcheck 0×3B is forced on x64 Windows platforms when an exception happens during a system service and unwind leads to a transition from a kernel to a user mode. Let’s see this in a complete memory dump:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff80001048a1d, Address of the exception record for the exception that caused the bugcheck
Arg3: fffffade643f6870, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

CONTEXT: fffffade643f6870 -- (.cxr 0xfffffade643f6870)
rax=005300450053005c rbx=0000000000000048 rcx=0000000000000020
rdx=fffffa8007c9da20 rsi=0000000000000048 rdi=fffffade643f71d0
rip=fffff80001048a1d rsp=fffffade643f7088 rbp=0000000000000000
 r8=0000000000000048 r9=0000000000000002 r10=00490046002d0054
r11=0000000000000000 r12=fffffadf19744010 r13=fffffade643f7a78
r14=0000000000000800 r15=fffffadf1da71ee8
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202
nt!memmove+0xbd:
fffff800`01048a1d 488941e0 mov qword ptr [rcx-20h],rax ds:002b:00000000`00000000=????????????????

0: kd> kL
Child-SP RetAddr Call Site
fffffade`643f5eb8 fffff800`0104e834 nt!KeBugCheckEx
fffffade`643f5ec0 fffff800`0104e2fb nt!KiBugCheckDispatch+0x74
fffffade`643f6040 fffff800`0105c09d nt!KiSystemServiceHandler+0x7b
fffffade`643f6080 fffff800`01031561 nt!RtlpExecuteHandlerForException+0xd
fffffade`643f60b0 fffff800`010174fa nt!RtlDispatchException+0x2c0
fffffade`643f6770 fffff800`0104e92f nt!KiDispatchException+0xd9
fffffade`643f6d70 fffff800`0104d7e1 nt!KiExceptionExit
fffffade`643f6ef0 fffff800`01048a1d nt!KiPageFault+0x1e1
fffffade`643f7088 fffff800`01025977 nt!memmove+0xbd
fffffade`643f7090 fffffadf`101f858d nt!RtlAppendUnicodeStringToString+0x67
fffffade`643f70c0 fffffadf`101f8a1d driver+0x558d
fffffade`643f7a20 fffff800`012c3b21 driver+0x5a1d
fffffade`643f7a70 fffff800`012c3bd6 nt!IopXxxControlFile+0xa6b
fffffade`643f7b90 fffff800`0104e5fd nt!NtDeviceIoControlFile+0x56
fffffade`643f7c00 00000000`77ef12ca nt!KiSystemServiceCopyEnd+0×3
00000000`00e6ba08 00000000`77d67963 ntdll!ZwDeviceIoControlFile+0xa
00000000`00e6ba10 00000000`6340239f kernel32!DeviceIoControl+0×237
00000000`00e6bbf0 00000000`0000000e application!DllUnregisterServer+0×40f
[…]

On x64 Windows platforms KiSystemServiceCopyEnd has the similar purpose as KiFastSystemCallRet on x86 platforms.

We see that the chain of exception handlers spans protection boundary where KiSystemServiceCopyEnd has KiSystemServiceHandler as its exception handler:

0: kd> !exchain
100 stack frames, scanning for handlers...
Frame 0x03: nt!RtlpExecuteHandlerForException+0xd (fffff800`0105c09d)
  ehandler nt!RtlpExceptionHandler (fffff800`0105c060)
Frame 0x05: nt!KiDispatchException+0xd9 (fffff800`010174fa)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0x0a: driver+0x558d (fffffadf`101f858d)
  ehandler driver+0x1242 (fffffadf`101f4242)
Frame 0x0c: nt!IopXxxControlFile+0xa6b (fffff800`012c3b21)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0×0e: nt!KiSystemServiceCopyEnd+0×3 (fffff800`0104e5fd)
  ehandler nt!KiSystemServiceHandler (fffff800`0104e280)

Frame 0×10: kernel32!DeviceIoControl+0×237 (00000000`77d67963)
  ehandler kernel32!_C_specific_handler (00000000`77d92200)

If we disassemble KiSystemServiceHandler we get this code with bugcheck 3B branch:

kd> uf nt!KiSystemServiceHandler
nt!KiSystemServiceHandler:
[...]
fffff800`01040ddc cmp     byte ptr [rax+153h],0
fffff800`01040de3 je      nt!KiSystemServiceHandler+0×7b (fffff800`01040dfb)

nt!KiSystemServiceHandler+0x65:
fffff800`01040de5 xor     r10,r10
fffff800`01040de8 mov     r9,r8
fffff800`01040deb mov     r8,qword ptr [rcx+10h]
fffff800`01040def mov     edx,dword ptr [rcx]
fffff800`01040df1 mov     ecx,3Bh
fffff800`01040df6 call    nt!KiBugCheckDispatch (fffff800`01041300)

nt!KiSystemServiceHandler+0x7b:
fffff800`01040dfb mov     eax,1
fffff800`01040e00 add     rsp,38h
fffff800`01040e04 ret
[...]

Here we see that the code checks if the previous mode for a thread was UserMode and if this is the case it bugchecks the system because transitioning back to  user space in exception unwind would have had disastrous consequences. The system wants to save a controlled crash dump for later problem analysis:

kd> dt _KTHREAD
ntdll!_KTHREAD
   +0x000 Header           : _DISPATCHER_HEADER
   +0x018 MutantListHead   : _LIST_ENTRY
   +0x028 InitialStack     : Ptr64 Void
   +0x030 StackLimit       : Ptr64 Void
   +0x038 KernelStack      : Ptr64 Void
[...]
   +0×153 PreviousMode     : Char
[…]

Note that _KTHREAD.PreviousMode should not be confused with _KTRAP_FRAME.PreviousMode. The latter has KernelMode value if an exception happened while CPU was in kernel mode but the former structure field shows the previous CPU mode of a thread, for example, it has UserMode value if a user space thread called a system service.

kd> dt _KTRAP_FRAME
ntdll!_KTRAP_FRAME
  +0x000 P1Home : Uint8B
  +0x008 P2Home : Uint8B
  +0x010 P3Home : Uint8B
  +0x018 P4Home : Uint8B
  +0x020 P5 : Uint8B
  +0×028 PreviousMode : Char
  +0×029 PreviousIrql : UChar
  +0×02a FaultIndicator : UChar
[…]

I put all of this on a colored sequence UML diagram:

 

- Dmitry Vostokov @ DumpAnalysis.org -

What was this process doing?

Monday, April 21st, 2008

This is a common question we have when faced with stack traces for which we don’t have symbols. Consider the following stack trace from a complete memory dump where a bugcheck thread belongs to one graphical application:

2: kd> kL 100
ChildEBP RetAddr 
aa1999b4 8082d800 nt!KeBugCheckEx+0x1b
aa199d78 8088a262 nt!KiDispatchException+0x3a2
aa199de0 8088a216 nt!CommonDispatchException+0x4a
aa199e5c bfe7e5b7 nt!KiExceptionExit+0x186
[...]
aa19a110 bf8b2fe6 win32k!GrePolyPatBlt+0x45
aa19a148 bf89422b win32k!FillRect+0x58
aa19a16c bf8942f7 win32k!xxxPaintRect+0x70
aa19a19c bf8942ac win32k!xxxFillWindow+0x3e
aa19a1b4 bf8adf6e win32k!xxxDWP_EraseBkgnd+0x51
aa19a214 bf884771 win32k!xxxRealDefWindowProc+0x318
aa19a22c bf8847a1 win32k!xxxWrapRealDefWindowProc+0x16
aa19a248 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
aa19a280 8088978c win32k!NtUserMessageCall+0xc0
aa19a280 7c8285ec nt!KiFastCallEntry+0xfc
0013f68c 7739d1ec ntdll!KiFastSystemCallRet
0013f6e0 7739c6ae USER32!NtUserMessageCall+0xc
0013f6fc 7739c718 USER32!RealDefWindowProcW+0x47
0013f744 3003a5b3 USER32!DefWindowProcW+0x72
0013f75c 300a0d72 Application+0x3a5b3
0013f7bc 300a0cb2 Application+0xa0d72
0013f7f4 7739b6e3 Application+0xa0cb2
0013f820 7739b874 USER32!InternalCallWinProc+0x28
0013f898 7739c8b8 USER32!UserCallWinProcCheckWow+0x151
0013f8f4 7739c9c6 USER32!DispatchClientMessage+0xd9
0013f91c 7c828536 USER32!__fnDWORD+0x24
0013f91c 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
aa19a564 8091d6d1 nt!KiCallUserMode+0x4
aa19a5bc bf8a2622 nt!KeUserModeCallback+0x8f
aa19a640 bf8a242d win32k!SfnDWORD+0xb4
aa19a688 bf8a13d9 win32k!xxxSendMessageToClient+0x176
aa19a6d4 bf8a12ee win32k!xxxSendMessageTimeout+0x1a6
aa19a6f8 bf8c1342 win32k!xxxSendMessage+0x1b
aa19a71c bf85e0a1 win32k!xxxSendEraseBkgnd+0x5c
aa19a73c bf85dee1 win32k!xxxSimpleDoSyncPaint+0xc6
aa19a758 bf8ae16d win32k!xxxInternalDoSyncPaint+0x12
aa19a7b4 bf884771 win32k!xxxRealDefWindowProc+0x753
aa19a7cc bf8847a1 win32k!xxxWrapRealDefWindowProc+0x16
aa19a7e8 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
aa19a820 8088978c win32k!NtUserMessageCall+0xc0
aa19a820 7c8285ec nt!KiFastCallEntry+0xfc
0013f91c 7c828536 ntdll!KiFastSystemCallRet
0013f91c 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
aa19ab00 8091d6d1 nt!KiCallUserMode+0x4
aa19ab58 bf8a2622 nt!KeUserModeCallback+0x8f
aa19abdc bf8a242d win32k!SfnDWORD+0xb4
aa19ac24 bf8c4177 win32k!xxxSendMessageToClient+0x176
aa19ac94 bf89b829 win32k!xxxReceiveMessage+0x2b5
aa19ace4 bf89c4d9 win32k!xxxRealInternalGetMessage+0x1da
aa19ad48 8088978c win32k!NtUserPeekMessage+0x42
aa19ad48 7c8285ec nt!KiFastCallEntry+0xfc
0013fbd8 7c828536 ntdll!KiFastSystemCallRet
0013fc04 7739bde5 ntdll!KiUserCallbackDispatcher+0x2e
0013fc30 7739be5e USER32!NtUserPeekMessage+0xc
0013fc5c 3002baa0 USER32!PeekMessageW+0xab
0013fc84 3002b556 Application+0x2baa0
0013fca8 3000abf5 Application+0x2b556
0013fcf4 30005dfd Application+0xabf5
0013ff34 3000248c Application+0x5dfd
0013ffc0 77e6f23b Application+0x248c
0013fff0 00000000 kernel32!BaseProcessStart+0x23

The thread seems to be doing some drawing in response to WM_ERASEBKGND message generated from the code processing WM_TIMER:

2: kd> kv 100
[...]
aa19a6f8 bf8c1342 be63f8b8 00000014 91010979 win32k!xxxSendMessage+0×1b
aa19a71c bf85e0a1 be63f8b8 00000000 00000001 win32k!xxxSendEraseBkgnd+0×5c
[…]
0013fc5c 3002baa0 0013fcc0 00000000 00000000 USER32!PeekMessageW+0xab
[…]

2: kd> dd 0013fcc0 l4
0013fcc0  00000000 00000113 000066c2 00000000

The first parameter to PeekMessage function is a pointer to MSG structure whose second member is a message code (from MSDN): 

BOOL PeekMessage(
    LPMSG lpMsg,
    HWND hWnd,
    UINT wMsgFilterMin,
    UINT wMsgFilterMax,
    UINT wRemoveMsg
);

typedef struct {
    HWND hwnd;
    UINT message;
    WPARAM wParam;
    LPARAM lParam;
    DWORD time;
    POINT pt;
} MSG, *PMSG;

In WinUser.h we can find message codes:

#define WM_ERASEBKGND  0x0014
#define WM_TIMER       0x0113

Now we can ask the next troubleshooting question: what was the application file loaded before the system crash? We know that the application uses EXT file extension for its data. If we look at the handle table we find the only one such instance of File object:

2: kd> !handle
processor number 2, process a31a4a08
PROCESS a31a4a08  SessionId: 1  Cid: 2440    Peb: 7ffd7000  ParentCid: 1180
    DirBase: bffca720  ObjectTable: ddc38eb8  HandleCount: 291.
    Image: Application.EXE

Handle table at dcb65000 with 291 Entries in use

[...]

03f4: Object: a2ee85b0  GrantedAccess: 00120089 Entry: dcb657e8
Object: a2ee85b0  Type: (a55c8ca0) File
    ObjectHeader: a2ee8598 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \Profiles\MYNAME\LOCALS~1\Temp\APPDATA\MyFile.ext {HarddiskVolume3}

[...]

Now we can check other crash dumps to see whether there is any consistency in file names.

- Dmitry Vostokov @ DumpAnalysis.org -

What does this function do?

Saturday, April 19th, 2008

Often I’m asked about what a particular function that we see on a stack trace does. Over the time I found the following function name and purpose mining techniques and resources useful:

  • - We might need to strip or replace prefixes and suffixes like

NtUserGetMessage

GetMessageW

ZwReadFile <-> NtReadFile

  • - Search in MSDN, Platform SDK and WDK (formerly DDK) help
  • - Various blogs like this excellent summary:

A catalog of NTDLL kernel mode to user mode callbacks

  • - Reverse engineering and logical deduction:

 What is KiFastSystemCallRet?

  • - Various books like this:

Windows NT/2000 Native API Reference

Buy from Amazon

  • - Win32 API emulators like WINE
  • - and finally Windows source code if you are a Microsoft source code licensee or a participant in Windows Academic Program.
  • - Sometimes Internet search finds the description of the whole stack trace collection from the class of common processes like this one:

Production Debugging for .NET Framework Applications 

- Dmitry Vostokov @ DumpAnalysis.org -

Microsoft DLL Help Database

Thursday, April 17th, 2008

Just to remind about this sometimes useful resource where we can check the product and indirectly any updates for the particular module if we have its file name and version from a crash dump, for example:

http://support.microsoft.com/dllhelp/

We can event see exports and component dependencies if we need to quickly check them without running depends.exe:

- Dmitry Vostokov @ DumpAnalysis.org -

Colorimetric Computer Memory Dating (Part 1)

Wednesday, April 16th, 2008

Similar to radiometric dating using isotopes we can use memory visualization techniques to see distribution of allocated buffers and their retention over time. The key is to allocate colored memory. For example, to append a red buffer that contains RGBA values 0xFF000000 to specific allocations. I call these colored memory marks isomemotopes

We can either inject a different isomemotope for a different data or change the isomemotope over time to mark specific allocation times. I created a test program that allocates buffers marked by a different amount of different isomemotopes every time: 

#include "stdafx.h"
#include <stdlib.h>
#include <memory.h>
#include <windows.h>

typedef unsigned int ISOMEMOTOPE;

void *alloc_and_mark_with_isomemotope(size_t size,
                                     ISOMEMOTOPE color,
                                     size_t amount)
{
  char *p = (char *)malloc(size+amount);

  for (char *isop = p+size;
       p && isop  < p+size+amount;
       isop+=sizeof(ISOMEMOTOPE))
  {
    *(ISOMEMOTOPE *)isop=color;
  }

  return p;
}

int _tmain(int argc, _TCHAR* argv[])
{
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFF000000, // red 
                                 0x10000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x00FF0000, // green
                                 0x20000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x0000FF00, // blue
                                 0x30000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFFFF00, // white
                                 0x40000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFF0000, // yellow
                                 0x50000);

  DebugBreak();

  return 0;
}

Corresponding Dump2Picture image is this (0×00000000 address is at the bottom):

 

:-)

- Dmitry Vostokov @ DumpAnalysis.org -

The First Windows® Memory Dump Analysis Book!

Tuesday, April 15th, 2008

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg as a Binary Editor

Tuesday, April 15th, 2008

Sometimes we have a binary file or even a text file where we want to alter some bytes but we don’t have any binary editor at hand. We can use WinDbg for this purpose. To illustrate this, I created hello.bin file with “Hello World!” in its contents. Suppose we want to change it to “Hello WinDbg!”. First, we need to open any available full process user dump file and then get the list of valid address ranges by either using !address or lm command:

0:000> lm
start             end                 module name
00000000`00400000 00000000`0044d000   TestDefaultDebugger64
00000000`77850000 00000000`77981000   kernel32
00000000`77990000 00000000`77a5a000   user32
00000000`77a60000 00000000`77bda000   ntdll
000007fe`f8940000 000007fe`f8997000   winspool
000007fe`fcb00000 000007fe`fccf0000   comctl32
000007fe`fcfc0000 000007fe`fd012000   uxtheme
000007fe`fe1d0000 000007fe`fe2d4000   msctf
000007fe`fe380000 000007fe`fe3f1000   shlwapi
000007fe`fe660000 000007fe`fe799000   rpcrt4
000007fe`fe9f0000 000007fe`feac8000   oleaut32
000007fe`fead0000 000007fe`ff704000   shell32
000007fe`ff880000 000007fe`ff91a000   usp10
000007fe`ff920000 000007fe`ff9c1000   msvcrt
000007fe`ff9d0000 000007fe`ff9f8000   imm32
000007fe`ffa00000 000007fe`ffbe0000   ole32
000007fe`ffbe0000 000007fe`ffcdf000   advapi32
000007fe`ffce0000 000007fe`ffcec000   lpk
000007fe`ffcf0000 000007fe`ffd51000   gdi32

Let’s choose 00000000`00400000 address. It points to the following memory data:

0:000> dc 00000000`00400000
00000000`00400000  00905a4d 00000003 00000004 0000ffff  MZ..............
00000000`00400010  000000b8 00000000 00000040 00000000  ........@.......
00000000`00400020  00000000 00000000 00000000 00000000  ................
00000000`00400030  00000000 00000000 00000000 000000e8  ................
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ........!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode....$.......

Now we load our hello.bin by specifying the this address and the number of bytes to load:

0:000> .readmem c:\dmitri\hello.bin 00000000`00400000 L0n12
Reading c bytes.

We see the new memory data immediately:

0:000> dc 00000000`00400000
00000000`00400000  6c6c6548 6f57206f 21646c72 0000ffff  Hello World!….
00000000`00400010  000000b8 00000000 00000040 00000000  ……..@…….
00000000`00400020  00000000 00000000 00000000 00000000  …………….
00000000`00400030  00000000 00000000 00000000 000000e8  …………….
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ……..!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode….$…….

Then we can change it immediately using any of e* commands: 

0:000> ea 00000000`00400000+6 "WinDbg!"

0:000> dc 00000000`00400000
00000000`00400000  6c6c6548 6957206f 6762446e 2100ffff  Hello WinDbg!
00000000`00400010  000000b8 00000000 00000040 00000000  ……..@…….
00000000`00400020  00000000 00000000 00000000 00000000  …………….
00000000`00400030  00000000 00000000 00000000 000000e8  …………….
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ……..!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode….$…….

Alternatively we can use GUI memory editor:

Now we can write memory contents back to our file:

0:000> .writemem c:\dmitri\hello.bin 00000000`00400000 L0n13
Writing d bytes.

- Dmitry Vostokov @ DumpAnalysis.org -

Final Back Cover for MDAA V1

Monday, April 14th, 2008

To avoid controversial pictures I decided to put an image of TestDefaultDebugger crash dump generated by Dump2Picture:

Final Back Cover for Memory Dump Analysis Anthology, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -

Controversial Book Cover?

Sunday, April 13th, 2008

Some people commented that by placing an image of a complete memory dump on the back cover of a book both violates copyright and intellectual property rights, as the picture is generated from copyrighted material. Instead they suggested to put a picture of a freeware program. Here is my response:

I disagree to the best of my understanding. This picture is just the visualized physical memory for illustration purposes only. What about disassembling a function to illustrate a bug? Or dumping memory, for example, a thread structure? Or printing a screenshot from Performance Monitor or Task Manager to illustrate CPU spike? Or a stack trace from a complete memory dump? Does it violate copyright and intellectual property rights because it is generated from copyrighted material? What about the front cover then, showing book spines of hundreds of copyrighted books? If Microsoft asks me to remove the picture, certainly, I’ll do it and reprint the book. And, surely, a memory dump of a freeware program will definitely contain portions of copyrighted material, like ntdll.dll, kernel32.dll or accidental 3rd-party hooks. Regarding a complete memory dump copyrighted material might have been paged out from physical memory and not included in file contents. Do you admit that printing a CRC number violates property rights because it was generated from copyrighted material? Due to the mathematical nature of involved algorithms it is not possible to reconstruct binary code from the printed cover picture which could have been created artificially as well.

What do you think?

- Dmitry Vostokov @ DumpAnalysis.org -

Final TOC for MDAA Volume 1

Saturday, April 12th, 2008

I’ve posted the final Table of Contents and additional information for the soon-to-be-published book:

Memory Dump Analysis Anthology, Volume 1

Note that the proposed back cover image is the picture of a 1Gb complete physical memory dump generated by Dump2Picture:

Back Cover for Memory Dump Analysis Anthology, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -