Archive for March 12th, 2011

Crash Dump Analysis Patterns (Part 132)

Saturday, March 12th, 2011

Sometimes simple comparison of crash signatures is not enough to find similar support incidents. We then traverse stack trace frames to find Least Common Frame matching similar stack traces in a database. For example, consider this signature:

0:026> r
eax=011349ec ebx=01136738 ecx=79f943e1 edx=00000000 esi=011349ec edi=0888f3b8
eip=00dfbef8 esp=0888f348 ebp=0888f3c8 iopl=0  nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00000246
00dfbef8 3902  cmp     dword ptr [edx],eax  ds:0023:00000000=????????

0:026> k 100
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0888f3c8 792a842f 0xdfbef8
0888f3e4 792a839b mscorlib_ni+0x1e842f
0888f3fc 79e71b4c mscorlib_ni+0x1e839b
0888f40c 79e821b9 mscorwks!CallDescrWorker+0×33
0888f48c 79e8281f mscorwks!CallDescrWorkerWithHandler+0xa3
0888f4ac 79e82860 mscorwks!DispatchCallBody+0×1e
0888f510 79e828d1 mscorwks!DispatchCallDebuggerWrapper+0×3d
0888f544 79ec50f5 mscorwks!DispatchCallNoEH+0×51
0888f5a0 79e9848f mscorwks!AddTimerCallback_Worker+0×66
0888f5b4 79e9842b mscorwks!Thread::DoADCallBack+0×32a
0888f648 79e98351 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0888f684 79e984dd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0888f6ac 79ec4a84 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
0888f6c4 79ec5075 mscorwks!ManagedThreadBase::ThreadPool+0×13
0888f70c 79ec50a4 mscorwks!AddTimerCallbackEx+0×83
0888f720 79ec514a mscorwks!AddTimerCallback+0×10
0888f75c 79ec4e0c mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
0888f7a8 79ec471e mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
0888f7bc 79ec4892 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0888f814 79f75715 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
0888ffb4 7c80b729 mscorwks!Thread::intermediateThreadProc+0×49
0888ffec 00000000 kernel32!BaseThreadStart+0×37

Most likely we won’t find any similar stack trace when searching for 0xdfbef8. The search for mscorlib_ni+0×1e842f brings several results but they are not crashes but hangs with the frame being in the middle of call stack. The same is for mscorlib_ni+0×1e839b. So we finally try searching for CallDescrWorker+0×33 but with a limit to results having the same application module name. And indeed we find the similar software incident with the same stack trace after our least common frame:

0:004> r
eax=00000024 ebx=03e6f738 ecx=738129d8 edx=00495ef0 esi=01a87c4c edi=019c5f1c
eip=00a92037 esp=03e6f6cc ebp=03e6f6e8 iopl=0 nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00010206
00a92037 ??              ???

0:004> k 100
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
03e6f6c8 737d6bb5 0xa92037
03e6f6e8 737a509f mscorlib_ni+0x216bb5
03e6f6f8 737a834c mscorlib_ni+0x1e509f
03e6f70c 74171b6c mscorlib_ni+0x1e834c
03e6f71c 74182209 mscorwks!CallDescrWorker+0×33
03e6f79c 7418286f mscorwks!CallDescrWorkerWithHandler+0xa3
03e6f7bc 741828b0 mscorwks!DispatchCallBody+0×1e
03e6f820 74182921 mscorwks!DispatchCallDebuggerWrapper+0×3d
03e6f854 742ced79 mscorwks!DispatchCallNoEH+0×51
03e6f8b0 7419846f mscorwks!AddTimerCallback_Worker+0×66
03e6f8c4 7419840b mscorwks!Thread::DoADCallBack+0×32a
03e6f958 74198331 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
03e6f994 741984bd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
03e6f9bc 742ce708 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
03e6f9d4 742cecf9 mscorwks!ManagedThreadBase::ThreadPool+0×13
03e6fa1c 742ced28 mscorwks!AddTimerCallbackEx+0×83
03e6fa30 742cedce mscorwks!AddTimerCallback+0×10
03e6fa6c 742cea90 mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
03e6fab8 742ce3a2 mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
03e6facc 742ce516 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
03e6fb64 74441ec9 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
03e6fc84 76813677 mscorwks!Thread::intermediateThreadProc+0×49
03e6fc90 77219d72 kernel32!BaseThreadInitThunk+0xe
03e6fcd0 77219d45 ntdll!__RtlUserThreadStart+0×70
03e6fce8 00000000 ntdll!_RtlUserThreadStart+0×1b

- Dmitry Vostokov @ + -