Archive for April 3rd, 2021

Crash Dump Analysis Patterns (Part 276)

Saturday, April 3rd, 2021

In simple exception cases, we have exception record, for example from Stored Exception corresponding to exception context, for example:

0:000> .exr -1
ExceptionAddress: 00000001400247ae (TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0×000000000000007e)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000

0:000> .ecxr
rax=0000000000000000 rbx=0000000000000001 rcx=000000000014fd20
rdx=00000000000003e8 rsi=000000000014fd20 rdi=000000014002daa0
rip=00000001400247ae rsp=000000000014efd0 rbp=0000000000000111
r8=0000000000000000  r9=0000000140024730 r10=0000000140024730
r11=000000000014f0d0 r12=0000000000000000 r13=00000000000003e8
r14=0000000000000110 r15=0000000000000001
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0×7e:
00000001`400247ae
c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=????????

In other cases, we may have missing context:

0:000> .excr
Minidump doesn't have an exception context
Unable to get exception context, HRESULT 0x80004002

invalid context (see also Invalid Exception Information) in the output of !analyze -v command:

CONTEXT:  00007ffb54bd1e60 -- (.cxr 0x7ffb54bd1e60)
rax=15ff480001191885 rbx=ff48c88b48000000 rcx=00441f0f00044c3c
rdx=08ba3824448d4c00 rsi=4838244c8b480001 rdi=0058b9413024448d
rip=00441f0f00044a04 rsp=441f0f00044bd315 rbp=18e4840fc0850000
r8=4c20244489480000  r9=244c89444024448d r10=15ff48a9518d4130
r11=00441f0f00044ebc r12=0118c1840fc08500 r13=8b4840244c8b4800
r14=d88b0000003ee8d7 r15=15ff4838244c8b48
iopl=0 vip vif ov dn ei pl nz na pe nc
cs=2183  ss=044c  ds=4800  es=f98b  fs=ff48  gs=5315             efl=441f0f00
00441f0f`00044a04 ??              ???
Resetting default scope

and valid context but not corresponding to stored exception record:

0:000> .ecxr
rax=00007ffe0a6a9618 rbx=0000024a3aa44020 rcx=0000000100000001
rdx=0000000000000001 rsi=0000024a3abff3e8 rdi=0000024a3abfb5c8
rip=00007ffe9768d759 rsp=000000dc0fd7caf0 rbp=000000dc0fd7d160
r8=0000024a00000007  r9=0000024a5ce8bc80 r10=0000000000000000
r11=0000000000000000 r12=0000024a3abfad90 r13=0000024a3aa44020
r14=0000000000000000 r15=0000024a3abfb5e0
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
KERNELBASE!RaiseException+0x69:
00007ffe`9768d759 0f1f440000      nop     dword ptr [rax+rax]

0:000> .exr -1
ExceptionAddress: 00007ffe0a6a9609
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000

However, Exception Stack Trace may be available with JIT Code address :

0:000> kL
# Child-SP          RetAddr           Call Site
00 000000dc`0fd7b558 00007ffe`976b0d40 ntdll!NtWaitForMultipleObjects+0x14
01 000000dc`0fd7b560 00007ffe`976b0c3e KERNELBASE!WaitForMultipleObjectsEx+0xf0
02 000000dc`0fd7b850 00007ffe`994cf6aa KERNELBASE!WaitForMultipleObjects+0xe
03 000000dc`0fd7b890 00007ffe`994cf0e6 kernel32!WerpReportFaultInternal+0x58a
04 000000dc`0fd7b9b0 00007ffe`9776c439 kernel32!WerpReportFault+0xbe
05 000000dc`0fd7b9f0 00007ffe`99cd4b63 KERNELBASE!UnhandledExceptionFilter+0x3d9
06 000000dc`0fd7bb10 00007ffe`99cbbb16 ntdll!RtlUserThreadStart$filt$0+0xa2
07 000000dc`0fd7bb50 00007ffe`99cd130f ntdll!_C_specific_handler+0x96
08 000000dc`0fd7bbc0 00007ffe`99c7b5e4 ntdll!RtlpExecuteHandlerForException+0xf
09 000000dc`0fd7bbf0 00007ffe`99c7b335 ntdll!RtlDispatchException+0x244
0a 000000dc`0fd7c300 00007ffe`9768d759 ntdll!RtlRaiseException+0x185
0b 000000dc`0fd7caf0 00007ffe`6986b259 KERNELBASE!RaiseException+0x69
0c 000000dc`0fd7cbd0 00007ffe`6986b28b coreclr!NakedThrowHelper2+0x9
0d 000000dc`0fd7cc00 00007ffe`6986b295 coreclr!NakedThrowHelper_RspAligned+0x1e
0e 000000dc`0fd7d128 00007ffe`0a6a9609 coreclr!NakedThrowHelper_FixRsp+0×5
0f 000000dc`0fd7d130 00007ffe`0a548023 0×00007ffe`0a6a9609
10 000000dc`0fd7d170 00007ffe`0a547734 0×00007ffe`0a548023
11 000000dc`0fd7d230 00000000`627311e5 0×00007ffe`0a547734
12 000000dc`0fd7d290 00007ffe`62b50fe7 PresentationCore+0×4011e5
13 000000dc`0fd7d2d0 00007ffe`62a35840 PresentationFramework+0xbb0fe7
14 000000dc`0fd7d310 00007ffe`62b51a60 PresentationFramework+0xa95840
15 000000dc`0fd7d350 00000000`62732e22 PresentationFramework+0xbb1a60
16 000000dc`0fd7d390 00000000`62757c42 PresentationCore+0×402e22
17 000000dc`0fd7d3d0 00007ffe`0a5448f3 PresentationCore+0×427c42
18 000000dc`0fd7d410 00007ffe`0a548023 0×00007ffe`0a5448f3
19 000000dc`0fd7d450 00000000`62740e19 0×00007ffe`0a548023
1a 000000dc`0fd7d510 00000000`62732b6a PresentationCore+0×410e19
1b 000000dc`0fd7d580 00000000`62757c42 PresentationCore+0×402b6a
1c 000000dc`0fd7d5c0 00007ffe`0a5448f3 PresentationCore+0×427c42
1d 000000dc`0fd7d600 00007ffe`0a548023 0×00007ffe`0a5448f3
1e 000000dc`0fd7d640 00007ffe`0a547734 0×00007ffe`0a548023
1f 000000dc`0fd7d700 00007ffe`0a550211 0×00007ffe`0a547734
20 000000dc`0fd7d760 00007ffe`0a558efd 0×00007ffe`0a550211
21 000000dc`0fd7d7a0 00007ffe`0a55ebb1 0×00007ffe`0a558efd
22 000000dc`0fd7d860 00007ffe`0a564474 0×00007ffe`0a55ebb1
23 000000dc`0fd7d8b0 00007ffe`0a550eff 0×00007ffe`0a564474
24 000000dc`0fd7d9e0 00007ffe`0a550692 0×00007ffe`0a550eff
25 000000dc`0fd7da70 00007ffe`0a54967d 0×00007ffe`0a550692
26 000000dc`0fd7dae0 00007ffe`0a549596 0×00007ffe`0a54967d
27 000000dc`0fd7db70 00007ffe`0a548ac7 0×00007ffe`0a549596
28 000000dc`0fd7dbc0 00007ffe`0a5488f5 0×00007ffe`0a548ac7
29 000000dc`0fd7dc20 00007ffe`0a54920c 0×00007ffe`0a5488f5
2a 000000dc`0fd7dc70 00007ffe`0a548f07 0×00007ffe`0a54920c
2b 000000dc`0fd7dd00 00007ffe`09d2d772 0×00007ffe`0a548f07
2c 000000dc`0fd7de00 00007ffe`995ae858 0×00007ffe`09d2d772
2d 000000dc`0fd7de80 00007ffe`995ae299 user32!UserCallWinProcCheckWow+0×2f8
2e 000000dc`0fd7e010 00007ffe`0a18011b user32!DispatchMessageWorker+0×249
2f 000000dc`0fd7e090 00007ffe`69557ec3 0×00007ffe`0a18011b
30 000000dc`0fd7e150 00007ffe`695553a1 WindowsBase+0×197ec3
31 000000dc`0fd7e1e0 00007ffe`6955534e WindowsBase+0×1953a1
32 000000dc`0fd7e210 00007ffe`6276966c WindowsBase+0×19534e
33 000000dc`0fd7e240 00007ffe`62767ccd PresentationFramework+0×7c966c
34 000000dc`0fd7e270 00007ffe`62764c5c PresentationFramework+0×7c7ccd
35 000000dc`0fd7e2c0 00007ffe`09d1618e PresentationFramework+0×7c4c5c
36 000000dc`0fd7e2f0 00007ffe`6986a2f3 0×00007ffe`09d1618e
37 000000dc`0fd7e340 00007ffe`697a2fcc coreclr!CallDescrWorkerInternal+0×83
38 000000dc`0fd7e380 00007ffe`697c22b3 coreclr!MethodDescCallSite::CallTargetWorker+0×268
39 (Inline Function) ——–`——– coreclr!MethodDescCallSite::Call+0xb
3a 000000dc`0fd7e4c0 00007ffe`697c207e coreclr!RunMainInternal+0×11f
3b 000000dc`0fd7e5f0 00007ffe`697c1be1 coreclr!RunMain+0xd2
3c 000000dc`0fd7e6a0 00007ffe`697c1908 coreclr!Assembly::ExecuteMainMethod+0×1cd
3d 000000dc`0fd7ea30 00007ffe`69789ad2 coreclr!CorHost2::ExecuteAssembly+0×1c8
3e 000000dc`0fd7eba0 00007ffe`7d502c72 coreclr!coreclr_execute_assembly+0xe2
3f (Inline Function) ——–`——– hostpolicy!coreclr_t::execute_assembly+0×2b
40 000000dc`0fd7ec40 00007ffe`7d502ed7 hostpolicy!run_app_for_context+0×3be
41 000000dc`0fd7edd0 00007ffe`7d503b6b hostpolicy!run_app+0×37
42 000000dc`0fd7ee10 00007ffe`7d5839ea hostpolicy!corehost_main+0xfb
43 000000dc`0fd7efd0 00007ffe`7d587358 hostfxr!execute_app+0×206
44 (Inline Function) ——–`——– hostfxr!?A0×83a23e19::read_config_and_execute+0×10a
45 000000dc`0fd7f0c0 00007ffe`7d585b5f hostfxr!fx_muxer_t::handle_exec_host_command+0×214
46 000000dc`0fd7f1b0 00007ffe`7d582029 hostfxr!fx_muxer_t::execute+0×39b
47 000000dc`0fd7f2f0 00007ff6`3aede0b0 hostfxr!hostfxr_main_startupinfo+0×89
48 000000dc`0fd7f3f0 00007ff6`3aede418 ApplicationA_exe!exe_start+0×620
49 000000dc`0fd7f5d0 00007ff6`3aedfef8 ApplicationA_exe!wmain+0×124
4a (Inline Function) ——–`——– ApplicationA_exe!invoke_main+0×22
4b 000000dc`0fd7f740 00007ffe`99477034 ApplicationA_exe!__scrt_common_main_seh+0×10c
4c 000000dc`0fd7f780 00007ffe`99c7d0d1 kernel32!BaseThreadInitThunk+0×14
4d 000000dc`0fd7f7b0 00000000`00000000 ntdll!RtlUserThreadStart+0×21

0:000> u 00007ffe`0a6a9609
00007ffe`0a6a9609 c70001000000    mov     dword ptr [rax],1

00007ffe`0a6a960f 90              nop
00007ffe`0a6a9610 90              nop
00007ffe`0a6a9611 488d6500        lea     rsp,[rbp]
00007ffe`0a6a9615 5d              pop     rbp
00007ffe`0a6a9616 c3              ret
00007ffe`0a6a9617 0019            add     byte ptr [rcx],bl
00007ffe`0a6a9619 0502000552      add     eax,52050002h

In the case of .NET Core dump, we can use Saved Exception Context to get the original exception:

0:000> dp coreclr!g_SavedExceptionInfo
00007ffe`69bd57f0  00000000`c0000005 00000000`00000000
00007ffe`69bd5800  00007ffe`0a6a9609 00000000`00000002
00007ffe`69bd5810  00000000`00000001 00000000`00000000
00007ffe`69bd5820  00000000`00000000 00000000`00000000
00007ffe`69bd5830  00000000`00000000 00000000`00000000
00007ffe`69bd5840  00000000`00000000 00000000`00000000
00007ffe`69bd5850  00000000`00000000 00000000`00000000
00007ffe`69bd5860  00000000`00000000 00000000`00000000

0:000> dt coreclr!g_SavedExceptionInfo
+0x000 m_ExceptionRecord : _EXCEPTION_RECORD
+0x0a0 m_ExceptionContext : _CONTEXT
+0x570 m_Crst           : CrstStatic

0:000> .cxr coreclr!g_SavedExceptionInfo+a0
rax=0000000000000000 rbx=0000024a3aa44020 rcx=0000024a3aa1d210
rdx=0000024a3aa44020 rsi=0000024a3abff3e8 rdi=0000024a3abfb5c8
rip=00007ffe0a6a9609 rsp=000000dc0fd7d130 rbp=000000dc0fd7d160
r8=0000024a3abff3e8  r9=0000000000000000 r10=0000000000000000
r11=000000dc0fd7d090 r12=0000024a3abfad90 r13=0000024a3aa44020
r14=0000000000000000 r15=0000024a3abfb5e0
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00007ffe`0a6a9609 c70001000000    mov     dword ptr [rax],1 ds:00000000`00000000=????????

This may also work in the case of invalid or missing exception information in .NET Core dumps:

0:000> .exr -1
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0

0:000> .excr
Minidump doesn't have an exception context
Unable to get exception context, HRESULT 0x80004002

0:000> .cxr coreclr!g_SavedExceptionInfo+a0
rax=0000000000000000 rbx=0000024a3aa44020 rcx=0000024a3aa1d210
rdx=0000024a3aa44020 rsi=0000024a3abff3e8 rdi=0000024a3abfb5c8
rip=00007ffe0a6a9609 rsp=000000dc0fd7d130 rbp=000000dc0fd7d160
r8=0000024a3abff3e8  r9=0000000000000000 r10=0000000000000000
r11=000000dc0fd7d090 r12=0000024a3abfad90 r13=0000024a3aa44020
r14=0000000000000000 r15=0000024a3abfb5e0
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00007ffe`0a6a9609 c70001000000    mov     dword ptr [rax],1 ds:00000000`00000000=????????

In some other unmanaged cases, we can use probe Execution Residue values around some exception processing symbols as in the case of Hidden Exceptions, but this may not work if such values are overwritten or no longer available.

A similar approach is available for .NET Framework despite the type not available:

0:000> x clr!g_SavedExceptionInfo
00007ffc`efc01f40 clr!g_SavedExceptionInfo = <no type information>

0:000> dt clr!g_SavedExceptionInfo
Symbol clr!g_SavedExceptionInfo not found.

0:000> .cxr clr!g_SavedExceptionInfo+a0
rax=0000000000000000 rbx=0000000002f8b8a0 rcx=0000000002f27ee8
rdx=0000000002f8a598 rsi=0000000002f8a598 rdi=0000000002fa1028
rip=00007ffc8fcb0829 rsp=000000000113e5b0 rbp=000000000113e5e0
r8=0000000002fa1028  r9=0000000000000000 r10=00007ff480140018
r11=00007ffc8fba8ae8 r12=0000000000000002 r13=0000000000000202
r14=0000000000000001 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00007ffc`8fcb0829 c70001000000    mov     dword ptr [rax],1 ds:00000000`00000000=????????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -