Crash Dump Analysis Patterns (Part 230)

Sometimes, a stack trace from Stack Trace Collection may look well-formed at the first sight like having an expected start frames, for example:

0:000> ~*k

[...]

# 19 Id: 16a4.21f4 Suspend: 0 Teb: 7e95b000 Unfrozen
ChildEBP RetAddr
0c2de6b0 74eb112f ntdll!NtWaitForMultipleObjects+0xc
0c2de83c 76ca7b89 KERNELBASE!WaitForMultipleObjectsEx+0xcc
0c2de858 76d007bf kernel32!WaitForMultipleObjects+0x19
0c2dec98 76d00295 kernel32!WerpReportFaultInternal+0x50b
0c2deca8 76ce1709 kernel32!WerpReportFault+0x74
0c2decb0 74f5f705 kernel32!BasepReportFault+0x19
0c2ded3c 76fb4f84 KERNELBASE!UnhandledExceptionFilter+0x1f4
0c2ded54 76fb5728 ntdll!TppExceptionFilter+0x30
0c2ded64 76f5c95a ntdll!TppWorkerpInnerExceptionFilter+0xe
0c2df914 76ca7c04 ntdll!TppWorkerThread+0×87f5a
0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0×24
0c2df970 76f1acea ntdll!__RtlUserThreadStart+0×2f
0c2df980 00000000 ntdll!_RtlUserThreadStart+0×1b

[...]

So, we may think something wrong happened in ntdll!TppWorkerThread code (although 0×87f5a offset looks suspicious). However, in reality, in this case due to exception filter logic (or some other reason in different cases) we have Hidden Stack Trace. When looking at UnhandledExceptionFilter parameters (or raw stack as in the case of Hidden Exceptions) we find an exception context:

0:019> kv
ChildEBP RetAddr Args to Child
0c2de6b0 74eb112f 00000003 0c2de880 00000001 ntdll!NtWaitForMultipleObjects+0xc
0c2de83c 76ca7b89 00000003 0c2de880 00000000 KERNELBASE!WaitForMultipleObjectsEx+0xcc
0c2de858 76d007bf 00000003 0c2de880 00000000 kernel32!WaitForMultipleObjects+0x19
0c2dec98 76d00295 00000000 00000001 00000000 kernel32!WerpReportFaultInternal+0x50b
0c2deca8 76ce1709 0c2ded3c 74f5f705 0c2ded94 kernel32!WerpReportFault+0x74
0c2decb0 74f5f705 0c2ded94 00000001 a79b7895 kernel32!BasepReportFault+0x19
0c2ded3c 76fb4f84 0c2ded94 0c2ded94 00000000 KERNELBASE!UnhandledExceptionFilter+0×1f4
0c2ded54 76fb5728 00000000 00000000 0c2df914 ntdll!TppExceptionFilter+0×30
0c2ded64 76f5c95a 0c2df8d0 76f00a70 0c2df914 ntdll!TppWorkerpInnerExceptionFilter+0xe
0c2df914 76ca7c04 0f79e380 76ca7be0 a5b45024 ntdll!TppWorkerThread+0×87f5a
0c2df928 76f1ad1f 0f79e380 a59141d7 00000000 kernel32!BaseThreadInitThunk+0×24
0c2df970 76f1acea ffffffff 76f00233 00000000 ntdll!__RtlUserThreadStart+0×2f
0c2df980 00000000 76ed4a00 0f79e380 00000000 ntdll!_RtlUserThreadStart+0×1b

0:019> dd 0c2ded94 L2
0c2ded94 0c2deef8 0c2def48

0:019> .cxr 0c2def48
eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10
eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdll!RtlEnterCriticalSection+0×12:
76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

0:019> k
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0c2df3ac 7407999c ntdll!RtlEnterCriticalSection+0x12
0c2df3cc 7407acd6 ModuleA!DoWork+0x1b
[...]
0c2df73c 76ee3aa7 ModuleA!ThreadPoolWorkCallback+0xa9
0c2df77c 76ee1291 ntdll!TppWorkpExecuteCallback+0x137
0c2df914 76ca7c04 ntdll!TppWorkerThread+0x48e
0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0x24
0c2df970 76f1acea ntdll!__RtlUserThreadStart+0x2f
0c2df980 00000000 ntdll!_RtlUserThreadStart+0x1b

We consider this a different pattern than Hidden Call because an entire stack (sub)trace is missing between UnhandledExceptionFilter and thread start frames:

ntdll!NtWaitForMultipleObjects
KERNELBASE!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!BasepReportFault
KERNELBASE!UnhandledExceptionFilter
[...]
ntdll!TppWorkerThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

This pattern is also different from Past Stack Trace pattern because Hidden Stack Trace belongs to PRESENT time zone. Our example is also different from Hidden Exception analysis pattern and its recovered stack trace because exception processing is not hidden and shows Exception Stack Trace albeit with a hidden part.

We were also fortunate to have Stored Exception (accessible by !analyze -v command):

0:019> .exr -1
ExceptionAddress: 76f00fb2 (ntdll!RtlEnterCriticalSection+0x00000012)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 15f237e5
Attempt to write to address 15f237e5

0:019> .ecxr
eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10
eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdll!RtlEnterCriticalSection+0x12:
76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

So this Hidden Stack Trace is detected straightforwardly. But in other cases, such as when we have Multiple Exceptions in a process dump or Stack Trace Collection from a complete memory dump, we need to pay attention to such a possibility.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply

You must be logged in to post a comment.