Incorrect stack trace, stack overflow, early crash dump, nested exception, problem exception handler and same vendor: pattern cooperation

This case study centers on 3 process dump files (two first chance exception and one second chance exception). To recall the difference between them please read first chance exceptions explained series. When we get first and second chance exception dumps together we usually open a second chance exception dump first. However, in this case, the second chance exception dump had an incorrect stack trace:

(f54.b80): Access violation - code c0000005 (!!! second chance !!!)
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

0:000> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

The default analysis command detected stack overflow pattern: 

0:000> !analyze -v

[...]

FAULTING_IP:
ntdll!RtlDispatchException+8
7c92a978 56              push    esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c92a978 (ntdll!RtlDispatchException+0x00000008)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00032fc0

DEFAULT_BUCKET_ID:  STACK_OVERFLOW

ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.

[...]

Indeed ESP was outside the stack region and that happened during unhandled exception processing:

0:000> r esp
esp=00030e4c

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        000310c4
    StackBase:            00130000
    StackLimit:           00031000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             00000f54 . 00000b80
    RpcHandle:            00000000
    Tls Storage:          001537a8
    PEB Address:          7ffdb000
    LastErrorValue:       2
    LastStatusValue:      c000000f
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dps esp l100
00030e4c  ????????
00030e50  ????????
[...]
00030ff8  ????????
00030ffc  ????????
00031000  00000000
00031004  00000000
00031008  00000000
0003100c  00000000
00031010  00000000
00031014  00000000
00031018  00000000
0003101c  00000000
00031020  00000000
00031024  7c910323 ntdll!RtlpImageNtHeader+0x56
00031028  004054e8 Application+0x54e8
0003102c  00400000 Application
00031030  00400000 Application
00031034  00400100 Application+0x100
00031038  00031028
0003103c  7e390000 USER32!_imp__GetClipRgn <PERF> (USER32+0x0)
00031040  00031060
00031044  7c910385 ntdll!RtlImageDirectoryEntryToData+0x57
00031048  00400000 Application
0003104c  00000001
00031050  0000000e
00031054  00031084
00031058  7c910323 ntdll!RtlpImageNtHeader+0x56
0003105c  004054e8 Application+0x54e8
00031060  7c900000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
00031064  0012ff00
00031068  7c9000d0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xd0)
0003106c  0003105c
00031070  0000000e
00031074  00114e88
00031078  7c90e920 ntdll!_except_handler3
0003107c  7c910328 ntdll!`string'+0x4
00031080  ffffffff
00031084  7c910323 ntdll!RtlpImageNtHeader+0x56
00031088  7c935f1c ntdll!RtlLookupFunctionTable+0xc5
0003108c  7c92ab3a ntdll!RtlLookupFunctionTable+0xf2
00031090  7c97e178 ntdll!LdrpLoaderLock
00031094  000310c4
00031098  7c809ad8 kernel32!_except_handler3
0003109c  7c833fd9 kernel32!UnhandledExceptionFilter+0xf
000310a0  7c834aa8 kernel32!`string’+0×1c
000310a4  000310d0
000310a8  0040550c Application+0×550c
000310ac  000310b4
000310b0  7c9032a8 ntdll!ExecuteHandler2+0×26
000310b4  00031198
000310b8  0012ffb4
000310bc  000311ac
 

Before we tried to reconstruct the stack trace we opened the earlier first-chance exception dump file:

0:000> .opendump 1stchance.dmp

Loading Dump File [1stchance.dmp]
User Dump File: Only application data is available

Opened '1stchance.dmp'

||0:0:000> g

(f54.b80): Stack overflow - code c00000fd (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0003332c ebx=00033040 ecx=00033054 edx=7c90e514 esi=000333a8 edi=00000000
eip=7c92a978 esp=00032fc4 ebp=00033028 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
ntdll!RtlDispatchException+0×8:
7c92a978 56              push    esi

Here we were able to get stack trace from the saved nested exception

||1:1:020> kL 1000
ChildEBP RetAddr 
00033028 7c90e48a ntdll!RtlDispatchException+0x8
00033028 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033390 7c90e48a ntdll!RtlDispatchException+0x133
00033390 7c95019e ntdll!KiUserExceptionDispatcher+0xe
000336f8 7c90e48a ntdll!RtlDispatchException+0x133
000336f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033a60 7c90e48a ntdll!RtlDispatchException+0x133
00033a60 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033dc8 7c90e48a ntdll!RtlDispatchException+0x133
00033dc8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034130 7c90e48a ntdll!RtlDispatchException+0x133
00034130 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034498 7c90e48a ntdll!RtlDispatchException+0x133
00034498 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034800 7c90e48a ntdll!RtlDispatchException+0x133
00034800 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034b68 7c90e48a ntdll!RtlDispatchException+0x133
00034b68 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034ed0 7c90e48a ntdll!RtlDispatchException+0x133
00034ed0 7c95019e ntdll!KiUserExceptionDispatcher+0xe
[...]
001143f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114760 7c90e48a ntdll!RtlDispatchException+0x133
00114760 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114ac8 7c90e48a ntdll!RtlDispatchException+0x133
00114ac8 7c7e2afb ntdll!KiUserExceptionDispatcher+0xe
00114e30 0057ad17 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
00114e54 0098ff95 Application+0x17ad17
[...]
00121fd8 7e398734 Application+0x313be
00122004 7e398816 USER32!InternalCallWinProc+0x28
0012206c 7e3a8ea0 USER32!UserCallWinProcCheckWow+0x150
001220c0 7e3aacd1 USER32!DispatchClientMessage+0xa3
001220f0 7c90e473 USER32!__fnINSTRING+0x37
0012212c 7e3993e9 ntdll!KiUserCallbackDispatcher+0x13
00122158 7e3aa43b USER32!NtUserPeekMessage+0xc
00122184 004794d9 USER32!PeekMessageA+0xeb
00122234 00461667 Application+0x794d9
[...]
0012ffc0 7c7e7077 Application+0x60610b
0012fff0 00000000 kernel32!BaseProcessStart+0x23

This all pointed to a problem exception handler:

||1:1:020> !analyze -v

[...]

CONTEXT:  00114b10 -- (.cxr 0x114b10)
eax=00114de0 ebx=0eedfade ecx=00000000 edx=001537a8 esi=00114e88 edi=00000007
eip=7c7e2afb esp=00114ddc ebp=00114e30 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
kernel32!RaiseException+0x53:
7c7e2afb 5e              pop     esi
Resetting default scope

[...]

||1:1:020> .cxr 0x114b10

||1:1:020> kv 1
ChildEBP RetAddr  Args to Child             
00114e30 0057ad17 0eedfade 00000001 00000007 kernel32!RaiseException+0×53

Being curious we also opened the second first chance exception dump and it pointed to the expected crash point (the same as seen in the second chance exception crash dump)

||1:1:020> .opendump 1stchance2.dmp

Loading Dump File [1stchance2.dmp]
User Dump File: Only application data is available

Opened '1stchance2.dmp'

||1:1:020> g

[...]

(f54.b80): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

||2:2:040> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

We found the similar past issue for a different process name but our main process module information included the same vendor name so it was easy to contact the corresponding vendor.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply

You must be logged in to post a comment.