Archive for the ‘x64 Windows’ Category

Crash Dump Analysis Patterns (Part 4b)

Friday, July 10th, 2020

One of the earliest memory analysis patterns, Lateral Damage now has a specialization for CPU mode. Due to some reasons, we may get a dump with a different default CPU mode, for example, WOW64 or even V86 segmented memory addressing mode. In such a case, most commands will not work or give incorrect output.

In one such a dump we see 32-bit bugcheck parameters in !analyze -v command output:

CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 00000003, Process
Arg2: 054d0450, Terminating object
Arg3: 054d0730, Process image file name
Arg4: 02b90db0, Explanatory message (ascii)

But the dump itself from x64 Windows:

Kernel base = 0xfffff800`0280d000 PsLoadedModuleList = 0xfffff800`02a52e90

We notice WOW64 prompt:

16.1: kd:x86> k
# ChildEBP          RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 00000000 00000000 0x0

We switch to x64 mode:

16.1: kd:x86> .effmach AMD64
Effective machine: x64 (AMD64)

Now we get correct bugcheck parameters:

16.1: kd> !analyze -v
[...]
Arguments:
Arg1: 0000000000000003, Process
Arg2: fffffa80054d0450, Terminating object
Arg3: fffffa80054d0730, Process image file name
Arg4: fffff80002b90db0, Explanatory message (ascii)

[…]

However, the stack trace is not available, all registers are zeroed, and stack region memory is not accessible:

16.1: kd> k
# Child-SP          RetAddr           Call Site
00 00000000`00000000 00000000`00000000 0x0

16.1: kd> !thread
THREAD fffffa800b2c6b60  Cid 0998.1a58  Teb: 000007ffffeee000 Win32Thread: 0000000000000000 RUNNING on processor 1
Not impersonating
DeviceMap                 fffff8a000008aa0
Owning Process            fffffa800425b820       Image:         App.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      28317542       Ticks: 0
Context Switch Count      2              IdealProcessor: 0
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x000007feee8080ec
Stack Init fffff880066f3c70 Current fffff880066f3440
Base fffff880066f4000 Limit fffff880066ee000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`00000000 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0×0

16.1: kd> r
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=0000000000000000 rsp=0000000000000000 rbp=0000000000000000
r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up di pl nz na pe nc
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000
00000000`00000000 ??              ???

16.1: kd> dp fffff880066f3440
0000:3440  ????????`???????? ????????`????????
0000:3450  ????????`???????? ????????`????????
0000:3460  ????????`???????? ????????`????????
0000:3470  ????????`???????? ????????`????????
0000:3480  ????????`???????? ????????`????????
0000:3490  ????????`???????? ????????`????????
0000:34a0  ????????`???????? ????????`????????
0000:34b0  ????????`???????? ????????`????????

We notice segmented memory addressing and apply .segmentation command that is still available (the hint is taken from here):

16.1: kd> .segmentation /V /X /a
In x86 v86 code: no
In x86 16-bit code: no
In amd64 64-bit code: yes

Although stack trace is not available we see the normal prompt and can get look at stack region Execution Residue and get Rough Stack Trace:

1: kd> k
# Child-SP          RetAddr           Call Site
00 00000000`00000000 00000000`00000000 0x0

1: kd> dpS fffff880066ee000 fffff880066f4000
fffff800`02b7e79e nt!PspGetSetContextInternal+0×2c6
fffff800`02b7e8e5 nt!PspGetSetContextInternal+0×40d
fffff800`0289e11e nt!MiResolveDemandZeroFault+0×3be
fffff800`02e04245 hal!HalpPCIPerformConfigAccess+0×55
fffff800`02e04b32 hal!HalpPciAccessMmConfigSpace+0×196
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`02e04000 hal!HalpPCIConfig+0×70
fffff880`051f3dbb dump_diskdump!ScsiPortNotification+0×107
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`04c086c2 dump_LSI_SAS!CheckInqFlagReplies+0×42e [e:\win7\drivers\oem\src\storage\lsi_sas\ca_util.c @ 6455]
fffff800`028cdf7e nt!iswctype_l+0xce
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`028cde19 nt!output_l+0×6e1
fffff880`051e9110 crashdmp!StrBeginningDump
fffff880`051f3dbb dump_diskdump!ScsiPortNotification+0×107
fffff800`028cde19 nt!output_l+0×6e1
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`04c12090 dump_LSI_SAS!LSImpiMSIIsr+0xf4 [e:\win7\drivers\oem\src\storage\lsi_sas\ca_int.c @ 208]
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`02e072ec hal!HalpTscStallExecutionProcessor+0xe8
fffff880`051f2401 dump_diskdump!AllocateScatterGatherList+0×5d
fffff880`051f257c dump_diskdump!ExecuteSrb+0×68
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e9370 crashdmp!Context+0×30
fffff800`0292810c nt!DisplayCharacter+0×5c
fffff880`051f3a9d dump_diskdump!ScsiPortInitialize+0×805
fffff880`051e9370 crashdmp!Context+0×30
fffff800`02929c33 nt!VidDisplayString+0×73
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e6440 crashdmp!IsBufferValid+0×28
fffff880`051e5f7a crashdmp!FilterCallback+0xae
fffff880`051f2a79 dump_diskdump!DiskDumpWrite+0×1a9
fffff880`051e5d76 crashdmp!CrashdmpWriteRoutine+0×4a
fffff880`051e4f48 crashdmp!WritePageSpanToDisk+0×180
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e4c95 crashdmp!WriteKernelDump+0×12d
fffff880`051e5d2c crashdmp!CrashdmpWriteRoutine
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff880`051e4ac5 crashdmp!DumpWrite+0×145
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e4187 crashdmp!CrashdmpWrite+0×57
fffff880`051e9a30 crashdmp!ContextCopy
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02974bc1 nt!IoWriteCrashDump+0×391
fffff800`02a898a0 nt!IopTriageDumpDataBlocks
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02936de0 nt!IoSetDumpRange
fffff800`02936d30 nt!IoFreeDumpRange
fffff800`02abe900 nt!KiProcessorBlock
fffff800`0280d000 nt!KiSelectNextThread <PERF> (nt+0×0)
fffff800`02975f26 nt!KeBugCheck2+0xac6
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`028a051e nt!SepNormalAccessCheck+0×18e
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02e0a501 hal!HalpSendFlatIpi+0×92
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`0288d744 nt!KeBugCheckEx+0×104
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02c15982 nt!PspCatchCriticalBreak+0×92
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02bc30ab nt! ?? ::NNGAKEGL::`string’+0×17ad6
fffff800`02b46698 nt!NtTerminateProcess+0xf4
fffff800`0288c8d3 nt!KiSystemServiceCopyEnd+0×13
????????`????????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 270)

Wednesday, June 17th, 2020

Inspecting memory region type to which a pointer points to may help diagnosing some coding mistakes. We call this analysis pattern Pointer Class. Below is a case study modeled on a real-life case.

The application was crashing sporadically and the memory dump was pointing to invalid objects reused after free. We recreated similar source code pattern and got the similar crash (PointerClass.exe.8752.dmp):

0:000> kL
# Child-SP          RetAddr           Call Site
00 000000e9`a10fe448 00007ff9`64e08037 ntdll!NtWaitForMultipleObjects+0x14
01 000000e9`a10fe450 00007ff9`64e07f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 000000e9`a10fe750 00007ff9`653c71fb KERNELBASE!WaitForMultipleObjects+0xe
03 000000e9`a10fe790 00007ff9`653c6ca8 kernel32!WerpReportFaultInternal+0x51b
04 000000e9`a10fe8b0 00007ff9`64eb00b8 kernel32!WerpReportFault+0xac
05 000000e9`a10fe8f0 00007ff9`672a4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 000000e9`a10fea10 00007ff9`6728c656 ntdll!RtlUserThreadStart$filt$0+0xa2
07 000000e9`a10fea50 00007ff9`672a11cf ntdll!_C_specific_handler+0x96
08 000000e9`a10feac0 00007ff9`6726a209 ntdll!RtlpExecuteHandlerForException+0xf
09 000000e9`a10feaf0 00007ff9`6729fe3e ntdll!RtlDispatchException+0x219
0a 000000e9`a10ff200 00007ff7`df32103a ntdll!KiUserExceptionDispatch+0×2e
0b 000000e9`a10ff918 00007ff7`df321081 PointerClass!Data::GetData+0xa
0c 000000e9`a10ff920 00007ff7`df32121a PointerClass!Work::DoWork+0×21
0d 000000e9`a10ff960 00007ff7`df321494 PointerClass!main+0×4a
0e (Inline Function) ——–`——– PointerClass!invoke_main+0×22
0f 000000e9`a10ff9b0 00007ff9`65377bd4 PointerClass!__scrt_common_main_seh+0×10c
10 000000e9`a10ff9f0 00007ff9`6726ce51 kernel32!BaseThreadInitThunk+0×14
11 000000e9`a10ffa20 00000000`00000000 ntdll!RtlUserThreadStart+0×21

0:000> .frame b
0b 000000e9`a10ff918 00007ff7`df321081 PointerClass!Data::GetData+0xa [C:\NewWork\PointerClass\PointerClass.cpp @ 7]

0:000> dv
this = 0×00000227`eb030000

0:000> dp poi(this)
00000227`eb030000  ????????`???????? ????????`????????
00000227`eb030010  ????????`???????? ????????`????????
00000227`eb030020  ????????`???????? ????????`????????
00000227`eb030030  ????????`???????? ????????`????????
00000227`eb030040  ????????`???????? ????????`????????
00000227`eb030050  ????????`???????? ????????`????????
00000227`eb030060  ????????`???????? ????????`????????
00000227`eb030070  ????????`???????? ????????`????????

struct Data
{
void SetData(int newData) { data = newData; }
int  GetData() { return data; }
private:
int data{};
};

An engineer found out that a pointer to an outside object was used and it was not updated when the object was freed:

struct Work
{
void SetData(Data* newData)
{
data = newData;
}

void DoWork()
{
if (data)
{
auto value = data->GetData();
++value;
data->SetData(value);
}
}

private:
Data* data{};
};

void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data* pData = model.GetData(); pData)
{
work.SetData(pData);
}

}

The solution was to use a double pointer but it also crashed (PointerClassFixNotWorking.exe.7452.dmp):

struct Work
{
void SetData(Data** newData)
{
data = newData;
}

void DoWork()
{
if (data && *data)
{
auto value = (*data)->GetData();
++value;
(*data)->SetData(value);
}
}

private:
Data** data{};
};

void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data* pData = model.GetData(); pData)
{
work.SetData(&pData);
}

}

0:000> .ecxr
*** WARNING: Unable to verify checksum for PointerClassFixNotWorking.exe
rax=0000019a4be10000 rbx=0000019a4bff29c0 rcx=0000019a4be10000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000019a4bff6cf0
rip=00007ff766d5103a rsp=0000007ede6ff958 rbp=0000000000000000
r8=0000007ede6ff938  r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
PointerClassFixNotWorking!Data::GetData+0xa:
00007ff7`66d5103a 8b00            mov     eax,dword ptr [rax] ds:0000019a`4be10000=????????

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP          RetAddr           Call Site
00 0000007e`de6ff958 00007ff7`66d51092 PointerClassFixNotWorking!Data::GetData+0xa
01 0000007e`de6ff960 00007ff7`66d5124a PointerClassFixNotWorking!Work::DoWork+0x32
02 0000007e`de6ff9a0 00007ff7`66d514d4 PointerClassFixNotWorking!main+0x4a
03 (Inline Function) --------`-------- PointerClassFixNotWorking!invoke_main+0x22
04 0000007e`de6ff9f0 00007ff9`65377bd4 PointerClassFixNotWorking!__scrt_common_main_seh+0x10c
05 0000007e`de6ffa30 00007ff9`6726ce51 kernel32!BaseThreadInitThunk+0x14
06 0000007e`de6ffa60 00000000`00000000 ntdll!RtlUserThreadStart+0x21

struct Data
{
void SetData(int newData) { data = newData; }
int  GetData() { return data; }
private:
int data{};
};

It was hypothesized that the object was also freed somewhere else and the debugging continued. However, the simple inspection of this->data Pointer Class would have revealed that it is pointing to a stack location (that was reused by subsequent calls to other functions):

0:000> .frame 1
01 0000007e`de6ff960 00007ff7`66d5124a PointerClassFixNotWorking!Work::DoWork+0x32 [C:\NewWork\PointerClassFixNotWorking\PointerClassFixNotWorking.cpp @ 24]

0:000> dv /i /v
prv local  0000007e`de6ff980           value = 0n1275013568
prv local  0000007e`de6ff9a0            this = 0x0000007e`de6ff9c0

0:000> dt this
Local var @ 0x7ede6ff9a0 Type Work*
0x0000007e`de6ff9c0
+0×000 data             : 0×0000007e`de6ff978  -> 0×0000019a`4be10000 Data

0:000> !address 0×0000007e`de6ff978
Usage:                  Stack
Base Address:           0000007e`de6fd000
End Address:            0000007e`de700000
Region Size:            00000000`00003000 (  12.000 kB)
State:                  00001000          MEM_COMMIT
Protect:                00000004          PAGE_READWRITE
Type:                   00020000          MEM_PRIVATE
Allocation Base:        0000007e`de600000
Allocation Protect:     00000004          PAGE_READWRITE
More info:              ~0k

So the correct fix should have been be to use an address (heap region in the original case) of a pointer stored inside an owner object (allocated on heap in the original case):

void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data** ppData = model.GetData(); ppData && *ppData)
{
work.SetData(ppData);
}

}

The example memory dumps, PDB files, and source code of applications be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 269)

Monday, June 15th, 2020

The collection of register values can be interpreted as Context Pointer to various memory locations:

0:000> !for_each_register -c:.if (${@#RegisterValue} > 0xFFFF) {.printf "${@#RegisterName}: %p -> %p\n", ${@#RegisterValue}, poi(${@#RegisterValue})}
rdx: 000000e4c0afe888 -> 00000000000000b4
rsp: 000000e4c0afe488 -> 00007ffecc888037
r9: 00000000ffffffff -> Memory access error at ')'
r11: 000000e4c0afdc30 -> 0000000000000090
r12: 00000000ffffffff -> Memory access error at ')'
r13: 000000e4c0afe888 -> 00000000000000b4
rip: 00007ffecf6bcbc4 -> 00841f0fc32ecdc3
edx: 00000000c0afe888 -> Memory access error at ')'
esp: 00000000c0afe488 -> Memory access error at ')'
r9d: 00000000ffffffff -> Memory access error at ')'
r11d: 00000000c0afdc30 -> Memory access error at ')'
r12d: 00000000ffffffff -> Memory access error at ')'
r13d: 00000000c0afe888 -> Memory access error at ')'
eip: 00000000cf6bcbc4 -> Memory access error at ')'

We filter pointer values to avoid a lot of registers that contain 0 values.

Individual pointers can also be Pointer Cones.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 268)

Sunday, June 14th, 2020

When interpreting a value as a pointer to a memory address we are usually interested in adjacent values pointed to:

0:000> ? rdx
Evaluate expression: 982485297288 = 000000e4`c0afe888

0:000> ? poi(000000e4`c0afe888)
Evaluate expression: 180 = 00000000`000000b4

0:000> ? poi(rdx)
Evaluate expression: 180 = 00000000`000000b4

0:000> dps rdx-10 rdx+10
000000e4`c0afe878 000000e4`c0afeac0
000000e4`c0afe880 000000e4`c0971000
000000e4`c0afe888 00000000`000000b4
000000e4`c0afe890 00000000`0000008c
000000e4`c0afe898 00000000`00000088

0:000> r $t0 = 0

0:000> dps rdx-@$t0 rdx+@$t0
000000e4`c0afe888 00000000`000000b4

0:000> r $t0 = 20

0:000> dps rdx-@$t0 rdx+@$t0
000000e4`c0afe868 00000000`00000001
000000e4`c0afe870 00001a38`00000001
000000e4`c0afe878 000000e4`c0afeac0
000000e4`c0afe880 000000e4`c0971000
000000e4`c0afe888 00000000`000000b4
000000e4`c0afe890 00000000`0000008c
000000e4`c0afe898 00000000`00000088
000000e4`c0afe8a0 00000000`00000088
000000e4`c0afe8a8 00000000`0000008c

We call this analysis pattern Pointer Cone by analogy with cones in category theory and our earlier attempts to use it. The reason for this pattern appearance is that it has been used in many times in other analysis pattern descriptions. We also intend to use this pattern language building block in our next analysis pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 267)

Saturday, June 13th, 2020

When developers look at crash dumps they are more interested at parameters and local variables in particular stack frames of interest. However, sometimes it is useful to look at all such frames especially to gather information that may be useful for technical support or to correlate to additional traces and logs (for example, Historical Information to establish additional Basic Facts and build Vocabulary Index).

Listing the parameters can be done, for example, by using Stack Trace command variant (kP WinDbg commend, but we use kPL to exclude source code references to reduce visual clutter):

0:000> kPL
# Child-SP          RetAddr           Call Site
00 000000e4`c0afe488 00007ffe`cc888037 ntdll!NtWaitForMultipleObjects+0x14
01 000000e4`c0afe490 00007ffe`cc887f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 000000e4`c0afe790 00007ffe`cd8271fb KERNELBASE!WaitForMultipleObjects+0xe
03 000000e4`c0afe7d0 00007ffe`cd826ca8 kernel32!WerpReportFaultInternal+0x51b
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 0,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0aff9d0 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×68
0d 000000e4`c0aff9b0 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 0,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affa30 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×60
0e 000000e4`c0affa10 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 1,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affa90 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×60
0f 000000e4`c0affa70 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 2,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affaf0 “Hello World! Hello World! Hello World! Hello World! “)+0×60
10 000000e4`c0affad0 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 3,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affb50 “Hello World! Hello World! “)+0×60
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo(
unsigned int64 num = 4,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affbb0 “Hello World! “)+0×60
12 000000e4`c0affb90 00007ff6`c8ab2b14 FrameTrace!main(void)+0×25
13 (Inline Function) ——–`——– FrameTrace!invoke_main+0×22
14 000000e4`c0affbe0 00007ffe`cd7d7bd4 FrameTrace!__scrt_common_main_seh(void)+0×10c
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0×14
16 000000e4`c0affc50 00000000`00000000 ntdll!RtlUserThreadStart+0×21

The stack trace comes from the following modeling application:

void foo(std::size_t num, const std::wstring& str)
{
if (std::wstring concatStr{ str }; num)
{
concatStr += str;

foo(–num, concatStr);
}
else
{
::DebugBreak();
}
}

int main()
{
foo(5, L”Hello World! “);
}

To list local variable we need to use !for_each_frame WinDbg command:

0:000> !for_each_frame "dv"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
00 000000e4`c0afe488 00007ffe`cc888037 ntdll!NtWaitForMultipleObjects+0x14
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
01 000000e4`c0afe490 00007ffe`cc887f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
02 000000e4`c0afe790 00007ffe`cd8271fb KERNELBASE!WaitForMultipleObjects+0xe
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
03 000000e4`c0afe7d0 00007ffe`cd826ca8 kernel32!WerpReportFaultInternal+0x51b
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 0
str = 0x000000e4`c0aff9d0 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0d 000000e4`c0aff9b0 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 0
str = 0x000000e4`c0affa30 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0e 000000e4`c0affa10 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 1
str = 0x000000e4`c0affa90 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0f 000000e4`c0affa70 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 2
str = 0x000000e4`c0affaf0 "Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
10 000000e4`c0affad0 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! "
num = 3
str = 0x000000e4`c0affb50 "Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! "
num = 4
str = 0x000000e4`c0affbb0 "Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
12 000000e4`c0affb90 00007ff6`c8ab2b14 FrameTrace!main+0x25 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 20]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
13 (Inline Function) --------`-------- FrameTrace!invoke_main+0x22 [d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
14 000000e4`c0affbe0 00007ffe`cd7d7bd4 FrameTrace!__scrt_common_main_seh+0x10c [d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
has_cctor = false
main_result = <value unavailable>
tls_init_callback = <value unavailable>
is_nested = <value unavailable>
tls_dtor_callback = <value unavailable>
main_result = <value unavailable>
__scrt_current_native_startup_state = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0x14
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
16 000000e4`c0affc50 00000000`00000000 ntdll!RtlUserThreadStart+0x21
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0x14

We can also apply “dv /i /V” command to each frame to get additional low-level frame details:

[...]
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
prv local  000000e4`c0affb50 @rsp+0x0020             concatStr = "Hello World! Hello World! "
prv param  000000e4`c0affb90 @rsp+0x0060                   num = 4
prv param  000000e4`c0affb98 @rsp+0x0068                   str = 0x000000e4`c0affbb0 "Hello World! "
[...]

We see this as a form of back tracing Execution Residue, for example:

0:000> !for_each_frame ".frame /c @$frame; dps rsp"
[...]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000003
rip=00007ffecd826ca8 rsp=000000e4c0afe8f0 rbp=0000000000000000
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0afeac0 r13=ffffffffffffffff
r14=000000e4c0afeac0 r15=0000000000001a38
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
kernel32!WerpReportFault+0xac:
00007ffe`cd826ca8 8bf8            mov     edi,eax
000000e4`c0afe8f0  00000000`00000000
000000e4`c0afe8f8  00000000`00000000
000000e4`c0afe900  00000000`00000003
000000e4`c0afe908  000000e4`c0afeac0
000000e4`c0afe910  00000000`00000004
000000e4`c0afe918  00000000`00000001
000000e4`c0afe920  00000000`00000000
000000e4`c0afe928  00007ffe`cc9300b8 KERNELBASE!UnhandledExceptionFilter+0x3b8
000000e4`c0afe930  00000000`00000000
000000e4`c0afe938  000000e4`c0affc50
000000e4`c0afe940  00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe948  00000207`5d660000
000000e4`c0afe950  00000000`00000000
000000e4`c0afe958  00007ffe`cf6660b9 ntdll!RtlpFindEntry+0x4d
000000e4`c0afe960  00000004`00000006
000000e4`c0afe968  00000001`00000000
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=00007ffecd7c0000 rdi=0000000000000000
rip=00007ffecc9300b8 rsp=000000e4c0afe930 rbp=000000e4c0affc50
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0afeac0 r13=ffffffffffffffff
r14=0000000000000001 r15=0000000000000004
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
KERNELBASE!UnhandledExceptionFilter+0x3b8:
00007ffe`cc9300b8 0f1f440000      nop     dword ptr [rax+rax]
000000e4`c0afe930  00000000`00000000
000000e4`c0afe938  000000e4`c0affc50
000000e4`c0afe940  00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe948  00000207`5d660000
000000e4`c0afe950  00000000`00000000
000000e4`c0afe958  00007ffe`cf6660b9 ntdll!RtlpFindEntry+0x4d
000000e4`c0afe960  00000004`00000006
000000e4`c0afe968  00000001`00000000
000000e4`c0afe970  00000000`00000001
000000e4`c0afe978  00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe980  00000207`5d662ff0
000000e4`c0afe988  00000000`00000000
000000e4`c0afe990  000000e4`c0afeac0
000000e4`c0afe998  00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe9a0  00000000`005a0058
000000e4`c0afe9a8  00007ffe`cca6ff70 KERNELBASE!`string'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
rax=000000000000005b rbx=00007ffecf764420 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000000
rip=00007ffecf6c4ab2 rsp=000000e4c0afea50 rbp=000000e4c0affc50
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0aff730 r13=000000e4c0affc50
r14=000000e4c0aff0c0 r15=00007ffecf620000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!RtlUserThreadStart$filt$0+0xa2:
00007ffe`cf6c4ab2 eb16            jmp     ntdll!RtlUserThreadStart$filt$0+0xba (00007ffe`cf6c4aca)
000000e4`c0afea50  00000000`00000000
000000e4`c0afea58  00007ffe`cf764420 ntdll!`string'+0x9aa8
000000e4`c0afea60  00000000`00000000
000000e4`c0afea68  000000e4`c0affbe0
000000e4`c0afea70  00000000`00000000
000000e4`c0afea78  00007ffe`cf6457d8 ntdll!LdrpAppendUnicodeStringToFilenameBuffer+0x50
000000e4`c0afea80  00000000`0006ce51
000000e4`c0afea88  00007ffe`cf6ac656 ntdll!_C_specific_handler+0x96
000000e4`c0afea90  000000e4`c0afeb40
000000e4`c0afea98  00007ffe`cf642930 ntdll!LdrpFindLoadedDllByNameLockHeld+0xe4
000000e4`c0afeaa0  000000e4`c0aff088
000000e4`c0afeaa8  000000e4`c0aff110
000000e4`c0afeab0  000000e4`c0aff240
000000e4`c0afeab8  00000000`00000000
000000e4`c0afeac0  000000e4`c0aff730
000000e4`c0afeac8  000000e4`c0aff240
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
rax=000000000000005b rbx=00007ffecf764420 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000000
rip=00007ffecf6ac656 rsp=000000e4c0afea90 rbp=000000000006ce51
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0aff730 r13=000000e4c0affc50
r14=000000e4c0aff0c0 r15=00007ffecf620000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!_C_specific_handler+0x96:
00007ffe`cf6ac656 85c0            test    eax,eax
000000e4`c0afea90  000000e4`c0afeb40
000000e4`c0afea98  00007ffe`cf642930 ntdll!LdrpFindLoadedDllByNameLockHeld+0xe4
000000e4`c0afeaa0  000000e4`c0aff088
000000e4`c0afeaa8  000000e4`c0aff110
000000e4`c0afeab0  000000e4`c0aff240
000000e4`c0afeab8  00000000`00000000
000000e4`c0afeac0  000000e4`c0aff730
000000e4`c0afeac8  000000e4`c0aff240
000000e4`c0afead0  00000000`00000000
000000e4`c0afead8  000000e4`c0afeb70
000000e4`c0afeae0  000000e4`c0aff240
000000e4`c0afeae8  00007ffe`cf6ac5c0 ntdll!_C_specific_handler
000000e4`c0afeaf0  00000000`00000000
000000e4`c0afeaf8  00007ffe`cf6c11cf ntdll!RtlpExecuteHandlerForException+0xf
000000e4`c0afeb00  00000000`00000000
000000e4`c0afeb08  000000e4`c0aff070
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=000000e4c0aff730 rdi=0000000000000000
rip=00007ffecf6c11cf rsp=000000e4c0afeb00 rbp=000000e4c0aff070
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=00007ffecf6ac5c0 r13=000000e4c0aff240
r14=000000e4c0afeb70 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!RtlpExecuteHandlerForException+0xf:
00007ffe`cf6c11cf 90              nop
000000e4`c0afeb00  00000000`00000000
000000e4`c0afeb08  000000e4`c0aff070
000000e4`c0afeb10  000000e4`c0aff730
000000e4`c0afeb18  000000e4`c0aff730
000000e4`c0afeb20  000000e4`c0aff0c0
000000e4`c0afeb28  00007ffe`cf68a209 ntdll!RtlDispatchException+0x219
000000e4`c0afeb30  000000e4`00000001
000000e4`c0afeb38  00007ffe`cf620000 ntdll!RtlStringCchCopyW <PERF> (ntdll+0x0)
000000e4`c0afeb40  00000000`00000000
000000e4`c0afeb48  00007ffe`cf78e9f0 ntdll!__PchSym_ <PERF> (ntdll+0x16e9f0)
000000e4`c0afeb50  000000e4`c0afeb70
000000e4`c0afeb58  000000e4`c0aff090
000000e4`c0afeb60  000000e4`c0aff080
000000e4`c0afeb68  00000000`00000000
000000e4`c0afeb70  000000e4`00000000
000000e4`c0afeb78  00007ffe`cc8300f0 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0xf0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=000000e4c0aff730 rdi=0000000000000000
rip=00007ffecf68a209 rsp=000000e4c0afeb30 rbp=000000e4c0aff070
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=00007ffecf6ac5c0 r13=000000e4c0aff240
r14=000000e4c0afeb70 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!RtlDispatchException+0x219:
00007ffe`cf68a209 8bd0            mov     edx,eax
000000e4`c0afeb30  000000e4`00000001
000000e4`c0afeb38  00007ffe`cf620000 ntdll!RtlStringCchCopyW <PERF> (ntdll+0x0)
000000e4`c0afeb40  00000000`00000000
000000e4`c0afeb48  00007ffe`cf78e9f0 ntdll!__PchSym_ <PERF> (ntdll+0x16e9f0)
000000e4`c0afeb50  000000e4`c0afeb70
000000e4`c0afeb58  000000e4`c0aff090
000000e4`c0afeb60  000000e4`c0aff080
000000e4`c0afeb68  00000000`00000000
000000e4`c0afeb70  000000e4`00000000
000000e4`c0afeb78  00007ffe`cc8300f0 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0xf0)
000000e4`c0afeb80  00000001`00000000
000000e4`c0afeb88  00000012`00000018
000000e4`c0afeb90  00000000`00000000
000000e4`c0afeb98  00360030`00300030
000000e4`c0afeba0  00001f80`0010000f
000000e4`c0afeba8  00000000`00000033
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ffecf6bfe3e rsp=000000e4c0aff240 rbp=0000000000000000
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!KiUserExceptionDispatch+0x2e:
00007ffe`cf6bfe3e 84c0            test    al,al
000000e4`c0aff240  00007ff6`c8ac32f0 FrameTrace!`string'
000000e4`c0aff248  00000000`000a0008
000000e4`c0aff250  00000207`5d662a10
000000e4`c0aff258  00007ff6`00200000
000000e4`c0aff260  000000e4`c0aff2f0
000000e4`c0aff268  000000e4`c0aff2f0
000000e4`c0aff270  00001f80`0010005f
000000e4`c0aff278  0053002b`002b0033
000000e4`c0aff280  00000246`002b002b
000000e4`c0aff288  00000000`00000000
000000e4`c0aff290  00000000`00000000
000000e4`c0aff298  00000000`00000000
000000e4`c0aff2a0  00000000`00000000
000000e4`c0aff2a8  00000000`00000000
000000e4`c0aff2b0  00000000`00000000
000000e4`c0aff2b8  000000e4`c0aff970
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ffecc8f0aa2 rsp=000000e4c0aff948 rbp=0000000000000000
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
KERNELBASE!wil::details::DebugBreak+0x2:
00007ffe`cc8f0aa2 cc              int     3
000000e4`c0aff948  00007ff6`c8ab1568 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
000000e4`c0aff950  000000e4`c0aff970
000000e4`c0aff958  000000e4`c0aff9d0
000000e4`c0aff960  00000000`000000d0
000000e4`c0aff968  00000207`5d66a990
000000e4`c0aff970  00000207`5d66b070
000000e4`c0aff978  00007ff6`c8ab15ed FrameTrace!std::basic_string<wchar_t,std::char_traits<wchar_t>, std::allocator<wchar_t> >::operator+=+0x1d [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.26.28801\include\xstring @ 2821]
000000e4`c0aff980  00000000`000001a0
000000e4`c0aff988  00000000`000001a7
000000e4`c0aff990  0000e8d4`e5494150
000000e4`c0aff998  0000e8d4`e5494150
000000e4`c0aff9a0  000000e4`c0affa30
000000e4`c0aff9a8  00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
000000e4`c0aff9b0  00000000`00000000
000000e4`c0aff9b8  000000e4`c0aff9d0
000000e4`c0aff9c0  00000000`00000068
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ff6c8ab1568 rsp=000000e4c0aff950 rbp=0000000000000000
r8=0000000000000000  r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
FrameTrace!foo+0x68:
00007ff6`c8ab1568 90              nop
000000e4`c0aff950  000000e4`c0aff970
000000e4`c0aff958  000000e4`c0aff9d0
000000e4`c0aff960  00000000`000000d0
000000e4`c0aff968  00000207`5d66a990
000000e4`c0aff970  00000207`5d66b070
000000e4`c0aff978  00007ff6`c8ab15ed FrameTrace!std::basic_string<wchar_t,std::char_traits<wchar_t>, std::allocator<wchar_t> >::operator+=+0x1d [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.26.28801\include\xstring @ 2821]
000000e4`c0aff980  00000000`000001a0
000000e4`c0aff988  00000000`000001a7
000000e4`c0aff990  0000e8d4`e5494150
000000e4`c0aff998  0000e8d4`e5494150
000000e4`c0aff9a0  000000e4`c0affa30
000000e4`c0aff9a8  00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
000000e4`c0aff9b0  00000000`00000000
000000e4`c0aff9b8  000000e4`c0aff9d0
000000e4`c0aff9c0  00000000`00000068
000000e4`c0aff9c8  00000207`5d66a8a0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[...]

We need to reset the current context after the command above since the last frame becomes the current:

0:000> kc
*** Stack trace for last set context - .thread/.cxr resets it
# Call Site
15 ntdll!RtlUserThreadStart

0:000> .cxr
Resetting default scope

0:000> kc
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 KERNELBASE!WaitForMultipleObjects
03 kernel32!WerpReportFaultInternal
04 kernel32!WerpReportFault
05 KERNELBASE!UnhandledExceptionFilter
06 ntdll!RtlUserThreadStart$filt$0
07 ntdll!_C_specific_handler
08 ntdll!RtlpExecuteHandlerForException
09 ntdll!RtlDispatchException
0a ntdll!KiUserExceptionDispatch
0b KERNELBASE!wil::details::DebugBreak
0c FrameTrace!foo
0d FrameTrace!foo
0e FrameTrace!foo
0f FrameTrace!foo
10 FrameTrace!foo
11 FrameTrace!foo
12 FrameTrace!main
13 FrameTrace!invoke_main
14 FrameTrace!__scrt_common_main_seh
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart

We call this analysis pattern Frame Trace.

The example memory dump, the application PDB file, and source code can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 266)

Saturday, May 2nd, 2020

When modeling Invalid Pointer (Objects) analysis pattern, we noticed that if we use MEM_RELEASE instead of MEM_DECOMMIT in VirtualFree API call, we see page memory contents despite an access violation Stored Exception pointing to that page. Moreover, the page contents were not corresponding to what should have been expected from source code. We had to do live kernel debugging in order to verify what was going on.

We launched InvalidPointerObject.exe that displayed the committed address allocated via VirtualAlloc API call:

The we broke into the system, found our process and inspected that address:

Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Opened \\.\pipe\com2
Waiting to reconnect...
Connected to Windows 10 18362 x64 target at (Fri May 1 22:46:00.982 2020 (UTC + 1:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 18362 MP (1 procs) Free x64
Built by: 18362.1.amd64fre.19h1_release.190318-1202
Machine Name:
Kernel base = 0xfffff800`74800000 PsLoadedModuleList = 0xfffff800`74c48190
System Uptime: 0 days 0:00:00.000
KDTARGET: Refreshing KD connection
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
* *
* You are seeing this message because you pressed either *
* CTRL+C (if you run console kernel debugger) or, *
* CTRL+BREAK (if you run GUI kernel debugger), *
* on your debugger machine's keyboard. *
* *
* THIS IS NOT A BUG OR A SYSTEM CRASH *
* *
* If you did not intend to break into the debugger, press the "g" key, then *
* press the "Enter" key now. This message might immediately reappear. If it *
* does, press "g" and "Enter" again. *
* *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff800`749c93a0 cc int 3

1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS ffffe00314e89300
SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 001ad002 ObjectTable: ffffc90314806d40 HandleCount: 3136.
Image: System

[...]

PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 33.
Image: InvalidPointerObject.exe

[...]

1: kd> !process ffffe00318d60080 3f
PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 33.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a78d1c0 Vads 22 Clone 0 Private 94. Modified 0. Locked 2.
DeviceMap ffffc903193e9bf0
Token ffffc9031c692060
ElapsedTime 00:01:13.571
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 20344
QuotaPoolUsage[NonPagedPool] 3256
Working Set Sizes (now,min,max) (497, 50, 345) (1988KB, 200KB, 1380KB)
PeakWorkingSetSize 465
VirtualSize 4139 Mb
PeakVirtualSize 4139 Mb
PageFaultCount 499
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 107
Job ffffe00317be8060

[...]

THREAD ffffe003198ba0c0 Cid 1a90.18bc Teb: 000000161ab74000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Alertable
ffffe0031a7b0238 NotificationEvent
IRP List:
ffffe00318a49510: (0006,0238) Flags: 00060900 Mdl: ffffe00319319470
Not impersonating
DeviceMap ffffc903193e9bf0
Owning Process ffffe00318d60080 Image: InvalidPointerObject.exe
Attached Process N/A Image: N/A
Wait Start TickCount 6673 Ticks: 4692 (0:00:01:13.312)
Context Switch Count 118 IdealProcessor: 1
UserTime 00:00:00.000
KernelTime 00:00:00.015
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
Win32 Start Address InvalidPointerObject!wmainCRTStartup (0x00007ff66357e044)
Stack Init ffff848c00a22c90 Current ffff848c00a22560
Base ffff848c00a23000 Limit ffff848c00a1d000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffff848c`00a225a0 fffff800`7483c7bd nt!KiSwapContext+0x76
ffff848c`00a226e0 fffff800`7483b644 nt!KiSwapThread+0xbfd
ffff848c`00a22780 fffff800`7483ade5 nt!KiCommitThreadWait+0x144
ffff848c`00a22820 fffff800`74de982a nt!KeWaitForSingleObject+0x255
ffff848c`00a22900 fffff800`74de595f nt!IopSynchronousServiceTail+0x24a
ffff848c`00a229a0 fffff800`749d2e15 nt!NtReadFile+0x59f
ffff848c`00a22a90 00007ffb`0ed3c184 nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff848c`00a22b00)
00000016`1a96f338 00007ffb`0c405227 ntdll!NtReadFile+0x14
00000016`1a96f340 00007ff6`6359b3b9 KERNELBASE!ReadFile+0x77
00000016`1a96f3c0 00000000`00000001 InvalidPointerObject!_read_nolock+0x2f5 [minkernel\crts\ucrt\src\appcrt\lowio\read.cpp @ 566]
00000016`1a96f3c8 00000000`00000000 0x1

1: kd> .thread /r /p ffffe003198ba0c0
Implicit thread is now ffffe003`198ba0c0
Implicit process is now ffffe003`18d60080
.cache forcedecodeuser done
Loading User Symbols
....

1: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 ffff848c`00a225a0 fffff800`7483c7bd nt!KiSwapContext+0x76
01 ffff848c`00a226e0 fffff800`7483b644 nt!KiSwapThread+0xbfd
02 ffff848c`00a22780 fffff800`7483ade5 nt!KiCommitThreadWait+0x144
03 ffff848c`00a22820 fffff800`74de982a nt!KeWaitForSingleObject+0x255
04 ffff848c`00a22900 fffff800`74de595f nt!IopSynchronousServiceTail+0x24a
05 ffff848c`00a229a0 fffff800`749d2e15 nt!NtReadFile+0x59f
06 ffff848c`00a22a90 00007ffb`0ed3c184 nt!KiSystemServiceCopyEnd+0x25
07 00000016`1a96f338 00007ffb`0c405227 ntdll!NtReadFile+0x14
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
08 00000016`1a96f340 00007ff6`6359b3b9 KERNELBASE!ReadFile+0x77
09 00000016`1a96f3c0 00000000`00000001 InvalidPointerObject!_read_nolock+0x2f5
0a 00000016`1a96f3c8 00000000`00000000 0x1

1: kd> !vad 146e3a70000 1

VAD @ ffffe0031a78eb10
Start VPN 146e3a70 End VPN 146e3a70 Control Area 0000000000000000
FirstProtoPte 0000000000000000 LastPte 0000000000000000 Commit Charge 1 (0n1)
Secured.Flink 0 Blink 0 Banked/Extend 0
File Offset 0
ViewUnmap MemCommit PrivateMemory READWRITE

1: kd> dc 146e3a70000
00000146`e3a70000 00000001 00000000 00000000 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….

We see the page memory contents show the correct counter value (1):

struct Resource
{
    void DoSomething()
    {
        ++m_usageCounter;
    }
    std::size_t m_usageCounter{};
}; 

We resume system execution and hit a key. The program crashes in the second DoSomething call after releasing memory that contained Resource object:

	::VirtualFree(pMem, 0, MEM_RELEASE);
	pResource->DoSomething();

We wait until WER dialog appears (we had to add DWORD DontShowUI (0) to \HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting):

We then break in the system again and inspect the same address:

1: kd> g
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
* *
* You are seeing this message because you pressed either *
* CTRL+C (if you run console kernel debugger) or, *
* CTRL+BREAK (if you run GUI kernel debugger), *
* on your debugger machine's keyboard. *
* *
* THIS IS NOT A BUG OR A SYSTEM CRASH *
* *
* If you did not intend to break into the debugger, press the "g" key, then *
* press the "Enter" key now. This message might immediately reappear. If it *
* does, press "g" and "Enter" again. *
* *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff800`749c93a0 cc int 3

0: kd> .thread /r /p ffffe003198ba0c0
Implicit thread is now ffffe003`198ba0c0
Implicit process is now ffffe003`18d60080
.cache forcedecodeuser done
Loading User Symbols
....

0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 ffff848c`00a21f70 fffff800`7483c7bd nt!KiSwapContext+0x76
01 ffff848c`00a220b0 fffff800`7483b644 nt!KiSwapThread+0xbfd
02 ffff848c`00a22150 fffff800`748884e7 nt!KiCommitThreadWait+0x144
03 ffff848c`00a221f0 fffff800`74e1ffe9 nt!KeWaitForMultipleObjects+0x287
04 ffff848c`00a22300 fffff800`74e1fd05 nt!ObWaitForMultipleObjects+0x2a9
05 ffff848c`00a22800 fffff800`749d2e15 nt!NtWaitForMultipleObjects+0x105
06 ffff848c`00a22a90 00007ffb`0ed3cc14 nt!KiSystemServiceCopyEnd+0x25
07 00000016`1a96e208 00007ffb`0c438027 ntdll!NtWaitForMultipleObjects+0x14
08 00000016`1a96e210 00007ffb`0c437f0e KERNELBASE!WaitForMultipleObjectsEx+0x107
09 00000016`1a96e510 00007ffb`0e0071fb KERNELBASE!WaitForMultipleObjects+0xe
0a 00000016`1a96e550 00007ffb`0e006ca8 KERNEL32!WerpReportFaultInternal+0x51b
0b 00000016`1a96e670 00007ffb`0c4df868 KERNEL32!WerpReportFault+0xac
0c 00000016`1a96e6b0 00007ffb`0ed44b32 KERNELBASE!UnhandledExceptionFilter+0x3b8
0d 00000016`1a96e7d0 00007ffb`0ed2c6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
0e 00000016`1a96e810 00007ffb`0ed4121f ntdll!_C_specific_handler+0x96
0f 00000016`1a96e880 00007ffb`0ed0a289 ntdll!RtlpExecuteHandlerForException+0xf
10 00000016`1a96e8b0 00007ffb`0ed3fe8e ntdll!RtlDispatchException+0x219
11 00000016`1a96efc0 00007ff6`6357378a ntdll!KiUserExceptionDispatch+0×2e
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
12 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa
13 00000016`1a96f6e0 00007ff6`6357dfd4 InvalidPointerObject!wmain+0xd5
14 (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
15 00000016`1a96f730 00007ffb`0dfb7bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
16 00000016`1a96f770 00007ffb`0ed0ced1 KERNEL32!BaseThreadInitThunk+0×14
17 00000016`1a96f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×21

0: kd> .frame 0n18;dv /t /v
12 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa [C:\NewWork\InvalidPointerObject\InvalidPointerObject.cpp @ 10]
00000016`1a96f6e0 struct Resource * this = 0×00000146`e3a70000

0: kd> !vad 146e3a70000 1

VAD @ ffffe0031ab91080
Start VPN 146e3a70 End VPN 146e3a70 Control Area ffffe0031a66f780
FirstProtoPte ffffc9031b8fdf50 LastPte ffffc9031b8fdf50 Commit Charge 0 (0n0)
Secured.Flink 0 Blink 0 Banked/Extend 0
File Offset 0
ViewShare READWRITE

ControlArea @ ffffe0031a66f780
Segment ffffc9031c7d55d0 Flink ffffe0031ab91f40 Blink ffffe0031ab910e0
Section Ref 1 Pfn Ref 0 Mapped Views 3
User Ref 4 WaitForDel 0 Flush Count 1
File Object 0000000000000000 ModWriteCount 0 System Views 0
WritableRefs 0 PartitionId 0
Flags (2000) Commit

Pagefile-backed section

Segment @ ffffc9031c7d55d0
ControlArea ffffe0031a66f780 ExtendInfo 0000000000000000
Total Ptes 1
Segment Size 1000 Committed 1
CreatingProcessId 1a90 FirstMappedVa 146e3a70000
ProtoPtes ffffc9031b8fdf50
Flags (80000) ProtectionMask

0: kd> !ca ffffe0031a66f780 4

ControlArea @ ffffe0031a66f780
Segment ffffc9031c7d55d0 Flink ffffe0031ab91f40 Blink ffffe0031ab910e0
Section Ref 1 Pfn Ref 0 Mapped Views 3
User Ref 4 WaitForDel 0 Flush Count 1
File Object 0000000000000000 ModWriteCount 0 System Views 0
WritableRefs 0 PartitionId 0
Flags (2000) Commit

Pagefile-backed section

3 mapped view(s):

ffffe0031ab91f40 - VAD ffffe0031ab91ee0, process ffffe0031a8d3080 WerFault.exe
ffffe0031ab93ca0 - VAD ffffe0031ab93c40, process ffffe0031acd3080 InvalidPointer
ffffe0031ab910e0 - VAD ffffe0031ab91080, process ffffe00318d60080 InvalidPointer

0: kd> dc 146e3a70000
00000146`e3a70000 000000f0 00001a90 000018bc 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….

We see the page contents changed (it now contains PID and TID) and also its pagefile-backed section lists 3 mapped views including 2 new processes, one is Zombie Process duplicate of the original InvalidPointerObject.exe process and another is WerFault.exe:

0: kd> !process ffffe0031acd3080
PROCESS ffffe0031acd3080
SessionId: 1 Cid: 06cc Peb: 161ab73000 ParentCid: 1a90
DirBase: 9d002002 ObjectTable: ffffc9031c03e5c0 HandleCount: 0.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a789d90 Vads 21 Clone ffffe00318770810 Private 41. Modified 0. Locked 0.
DeviceMap 0000000000000000
Token ffffc9031d32a770
ElapsedTime 00:00:26.741
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 17776
QuotaPoolUsage[NonPagedPool] 6024
Working Set Sizes (now,min,max) (28, 50, 345) (112KB, 200KB, 1380KB)
PeakWorkingSetSize 10
VirtualSize 4138 Mb
PeakVirtualSize 4138 Mb
PageFaultCount 28
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 51

No active threads

0: kd> !process 1a90
Searching for Process with Cid == 1a90
PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 39.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a78d1c0 Vads 22 Clone ffffe00318770590 Private 43. Modified 18. Locked 0.
DeviceMap ffffc903193e9bf0
Token ffffc9031c692060
ElapsedTime 00:38:20.130
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 20352
QuotaPoolUsage[NonPagedPool] 6328
Working Set Sizes (now,min,max) (547, 50, 345) (2188KB, 200KB, 1380KB)
PeakWorkingSetSize 515
VirtualSize 4139 Mb
PeakVirtualSize 4139 Mb
PageFaultCount 552
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 106
Job ffffe00317be8060

[...]

We resume system execution and collect the process crash dump. When we look at the crash address we see the same unexpected False Memory contents:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1a90.18bc): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ffb`0ed3cc14 c3 ret

0:000> kL
# Child-SP RetAddr Call Site
00 00000016`1a96e208 00007ffb`0c438027 ntdll!NtWaitForMultipleObjects+0x14
01 00000016`1a96e210 00007ffb`0c437f0e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 00000016`1a96e510 00007ffb`0e0071fb KERNELBASE!WaitForMultipleObjects+0xe
03 00000016`1a96e550 00007ffb`0e006ca8 kernel32!WerpReportFaultInternal+0x51b
04 00000016`1a96e670 00007ffb`0c4df868 kernel32!WerpReportFault+0xac
05 00000016`1a96e6b0 00007ffb`0ed44b32 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 00000016`1a96e7d0 00007ffb`0ed2c6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
07 00000016`1a96e810 00007ffb`0ed4121f ntdll!_C_specific_handler+0x96
08 00000016`1a96e880 00007ffb`0ed0a289 ntdll!RtlpExecuteHandlerForException+0xf
09 00000016`1a96e8b0 00007ffb`0ed3fe8e ntdll!RtlDispatchException+0x219
0a 00000016`1a96efc0 00007ff6`6357378a ntdll!KiUserExceptionDispatch+0×2e
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
0b 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa
0c 00000016`1a96f6e0 00007ff6`6357dfd4 InvalidPointerObject!wmain+0xd5
0d (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
0e 00000016`1a96f730 00007ffb`0dfb7bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
0f 00000016`1a96f770 00007ffb`0ed0ced1 kernel32!BaseThreadInitThunk+0×14
10 00000016`1a96f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×21

0:000> dx Debugger.Sessions[0].Processes[6800].Threads[6332].Stack.Frames[11].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[6800].Threads[6332].Stack.Frames[11].SwitchTo()
00000016`1a96f6e0 struct Resource * this = 0×00000146`e3a70000

0:000> !address 0×00000146`e3a70000

Usage: <unknown>
Base Address: 00000146`e3a70000
End Address: 00000146`e3a71000
Region Size: 00000000`00001000 ( 4.000 kB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00040000 MEM_MAPPED

Allocation Base: 00000146`e3a70000
Allocation Protect: 00000004 PAGE_READWRITE

Content source: 1 (target), length: 1000

0:000> dc 0×00000146`e3a70000
00000146`e3a70000 000000f0 00001a90 000018bc 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….

0:000> ~
. 0 Id: 1a90.18bc Suspend: 0 Teb: 00000016`1ab74000 Unfrozen

0:000> dx -r1 ((InvalidPointerObject!Resource *)0x146e3a70000)
((InvalidPointerObject!Resource *)0x146e3a70000) : 0×146e3a70000 [Type: Resource *]
[+0×000] m_usageCounter : 0×1a90000000f0 [Type: unsigned __int64]

0:000> .ecxr
rax=00000146e3a70000 rbx=00000146e3aa5bf0 rcx=00000146e3a70000
rdx=0000000000000000 rsi=0000000000000000 rdi=00000146e3aa5c70
rip=00007ff66357378a rsp=000000161a96f6d8 rbp=0000000000000000
r8=000000161a96f6a8 r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
InvalidPointerObject!Resource::DoSomething+0xa:
00007ff6`6357378a 488b00 mov rax,qword ptr [rax] ds:00000146`e3a70000=00001a90000000f0

Such False Memory may complicate the analysis of process crash dumps when we want to examine memory contents prior to exception.

The example memory dump, the application PDB file, and source code can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 265)

Thursday, April 30th, 2020

In addition to generic Invalid Pointer pattern that maps to visible pointer dereference in C and C++ code, plain NULL Code Pointers and NULL Data Pointers that are visible Small Values, and Wild Pointers showing ASCII or Regular Data (such as UNICODE fragments), we have implicit dereference (from C++ source code perspective) crash dump analysis patterns that we call Invalid Pointer (Objects). When seeing them in a high-level debugger (could be just an exception during debugging) developers are confused since they do not see the usual pointer dereference:

struct Resource
{
    void DoSomething()
    {
        ++m_usageCounter;
    }
    std::size_t m_usageCounter{};
}; 

However, the function call was ordinary (not virtual, otherwise we would have NULL Code Pointer), and the object address to access its members was passed via RCX register, but the memory of the object was invalid, hence we have an exception inside the method call when trying to access object members:

0:000> .ecxr
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
rax=0000022c837e0000 rbx=0000022c83905ca0 rcx=0000022c837e0000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000022c83907540
rip=00007ff6d65630ba rsp=00000098812ffc18 rbp=0000000000000000
r8=00000098812ffbe8 r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
InvalidPointerObject!Resource::DoSomething+0xa:
00007ff6`d65630ba 488b00 mov rax,qword ptr [rax] ds:0000022c`837e0000=????????????????

0:000> !address @rax

Usage:
Base Address: 0000022c`837e0000
End Address: 0000022c`837e1000
Region Size: 00000000`00001000 ( 4.000 kB)
State: 00002000 MEM_RESERVE
Protect:
Type: 00020000 MEM_PRIVATE
Allocation Base: 0000022c`837e0000
Allocation Protect: 00000004 PAGE_READWRITE

0:000> kL
# Child-SP RetAddr Call Site
00 00000098`812fe748 00007ffd`62278027 ntdll!NtWaitForMultipleObjects+0×14
01 00000098`812fe750 00007ffd`62277f0e KERNELBASE!WaitForMultipleObjectsEx+0×107
02 00000098`812fea50 00007ffd`63d871fb KERNELBASE!WaitForMultipleObjects+0xe
03 00000098`812fea90 00007ffd`63d86ca8 kernel32!WerpReportFaultInternal+0×51b
04 00000098`812febb0 00007ffd`6231f868 kernel32!WerpReportFault+0xac
05 00000098`812febf0 00007ffd`64ee4b32 KERNELBASE!UnhandledExceptionFilter+0×3b8
06 00000098`812fed10 00007ffd`64ecc6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
07 00000098`812fed50 00007ffd`64ee121f ntdll!_C_specific_handler+0×96
08 00000098`812fedc0 00007ffd`64eaa289 ntdll!RtlpExecuteHandlerForException+0xf
09 00000098`812fedf0 00007ffd`64edfe8e ntdll!RtlDispatchException+0×219
0a 00000098`812ff500 00007ff6`d65630ba ntdll!KiUserExceptionDispatch+0×2e
0b 00000098`812ffc18 00007ff6`d656313c InvalidPointerObject!Resource::DoSomething+0xa
0c 00000098`812ffc20 00007ff6`d6568454 InvalidPointerObject!wmain+0×6c
0d (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
0e 00000098`812ffc70 00007ffd`63d37bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
0f 00000098`812ffcb0 00007ffd`64eaced1 kernel32!BaseThreadInitThunk+0×14
10 00000098`812ffce0 00000000`00000000 ntdll!RtlUserThreadStart+0×21

0:000> ub 00007ff6`d656313c
InvalidPointerObject!wmain+0×45:
00007ff6`d6563115 488b4c2428 mov rcx,qword ptr [rsp+28h]
00007ff6`d656311a e891ffffff call InvalidPointerObject!Resource::DoSomething (00007ff6`d65630b0)
00007ff6`d656311f 41b800400000 mov r8d,4000h
00007ff6`d6563125 33d2 xor edx,edx
00007ff6`d6563127 488b4c2420 mov rcx,qword ptr [rsp+20h]
00007ff6`d656312c ff15ce0e0200 call qword ptr [InvalidPointerObject!_imp_VirtualFree (00007ff6`d6584000)]
00007ff6`d6563132 488b4c2428 mov rcx,qword ptr [rsp+28h]
00007ff6`d6563137 e874ffffff call InvalidPointerObject!Resource::DoSomething (00007ff6`d65630b0)

0:000> u InvalidPointerObject!Resource::DoSomething
InvalidPointerObject!Resource::DoSomething:
00007ff6`d65630b0 48894c2408 mov qword ptr [rsp+8],rcx
00007ff6`d65630b5 488b442408 mov rax,qword ptr [rsp+8]
00007ff6`d65630ba 488b00 mov rax,qword ptr [rax]
00007ff6`d65630bd 48ffc0 inc rax
00007ff6`d65630c0 488b4c2408 mov rcx,qword ptr [rsp+8]
00007ff6`d65630c5 488901 mov qword ptr [rcx],rax
00007ff6`d65630c8 c3 ret
00007ff6`d65630c9 cc int 3

The example memory dump, the application PDB file, and source code can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 264)

Thursday, February 27th, 2020

Interrupts can happen in either kernel or user mode. In the latter case, upon transition to kernel mode, a special memory region in is used for interrupt processing in kernel space, distinct from the thread’s kernel stack, that we call Interrupt Stack. It can also be used for mining Execution Residue.

2: kd> !thread -1 1f
THREAD fffffa801a9fa3e0  Cid 0f74.0804  Teb: 000007ffffdf8000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff88000007400
Owning Process            fffffa801a949c10       Image:         App.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      81642662       Ticks: 0
Context Switch Count      58671950       IdealProcessor: 4
UserTime                  01:33:39.702
KernelTime                00:01:11.401
Win32 Start Address 0x000007fef9b1050c
Stack Init fffffa6005af4db0 Current fffffa6005af4950
Base fffffa6005af5000 Limit fffffa6005aef000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`01793b98 fffff800`01a58eee nt!KeBugCheckEx
fffffa60`01793ba0 fffff800`01a57dcb nt!KiBugCheckDispatch+0×6e
fffffa60`01793ce0 fffffa60`00eb279b nt!KiPageFault+0×20b (TrapFrame @ fffffa60`01793ce0)
fffffa60`01793e70 fffffa60`00e62739 tcpip! ?? ::FNODOBFM::`string’+0×3883b
fffffa60`01794020 fffffa60`00e62194 tcpip!TcpMatchReceive+0×1b9
fffffa60`01794120 fffffa60`00e52ddd tcpip!TcpPreValidatedReceive+0×2e4
fffffa60`017941c0 fffffa60`00e52e89 tcpip!IppDeliverListToProtocol+0×4d
fffffa60`01794280 fffffa60`00e52463 tcpip!IppProcessDeliverList+0×59
fffffa60`017942f0 fffffa60`00e5176c tcpip!IppReceiveHeaderBatch+0×223
fffffa60`017943d0 fffffa60`00e50d54 tcpip!IpFlcReceivePackets+0×8dc
fffffa60`017945d0 fffffa60`00e61133 tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0×264
fffffa60`017946b0 fffffa60`009a40bc tcpip!FlReceiveNetBufferListChain+0xd3
fffffa60`01794700 fffffa60`0096c8c9 NDIS!ndisMIndicateNetBufferListsToOpen+0xac
fffffa60`01794750 fffffa60`008016f7 NDIS!ndisMDispatchReceiveNetBufferLists+0×1d9
fffffa60`01794bd0 fffffa60`02b4e2d3 NDIS!NdisMIndicateReceiveNetBufferLists+0×67
fffffa60`01794c10 fffffa60`02b3de0c Driver+0×152d3
fffffa60`01794de0 fffffa60`02b3df6b Driver+0×4e0c
fffffa60`01794e20 fffffa60`02b3e0b3 Driver+0×4f6b
fffffa60`01794e60 fffffa60`00801670 Driver+0×50b3
fffffa60`01794ec0 fffff800`01a5d367 NDIS!ndisInterruptDpc+0xc0
fffffa60`01794f40 fffff800`01a5bc35 nt!KiRetireDpcList+0×117
fffffa60`01794fb0 fffff800`01a5ba47 nt!KyRetireDpcList+0×5 (TrapFrame @ fffffa60`01794e70)
fffffa60`05af4bf0 fffff800`01aa1b28 nt!KiDispatchInterruptContinue
fffffa60`05af4c20 000007fe`f7e5c55a nt!KiDpcInterrupt+0xf8 (TrapFrame @ fffffa60`05af4c20)
00000000`4deae430 00000000`00000000 0×000007fe`f7e5c55a

2: kd> !address fffffa60`01794e60
Usage:
Base Address:           fffffa60`011ff000
End Address:            fffffa60`019dc000
Region Size:            00000000`007dd000

VA Type:                SystemDynamicSpace
VAD Address:            0×27676e69727473
Commit Charge:          0×244a0f51940
Protection:             0×244a0f51940 []
Memory Usage:           Private
No Change:              yes
More info:              !vad 0xfffffa60011ff000

2: kd> !address fffffa60`05af4c20
Usage:                  Stack
Base Address:           fffffa60`05aef000
End Address:            fffffa60`05af5000
Region Size:            00000000`00006000

VA Type:                SystemDynamicSpace

2: kd> dpS fffffa60`01793b98 fffffa60`01794fb0
[…]
fffffa60`05657c3f Driver2+0×4c3f
fffffa60`05656369 Driver2+0×3369

[…]
fffffa60`00801670 NDIS!ndisInterruptDpc+0xc0
fffff800`01a5d367 nt!KiRetireDpcList+0×117
fffff800`01a5bc35 nt!KyRetireDpcList+0×5
fffffa60`008015b0 NDIS!ndisInterruptDpc

2: kd> ub fffffa60`05657c3f
Driver2+0×4c25:
fffffa60`05657c25 8bf2            mov     esi,edx
fffffa60`05657c27 33d2            xor     edx,edx
fffffa60`05657c29 418be8          mov     ebp,r8d
fffffa60`05657c2c 488bd9          mov     rbx,rcx
fffffa60`05657c2f 448d4240        lea     r8d,[rdx+40h]
fffffa60`05657c33 488d48b8        lea     rcx,[rax-48h]
fffffa60`05657c37 418bf9          mov     edi,r9d
fffffa60`05657c3a e8010e0000      call    Driver2+0×5a40 (fffffa60`05658a40)

2: kd> ub fffffa60`05656369
Driver2+0×334d:
fffffa60`0565634d cc              int     3
fffffa60`0565634e cc              int     3
fffffa60`0565634f cc              int     3
fffffa60`05656350 4889542410      mov     qword ptr [rsp+10h],rdx
fffffa60`05656355 48894c2408      mov     qword ptr [rsp+8],rcx
fffffa60`0565635a 4883ec58        sub     rsp,58h
fffffa60`0565635e 488d4c2428      lea     rcx,[rsp+28h]
fffffa60`05656363 ff15972c0000    call    qword ptr [Driver2+0×6000 (fffffa60`05659000)]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 263)

Friday, February 21st, 2020

Sometimes, when we have One-Thread Process memory dumps, it is possible to get other stack regions indirectly through the analysis of virtual memory regions. Consider, for example, this dump that has only one process exit thread:

0:000> kL
# Child-SP          RetAddr           Call Site
00 000000f1`828ff848 00007ff9`d29aa9b8 ntdll!NtTerminateProcess+0x14
01 000000f1`828ff850 00007ff9`d113cd8a ntdll!RtlExitUserProcess+0xb8
02 000000f1`828ff880 00007ff7`fbb91231 kernel32!ExitProcessImplementation+0xa
03 000000f1`828ff8b0 00007ff7`fbb9125f HiddenStack!bar1+0x41
04 000000f1`828ffa80 00007ff7`fbb91cb5 HiddenStack!foo1+0x1f
05 000000f1`828ffc40 00007ff7`fbb91b1b HiddenStack!std::_Invoker_functor::_Call<void (__cdecl*)(void)>+0x15
06 000000f1`828ffc70 00007ff7`fbb917c4 HiddenStack!std::invoke<void (__cdecl*)(void)>+0x1b
07 000000f1`828ffca0 00007ff7`fbb99728 HiddenStack!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0x64
08 000000f1`828ffcf0 00007ff9`d1137bd4 HiddenStack!thread_start<unsigned int (__cdecl*)(void *),1>+0x50
09 000000f1`828ffd20 00007ff9`d29aced1 kernel32!BaseThreadInitThunk+0x14
0a 000000f1`828ffd50 00000000`00000000 ntdll!RtlUserThreadStart+0x21

There are no more thread stack traces:

0:000> ~
. 0 Id: 27d4.22a4 Suspend: -1 Teb: 000000f1`8266a000 Unfrozen

However, in addition to thread #0, we can find several regions having PAGE_GUARD protection:

0:000> !address
[...]
+       f1`82800000       f1`828fb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~0; 27d4.22a4]
f1`828fb000       f1`828fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          Stack      [~0; 27d4.22a4]
f1`828fe000       f1`82900000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~0; 27d4.22a4]
+       f1`82900000       f1`829fb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`829fb000       f1`829fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`829fe000       f1`82a00000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82a00000       f1`82afc000        0`000fc000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`82afc000       f1`82aff000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`82aff000       f1`82b00000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82b00000       f1`82bfb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`82bfb000       f1`82bfe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`82bfe000       f1`82c00000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82c00000      1fe`828f0000      10c`ffcf0000             MEM_FREE    PAGE_NOACCESS                      Free
[…]

We then can get Rough Stack Traces out of them:

0:000> .lines -d
Line number information will not be loaded

0:000> dpS f1`829fe000       f1`82a00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d0070000 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0×0)
00007ff9`d007b4b1 KERNELBASE!SetTEBLangID+0×2d
00007ff9`d007ac70 KERNELBASE!_KernelBaseBaseDllInitialize+0×90
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d19e7890 msvcrt!CRTDLL_INIT
00007ff9`d29db5a3 ntdll!RTL_BINARY_ARRAY<RTLP_FLS_SLOT,8,4>::ChunkAllocate+0×67
00007ff9`d19e0000 msvcrt!`dynamic initializer for ‘__ExceptionPtr::m_badAllocExceptionPtr” <PERF> (msvcrt+0×0)
00007ff9`d29db65d ntdll!RTL_BINARY_ARRAY<RTLP_FLS_SLOT,8,4>::SetValue+0×39
00007ff9`d2964ef7 ntdll!RtlDeactivateActivationContextUnsafeFast+0xc7
00007ff9`d299439c ntdll!RtlFlsSetValue+0xec
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00000000`7ffe0301 SharedUserData+0×301
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00000000`7ffe0358 SharedUserData+0×358
00007ff7`fbb923ca HiddenStack!std::chrono::duration_cast<std::chrono::duration<double,
std::ratio<1,1000000000> >,__int64,std::ratio<1,1000000000>,void>+0×4a
00000000`7ffe0358 SharedUserData+0×358
00007ff9`d294bb47 ntdll!RtlGetSystemTimePrecise+0×57
00007ff9`d00b6931 KERNELBASE!SleepEx+0xa1
00007ff9`d00d3890 KERNELBASE!GetSystemTimePreciseAsFileTime+0×10
00007ff7`fbb931b4 HiddenStack!_Thrd_sleep+0×3c
00007ff7`fbb916c5 HiddenStack!std::this_thread::sleep_until<std::chrono::steady_clock,
std::chrono::duration<__int64,std::ratio<1,1000000000> > >+0×65
00007ff7`fbb91651 HiddenStack!std::chrono::operator+<std::chrono::steady_clock,
std::chrono::duration<__int64,std::ratio<1,1000000000> >,__int64,std::ratio<1,1> >+0×41
00007ff7`fbb913fd HiddenStack!std::this_thread::sleep_for<__int64,std::ratio<1,1> >+0×2d
00007ff7`fbb912a9 HiddenStack!bar2+0×39
00007ff7`fbb912df HiddenStack!foo2+0×1f
00007ff7`fbb91cb5 HiddenStack!std::_Invoker_functor::_Call<void (__cdecl*)(void)>+0×15
00007ff7`fbb91aec HiddenStack!std::unique_ptr<std::tuple<void (__cdecl*)(void)>,std::default_delete<std::tuple<void (__cdecl*)(void)> > >::unique_ptr<std::tuple<void (__cdecl*)(void)>,std::default_delete<std::tuple<void (__cdecl*)(void)> > ><std::default_delete<std::tuple<void (__cdecl*)(void)> >,0>+0×2c
00007ff7`fbb91b1b HiddenStack!std::invoke<void (__cdecl*)(void)>+0×1b
00007ff7`fbb917c4 HiddenStack!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0×64
00007ff7`fbb9c1d7 HiddenStack!__acrt_getptd+0xb3
00007ff7`fbb99728 HiddenStack!thread_start<unsigned int (__cdecl*)(void *),1>+0×50
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

0:000> dpS f1`82aff000       f1`82b00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d2962da8 ntdll!LdrpInitializeThread+0×40
00007ff9`d297562f ntdll!TppCallbackCheckThreadAfterCallback+0×9f
00007ff9`d29700e5 ntdll!RtlRegisterThreadWithCsrss+0×35
00007ff9`d29b18f5 ntdll!_LdrpInitialize+0×89
00007ff9`d2975394 ntdll!TppCallbackEpilog+0×144
00007ff9`d29701d6 ntdll!TppCritSetThread+0×7a
00007ff9`d2973155 ntdll!TppWorkCallbackPrologRelease+0×1c9
00007ff9`d296e2c3 ntdll!LdrpWorkCallback+0×63
00007ff9`d2aa52f0 ntdll!LdrpWorkQueue
00007ff9`d29708a2 ntdll!TppWorkpExecuteCallback+0xb2
00000000`7ffe0386 SharedUserData+0×386
00007ff9`d2974060 ntdll!TppWorkerThread+0×300
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

0:000> dpS f1`82bfe000       f1`82c00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d2962da8 ntdll!LdrpInitializeThread+0×40
00007ff9`d29700e5 ntdll!RtlRegisterThreadWithCsrss+0×35
00007ff9`d29b18f5 ntdll!_LdrpInitialize+0×89
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d29701d6 ntdll!TppCritSetThread+0×7a
00007ff9`d2970098 ntdll!TppPoolAddWorker+0×68
00007ff9`d2974060 ntdll!TppWorkerThread+0×300
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

We call such analysis pattern Hidden Stack as another way to get Historical Information from memory dumps.

The example memory dump, the application PDB file, and source code can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 261)

Sunday, October 13th, 2019

Raw stack memory region analysis is more productive with Region Clusters analysis pattern. Here we apply simple clustering techniques to organize various region values into disjoint sets with chosen semantics. For our purposes simple sort suffices to generate such clusters that can be visually inspected. We take the same stack.csv file from Region Profile analysis pattern. It’s values are sorted and the results are shown in sorted order with corresponding count of occurrences and symbolic references (we use the internal version of Narrascope written in C++, a narrative debugger, but you can use your favorite scripting language here):

0 count: 13718
1 count: 273
2 count: 23
3 count: 22
4 count: 28
5 count: 9
6 count: 5
7 count: 18
8 count: 35
9 count: 5
a count: 24
b count: 12
c count: 4
d count: 3
e count: 1
f count: 28
10 count: 14
...
c0000034 count: 2
c0000388 count: 2
c01c0001 count: 1
c0a70000 count: 1
d0908070 count: 1
dcae0fa0 count: 1
e30000e3 count: 1
f80004fc count: 2
ffff5815 count: 2
fffffed3 count: 2
fffffffd count: 2
ffffffff count: 18
100000000 count: 6
100000001 count: 4
100001f80 count: 1
100001fa0 count: 16
100001fa4 count: 2
100003033 count: 2
100010000 count: 1
...
7ff700000000 count: 1
7ff700000001 count: 2
7ff70000000d count: 1
7ff747390000 Photoshop_exe count: 1
7ff74ebd4ec0 Photoshop_exe+0x7844ec0 count: 1
7ff74ef351c7 Photoshop_exe+0x7ba51c7 count: 1
7ff74ef4e2f0 Photoshop_exe+0x7bbe2f0 count: 1
7ff74ef4e5a9 Photoshop_exe+0x7bbe5a9 count: 1
...
7fff00000000 count: 21
7fff00000001 count: 7
7fff00000002 count: 1
7fff00000003 count: 1
7fff00000004 count: 1
7fff00000011 count: 1
7fff00000020 count: 1
7fff00000040 count: 3
7fff00000102 count: 1
7fff0000029e count: 3
7fff00140000 count: 1
7fff02000002 count: 1
7fff4782c33b libcef!GetHandleVerifier+0x61d7b count: 1
7fff4782c884 libcef!GetHandleVerifier+0x622c4 count: 1
7fff493749cc libcef!cef_time_to_timet+0x1a9228 count: 2
...
7fff9a0c1e57 GdiPlus!GpGraphics::MeasureString+0x333 count: 1
7fff9a128c2a GdiPlus!FastTextImager::MeasureString+0x32 count: 1
7fff9a174e18 GdiPlus!GpFontFamily::vftable' count: 2
7fff9b6055b3 DWrite!FontFace::GetDesignGlyphAdvances+0x57 count: 1
7fffa7e6c260 comctl32!ListBox_WndProc count: 5
7fffa7e6c357 comctl32!ListBox_WndProc+0xf7 count: 2
7fffb1373c18 npmproxy!INotifyNetworkListManagerEventsProxyVtbl+0x1b8 count: 1
7fffb2c14e96 msvcp140!_Mbrtowc+0x66 [f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c @ 156] count: 1
...
7fffc09f0359 ntdll!qsort+0x379 count: 1
7fffc09fa1e4 ntdll!woutput_s+0x8e8 count: 1
7fffc09fa297 ntdll!write_string+0x3f count: 1
7fffc09fbd30 ntdll!NtdllDefWindowProc_W count: 2
7fffc09fbf10 ntdll!NtdllDispatchHook_W count: 2
7fffc09ffc54 ntdll!KiUserCallForwarder+0x24 count: 1
7fffc09ffdb4 ntdll!KiUserCallbackDispatcherContinue count: 2
800000000000 count: 1
800000000001 count: 2
800063640000 count: 36
800066660000 count: 38
80006f6f0000 count: 2
800072720000 count: 8
800075750000 count: 1
974b00000000 count: 1
974b8118d10d count: 1
a76b00000000 count: 1
a76bb8365307 count: 1
a76bb8378c47 count: 1
a76bb8378f77 count: 1
a76bb837bfd7 count: 1
a8c300000000 count: 1
a8c311cf265f count: 1
...
30000000000000 count: 1
30000000310030 count: 1
30000300470048 count: 1
30002000100000 count: 1
3000300030007b count: 1
3000300031002d count: 1
30003000310031 count: 2
300031002d0037 count: 1
30003800390032 count: 3
31000000000000 count: 1
310000007d0036 count: 1
31002d00310037 count: 1
310032002d0035 count: 1
...
7fdf7fbd7f9c7f7b count: 2
8000800000000001 count: 1
8000800000001fa0 count: 1
8000800080000000 count: 6
8000800080008000 count: 52
80121a254b25250a count: 1
923800003f000000 count: 2
bf000000bf000000 count: 1
bff0000000000000 count: 2
e5b2a56118358cbe count: 2
ffff0072656c6c6f count: 1
fffffdb773438b57 count: 3
ffffff0000000005 count: 1
ffffff7bc010786f count: 1
ffffff7bc010787f count: 1
fffffffb00000000 count: 1
ffffffff00000000 count: 4
ffffffff00000001 count: 3
ffffffff00000005 count: 1
ffffffff00001fa0 count: 2
ffffffff4c494146 count: 2
ffffffffffffc3ce count: 1
fffffffffffffef6 count: 1
ffffffffffffff00 count: 2
ffffffffffffff01 count: 2
fffffffffffffffe count: 166
ffffffffffffffff count: 38

We can easily identify error values, module boundaries, and Regular Data. The sorting can also be done for double word or word values, for example to isolate errors or wide character values, but this will have to be seen whether it is useful.

This clustering approach can be depicted in the following idealized diagram:

The full output can be found here: stack-clusters.txt for stack.csv file.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 260)

Friday, September 27th, 2019

Manual analysis of Execution Residue in stack regions can be quite daunting so some basic statistics on the distribution of address, value, and symbolic information can be useful. To automate this process we use Pandas python library and interpret preprocessed WinDbg output of dps and dpp commands as DataFrame object:

import pandas as pd
import pandas_profiling

df = pd.read_csv("stack.csv")
html_file = open("stack.html", "w")
html_file.write (pandas_profiling.ProfileReport(df).to_html())
html_file.close()

We get a rudimentary profile report: stack.html for stack.csv file. The same was also done for Address, Address/Value, Value, Symbol output of dpp command: stack4columns.html for stack4columns.csv file.

We call this analysis pattern Region Profile since any Memory Region can be used. This analysis pattern is not limited to Python/Pandas and different libraries/scripts/script languages can also be used for statistical and correlational profiling. We plan to provide more examples here in the future.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 259)

Sunday, September 1st, 2019

Sometimes we have complex plugins or subsystems consisting from several modules that are loaded in the same process address space (and this is also possible in kernel space as well). Usually subsystem components reside in the same persistent folder (including its possible subfolders):

0:000> lmf
start             end                 module name
...
00007fff`46ee0000 00007fff`4cde8000 libcef C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\libcef.dll

00007fff`7fa40000 00007fff`7fbc2000 Spaces C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\Spaces.8li

00007fff`8ba50000 00007fff`8bae3000 chrome_elf C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\chrome_elf.dll

When we know product architecture we can group modules according to the known design not only by their physical locations.

This analysis pattern, that we call Subsystem Modules, is useful for the analysis of possibles relationships of Stack Traces from Stack Trace Collections and other Historical Information during the root cause analysis of various crash and hang issues. This can also be applicable to .NET managed space analysis that includes various multi-file assemblies.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 16d)

Thursday, May 23rd, 2019

Stack Overflow (user mode) analysis pattern relied on prominent stack trace signs. However, in cases of low memory condition there may not be an opportunity to create a new guard page for the expanding stack region and the exception itself may be Hidden Exception:

0:003> kc
# Call Site
00 KERNELBASE!GetModuleHandleExW
01 EdgeContent!wil::details::GetModuleInformation
02 EdgeContent!wil::details::GetCurrentModuleName
03 EdgeContent!wil::details::LogFailure
04 EdgeContent!wil::details::ReportFailure
05 EdgeContent!wil::details::ReportFailure_Hr
06 EdgeContent!wil::details::in1diag3::FailFast_Hr
07 EdgeContent!`anonymous namespace'::MemoryLimitWatchdogThreadProc
08 kernel32!BaseThreadInitThunk
09 ntdll!RtlUserThreadStart

0:003> .exr -1
ExceptionAddress: 00007ff992f6391b (KERNELBASE!GetModuleHandleExW+0x000000000000007b)
ExceptionCode: c00000fd (Stack overflow)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 000000fe709fdfc8

0:003> r
rax=0000000000000001 rbx=0000000000000006 rcx=00007ff945e3fa80
rdx=000000fe709fe020 rsi=00007ff945e3fa80 rdi=000000fe709fe030
rip=00007ff992f6391b rsp=000000fe709fdfd0 rbp=0000000000000040
r8=000000fe709fe030  r9=0000000000000040 r10=00000fff28bc7dba
r11=0440000001000010 r12=00007ff94606be60 r13=000000000000013f
r14=0000000000000003 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010200
KERNELBASE!GetModuleHandleExW+0x7b:
00007ff9`92f6391b ff1597f31200    call    qword ptr [KERNELBASE!_imp_RtlPcToFileHeader (00007ff9`93092cb8)] ds:00007ff9`93092cb8={ntdll!RtlPcToFileHeader (00007ff9`960d0f50)}

The stack trace is short and the exception seem happened during call so we examine the stack pointer:

0:003> !address 000000fe709fdfd0
Usage:                  Stack
Base Address:           000000fe`709fb000
End Address:            000000fe`70a00000
Region Size:            00000000`00005000 (  20.000 kB)
State:                  00001000          MEM_COMMIT

Protect:                00000004          PAGE_READWRITE
Type:                   00020000          MEM_PRIVATE
Allocation Base:        000000fe`70900000
Allocation Protect:     00000004          PAGE_READWRITE

The committed stack region is very small so we expect its surrounding from the output of !address WinDbg command (without any address parameters):

Opened log file 'c:\MemoryDumps\stack.txt'
0:003> !address 000000fe709fdfd0
[...]
+ fe`70600000 fe`706fb000 0`000fb000 MEM_PRIVATE MEM_RESERVE Stack [~1; 1f50.21d4]
fe`706fb000 fe`706fe000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~1; 1f50.21d4]
fe`706fe000 fe`70700000 0`00002000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~1; 1f50.21d4]
+ fe`70700000 fe`70800000 0`00100000 MEM_FREE PAGE_NOACCESS Free
+ fe`70800000 fe`708f9000 0`000f9000 MEM_PRIVATE MEM_RESERVE Stack [~2; 1f50.460c]
fe`708f9000 fe`708fc000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~2; 1f50.460c]
fe`708fc000 fe`70900000 0`00004000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~2; 1f50.460c]
+ fe`70900000 fe`709fb000 0`000fb000 MEM_PRIVATE MEM_RESERVE Stack [~3; 1f50.4f80]
fe`709fb000 fe`70a00000 0`00005000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~3; 1f50.4f80]

+ fe`70a00000 fe`70afc000 0`000fc000 MEM_PRIVATE MEM_RESERVE Stack [~4; 1f50.4ff4]
fe`70afc000 fe`70aff000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~4; 1f50.4ff4]
fe`70aff000 fe`70b00000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~4; 1f50.4ff4]
+ fe`70b00000 fe`70bf9000 0`000f9000 MEM_PRIVATE MEM_RESERVE Stack [~5; 1f50.5ea0]
fe`70bf9000 fe`70bfc000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~5; 1f50.5ea0]
fe`70bfc000 fe`70c00000 0`00004000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~5; 1f50.5ea0]
[…]

We see that there is no guard page for the stack ~3 compared to other stacks and there is plenty of reserved space before that to create a new guard page in the normal circumstances. We also recall that the stack overflow exception is about the inability to create a guard page:

0:003> !error c00000fd
Error code: (NTSTATUS) 0xc00000fd (3221225725) - A new guard page for the stack cannot be created.

Looking at Stack Trace Collection we find the thread ~11 that was doing System Call:

0:003> ~*kc
[...]

11  Id: 1f50.2424 Suspend: 1 Teb: 000000fe`6fec8000 Unfrozen
# Call Site
00 ntdll!NtAllocateVirtualMemory
01 KERNELBASE!VirtualAlloc
02 eShims!NS_ACGLockdownTelemetry::APIHook_VirtualAlloc
03 Chakra!Memory::VirtualAllocWrapper::AllocPages
[…]

For this thread the last error value show that there was low memory condition, perhaps the page file was full to create new committed pages:

0:003> ~11s
ntdll!NtAllocateVirtualMemory+0x14:
00007ff9`9615aca4 c3              ret

0:011> !teb
TEB at 000000fe6fec8000
ExceptionList:        0000000000000000
StackBase:            000000fe71e00000
StackLimit:           000000fe7140a000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 000000fe6fec8000
EnvironmentPointer:   0000000000000000
ClientId:             0000000000001f50 . 0000000000002424
RpcHandle:            0000000000000000
Tls Storage:          0000023c84be2c70
PEB Address:          000000fe6fe97000
LastErrorValue:       1455
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

0:011> !error c000012d
Error code: (NTSTATUS) 0xc000012d (3221225773) - {Out of Virtual Memory}  Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

0:011> !error 0n1455
Error code: (Win32) 0x5af (1455) - The paging file is too small for this operation to complete.

Because the original analysis you see above was done on the memory dump from the working computer we are not able to provide that memory dump for security reasons. So, to model this behavior, we created an application that has two threads: one leaks committed virtual memory and when the next allocation fails it creates a separate thread that attempts to do stack overflow. We then launched 20 copies of that process to limit the size of the individual crash dump. The example memory dump and the application PDB file can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 258)

Sunday, February 3rd, 2019

When we have Active Thread:

0:001> kL
# Child-SP RetAddr Call Site
00 (Inline Function) --------`-------- Windows_Web!JsonParser::ParseString+0x96
01 0000007f`56efdb00 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0x3b7
02 0000007f`56efdb80 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
03 0000007f`56efdbd0 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x663
04 0000007f`56efdc50 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
05 0000007f`56efdda0 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0x2ab
06 0000007f`56efde20 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
07 0000007f`56efde70 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x663
08 0000007f`56efdef0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
09 0000007f`56efe040 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x2ab
0a 0000007f`56efe0c0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
0b 0000007f`56efe210 00007fff`70011a69 Windows_Web!JsonParser::ParseValue+0x2ab
0c 0000007f`56efe290 00007fff`700119d2 Windows_Web!JsonValueImpl::ParseInternal+0x75
0d 0000007f`56efe2d0 00007fff`700118e8 Windows_Web!JsonValueImpl::InitializeFromString+0x16
0e 0000007f`56efe300 00007fff`70011876 Windows_Web!JsonValueFactory::TryParseInternal+0x50
0f 0000007f`56efe330 00007fff`3a6fe48e Windows_Web!JsonValueFactory::TryParse+0x36
10 0000007f`56efe360 00007fff`3a78f95c eModel!JsonUtils::GetJsonObject+0x82
[...]

which is also Spiking Thread:

0:001> !runaway f
User Mode Time
Thread Time
1:1a2c 0 days 0:03:37.562
8:52a8 0 days 0:00:36.890
[…]

Kernel Mode Time
Thread Time
8:52a8 0 days 0:00:10.625
1:1a2c 0 days 0:00:18.078
[...]

we can look at the relative proportion of User Mode / Kernel Mode times to infer Active Space. This can be further confirmed if we have Step Dumps, for example, in our case we had an earlier memory dump with the same Active Thread and Constant Subtrace (with Variable Subtraces in the same Active Space):

0:001> kL
# Child-SP RetAddr Call Site
00 0000007f`56efd8d0 00007fff`a1109453 ntdll!RtlpHpLfhSlotAllocate+0×126
01 0000007f`56efd9c0 00007fff`a110a8db ntdll!RtlpAllocateHeapInternal+0×313
02 0000007f`56efdaa0 00007fff`9ea09960 ntdll!RtlpHpAllocWithExceptionProtection+0×3b
03 0000007f`56efdb70 00007fff`7001ba1b msvcrt!malloc+0×70
04 0000007f`56efdba0 00007fff`700187ca Windows_Web!operator new+0×23
05 (Inline Function) ——–`——– Windows_Web!StringBuilder::EnsureBufferSpace+0×18
06 (Inline Function) ——–`——– Windows_Web!JsonParser::ParseString+0×59
07 0000007f`56efdbd0 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×37a

08 0000007f`56efdc50 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
09 0000007f`56efdda0 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0×2ab
0a 0000007f`56efde20 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
0b 0000007f`56efde70 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×663
0c 0000007f`56efdef0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
0d 0000007f`56efe040 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×2ab
0e 0000007f`56efe0c0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
0f 0000007f`56efe210 00007fff`70011a69 Windows_Web!JsonParser::ParseValue+0×2ab
10 0000007f`56efe290 00007fff`700119d2 Windows_Web!JsonValueImpl::ParseInternal+0×75
11 0000007f`56efe2d0 00007fff`700118e8 Windows_Web!JsonValueImpl::InitializeFromString+0×16
12 0000007f`56efe300 00007fff`70011876 Windows_Web!JsonValueFactory::TryParseInternal+0×50
13 0000007f`56efe330 00007fff`3a6fe48e Windows_Web!JsonValueFactory::TryParse+0×36
14 0000007f`56efe360 00007fff`3a78f95c eModel!JsonUtils::GetJsonObject+0×82
[…]

Even in the absence of Active Threads in process memory dumps it is useful to look at time spent in kernel and user modes to see any anomalies, especially when threads are blocked in System Calls. Then the high value of kernel mode time should suggest to inspect kernel Activity Space (for example, via a kernel dump). If we have an indication of high values of user mode time in a kernel memory dump we may need to obtain process memory Step Dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 257)

Tuesday, November 20th, 2018

In addition to Exception Thread analysis pattern (or several threads when we have Multiple Exceptions) that we named Exception Stack Trace there may be a separated thread that reports such exceptions (Exception Reporting Thread):

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(4a0c.2ab4): Application hang - code cfffffff (first/second chance not available)
win32u!NtUserCreateWindowEx+0x14:

0:002> kc
# Call Site
00 win32u!NtUserCreateWindowEx
01 user32!VerNtUserCreateWindowEx
02 user32!CreateWindowInternal
03 user32!CreateWindowExW
04 CoreMessaging!Microsoft::CoreUI::Dispatch::UserAdapter:: InitializeWindow
05 CoreMessaging!Microsoft::CoreUI::Dispatch::UserAdapter::Initialize
06 CoreMessaging!Microsoft::CoreUI::Dispatch::EventLoop::Callback_Run
07 CoreMessaging!Microsoft::CoreUI::Messaging::MessageSession$R:: Microsoft__CoreUI__IExportMessageSession_Impl::Run
08 CoreMessaging!Microsoft::CoreUI:: IExportMessageSession$X__ExportAdapter::Run
09 CoreMessaging!Windows::System::DispatcherQueue::RunLoop
0a CoreMessaging!Windows::System::DispatcherQueueController:: DispatcherQueueThreadProc
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

0:002> ~42kc
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!WerpWaitForCrashReporting
02 ntdll!RtlReportExceptionEx
03 eModel!s_ReportAsyncBoundaryHangOnSuspendedThread
04 kernel32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

We can spot such a thread in Stack Trace Collection.

Although the example above is for application hang (Blocked Thread) and also illustrates Self-Diagnosis, similar threads may report Handled Exceptions. Usually, Exception Threads are also Exception Reporting Threads as shown in Invalid Parameter analysis pattern example.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 13l)

Saturday, October 6th, 2018

Insufficient stack memory may not result in Stack Overflow if there are internal thread checks or external watchdog threads. In some way this is a specialization of a more general Insufficient Memory (region) analysis pattern.

Consider this thread stack trace we see when we open a crash dump in WinDbg:

0:003> kc
# Call Site
00 EdgeContent!wil::details::ReportFailure
01 EdgeContent!wil::details::ReportFailure_Hr
02 EdgeContent!wil::details::in1diag3::FailFast_Hr
03 EdgeContent!`anonymous namespace'::MemoryLimitWatchdogThreadProc
04 kernel32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

The reported error is related to general security checks and doesn’t reveal much:

0:003> .lastevent
Last event: 10e0.25fc: Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
debugger time: Sat Sep 29 21:30:31.531 2018 (UTC + 1:00)

However, from Stack Trace Motif we may infer that the failure was related to some memory limit.

When looking at Stack Trace Collection we notice the thread #11 which has the long sequence of frames:

11 Id: 10e0.d68 Suspend: 2 Teb: 000000ca`e96e3000 Unfrozen
# Call Site
00 ntdll!NtWaitForSingleObject
01 KERNELBASE!WaitForSingleObjectEx
02 Chakra!Memory::Recycler::WaitForConcurrentThread
03 Chakra!Memory::Recycler::FinishConcurrentCollect
04 Chakra!ThreadContext::ExecuteRecyclerCollectionFunction
05 Chakra!Memory::Recycler::FinishConcurrentCollectWrapped
06 Chakra!Memory::Recycler::LargeAlloc<0>
07 Chakra!Memory::Recycler::AllocLeaf
08 Chakra!Js::CompoundString::GetSz
09 Chakra!Js::JSONStringifier::ReadProperty
0a Chakra!Js::JSONStringifier::ReadObject
0b Chakra!Js::JSONStringifier::ReadProperty
0c Chakra!Js::JSONStringifier::Stringify
0d Chakra!JSON::Stringify
0e Chakra!amd64_CallFunction
0f Chakra!Js::InterpreterStackFrame::OP_CallCommon<Js::OpLayoutDynamicProfile<
Js::OpLayoutT_CallIWithICIndex<Js::LayoutSizePolicy<0> > > >
10 Chakra!Js::InterpreterStackFrame::ProcessUnprofiled
11 Chakra!Js::InterpreterStackFrame::Process
12 Chakra!Js::InterpreterStackFrame::InterpreterHelper
13 Chakra!Js::InterpreterStackFrame::InterpreterThunk
[...]
fa Chakra!ThreadContext::ExecuteImplicitCall<<
lambda_5a46706206cf607f01fe0fb33b6e8acf> >
fb Chakra!Js::DynamicObject::CallToPrimitiveFunction
fc Chakra!Js::DynamicObject::ToPrimitiveImpl<409>
fd Chakra!Js::DynamicObject::ToPrimitive
fe Chakra!Js::JavascriptConversion::OrdinaryToPrimitive
ff Chakra!Js::JavascriptConversion::MethodCallToPrimitive

It looks like the thread stack is really long that we suspect the possibility of overflow:

0:003> ~11kc 0xffff
[...]
c473 Chakra!amd64_CallFunction
c474 Chakra!Js::JavascriptFunction::CallFunction<1>
c475 Chakra!Js::JavascriptFunction::CallRootFunctionInternal
c476 Chakra!Js::JavascriptFunction::CallRootFunction
c477 Chakra!ScriptSite::CallRootFunction
c478 Chakra!ScriptSite::Execute
c479 Chakra!ScriptEngineBase::Execute
c47a edgehtml!CMutationObserver::PerformMicrotaskCheckpoint
c47b edgehtml!CObserverManager::InvokeObserversForCheckpoint
c47c edgehtml!CObserverManager::MicroTaskExecutionCallback
c47d edgehtml!HTML5TaskScheduler::RunReadiedTask
c47e edgehtml!HTML5TaskScheduler::PerformMicrotaskCheckpoint
c47f edgehtml!CJScript9Holder::ExecuteCallback
c480 edgehtml!CScriptTimer::ExecuteTimer
c481 edgehtml!CScriptTimerManager::OnTick
c482 edgehtml!CPaintAlignedTimerManager<CPaintBeat>::ProcessTimers
c483 edgehtml!CPaintBeat::OnProcessTimersTask
c484 edgehtml!GWndAsyncTask::Run
c485 edgehtml!HTML5TaskScheduler::RunReadiedTask
c486 edgehtml!TaskSchedulerBase::RunReadiedTasksInTaskQueueWithCallback
c487 edgehtml!HTML5TaskScheduler::RunReadiedTasks
c488 edgehtml!HTML5EventLoopDriver::DriveLowPriorityTaskExecution
c489 edgehtml!GlobalWndOnPaintPriorityMethodCall
c48a edgehtml!GlobalWndProc
c48b user32!FilteredProcessRedirectingWndProcW
c48c user32!UserCallWinProcCheckWow
c48d user32!DispatchClientMessage
c48e user32!_fnDWORD
c48f ntdll!KiUserCallbackDispatcherContinue
c490 win32u!NtUserDispatchMessage
c491 user32!DispatchMessageWorker
c492 EdgeContent!CBrowserTab::_TabWindowThreadProc
c493 EdgeContent!LCIETab_ThreadProc
c494 edgeIso!_IsoThreadProc_WrapperToReleaseScope
c495 kernel32!BaseThreadInitThunk
c496 ntdll!RtlUserThreadStart

However, the current stack pointer is the way above the stack region limit:

0:003> ~11s
ntdll!NtWaitForSingleObject+0x14:
00007ffa`b0d99f74 c3              ret

0:011> r rsp
rsp=000000caeb033348

0:011> !teb
TEB at 000000cae96e3000
ExceptionList:        0000000000000000
StackBase:            000000caeba00000
StackLimit:           000000caeb00b000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 000000cae96e3000
EnvironmentPointer:   0000000000000000
ClientId:             00000000000010e0 . 0000000000000d68
RpcHandle:            0000000000000000
Tls Storage:          000001e915185590
PEB Address:          000000cae96a8000
LastErrorValue:       1455
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

The last error code shows possible Insufficient Memory (committed memory) pattern but there were enough space in the page file and the same crashes were observed in a much more spacious memory environments:

0:011> !error c000012d
Error code: (NTSTATUS) 0xc000012d (3221225773) - {Out of Virtual Memory}  Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

0:011> !error 0n1455
Error code: (Win32) 0x5af (1455) - The paging file is too small for this operation to complete.

When we look at a raw stack trace region though, we see it was used to the fullest:

0:011> dps 000000caeb00b000 L30
000000ca`eb00b000 00000000`00000000
000000ca`eb00b008 00000000`00000000
000000ca`eb00b010 00000000`00000000
000000ca`eb00b018 00000000`00000000
000000ca`eb00b020 00000000`00000000
000000ca`eb00b028 00000000`00000000
000000ca`eb00b030 00000000`00000000
000000ca`eb00b038 00000000`00000000
000000ca`eb00b040 00000000`00000000
000000ca`eb00b048 00000000`00000000
000000ca`eb00b050 00000000`00000000
000000ca`eb00b058 00000000`00000000
000000ca`eb00b060 00000000`00000000
000000ca`eb00b068 00000000`00000000
000000ca`eb00b070 00000000`00000000
000000ca`eb00b078 00000000`00000000
000000ca`eb00b080 00000000`00000000
000000ca`eb00b088 00000000`00000000
000000ca`eb00b090 00000000`00000000
000000ca`eb00b098 000001e9`258805c0
000000ca`eb00b0a0 00000000`00000000
000000ca`eb00b0a8 00000000`00000000
000000ca`eb00b0b0 00000000`00000131
000000ca`eb00b0b8 00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned>::GetProperty+0×119
000000ca`eb00b0c0 00000000`00000006
000000ca`eb00b0c8 000001e9`258805c0
000000ca`eb00b0d0 000001e9`258805c0
000000ca`eb00b0d8 000001e9`283d4e70
000000ca`eb00b0e0 000000ca`eb00b210
000000ca`eb00b0e8 00000000`00000000
000000ca`eb00b0f0 00000000`00000131
000000ca`eb00b0f8 000001f1`71863c70
000000ca`eb00b100 00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
000000ca`eb00b108 00000000`00000000
000000ca`eb00b110 000001f1`71863c70
000000ca`eb00b118 000000ca`eb00b210
000000ca`eb00b120 000001e9`258805c0
000000ca`eb00b128 00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0×53
000000ca`eb00b130 000001e9`24cc65b0
000000ca`eb00b138 000001e9`258805c0
000000ca`eb00b140 000001e9`258805c0
000000ca`eb00b148 00000000`00000131
000000ca`eb00b150 000000ca`eb00b210
000000ca`eb00b158 00000000`00000000
000000ca`eb00b160 000001f1`71863c70
000000ca`eb00b168 00000000`00000000
000000ca`eb00b170 00000000`00000000
000000ca`eb00b178 00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0×64

We also see Execution Residue possibly indicating catch exception processing:

0:011> dpS 000000caeb00b000 L100
00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned short>::GetProperty+0x119
00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0x53
00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0x64
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b048a5da msvcrt!getptd_noexit+0x6e
00007ffa`8dfb36d6 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
+0x50
00007ffa`b048a553 msvcrt!getptd+0xb
00007ffa`b045aad6 msvcrt!_DestructExceptionObject+0x46
00007ffa`b0463798 msvcrt!_CxxCallCatchBlock+0x1e8
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4
00007ffa`8e324798 Chakra!ValueType::PrimitiveOrObject+0xa80
00007ffa`8dfb3686 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
00007ffa`8dd20000 Chakra!_jscriptinfo_IID_Lookup <PERF> (Chakra+0x0)
00007ffa`b04635b0 msvcrt!_CxxCallCatchBlock
00007ffa`b0d9e2b3 ntdll!RcConsolidateFrames+0x3
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4
00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned short>::GetProperty+0x119
00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0x53
00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0x64
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b048a5da msvcrt!getptd_noexit+0x6e
00007ffa`8dfb36d6 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
+0x50
00007ffa`b048a553 msvcrt!getptd+0xb
00007ffa`b045aad6 msvcrt!_DestructExceptionObject+0x46
00007ffa`b0463798 msvcrt!_CxxCallCatchBlock+0x1e8
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4

Another similar crash on a different machine with much more memory available had Active Thread Exception Stack Trace:

0:013> kc
# Call Site
00 ntdll!NtQueryWnfStateNameInformation
01 ntdll!SignalStartWerSvc
02 ntdll!SendMessageToWERService
03 ntdll!ReportExceptionInternal
04 kernel32!WerpReportFaultInternal
05 kernel32!WerpReportFault
06 KERNELBASE!UnhandledExceptionFilter
07 Chakra!FatalExceptionFilter
08 Chakra!ReportFatalException$filt$0
09 msvcrt!_C_specific_handler
0a ntdll!RtlpExecuteHandlerForException
0b ntdll!RtlDispatchException
0c ntdll!RtlRaiseException
0d KERNELBASE!RaiseException
0e Chakra!ReportFatalException
0f Chakra!OutOfMemory_fatal_error
10 Chakra!Js::Exception::RaiseIfScriptActive
11 Chakra!Js::Throw::OutOfMemory
12 Chakra!Memory::Recycler::LargeAlloc<0>
13 Chakra!Memory::Recycler::AllocLeaf
14 Chakra!Js::LazyJSONString::GetSz
15 Chakra!Js::CompoundString::AppendGeneric<Js::CompoundString>
16 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor
17 Chakra!Js::InlineCache::TryGetProperty<1,1,1,1,0>
18 Chakra!Js::CacheOperators::TryGetProperty<1,1,1,1,1,1,1,0,0>
19 Chakra!Js::JavascriptOperators::PatchGetValue<1,Js::InlineCache>
1a js!d.toString
[...]

which actually showed the Javascript URL:

0:013> .frame 1a
1a 00000037`4671e5e0 00007ffe`1753f1a1 js!d.toString [https://yastatic.net/awaps-ad-sdk-js-bundles/1.0-889/bundles/inpage.bundle.js @ 8,36454]

The stack region was also almost full with execution residue including exception catch processing. But this time it show “stack space” hint:

0:013> !teb
TEB at 00000037456dc000
ExceptionList:        0000000000000000
StackBase:            0000003747100000
StackLimit:           000000374670a000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 00000037456dc000
EnvironmentPointer:   0000000000000000
ClientId:             0000000000005914 . 0000000000001150
RpcHandle:            0000000000000000
Tls Storage:          000001a5bcfd88d0
PEB Address:          00000037456bd000
LastErrorValue:       0
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

0:013> s-su 000000374670a000 00000037`46720000
[…]
00000037`4670d210 “tack space”
[…]

Note: Regardless of the actual root cause and mechanism for these crash dumps, such stack region monitoring can also be used for software prognostics purposes.

The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 256)

Saturday, August 4th, 2018

Backwards disassembling used in memory analysis patterns such as Coincidental Symbolic Information may be ambiguous and can show Wild Code output. This may also be debugger disassembling algorithm dependent. For example, default 8-instruction backwards disassembly shows this code:

0:011> ub 00007ff8`cdc9b4bf
00007ff8`cdc9b4ab 855948 test dword ptr [rcx+48h],ebx
00007ff8`cdc9b4ae b988bf03a8 mov ecx,0A803BF88h
00007ff8`cdc9b4b3 f4 hlt
00007ff8`cdc9b4b4 0100 add dword ptr [rax],eax
00007ff8`cdc9b4b6 00488b add byte ptr [rax-75h],cl
00007ff8`cdc9b4b9 09e8 or eax,ebp
00007ff8`cdc9b4bb 117236 adc dword ptr [rdx+36h],esi
00007ff8`cdc9b4be 5f pop rdi

However, if we specify the number of instructions to disassemble except 7 and 8 we get a different result (which is more correct from the forward code execution view since we disassembled the saved return address from the stack region):

0:011> ub 00007ff8`cdc9b4bf L1
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> ub 00007ff8`cdc9b4bf L2
00007ff8`cdc9b4b7 488b09 mov rcx,qword ptr [rcx]
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> k L10
# Child-SP RetAddr Call Site
00 0000002a`fc23e308 00007ff9`53d06099 ntdll!NtWaitForMultipleObjects+0x14
01 0000002a`fc23e310 00007ff9`2d1a96be KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 0000002a`fc23e610 00007ff9`2d1a951c clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62
03 0000002a`fc23e670 00007ff9`2d1a9315 clr!Thread::DoAppropriateWaitWorker+0x1e4
04 0000002a`fc23e770 00007ff9`2d0c2b7f clr!Thread::DoAppropriateWait+0x7d
05 0000002a`fc23e7f0 00007ff9`2d1aa491 clr!CLREventBase::WaitEx+0xc4
06 0000002a`fc23e880 00007ff9`2d1aa39e clr!AwareLock::EnterEpilogHelper+0xc2
07 0000002a`fc23e940 00007ff9`2d1c1a92 clr!AwareLock::EnterEpilog+0x62
08 0000002a`fc23e9a0 00007ff8`cdc9b4bf clr!JITutil_MonEnterWorker+0xe2
09 0000002a`fc23eb40 00007ff9`275231d3 0×00007ff8`cdc9b4bf
0a 0000002a`fc23eb80 00007ff9`27523064 mscorlib_ni+0×5031d3
0b 0000002a`fc23ec50 00007ff9`27523032 mscorlib_ni+0×503064
0c 0000002a`fc23ec80 00007ff9`2751c812 mscorlib_ni+0×503032
0d 0000002a`fc23ecd0 00007ff9`2d006bb3 mscorlib_ni+0×4fc812
0e 0000002a`fc23ed10 00007ff9`2d006a70 clr!CallDescrWorkerInternal+0×83
0f 0000002a`fc23ed50 00007ff9`2d00735d clr!CallDescrWorkerWithHandler+0×4e

We call this analysis pattern Disassembly Ambiguity. The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 255)

Sunday, March 11th, 2018

A virtual memory may contain regions that are memories of some other processes or systems. We do not consider the ordinary case of memory-mapped regions here but the case of type 2 hypervisor. In such a case, memory regions may be “physical memories” of Virtualized Systems. For example, we discovered such a region in crashed vmware-vmx.exe process memory dump:

0:007> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 231 7ffe`d009b000 ( 127.995 TB) 100.00%
<unknown> 518 1`2508e000 ( 4.579 GB) 96.41% 0.00%
Image 547 0`07056000 ( 112.336 MB) 2.31% 0.00%
Heap 73 0`0216a000 ( 33.414 MB) 0.69% 0.00%
Stack 81 0`01b00000 ( 27.000 MB) 0.56% 0.00%
Other 11 0`001d0000 ( 1.813 MB) 0.04% 0.00%
TEB 27 0`00036000 ( 216.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 88 1`0e25a000 ( 4.221 GB) 88.88% 0.00%
MEM_PRIVATE 623 0`1aca5000 ( 428.645 MB) 8.81% 0.00%
MEM_IMAGE 547 0`07056000 ( 112.336 MB) 2.31% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 231 7ffe`d009b000 ( 127.995 TB) 100.00%
MEM_COMMIT 1185 1`27657000 ( 4.616 GB) 97.18% 0.00%
MEM_RESERVE 73 0`088fe000 ( 136.992 MB) 2.82% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 473 1`1f38b000 ( 4.488 GB) 94.49% 0.00%
PAGE_READONLY 400 0`04a05000 ( 74.020 MB) 1.52% 0.00%
PAGE_EXECUTE_READ 196 0`0367a000 ( 54.477 MB) 1.12% 0.00%
PAGE_WRITECOPY 59 0`001de000 ( 1.867 MB) 0.04% 0.00%
PAGE_READWRITE|PAGE_GUARD 27 0`00051000 ( 324.000 kB) 0.01% 0.00%
PAGE_NOACCESS 27 0`0001b000 ( 108.000 kB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 3 0`00003000 ( 12.000 kB) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 290`ffe50000 7d66`9b210000 ( 125.401 TB)
<unknown> 28f`f8f90000 1`00000000 ( 4.000 GB)
Image 7ffa`9969f000 0`00e47000 ( 14.277 MB)
Heap 28f`95c7b000 0`00ae4000 ( 10.891 MB)
Stack b8`f7b00000 0`000fc000 (1008.000 kB)
Other 28f`f2050000 0`00181000 ( 1.504 MB)
TEB b8`f7147000 0`00002000 ( 8.000 kB)
PEB b8`f7146000 0`00001000 ( 4.000 kB)

The size of the region is 4 GB which coincides with the size of Windows VM:

We assume that the whole VM physical space was placed there and we had an instance of a physical memory dump inside a process memory dump. Whatever is such a physical memory dump internal organization, most likely the pages correspond to 4 Kb memory chunks inside. We can employ WinDbg commands that allow the address parameter. For example, we can look for Hidden Modules:

0:007> .imgscan /r 28f`f8f90000 L?1`00000000
[…]
MZ at 00000290`f5867000 - size 7f000
Name: HAL.dll
[…]
MZ at 00000290`a089b000 - size 3000
Name: TDI.SYS
[…]

0:007> !dh 00000290`a089b000

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
2 number of sections
592AD310 time date stamp Sun May 28 06:39:28 2017

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2122 characteristics
Executable
App can handle >2gb addresses
32 bit word machine
DLL

OPTIONAL HEADER VALUES
10B magic #
9.00 linker version
A00 size of code
400 size of initialized data
0 size of uninitialized data
0 address of entry point
1000 base of code
----- new -----
ffffffff8a7d0000 image base
1000 section alignment
200 file alignment
3 subsystem (Windows CUI)
10.00 operating system version
10.00 image version
5.01 subsystem version
3000 size of image
400 size of headers
10F33 checksum
0000000000040000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
540 DLL characteristics
Dynamic base
NX compatible
No structured exception handler
1140 [ 73A] address [size] of Export Directory
0 [ 0] address [size] of Import Directory
2000 [ 3E8] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
0 [ 0] address [size] of Base Relocation Directory
1000 [ 1C] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
0 [ 0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
0 [ 0] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory

SECTION HEADER #1
.text name
87A virtual size
1000 virtual address
A00 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read

Debug Directories(1)
Type Size Address Pointer
cv 20 101c 41c Format: RSDS, guid, 1, tdi.pdb

SECTION HEADER #2
.rsrc name
3E8 virtual size
2000 virtual address
400 size of raw data
E00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only
[...]

We call such pattern Hyperdump:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 254)

Wednesday, January 31st, 2018

Sometimes we can diagnose potential data races by looking at Stack Trace Collection (from all processes and threads or from CPUs). For example, if we have one Exception Stack Trace and a very similar stack trace, for example, with Constant Subtrace and the same Stack Trace Motif, it may point to such a situation with potential diagnosis of related components. Here’s an example from a kernel memory dump where it was suggested as a workaround to close ProcessA before hibernating a system (the problem repeatedly happened on wake up):

0: kd> !analyze -v

[...]

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure. The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000003, A LIST_ENTRY has been corrupted (i.e. double remove).
Arg2: ffff8900ea8fae40, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffff8900ea8fad98, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

[...]

EXCEPTION_RECORD: ffff8900ea8fad98 -- (.exr 0xffff8900ea8fad98)
ExceptionAddress: fffff8034a9e2e93 (nt!CmpFreeKeyControlBlock+0x0000000000106a73)
ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 0000000000000003
Subcode: 0x3 FAST_FAIL_CORRUPT_LIST_ENTRY

[...]

0: kd> k
# Child-SP RetAddr Call Site
00 ffff8900`ea8fab18 fffff803`4a576ba9 nt!KeBugCheckEx
01 ffff8900`ea8fab20 fffff803`4a576f50 nt!KiBugCheckDispatch+0x69
02 ffff8900`ea8fac60 fffff803`4a575286 nt!KiFastFailDispatch+0xd0
03 ffff8900`ea8fae40 fffff803`4a9e2e93 nt!KiRaiseSecurityCheckFailure+0x3c6
04 ffff8900`ea8fafd0 fffff803`4a8c444b nt!CmpFreeKeyControlBlock+0×106a73
05 ffff8900`ea8fb000 fffff803`4a8e1392 nt!CmpDoParseKey+0×2adb
06 ffff8900`ea8fb3d0 fffff803`4a8bdd61 nt!CmpParseKey+0×302
07 ffff8900`ea8fb570 fffff803`4a8d3a2d nt!ObpLookupObjectName+0xb71
08 ffff8900`ea8fb740 fffff803`4a8d370d nt!ObOpenObjectByNameEx+0×1dd
09 ffff8900`ea8fb880 fffff803`4a8cfc5f nt!CmOpenKey+0×29d
0a ffff8900`ea8fba40 fffff803`4a576683 nt!NtOpenKeyEx+0xf
0b ffff8900`ea8fba80 00007ffc`ac3b82b4 nt!KiSystemServiceCopyEnd+0×13
0c 00000001`2c0feb28 00000000`00000000 0×00007ffc`ac3b82b4

0: kd> !stacks 2 nt!Cmp

[...]

[ffffaa0a35ccb2c0 winlogon.exe]
13dc.002668 ffffaa0a37e02080 fff4e8f4 RUNNING nt!KeBugCheckEx
nt!KiBugCheckDispatch+0×69
nt!KiFastFailDispatch+0xd0
nt!KiRaiseSecurityCheckFailure+0×3c6

nt!CmpFreeKeyControlBlock+0×106a73
nt!CmpDoParseKey+0×2adb
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bda780 ProcessA.exe]
1794.0017e4 ffffaa0a35c06080 fff4e8f4 RUNNING nt!CmpLockKcbStackShared+0×2
nt!CmpWalkOneLevel+0×93
nt!CmpDoParseKey+0×1c36
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bb8780 svchost.exe]
1754.001fec ffffaa0a3636a7c0 fff4e8f4 READY nt!KxDispatchInterrupt+0x122
nt!KiDpcInterrupt+0x3a6
nt!KeAbPreAcquire+0xd7
nt!ExfAcquirePushLockExclusiveEx+0x10b
nt!CmpDoParseKey+0×294e
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf

nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

We call this pattern Stack Trace Race.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 134b)

Saturday, November 25th, 2017

This is a variant of Data Correlation (function parameters) analysis pattern where we look at correlations across memory structures. Simple arithmetical ratios may link such structures and correlate corresponding behavioral processes. Here we look at a recent instance of calc.exe consuming a lot of CPU. Upon the discovery of that process we were curious and saved its full process memory dump via Task Manager. In the dump we discovered 4 Spiking Threads:

0:000> !runaway f
User Mode Time
Thread       Time
13:1b68      0 days 1:51:39.906
10:23a8      0 days 1:51:37.796

11:1b98      0 days 0:00:09.890
14:88c       0 days 0:00:09.828

1:2eb4      0 days 0:00:00.390
18:2a44      0 days 0:00:00.015
19:28f0      0 days 0:00:00.000
17:22c0      0 days 0:00:00.000
16:232c      0 days 0:00:00.000
15:2008      0 days 0:00:00.000
12:2880      0 days 0:00:00.000
9:2f38      0 days 0:00:00.000
8:1a98      0 days 0:00:00.000
7:1dcc      0 days 0:00:00.000
6:c58       0 days 0:00:00.000
5:1550      0 days 0:00:00.000
4:2938      0 days 0:00:00.000
3:2b64      0 days 0:00:00.000
2:2f90      0 days 0:00:00.000
0:dc4       0 days 0:00:00.000
[…]

We see that 10/13 approx. equals 1 and #11/#14 too, or #10/#11 approx. equals #13/#14 in user mode CPU consumption. If we look at kernel times we see the same ratios:

[...]
Kernel Mode Time
Thread Time
10:23a8 0 days 0:10:36.718
13:1b68 0 days 0:10:32.968

14:88c 0 days 0:00:23.859
11:1b98 0 days 0:00:23.812

1:2eb4 0 days 0:00:00.218
2:2f90 0 days 0:00:00.015
0:dc4 0 days 0:00:00.015
19:28f0 0 days 0:00:00.000
18:2a44 0 days 0:00:00.000
17:22c0 0 days 0:00:00.000
16:232c 0 days 0:00:00.000
15:2008 0 days 0:00:00.000
12:2880 0 days 0:00:00.000
9:2f38 0 days 0:00:00.000
8:1a98 0 days 0:00:00.000
7:1dcc 0 days 0:00:00.000
6:c58 0 days 0:00:00.000
5:1550 0 days 0:00:00.000
4:2938 0 days 0:00:00.000
3:2b64 0 days 0:00:00.000
[…]

Elapsed times are also correlated and we see that correlated threads were created in pairs {#10, #11} and {#13, #14}:

[...]
Elapsed Time
Thread Time
0:dc4 0 days 18:20:55.778
1:2eb4 0 days 18:20:55.731
2:2f90 0 days 18:20:55.725
3:2b64 0 days 18:20:55.721
4:2938 0 days 18:20:55.715
5:1550 0 days 18:20:55.582
6:c58 0 days 18:20:55.522
7:1dcc 0 days 18:20:55.522
8:1a98 0 days 18:20:55.522
9:2f38 0 days 18:20:55.522
10:23a8 0 days 16:12:52.330
11:1b98 0 days 16:12:52.329
12:2880 0 days 16:12:52.195
13:1b68 0 days 16:11:44.822
14:88c 0 days 16:11:44.821
15:2008 0 days 16:11:44.693
16:232c 0 days 2:09:35.021
17:22c0 0 days 2:05:13.038
18:2a44 0 days 0:23:38.000
19:28f0 0 days 0:00:24.261

This suggests that the threads are related. We call such analysis pattern variant Data Correlation (CPU times). It may also help in finding weak Coupled Processes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -