If we take Combed Trace for Thread of Activity or some Adjoint Thread of Activity, strip other message content, and then trace all non-empty values we get Trace Contour:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Traces and logs from diverse software systems doing different things may have similar Trace Shape despite completely different message content, especially for specific Threads of Activity or Adjoint Threads of Activity:
This may be apparent when we compare Trace Shape of Quotient Trace.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
If we have an attribute we can sort messages based on that attribute values and get Sorted Trace. If that attribute is TID or ATID we get the sequence of Threads of Activity or Adjoint Threads of Activity:
If we sort by message types or Message Invariants or some message data we get a sequence of Fibers of Activity.
The diagram above also shows on the right Quotient Trace by message type equivalence after additional sorting inside each Adjoint Thread of Activity.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Causal History messages (black circles) pass through Activity Regions which can be marked as hollow circles:
We call this analysis pattern Trace D’Enfant by analogy with dessin d’enfant in mathematics, a bipartite graph embedded in an oriented surface, so in theory Traces D’Enfants can be studied algebraically.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The relations between Causal History messages (0-chains) can be abstracted as Causal Chains (1-chains). Two relations can be linked if an endpoint of one is also a beginning point of another:
The relations of 1-chains can be abstracted as 2-chains and so on (n-chains):
We took the idea of relation spaces and chains from already quoted “Discreet Causal Theory” book. Causal chain terminology is also used in philosophy.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
When looking at Causal History we can choose Causal Messages (not necesseraly the top ones):
Causal Messages may not overlap with the trace Defect Group which may not have any causal relevance being only correlation messages.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Trace Paths and Back Traces form Causal History of the log where arrows point in the direction of possible causation:
Here we borrow the notion of causal sets from physics and corresponding mathematics. The left diagramming idea was taken from Discrete Causal Theory book and Hasse diagrams (which is inverted in our picture). Also, such graphs are internal to software narratives compared to the more general external space we proposed earlier.
We omit Time arrow as it is possible to consider general traces and logs with their causality markers.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Some trace acquisition methods and analysis workflows may require Trace Summaries having some timing and other statistical information to play the role of Indexical Trace when combined together:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Trace analysis gestures that result in CoTrace also produce Trace Path between messages of interest:
Such Trace Paths can also be useful for Trace Homotopy analysis. They also provide the basis for Explanation Traces.
Note that Trace Path is also a reverse for Back Trace analysis pattern. Both are usually selected from Working Set.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We call the association of external global variables, for example, the number of threads in the system and context switches, Trace Flux analysis pattern. Such variables can be discreet or continuous. Here we adopt the definition of flux as “a global physical variable associated with a surface and a time instant” from Enzo Tonti’s book “The Mathematical Structure of Classical and Relativistic Physics”.
This is different than the internal functional association, Thread Field.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
One of the earliest memory analysis patterns, Lateral Damage now has a specialization for CPU mode. Due to some reasons, we may get a dump with a different default CPU mode, for example, WOW64 or even V86 segmented memory addressing mode. In such a case, most commands will not work or give incorrect output.
In one such a dump we see 32-bit bugcheck parameters in !analyze -v command output:
CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 00000003, Process
Arg2: 054d0450, Terminating object
Arg3: 054d0730, Process image file name
Arg4: 02b90db0, Explanatory message (ascii)
But the dump itself from x64 Windows:
Kernel base = 0xfffff800`0280d000 PsLoadedModuleList = 0xfffff800`02a52e90
We notice WOW64 prompt:
16.1: kd:x86> k
# ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 00000000 00000000 0x0
We switch to x64 mode:
16.1: kd:x86> .effmach AMD64
Effective machine: x64 (AMD64)
Now we get correct bugcheck parameters:
16.1: kd> !analyze -v
[...]
Arguments:
Arg1: 0000000000000003, Process
Arg2: fffffa80054d0450, Terminating object
Arg3: fffffa80054d0730, Process image file name
Arg4: fffff80002b90db0, Explanatory message (ascii)
[…]
However, the stack trace is not available, all registers are zeroed, and stack region memory is not accessible:
16.1: kd> k
# Child-SP RetAddr Call Site
00 00000000`00000000 00000000`00000000 0x0
16.1: kd> !thread
THREAD fffffa800b2c6b60 Cid 0998.1a58 Teb: 000007ffffeee000 Win32Thread: 0000000000000000 RUNNING on processor 1
Not impersonating
DeviceMap fffff8a000008aa0
Owning Process fffffa800425b820 Image: App.exe
Attached Process N/A Image: N/A
Wait Start TickCount 28317542 Ticks: 0
Context Switch Count 2 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x000007feee8080ec
Stack Init fffff880066f3c70 Current fffff880066f3440
Base fffff880066f4000 Limit fffff880066ee000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
00000000`00000000 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0×0
16.1: kd> r
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=0000000000000000 rsp=0000000000000000 rbp=0000000000000000
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up di pl nz na pe nc
cs=0000 ss=0000 ds=0000 es=0000 fs=0000 gs=0000 efl=00000000
00000000`00000000 ?? ???
16.1: kd> dp fffff880066f3440
0000:3440 ????????`???????? ????????`????????
0000:3450 ????????`???????? ????????`????????
0000:3460 ????????`???????? ????????`????????
0000:3470 ????????`???????? ????????`????????
0000:3480 ????????`???????? ????????`????????
0000:3490 ????????`???????? ????????`????????
0000:34a0 ????????`???????? ????????`????????
0000:34b0 ????????`???????? ????????`????????
We notice segmented memory addressing and apply .segmentation command that is still available (the hint is taken from here):
16.1: kd> .segmentation /V /X /a
In x86 v86 code: no
In x86 16-bit code: no
In amd64 64-bit code: yes
Although stack trace is not available we see the normal prompt and can get look at stack region Execution Residue and get Rough Stack Trace:
1: kd> k
# Child-SP RetAddr Call Site
00 00000000`00000000 00000000`00000000 0x0
1: kd> dpS fffff880066ee000 fffff880066f4000
fffff800`02b7e79e nt!PspGetSetContextInternal+0×2c6
fffff800`02b7e8e5 nt!PspGetSetContextInternal+0×40d
fffff800`0289e11e nt!MiResolveDemandZeroFault+0×3be
fffff800`02e04245 hal!HalpPCIPerformConfigAccess+0×55
fffff800`02e04b32 hal!HalpPciAccessMmConfigSpace+0×196
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`02e04000 hal!HalpPCIConfig+0×70
fffff880`051f3dbb dump_diskdump!ScsiPortNotification+0×107
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`04c086c2 dump_LSI_SAS!CheckInqFlagReplies+0×42e [e:\win7\drivers\oem\src\storage\lsi_sas\ca_util.c @ 6455]
fffff800`028cdf7e nt!iswctype_l+0xce
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`028cde19 nt!output_l+0×6e1
fffff880`051e9110 crashdmp!StrBeginningDump
fffff880`051f3dbb dump_diskdump!ScsiPortNotification+0×107
fffff800`028cde19 nt!output_l+0×6e1
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`04c12090 dump_LSI_SAS!LSImpiMSIIsr+0xf4 [e:\win7\drivers\oem\src\storage\lsi_sas\ca_int.c @ 208]
fffff880`04c0386f dump_LSI_SAS!BuildScatterGather+0xcf [e:\win7\drivers\oem\src\storage\lsi_sas\ca_init.c @ 2468]
fffff880`051f224f dump_diskdump!WorkHorseDpc+0×18f
fffff800`02e072ec hal!HalpTscStallExecutionProcessor+0xe8
fffff880`051f2401 dump_diskdump!AllocateScatterGatherList+0×5d
fffff880`051f257c dump_diskdump!ExecuteSrb+0×68
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e9370 crashdmp!Context+0×30
fffff800`0292810c nt!DisplayCharacter+0×5c
fffff880`051f3a9d dump_diskdump!ScsiPortInitialize+0×805
fffff880`051e9370 crashdmp!Context+0×30
fffff800`02929c33 nt!VidDisplayString+0×73
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e6440 crashdmp!IsBufferValid+0×28
fffff880`051e5f7a crashdmp!FilterCallback+0xae
fffff880`051f2a79 dump_diskdump!DiskDumpWrite+0×1a9
fffff880`051e5d76 crashdmp!CrashdmpWriteRoutine+0×4a
fffff880`051e4f48 crashdmp!WritePageSpanToDisk+0×180
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e4c95 crashdmp!WriteKernelDump+0×12d
fffff880`051e5d2c crashdmp!CrashdmpWriteRoutine
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff880`051e4ac5 crashdmp!DumpWrite+0×145
fffff880`051e9370 crashdmp!Context+0×30
fffff880`051e4187 crashdmp!CrashdmpWrite+0×57
fffff880`051e9a30 crashdmp!ContextCopy
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02974bc1 nt!IoWriteCrashDump+0×391
fffff800`02a898a0 nt!IopTriageDumpDataBlocks
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02a85b60 nt!KeBugCheckAddPagesCallbackListHead
fffff800`02936de0 nt!IoSetDumpRange
fffff800`02936d30 nt!IoFreeDumpRange
fffff800`02abe900 nt!KiProcessorBlock
fffff800`0280d000 nt!KiSelectNextThread <PERF> (nt+0×0)
fffff800`02975f26 nt!KeBugCheck2+0xac6
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`028a051e nt!SepNormalAccessCheck+0×18e
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02e0a501 hal!HalpSendFlatIpi+0×92
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`0288d744 nt!KeBugCheckEx+0×104
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02c15982 nt!PspCatchCriticalBreak+0×92
fffff800`02b90db0 nt! ?? ::NNGAKEGL::`string’
fffff800`02bc30ab nt! ?? ::NNGAKEGL::`string’+0×17ad6
fffff800`02b46698 nt!NtTerminateProcess+0xf4
fffff800`0288c8d3 nt!KiSystemServiceCopyEnd+0×13
????????`????????
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Defect Group analysis pattern addresses messages related to source code defects (PLOTs), problem descriptions, and Inter-Correlation with wrong configuration files (Small DA+TA). It differs from Message Set analysis pattern as a predicate to group them may not be easily available.
Such Defect Groups can be results of previous analyses activities. The name of the analysis pattern came from representation theory defect group of a block but at present, it is only name analogy.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
A page to reference all different kinds of analysis patterns related to pointers is necessary, so I created this post:
I update it as soon as I add more similar patterns.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Inspecting memory region type to which a pointer points to may help diagnosing some coding mistakes. We call this analysis pattern Pointer Class. Below is a case study modeled on a real-life case.
The application was crashing sporadically and the memory dump was pointing to invalid objects reused after free. We recreated similar source code pattern and got the similar crash (PointerClass.exe.8752.dmp):
0:000> kL
# Child-SP RetAddr Call Site
00 000000e9`a10fe448 00007ff9`64e08037 ntdll!NtWaitForMultipleObjects+0x14
01 000000e9`a10fe450 00007ff9`64e07f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 000000e9`a10fe750 00007ff9`653c71fb KERNELBASE!WaitForMultipleObjects+0xe
03 000000e9`a10fe790 00007ff9`653c6ca8 kernel32!WerpReportFaultInternal+0x51b
04 000000e9`a10fe8b0 00007ff9`64eb00b8 kernel32!WerpReportFault+0xac
05 000000e9`a10fe8f0 00007ff9`672a4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 000000e9`a10fea10 00007ff9`6728c656 ntdll!RtlUserThreadStart$filt$0+0xa2
07 000000e9`a10fea50 00007ff9`672a11cf ntdll!_C_specific_handler+0x96
08 000000e9`a10feac0 00007ff9`6726a209 ntdll!RtlpExecuteHandlerForException+0xf
09 000000e9`a10feaf0 00007ff9`6729fe3e ntdll!RtlDispatchException+0x219
0a 000000e9`a10ff200 00007ff7`df32103a ntdll!KiUserExceptionDispatch+0×2e
0b 000000e9`a10ff918 00007ff7`df321081 PointerClass!Data::GetData+0xa
0c 000000e9`a10ff920 00007ff7`df32121a PointerClass!Work::DoWork+0×21
0d 000000e9`a10ff960 00007ff7`df321494 PointerClass!main+0×4a
0e (Inline Function) ——–`——– PointerClass!invoke_main+0×22
0f 000000e9`a10ff9b0 00007ff9`65377bd4 PointerClass!__scrt_common_main_seh+0×10c
10 000000e9`a10ff9f0 00007ff9`6726ce51 kernel32!BaseThreadInitThunk+0×14
11 000000e9`a10ffa20 00000000`00000000 ntdll!RtlUserThreadStart+0×21
0:000> .frame b
0b 000000e9`a10ff918 00007ff7`df321081 PointerClass!Data::GetData+0xa [C:\NewWork\PointerClass\PointerClass.cpp @ 7]
0:000> dv
this = 0×00000227`eb030000
0:000> dp poi(this)
00000227`eb030000 ????????`???????? ????????`????????
00000227`eb030010 ????????`???????? ????????`????????
00000227`eb030020 ????????`???????? ????????`????????
00000227`eb030030 ????????`???????? ????????`????????
00000227`eb030040 ????????`???????? ????????`????????
00000227`eb030050 ????????`???????? ????????`????????
00000227`eb030060 ????????`???????? ????????`????????
00000227`eb030070 ????????`???????? ????????`????????
struct Data
{
void SetData(int newData) { data = newData; }
int GetData() { return data; }
private:
int data{};
};
An engineer found out that a pointer to an outside object was used and it was not updated when the object was freed:
struct Work
{
void SetData(Data* newData)
{
data = newData;
}
void DoWork()
{
if (data)
{
auto value = data->GetData();
++value;
data->SetData(value);
}
}
private:
Data* data{};
};
void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data* pData = model.GetData(); pData)
{
work.SetData(pData);
}
}
The solution was to use a double pointer but it also crashed (PointerClassFixNotWorking.exe.7452.dmp):
struct Work
{
void SetData(Data** newData)
{
data = newData;
}
void DoWork()
{
if (data && *data)
{
auto value = (*data)->GetData();
++value;
(*data)->SetData(value);
}
}
private:
Data** data{};
};
void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data* pData = model.GetData(); pData)
{
work.SetData(&pData);
}
}
0:000> .ecxr
*** WARNING: Unable to verify checksum for PointerClassFixNotWorking.exe
rax=0000019a4be10000 rbx=0000019a4bff29c0 rcx=0000019a4be10000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000019a4bff6cf0
rip=00007ff766d5103a rsp=0000007ede6ff958 rbp=0000000000000000
r8=0000007ede6ff938 r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
PointerClassFixNotWorking!Data::GetData+0xa:
00007ff7`66d5103a 8b00 mov eax,dword ptr [rax] ds:0000019a`4be10000=????????
0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 0000007e`de6ff958 00007ff7`66d51092 PointerClassFixNotWorking!Data::GetData+0xa
01 0000007e`de6ff960 00007ff7`66d5124a PointerClassFixNotWorking!Work::DoWork+0x32
02 0000007e`de6ff9a0 00007ff7`66d514d4 PointerClassFixNotWorking!main+0x4a
03 (Inline Function) --------`-------- PointerClassFixNotWorking!invoke_main+0x22
04 0000007e`de6ff9f0 00007ff9`65377bd4 PointerClassFixNotWorking!__scrt_common_main_seh+0x10c
05 0000007e`de6ffa30 00007ff9`6726ce51 kernel32!BaseThreadInitThunk+0x14
06 0000007e`de6ffa60 00000000`00000000 ntdll!RtlUserThreadStart+0x21
struct Data
{
void SetData(int newData) { data = newData; }
int GetData() { return data; }
private:
int data{};
};
It was hypothesized that the object was also freed somewhere else and the debugging continued. However, the simple inspection of this->data Pointer Class would have revealed that it is pointing to a stack location (that was reused by subsequent calls to other functions):
0:000> .frame 1
01 0000007e`de6ff960 00007ff7`66d5124a PointerClassFixNotWorking!Work::DoWork+0x32 [C:\NewWork\PointerClassFixNotWorking\PointerClassFixNotWorking.cpp @ 24]
0:000> dv /i /v
prv local 0000007e`de6ff980 value = 0n1275013568
prv local 0000007e`de6ff9a0 this = 0x0000007e`de6ff9c0
0:000> dt this
Local var @ 0x7ede6ff9a0 Type Work*
0x0000007e`de6ff9c0
+0×000 data : 0×0000007e`de6ff978 -> 0×0000019a`4be10000 Data
0:000> !address 0×0000007e`de6ff978
Usage: Stack
Base Address: 0000007e`de6fd000
End Address: 0000007e`de700000
Region Size: 00000000`00003000 ( 12.000 kB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 0000007e`de600000
Allocation Protect: 00000004 PAGE_READWRITE
More info: ~0k
So the correct fix should have been be to use an address (heap region in the original case) of a pointer stored inside an owner object (allocated on heap in the original case):
void Init(Work& work, Model& model)
{
unsigned long long dummy{};
if (Data** ppData = model.GetData(); ppData && *ppData)
{
work.SetData(ppData);
}
}
The example memory dumps, PDB files, and source code of applications can be downloaded from here.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
The collection of register values can be interpreted as Context Pointer to various memory locations:
0:000> !for_each_register -c:.if (${@#RegisterValue} > 0xFFFF) {.printf "${@#RegisterName}: %p -> %p\n", ${@#RegisterValue}, poi(${@#RegisterValue})}
rdx: 000000e4c0afe888 -> 00000000000000b4
rsp: 000000e4c0afe488 -> 00007ffecc888037
r9: 00000000ffffffff -> Memory access error at ')'
r11: 000000e4c0afdc30 -> 0000000000000090
r12: 00000000ffffffff -> Memory access error at ')'
r13: 000000e4c0afe888 -> 00000000000000b4
rip: 00007ffecf6bcbc4 -> 00841f0fc32ecdc3
edx: 00000000c0afe888 -> Memory access error at ')'
esp: 00000000c0afe488 -> Memory access error at ')'
r9d: 00000000ffffffff -> Memory access error at ')'
r11d: 00000000c0afdc30 -> Memory access error at ')'
r12d: 00000000ffffffff -> Memory access error at ')'
r13d: 00000000c0afe888 -> Memory access error at ')'
eip: 00000000cf6bcbc4 -> Memory access error at ')'
We filter pointer values to avoid a lot of registers that contain 0 values.
Individual pointers can also be Pointer Cones.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
When interpreting a value as a pointer to a memory address we are usually interested in adjacent values pointed to:
0:000> ? rdx
Evaluate expression: 982485297288 = 000000e4`c0afe888
0:000> ? poi(000000e4`c0afe888)
Evaluate expression: 180 = 00000000`000000b4
0:000> ? poi(rdx)
Evaluate expression: 180 = 00000000`000000b4
0:000> dps rdx-10 rdx+10
000000e4`c0afe878 000000e4`c0afeac0
000000e4`c0afe880 000000e4`c0971000
000000e4`c0afe888 00000000`000000b4
000000e4`c0afe890 00000000`0000008c
000000e4`c0afe898 00000000`00000088
0:000> r $t0 = 0
0:000> dps rdx-@$t0 rdx+@$t0
000000e4`c0afe888 00000000`000000b4
0:000> r $t0 = 20
0:000> dps rdx-@$t0 rdx+@$t0
000000e4`c0afe868 00000000`00000001
000000e4`c0afe870 00001a38`00000001
000000e4`c0afe878 000000e4`c0afeac0
000000e4`c0afe880 000000e4`c0971000
000000e4`c0afe888 00000000`000000b4
000000e4`c0afe890 00000000`0000008c
000000e4`c0afe898 00000000`00000088
000000e4`c0afe8a0 00000000`00000088
000000e4`c0afe8a8 00000000`0000008c
We call this analysis pattern Pointer Cone by analogy with cones in category theory and our earlier attempts to use it. The reason for this pattern appearance is that it has been used in many times in other analysis pattern descriptions. We also intend to use this pattern language building block in our next analysis pattern.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
When developers look at crash dumps they are more interested at parameters and local variables in particular stack frames of interest. However, sometimes it is useful to look at all such frames especially to gather information that may be useful for technical support or to correlate to additional traces and logs (for example, Historical Information to establish additional Basic Facts and build Vocabulary Index).
Listing the parameters can be done, for example, by using Stack Trace command variant (kP WinDbg commend, but we use kPL to exclude source code references to reduce visual clutter):
0:000> kPL
# Child-SP RetAddr Call Site
00 000000e4`c0afe488 00007ffe`cc888037 ntdll!NtWaitForMultipleObjects+0x14
01 000000e4`c0afe490 00007ffe`cc887f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 000000e4`c0afe790 00007ffe`cd8271fb KERNELBASE!WaitForMultipleObjects+0xe
03 000000e4`c0afe7d0 00007ffe`cd826ca8 kernel32!WerpReportFaultInternal+0x51b
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 0,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0aff9d0 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×68
0d 000000e4`c0aff9b0 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 0,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affa30 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×60
0e 000000e4`c0affa10 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 1,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affa90 “Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! “)+0×60
0f 000000e4`c0affa70 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 2,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affaf0 “Hello World! Hello World! Hello World! Hello World! “)+0×60
10 000000e4`c0affad0 00007ff6`c8ab1560 FrameTrace!foo(
unsigned int64 num = 3,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affb50 “Hello World! Hello World! “)+0×60
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo(
unsigned int64 num = 4,
class std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > * str = 0×000000e4`c0affbb0 “Hello World! “)+0×60
12 000000e4`c0affb90 00007ff6`c8ab2b14 FrameTrace!main(void)+0×25
13 (Inline Function) ——–`——– FrameTrace!invoke_main+0×22
14 000000e4`c0affbe0 00007ffe`cd7d7bd4 FrameTrace!__scrt_common_main_seh(void)+0×10c
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0×14
16 000000e4`c0affc50 00000000`00000000 ntdll!RtlUserThreadStart+0×21
The stack trace comes from the following modeling application:
void foo(std::size_t num, const std::wstring& str)
{
if (std::wstring concatStr{ str }; num)
{
concatStr += str;
foo(–num, concatStr);
}
else
{
::DebugBreak();
}
}
int main()
{
foo(5, L”Hello World! “);
}
To list local variable we need to use !for_each_frame WinDbg command:
0:000> !for_each_frame "dv"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
00 000000e4`c0afe488 00007ffe`cc888037 ntdll!NtWaitForMultipleObjects+0x14
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
01 000000e4`c0afe490 00007ffe`cc887f1e KERNELBASE!WaitForMultipleObjectsEx+0x107
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
02 000000e4`c0afe790 00007ffe`cd8271fb KERNELBASE!WaitForMultipleObjects+0xe
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
03 000000e4`c0afe7d0 00007ffe`cd826ca8 kernel32!WerpReportFaultInternal+0x51b
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 0
str = 0x000000e4`c0aff9d0 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0d 000000e4`c0aff9b0 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 0
str = 0x000000e4`c0affa30 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0e 000000e4`c0affa10 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 1
str = 0x000000e4`c0affa90 "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0f 000000e4`c0affa70 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! "
num = 2
str = 0x000000e4`c0affaf0 "Hello World! Hello World! Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
10 000000e4`c0affad0 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! Hello World! Hello World! "
num = 3
str = 0x000000e4`c0affb50 "Hello World! Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
concatStr = "Hello World! Hello World! "
num = 4
str = 0x000000e4`c0affbb0 "Hello World! "
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
12 000000e4`c0affb90 00007ff6`c8ab2b14 FrameTrace!main+0x25 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 20]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
13 (Inline Function) --------`-------- FrameTrace!invoke_main+0x22 [d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
14 000000e4`c0affbe0 00007ffe`cd7d7bd4 FrameTrace!__scrt_common_main_seh+0x10c [d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
has_cctor = false
main_result = <value unavailable>
tls_init_callback = <value unavailable>
is_nested = <value unavailable>
tls_dtor_callback = <value unavailable>
main_result = <value unavailable>
__scrt_current_native_startup_state = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0x14
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
16 000000e4`c0affc50 00000000`00000000 ntdll!RtlUserThreadStart+0x21
Unable to enumerate locals, Win32 error 0n87
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
15 000000e4`c0affc20 00007ffe`cf68ce51 kernel32!BaseThreadInitThunk+0x14
We can also apply “dv /i /V” command to each frame to get additional low-level frame details:
[...]
11 000000e4`c0affb30 00007ff6`c8ab15b5 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
prv local 000000e4`c0affb50 @rsp+0x0020 concatStr = "Hello World! Hello World! "
prv param 000000e4`c0affb90 @rsp+0x0060 num = 4
prv param 000000e4`c0affb98 @rsp+0x0068 str = 0x000000e4`c0affbb0 "Hello World! "
[...]
We see this as a form of back tracing Execution Residue, for example:
0:000> !for_each_frame ".frame /c @$frame; dps rsp"
[...]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
04 000000e4`c0afe8f0 00007ffe`cc9300b8 kernel32!WerpReportFault+0xac
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000003
rip=00007ffecd826ca8 rsp=000000e4c0afe8f0 rbp=0000000000000000
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0afeac0 r13=ffffffffffffffff
r14=000000e4c0afeac0 r15=0000000000001a38
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
kernel32!WerpReportFault+0xac:
00007ffe`cd826ca8 8bf8 mov edi,eax
000000e4`c0afe8f0 00000000`00000000
000000e4`c0afe8f8 00000000`00000000
000000e4`c0afe900 00000000`00000003
000000e4`c0afe908 000000e4`c0afeac0
000000e4`c0afe910 00000000`00000004
000000e4`c0afe918 00000000`00000001
000000e4`c0afe920 00000000`00000000
000000e4`c0afe928 00007ffe`cc9300b8 KERNELBASE!UnhandledExceptionFilter+0x3b8
000000e4`c0afe930 00000000`00000000
000000e4`c0afe938 000000e4`c0affc50
000000e4`c0afe940 00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe948 00000207`5d660000
000000e4`c0afe950 00000000`00000000
000000e4`c0afe958 00007ffe`cf6660b9 ntdll!RtlpFindEntry+0x4d
000000e4`c0afe960 00000004`00000006
000000e4`c0afe968 00000001`00000000
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
05 000000e4`c0afe930 00007ffe`cf6c4ab2 KERNELBASE!UnhandledExceptionFilter+0x3b8
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=00007ffecd7c0000 rdi=0000000000000000
rip=00007ffecc9300b8 rsp=000000e4c0afe930 rbp=000000e4c0affc50
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0afeac0 r13=ffffffffffffffff
r14=0000000000000001 r15=0000000000000004
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
KERNELBASE!UnhandledExceptionFilter+0x3b8:
00007ffe`cc9300b8 0f1f440000 nop dword ptr [rax+rax]
000000e4`c0afe930 00000000`00000000
000000e4`c0afe938 000000e4`c0affc50
000000e4`c0afe940 00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe948 00000207`5d660000
000000e4`c0afe950 00000000`00000000
000000e4`c0afe958 00007ffe`cf6660b9 ntdll!RtlpFindEntry+0x4d
000000e4`c0afe960 00000004`00000006
000000e4`c0afe968 00000001`00000000
000000e4`c0afe970 00000000`00000001
000000e4`c0afe978 00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe980 00000207`5d662ff0
000000e4`c0afe988 00000000`00000000
000000e4`c0afe990 000000e4`c0afeac0
000000e4`c0afe998 00007ffe`cd7c0000 kernel32!RtlVirtualUnwindStub <PERF> (kernel32+0x0)
000000e4`c0afe9a0 00000000`005a0058
000000e4`c0afe9a8 00007ffe`cca6ff70 KERNELBASE!`string'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
06 000000e4`c0afea50 00007ffe`cf6ac656 ntdll!RtlUserThreadStart$filt$0+0xa2
rax=000000000000005b rbx=00007ffecf764420 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000000
rip=00007ffecf6c4ab2 rsp=000000e4c0afea50 rbp=000000e4c0affc50
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0aff730 r13=000000e4c0affc50
r14=000000e4c0aff0c0 r15=00007ffecf620000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!RtlUserThreadStart$filt$0+0xa2:
00007ffe`cf6c4ab2 eb16 jmp ntdll!RtlUserThreadStart$filt$0+0xba (00007ffe`cf6c4aca)
000000e4`c0afea50 00000000`00000000
000000e4`c0afea58 00007ffe`cf764420 ntdll!`string'+0x9aa8
000000e4`c0afea60 00000000`00000000
000000e4`c0afea68 000000e4`c0affbe0
000000e4`c0afea70 00000000`00000000
000000e4`c0afea78 00007ffe`cf6457d8 ntdll!LdrpAppendUnicodeStringToFilenameBuffer+0x50
000000e4`c0afea80 00000000`0006ce51
000000e4`c0afea88 00007ffe`cf6ac656 ntdll!_C_specific_handler+0x96
000000e4`c0afea90 000000e4`c0afeb40
000000e4`c0afea98 00007ffe`cf642930 ntdll!LdrpFindLoadedDllByNameLockHeld+0xe4
000000e4`c0afeaa0 000000e4`c0aff088
000000e4`c0afeaa8 000000e4`c0aff110
000000e4`c0afeab0 000000e4`c0aff240
000000e4`c0afeab8 00000000`00000000
000000e4`c0afeac0 000000e4`c0aff730
000000e4`c0afeac8 000000e4`c0aff240
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
07 000000e4`c0afea90 00007ffe`cf6c11cf ntdll!_C_specific_handler+0x96
rax=000000000000005b rbx=00007ffecf764420 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=0000000000000000
rip=00007ffecf6ac656 rsp=000000e4c0afea90 rbp=000000000006ce51
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=000000e4c0aff730 r13=000000e4c0affc50
r14=000000e4c0aff0c0 r15=00007ffecf620000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!_C_specific_handler+0x96:
00007ffe`cf6ac656 85c0 test eax,eax
000000e4`c0afea90 000000e4`c0afeb40
000000e4`c0afea98 00007ffe`cf642930 ntdll!LdrpFindLoadedDllByNameLockHeld+0xe4
000000e4`c0afeaa0 000000e4`c0aff088
000000e4`c0afeaa8 000000e4`c0aff110
000000e4`c0afeab0 000000e4`c0aff240
000000e4`c0afeab8 00000000`00000000
000000e4`c0afeac0 000000e4`c0aff730
000000e4`c0afeac8 000000e4`c0aff240
000000e4`c0afead0 00000000`00000000
000000e4`c0afead8 000000e4`c0afeb70
000000e4`c0afeae0 000000e4`c0aff240
000000e4`c0afeae8 00007ffe`cf6ac5c0 ntdll!_C_specific_handler
000000e4`c0afeaf0 00000000`00000000
000000e4`c0afeaf8 00007ffe`cf6c11cf ntdll!RtlpExecuteHandlerForException+0xf
000000e4`c0afeb00 00000000`00000000
000000e4`c0afeb08 000000e4`c0aff070
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
08 000000e4`c0afeb00 00007ffe`cf68a209 ntdll!RtlpExecuteHandlerForException+0xf
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=000000e4c0aff730 rdi=0000000000000000
rip=00007ffecf6c11cf rsp=000000e4c0afeb00 rbp=000000e4c0aff070
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=00007ffecf6ac5c0 r13=000000e4c0aff240
r14=000000e4c0afeb70 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!RtlpExecuteHandlerForException+0xf:
00007ffe`cf6c11cf 90 nop
000000e4`c0afeb00 00000000`00000000
000000e4`c0afeb08 000000e4`c0aff070
000000e4`c0afeb10 000000e4`c0aff730
000000e4`c0afeb18 000000e4`c0aff730
000000e4`c0afeb20 000000e4`c0aff0c0
000000e4`c0afeb28 00007ffe`cf68a209 ntdll!RtlDispatchException+0x219
000000e4`c0afeb30 000000e4`00000001
000000e4`c0afeb38 00007ffe`cf620000 ntdll!RtlStringCchCopyW <PERF> (ntdll+0x0)
000000e4`c0afeb40 00000000`00000000
000000e4`c0afeb48 00007ffe`cf78e9f0 ntdll!__PchSym_ <PERF> (ntdll+0x16e9f0)
000000e4`c0afeb50 000000e4`c0afeb70
000000e4`c0afeb58 000000e4`c0aff090
000000e4`c0afeb60 000000e4`c0aff080
000000e4`c0afeb68 00000000`00000000
000000e4`c0afeb70 000000e4`00000000
000000e4`c0afeb78 00007ffe`cc8300f0 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0xf0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
09 000000e4`c0afeb30 00007ffe`cf6bfe3e ntdll!RtlDispatchException+0x219
rax=000000000000005b rbx=0000000000000000 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=000000e4c0aff730 rdi=0000000000000000
rip=00007ffecf68a209 rsp=000000e4c0afeb30 rbp=000000e4c0aff070
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=00007ffecf6ac5c0 r13=000000e4c0aff240
r14=000000e4c0afeb70 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!RtlDispatchException+0x219:
00007ffe`cf68a209 8bd0 mov edx,eax
000000e4`c0afeb30 000000e4`00000001
000000e4`c0afeb38 00007ffe`cf620000 ntdll!RtlStringCchCopyW <PERF> (ntdll+0x0)
000000e4`c0afeb40 00000000`00000000
000000e4`c0afeb48 00007ffe`cf78e9f0 ntdll!__PchSym_ <PERF> (ntdll+0x16e9f0)
000000e4`c0afeb50 000000e4`c0afeb70
000000e4`c0afeb58 000000e4`c0aff090
000000e4`c0afeb60 000000e4`c0aff080
000000e4`c0afeb68 00000000`00000000
000000e4`c0afeb70 000000e4`00000000
000000e4`c0afeb78 00007ffe`cc8300f0 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0xf0)
000000e4`c0afeb80 00000001`00000000
000000e4`c0afeb88 00000012`00000018
000000e4`c0afeb90 00000000`00000000
000000e4`c0afeb98 00360030`00300030
000000e4`c0afeba0 00001f80`0010000f
000000e4`c0afeba8 00000000`00000033
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
0a 000000e4`c0aff240 00007ffe`cc8f0aa2 ntdll!KiUserExceptionDispatch+0x2e
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ffecf6bfe3e rsp=000000e4c0aff240 rbp=0000000000000000
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!KiUserExceptionDispatch+0x2e:
00007ffe`cf6bfe3e 84c0 test al,al
000000e4`c0aff240 00007ff6`c8ac32f0 FrameTrace!`string'
000000e4`c0aff248 00000000`000a0008
000000e4`c0aff250 00000207`5d662a10
000000e4`c0aff258 00007ff6`00200000
000000e4`c0aff260 000000e4`c0aff2f0
000000e4`c0aff268 000000e4`c0aff2f0
000000e4`c0aff270 00001f80`0010005f
000000e4`c0aff278 0053002b`002b0033
000000e4`c0aff280 00000246`002b002b
000000e4`c0aff288 00000000`00000000
000000e4`c0aff290 00000000`00000000
000000e4`c0aff298 00000000`00000000
000000e4`c0aff2a0 00000000`00000000
000000e4`c0aff2a8 00000000`00000000
000000e4`c0aff2b0 00000000`00000000
000000e4`c0aff2b8 000000e4`c0aff970
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
0b 000000e4`c0aff948 00007ff6`c8ab1568 KERNELBASE!wil::details::DebugBreak+0x2
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ffecc8f0aa2 rsp=000000e4c0aff948 rbp=0000000000000000
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
KERNELBASE!wil::details::DebugBreak+0x2:
00007ffe`cc8f0aa2 cc int 3
000000e4`c0aff948 00007ff6`c8ab1568 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
000000e4`c0aff950 000000e4`c0aff970
000000e4`c0aff958 000000e4`c0aff9d0
000000e4`c0aff960 00000000`000000d0
000000e4`c0aff968 00000207`5d66a990
000000e4`c0aff970 00000207`5d66b070
000000e4`c0aff978 00007ff6`c8ab15ed FrameTrace!std::basic_string<wchar_t,std::char_traits<wchar_t>, std::allocator<wchar_t> >::operator+=+0x1d [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.26.28801\include\xstring @ 2821]
000000e4`c0aff980 00000000`000001a0
000000e4`c0aff988 00000000`000001a7
000000e4`c0aff990 0000e8d4`e5494150
000000e4`c0aff998 0000e8d4`e5494150
000000e4`c0aff9a0 000000e4`c0affa30
000000e4`c0aff9a8 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
000000e4`c0aff9b0 00000000`00000000
000000e4`c0aff9b8 000000e4`c0aff9d0
000000e4`c0aff9c0 00000000`00000068
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
0c 000000e4`c0aff950 00007ff6`c8ab1560 FrameTrace!foo+0x68 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 14]
rax=000000000000005b rbx=000002075d662a10 rcx=0000000000000003
rdx=000000e4c0afe888 rsi=0000000000000000 rdi=000002075d666e40
rip=00007ff6c8ab1568 rsp=000000e4c0aff950 rbp=0000000000000000
r8=0000000000000000 r9=00000000ffffffff r10=0000000000000000
r11=000000e4c0afdc30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
FrameTrace!foo+0x68:
00007ff6`c8ab1568 90 nop
000000e4`c0aff950 000000e4`c0aff970
000000e4`c0aff958 000000e4`c0aff9d0
000000e4`c0aff960 00000000`000000d0
000000e4`c0aff968 00000207`5d66a990
000000e4`c0aff970 00000207`5d66b070
000000e4`c0aff978 00007ff6`c8ab15ed FrameTrace!std::basic_string<wchar_t,std::char_traits<wchar_t>, std::allocator<wchar_t> >::operator+=+0x1d [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.26.28801\include\xstring @ 2821]
000000e4`c0aff980 00000000`000001a0
000000e4`c0aff988 00000000`000001a7
000000e4`c0aff990 0000e8d4`e5494150
000000e4`c0aff998 0000e8d4`e5494150
000000e4`c0aff9a0 000000e4`c0affa30
000000e4`c0aff9a8 00007ff6`c8ab1560 FrameTrace!foo+0x60 [C:\NewWork\FrameTrace\FrameTrace.cpp @ 11]
000000e4`c0aff9b0 00000000`00000000
000000e4`c0aff9b8 000000e4`c0aff9d0
000000e4`c0aff9c0 00000000`00000068
000000e4`c0aff9c8 00000207`5d66a8a0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[...]
We need to reset the current context after the command above since the last frame becomes the current:
0:000> kc
*** Stack trace for last set context - .thread/.cxr resets it
# Call Site
15 ntdll!RtlUserThreadStart
0:000> .cxr
Resetting default scope
0:000> kc
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 KERNELBASE!WaitForMultipleObjects
03 kernel32!WerpReportFaultInternal
04 kernel32!WerpReportFault
05 KERNELBASE!UnhandledExceptionFilter
06 ntdll!RtlUserThreadStart$filt$0
07 ntdll!_C_specific_handler
08 ntdll!RtlpExecuteHandlerForException
09 ntdll!RtlDispatchException
0a ntdll!KiUserExceptionDispatch
0b KERNELBASE!wil::details::DebugBreak
0c FrameTrace!foo
0d FrameTrace!foo
0e FrameTrace!foo
0f FrameTrace!foo
10 FrameTrace!foo
11 FrameTrace!foo
12 FrameTrace!main
13 FrameTrace!invoke_main
14 FrameTrace!__scrt_common_main_seh
15 kernel32!BaseThreadInitThunk
16 ntdll!RtlUserThreadStart
We call this analysis pattern Frame Trace.
The example memory dump, the application PDB file, and source code can be downloaded from here.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
When modeling Invalid Pointer (Objects) analysis pattern, we noticed that if we use MEM_RELEASE instead of MEM_DECOMMIT in VirtualFree API call, we see page memory contents despite an access violation Stored Exception pointing to that page. Moreover, the page contents were not corresponding to what should have been expected from source code. We had to do live kernel debugging in order to verify what was going on.
We launched InvalidPointerObject.exe that displayed the committed address allocated via VirtualAlloc API call:
The we broke into the system, found our process and inspected that address:
Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Opened \\.\pipe\com2
Waiting to reconnect...
Connected to Windows 10 18362 x64 target at (Fri May 1 22:46:00.982 2020 (UTC + 1:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 18362 MP (1 procs) Free x64
Built by: 18362.1.amd64fre.19h1_release.190318-1202
Machine Name:
Kernel base = 0xfffff800`74800000 PsLoadedModuleList = 0xfffff800`74c48190
System Uptime: 0 days 0:00:00.000
KDTARGET: Refreshing KD connection
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
* *
* You are seeing this message because you pressed either *
* CTRL+C (if you run console kernel debugger) or, *
* CTRL+BREAK (if you run GUI kernel debugger), *
* on your debugger machine's keyboard. *
* *
* THIS IS NOT A BUG OR A SYSTEM CRASH *
* *
* If you did not intend to break into the debugger, press the "g" key, then *
* press the "Enter" key now. This message might immediately reappear. If it *
* does, press "g" and "Enter" again. *
* *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff800`749c93a0 cc int 3
1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS ffffe00314e89300
SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 001ad002 ObjectTable: ffffc90314806d40 HandleCount: 3136.
Image: System
[...]
PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 33.
Image: InvalidPointerObject.exe
[...]
1: kd> !process ffffe00318d60080 3f
PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 33.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a78d1c0 Vads 22 Clone 0 Private 94. Modified 0. Locked 2.
DeviceMap ffffc903193e9bf0
Token ffffc9031c692060
ElapsedTime 00:01:13.571
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 20344
QuotaPoolUsage[NonPagedPool] 3256
Working Set Sizes (now,min,max) (497, 50, 345) (1988KB, 200KB, 1380KB)
PeakWorkingSetSize 465
VirtualSize 4139 Mb
PeakVirtualSize 4139 Mb
PageFaultCount 499
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 107
Job ffffe00317be8060
[...]
THREAD ffffe003198ba0c0 Cid 1a90.18bc Teb: 000000161ab74000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Alertable
ffffe0031a7b0238 NotificationEvent
IRP List:
ffffe00318a49510: (0006,0238) Flags: 00060900 Mdl: ffffe00319319470
Not impersonating
DeviceMap ffffc903193e9bf0
Owning Process ffffe00318d60080 Image: InvalidPointerObject.exe
Attached Process N/A Image: N/A
Wait Start TickCount 6673 Ticks: 4692 (0:00:01:13.312)
Context Switch Count 118 IdealProcessor: 1
UserTime 00:00:00.000
KernelTime 00:00:00.015
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
Win32 Start Address InvalidPointerObject!wmainCRTStartup (0x00007ff66357e044)
Stack Init ffff848c00a22c90 Current ffff848c00a22560
Base ffff848c00a23000 Limit ffff848c00a1d000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffff848c`00a225a0 fffff800`7483c7bd nt!KiSwapContext+0x76
ffff848c`00a226e0 fffff800`7483b644 nt!KiSwapThread+0xbfd
ffff848c`00a22780 fffff800`7483ade5 nt!KiCommitThreadWait+0x144
ffff848c`00a22820 fffff800`74de982a nt!KeWaitForSingleObject+0x255
ffff848c`00a22900 fffff800`74de595f nt!IopSynchronousServiceTail+0x24a
ffff848c`00a229a0 fffff800`749d2e15 nt!NtReadFile+0x59f
ffff848c`00a22a90 00007ffb`0ed3c184 nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff848c`00a22b00)
00000016`1a96f338 00007ffb`0c405227 ntdll!NtReadFile+0x14
00000016`1a96f340 00007ff6`6359b3b9 KERNELBASE!ReadFile+0x77
00000016`1a96f3c0 00000000`00000001 InvalidPointerObject!_read_nolock+0x2f5 [minkernel\crts\ucrt\src\appcrt\lowio\read.cpp @ 566]
00000016`1a96f3c8 00000000`00000000 0x1
1: kd> .thread /r /p ffffe003198ba0c0
Implicit thread is now ffffe003`198ba0c0
Implicit process is now ffffe003`18d60080
.cache forcedecodeuser done
Loading User Symbols
....
1: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 ffff848c`00a225a0 fffff800`7483c7bd nt!KiSwapContext+0x76
01 ffff848c`00a226e0 fffff800`7483b644 nt!KiSwapThread+0xbfd
02 ffff848c`00a22780 fffff800`7483ade5 nt!KiCommitThreadWait+0x144
03 ffff848c`00a22820 fffff800`74de982a nt!KeWaitForSingleObject+0x255
04 ffff848c`00a22900 fffff800`74de595f nt!IopSynchronousServiceTail+0x24a
05 ffff848c`00a229a0 fffff800`749d2e15 nt!NtReadFile+0x59f
06 ffff848c`00a22a90 00007ffb`0ed3c184 nt!KiSystemServiceCopyEnd+0x25
07 00000016`1a96f338 00007ffb`0c405227 ntdll!NtReadFile+0x14
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
08 00000016`1a96f340 00007ff6`6359b3b9 KERNELBASE!ReadFile+0x77
09 00000016`1a96f3c0 00000000`00000001 InvalidPointerObject!_read_nolock+0x2f5
0a 00000016`1a96f3c8 00000000`00000000 0x1
1: kd> !vad 146e3a70000 1
VAD @ ffffe0031a78eb10
Start VPN 146e3a70 End VPN 146e3a70 Control Area 0000000000000000
FirstProtoPte 0000000000000000 LastPte 0000000000000000 Commit Charge 1 (0n1)
Secured.Flink 0 Blink 0 Banked/Extend 0
File Offset 0
ViewUnmap MemCommit PrivateMemory READWRITE
1: kd> dc 146e3a70000
00000146`e3a70000 00000001 00000000 00000000 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….
We see the page memory contents show the correct counter value (1):
struct Resource
{
void DoSomething()
{
++m_usageCounter;
} std::size_t m_usageCounter{};
}; We resume system execution and hit a key. The program crashes in the second DoSomething call after releasing memory that contained Resource object:
::VirtualFree(pMem, 0, MEM_RELEASE); pResource->DoSomething();We wait until WER dialog appears (we had to add DWORD DontShowUI (0) to \HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting):
We then break in the system again and inspect the same address:
1: kd> g
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
* *
* You are seeing this message because you pressed either *
* CTRL+C (if you run console kernel debugger) or, *
* CTRL+BREAK (if you run GUI kernel debugger), *
* on your debugger machine's keyboard. *
* *
* THIS IS NOT A BUG OR A SYSTEM CRASH *
* *
* If you did not intend to break into the debugger, press the "g" key, then *
* press the "Enter" key now. This message might immediately reappear. If it *
* does, press "g" and "Enter" again. *
* *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff800`749c93a0 cc int 3
0: kd> .thread /r /p ffffe003198ba0c0
Implicit thread is now ffffe003`198ba0c0
Implicit process is now ffffe003`18d60080
.cache forcedecodeuser done
Loading User Symbols
....
0: kd> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 ffff848c`00a21f70 fffff800`7483c7bd nt!KiSwapContext+0x76
01 ffff848c`00a220b0 fffff800`7483b644 nt!KiSwapThread+0xbfd
02 ffff848c`00a22150 fffff800`748884e7 nt!KiCommitThreadWait+0x144
03 ffff848c`00a221f0 fffff800`74e1ffe9 nt!KeWaitForMultipleObjects+0x287
04 ffff848c`00a22300 fffff800`74e1fd05 nt!ObWaitForMultipleObjects+0x2a9
05 ffff848c`00a22800 fffff800`749d2e15 nt!NtWaitForMultipleObjects+0x105
06 ffff848c`00a22a90 00007ffb`0ed3cc14 nt!KiSystemServiceCopyEnd+0x25
07 00000016`1a96e208 00007ffb`0c438027 ntdll!NtWaitForMultipleObjects+0x14
08 00000016`1a96e210 00007ffb`0c437f0e KERNELBASE!WaitForMultipleObjectsEx+0x107
09 00000016`1a96e510 00007ffb`0e0071fb KERNELBASE!WaitForMultipleObjects+0xe
0a 00000016`1a96e550 00007ffb`0e006ca8 KERNEL32!WerpReportFaultInternal+0x51b
0b 00000016`1a96e670 00007ffb`0c4df868 KERNEL32!WerpReportFault+0xac
0c 00000016`1a96e6b0 00007ffb`0ed44b32 KERNELBASE!UnhandledExceptionFilter+0x3b8
0d 00000016`1a96e7d0 00007ffb`0ed2c6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
0e 00000016`1a96e810 00007ffb`0ed4121f ntdll!_C_specific_handler+0x96
0f 00000016`1a96e880 00007ffb`0ed0a289 ntdll!RtlpExecuteHandlerForException+0xf
10 00000016`1a96e8b0 00007ffb`0ed3fe8e ntdll!RtlDispatchException+0x219
11 00000016`1a96efc0 00007ff6`6357378a ntdll!KiUserExceptionDispatch+0×2e
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
12 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa
13 00000016`1a96f6e0 00007ff6`6357dfd4 InvalidPointerObject!wmain+0xd5
14 (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
15 00000016`1a96f730 00007ffb`0dfb7bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
16 00000016`1a96f770 00007ffb`0ed0ced1 KERNEL32!BaseThreadInitThunk+0×14
17 00000016`1a96f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×21
0: kd> .frame 0n18;dv /t /v
12 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa [C:\NewWork\InvalidPointerObject\InvalidPointerObject.cpp @ 10]
00000016`1a96f6e0 struct Resource * this = 0×00000146`e3a70000
0: kd> !vad 146e3a70000 1
VAD @ ffffe0031ab91080
Start VPN 146e3a70 End VPN 146e3a70 Control Area ffffe0031a66f780
FirstProtoPte ffffc9031b8fdf50 LastPte ffffc9031b8fdf50 Commit Charge 0 (0n0)
Secured.Flink 0 Blink 0 Banked/Extend 0
File Offset 0
ViewShare READWRITE
ControlArea @ ffffe0031a66f780
Segment ffffc9031c7d55d0 Flink ffffe0031ab91f40 Blink ffffe0031ab910e0
Section Ref 1 Pfn Ref 0 Mapped Views 3
User Ref 4 WaitForDel 0 Flush Count 1
File Object 0000000000000000 ModWriteCount 0 System Views 0
WritableRefs 0 PartitionId 0
Flags (2000) Commit
Pagefile-backed section
Segment @ ffffc9031c7d55d0
ControlArea ffffe0031a66f780 ExtendInfo 0000000000000000
Total Ptes 1
Segment Size 1000 Committed 1
CreatingProcessId 1a90 FirstMappedVa 146e3a70000
ProtoPtes ffffc9031b8fdf50
Flags (80000) ProtectionMask
0: kd> !ca ffffe0031a66f780 4
ControlArea @ ffffe0031a66f780
Segment ffffc9031c7d55d0 Flink ffffe0031ab91f40 Blink ffffe0031ab910e0
Section Ref 1 Pfn Ref 0 Mapped Views 3
User Ref 4 WaitForDel 0 Flush Count 1
File Object 0000000000000000 ModWriteCount 0 System Views 0
WritableRefs 0 PartitionId 0
Flags (2000) Commit
Pagefile-backed section
3 mapped view(s):
ffffe0031ab91f40 - VAD ffffe0031ab91ee0, process ffffe0031a8d3080 WerFault.exe
ffffe0031ab93ca0 - VAD ffffe0031ab93c40, process ffffe0031acd3080 InvalidPointer
ffffe0031ab910e0 - VAD ffffe0031ab91080, process ffffe00318d60080 InvalidPointer
0: kd> dc 146e3a70000
00000146`e3a70000 000000f0 00001a90 000018bc 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….
We see the page contents changed (it now contains PID and TID) and also its pagefile-backed section lists 3 mapped views including 2 new processes, one is Zombie Process duplicate of the original InvalidPointerObject.exe process and another is WerFault.exe:
0: kd> !process ffffe0031acd3080
PROCESS ffffe0031acd3080
SessionId: 1 Cid: 06cc Peb: 161ab73000 ParentCid: 1a90
DirBase: 9d002002 ObjectTable: ffffc9031c03e5c0 HandleCount: 0.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a789d90 Vads 21 Clone ffffe00318770810 Private 41. Modified 0. Locked 0.
DeviceMap 0000000000000000
Token ffffc9031d32a770
ElapsedTime 00:00:26.741
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 17776
QuotaPoolUsage[NonPagedPool] 6024
Working Set Sizes (now,min,max) (28, 50, 345) (112KB, 200KB, 1380KB)
PeakWorkingSetSize 10
VirtualSize 4138 Mb
PeakVirtualSize 4138 Mb
PageFaultCount 28
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 51
No active threads
0: kd> !process 1a90
Searching for Process with Cid == 1a90
PROCESS ffffe00318d60080
SessionId: 1 Cid: 1a90 Peb: 161ab73000 ParentCid: 1474
DirBase: af7ee002 ObjectTable: ffffc9031c02a0c0 HandleCount: 39.
Image: InvalidPointerObject.exe
VadRoot ffffe0031a78d1c0 Vads 22 Clone ffffe00318770590 Private 43. Modified 18. Locked 0.
DeviceMap ffffc903193e9bf0
Token ffffc9031c692060
ElapsedTime 00:38:20.130
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 20352
QuotaPoolUsage[NonPagedPool] 6328
Working Set Sizes (now,min,max) (547, 50, 345) (2188KB, 200KB, 1380KB)
PeakWorkingSetSize 515
VirtualSize 4139 Mb
PeakVirtualSize 4139 Mb
PageFaultCount 552
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 106
Job ffffe00317be8060
[...]
We resume system execution and collect the process crash dump. When we look at the crash address we see the same unexpected False Memory contents:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1a90.18bc): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ffb`0ed3cc14 c3 ret
0:000> kL
# Child-SP RetAddr Call Site
00 00000016`1a96e208 00007ffb`0c438027 ntdll!NtWaitForMultipleObjects+0x14
01 00000016`1a96e210 00007ffb`0c437f0e KERNELBASE!WaitForMultipleObjectsEx+0x107
02 00000016`1a96e510 00007ffb`0e0071fb KERNELBASE!WaitForMultipleObjects+0xe
03 00000016`1a96e550 00007ffb`0e006ca8 kernel32!WerpReportFaultInternal+0x51b
04 00000016`1a96e670 00007ffb`0c4df868 kernel32!WerpReportFault+0xac
05 00000016`1a96e6b0 00007ffb`0ed44b32 KERNELBASE!UnhandledExceptionFilter+0x3b8
06 00000016`1a96e7d0 00007ffb`0ed2c6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
07 00000016`1a96e810 00007ffb`0ed4121f ntdll!_C_specific_handler+0x96
08 00000016`1a96e880 00007ffb`0ed0a289 ntdll!RtlpExecuteHandlerForException+0xf
09 00000016`1a96e8b0 00007ffb`0ed3fe8e ntdll!RtlDispatchException+0x219
0a 00000016`1a96efc0 00007ff6`6357378a ntdll!KiUserExceptionDispatch+0×2e
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
0b 00000016`1a96f6d8 00007ff6`63573875 InvalidPointerObject!Resource::DoSomething+0xa
0c 00000016`1a96f6e0 00007ff6`6357dfd4 InvalidPointerObject!wmain+0xd5
0d (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
0e 00000016`1a96f730 00007ffb`0dfb7bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
0f 00000016`1a96f770 00007ffb`0ed0ced1 kernel32!BaseThreadInitThunk+0×14
10 00000016`1a96f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×21
0:000> dx Debugger.Sessions[0].Processes[6800].Threads[6332].Stack.Frames[11].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[6800].Threads[6332].Stack.Frames[11].SwitchTo()
00000016`1a96f6e0 struct Resource * this = 0×00000146`e3a70000
0:000> !address 0×00000146`e3a70000
Usage: <unknown>
Base Address: 00000146`e3a70000
End Address: 00000146`e3a71000
Region Size: 00000000`00001000 ( 4.000 kB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00040000 MEM_MAPPED
Allocation Base: 00000146`e3a70000
Allocation Protect: 00000004 PAGE_READWRITE
Content source: 1 (target), length: 1000
0:000> dc 0×00000146`e3a70000
00000146`e3a70000 000000f0 00001a90 000018bc 00000000 …………….
00000146`e3a70010 00000000 00000000 00000000 00000000 …………….
00000146`e3a70020 00000000 00000000 00000000 00000000 …………….
00000146`e3a70030 00000000 00000000 00000000 00000000 …………….
00000146`e3a70040 00000000 00000000 00000000 00000000 …………….
00000146`e3a70050 00000000 00000000 00000000 00000000 …………….
00000146`e3a70060 00000000 00000000 00000000 00000000 …………….
00000146`e3a70070 00000000 00000000 00000000 00000000 …………….
0:000> ~
. 0 Id: 1a90.18bc Suspend: 0 Teb: 00000016`1ab74000 Unfrozen
0:000> dx -r1 ((InvalidPointerObject!Resource *)0x146e3a70000)
((InvalidPointerObject!Resource *)0x146e3a70000) : 0×146e3a70000 [Type: Resource *]
[+0×000] m_usageCounter : 0×1a90000000f0 [Type: unsigned __int64]
0:000> .ecxr
rax=00000146e3a70000 rbx=00000146e3aa5bf0 rcx=00000146e3a70000
rdx=0000000000000000 rsi=0000000000000000 rdi=00000146e3aa5c70
rip=00007ff66357378a rsp=000000161a96f6d8 rbp=0000000000000000
r8=000000161a96f6a8 r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
InvalidPointerObject!Resource::DoSomething+0xa:
00007ff6`6357378a 488b00 mov rax,qword ptr [rax] ds:00000146`e3a70000=00001a90000000f0
Such False Memory may complicate the analysis of process crash dumps when we want to examine memory contents prior to exception.
The example memory dump, the application PDB file, and source code can be downloaded from here.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In addition to generic Invalid Pointer pattern that maps to visible pointer dereference in C and C++ code, plain NULL Code Pointers and NULL Data Pointers that are visible Small Values, and Wild Pointers showing ASCII or Regular Data (such as UNICODE fragments), we have implicit dereference (from C++ source code perspective) crash dump analysis patterns that we call Invalid Pointer (Objects). When seeing them in a high-level debugger (could be just an exception during debugging) developers are confused since they do not see the usual pointer dereference:
struct Resource
{
void DoSomething()
{
++m_usageCounter;
} std::size_t m_usageCounter{};
}; However, the function call was ordinary (not virtual, otherwise we would have NULL Code Pointer), and the object address to access its members was passed via RCX register, but the memory of the object was invalid, hence we have an exception inside the method call when trying to access object members:
0:000> .ecxr
*** WARNING: Unable to verify checksum for InvalidPointerObject.exe
rax=0000022c837e0000 rbx=0000022c83905ca0 rcx=0000022c837e0000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000022c83907540
rip=00007ff6d65630ba rsp=00000098812ffc18 rbp=0000000000000000
r8=00000098812ffbe8 r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
InvalidPointerObject!Resource::DoSomething+0xa:
00007ff6`d65630ba 488b00 mov rax,qword ptr [rax] ds:0000022c`837e0000=????????????????
0:000> !address @rax
Usage:
Base Address: 0000022c`837e0000
End Address: 0000022c`837e1000
Region Size: 00000000`00001000 ( 4.000 kB)
State: 00002000 MEM_RESERVE
Protect:
Type: 00020000 MEM_PRIVATE
Allocation Base: 0000022c`837e0000
Allocation Protect: 00000004 PAGE_READWRITE
0:000> kL
# Child-SP RetAddr Call Site
00 00000098`812fe748 00007ffd`62278027 ntdll!NtWaitForMultipleObjects+0×14
01 00000098`812fe750 00007ffd`62277f0e KERNELBASE!WaitForMultipleObjectsEx+0×107
02 00000098`812fea50 00007ffd`63d871fb KERNELBASE!WaitForMultipleObjects+0xe
03 00000098`812fea90 00007ffd`63d86ca8 kernel32!WerpReportFaultInternal+0×51b
04 00000098`812febb0 00007ffd`6231f868 kernel32!WerpReportFault+0xac
05 00000098`812febf0 00007ffd`64ee4b32 KERNELBASE!UnhandledExceptionFilter+0×3b8
06 00000098`812fed10 00007ffd`64ecc6d6 ntdll!RtlUserThreadStart$filt$0+0xa2
07 00000098`812fed50 00007ffd`64ee121f ntdll!_C_specific_handler+0×96
08 00000098`812fedc0 00007ffd`64eaa289 ntdll!RtlpExecuteHandlerForException+0xf
09 00000098`812fedf0 00007ffd`64edfe8e ntdll!RtlDispatchException+0×219
0a 00000098`812ff500 00007ff6`d65630ba ntdll!KiUserExceptionDispatch+0×2e
0b 00000098`812ffc18 00007ff6`d656313c InvalidPointerObject!Resource::DoSomething+0xa
0c 00000098`812ffc20 00007ff6`d6568454 InvalidPointerObject!wmain+0×6c
0d (Inline Function) ——–`——– InvalidPointerObject!invoke_main+0×22
0e 00000098`812ffc70 00007ffd`63d37bd4 InvalidPointerObject!__scrt_common_main_seh+0×10c
0f 00000098`812ffcb0 00007ffd`64eaced1 kernel32!BaseThreadInitThunk+0×14
10 00000098`812ffce0 00000000`00000000 ntdll!RtlUserThreadStart+0×21
0:000> ub 00007ff6`d656313c
InvalidPointerObject!wmain+0×45:
00007ff6`d6563115 488b4c2428 mov rcx,qword ptr [rsp+28h]
00007ff6`d656311a e891ffffff call InvalidPointerObject!Resource::DoSomething (00007ff6`d65630b0)
00007ff6`d656311f 41b800400000 mov r8d,4000h
00007ff6`d6563125 33d2 xor edx,edx
00007ff6`d6563127 488b4c2420 mov rcx,qword ptr [rsp+20h]
00007ff6`d656312c ff15ce0e0200 call qword ptr [InvalidPointerObject!_imp_VirtualFree (00007ff6`d6584000)]
00007ff6`d6563132 488b4c2428 mov rcx,qword ptr [rsp+28h]
00007ff6`d6563137 e874ffffff call InvalidPointerObject!Resource::DoSomething (00007ff6`d65630b0)
0:000> u InvalidPointerObject!Resource::DoSomething
InvalidPointerObject!Resource::DoSomething:
00007ff6`d65630b0 48894c2408 mov qword ptr [rsp+8],rcx
00007ff6`d65630b5 488b442408 mov rax,qword ptr [rsp+8]
00007ff6`d65630ba 488b00 mov rax,qword ptr [rax]
00007ff6`d65630bd 48ffc0 inc rax
00007ff6`d65630c0 488b4c2408 mov rcx,qword ptr [rsp+8]
00007ff6`d65630c5 488901 mov qword ptr [rcx],rax
00007ff6`d65630c8 c3 ret
00007ff6`d65630c9 cc int 3
The example memory dump, the application PDB file, and source code can be downloaded from here.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Source code can be considered as a type of a general trace from a corresponding generative narrative plane. We call it Generative Trace since it can generate different traces of execution. If such a trace contains logging code statements, then they form Declarative Trace as a subset of messages. Generative Trace also overlaps with the corresponding Moduli Trace. We can apply many trace and log analysis patterns and even consider line number axis as pseudo-time. The following diagram illustrates Linked Messages analysis pattern in the context of Generative Trace and generated traces:
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -