Software Diagnostics Library

Trace Network is an analysis pattern in which traces and logs are treated as evidence for constructing an attributed interaction network N=(V, E), where vertices V are Motives, (Adjoint) Threads or Features of Activity, and their combinations, and directed edges E are created by an explicit correspondence rule between them, for example, request/response, causality, correlation propagation, spawn/join relation, or shared resource usage. A scope such as Time Delta or some filtering for Message Patterns may also be applied before the network construction.

Edge aggregation, weighting, and labels are part of the construction specification, so the result is not merely a drawing but a diagnostic network on which structural properties such as fan-in, fan-out, hubs, components, and derived measures such as Trace Divergence can be computed. This differs from Trace Graph, whose primary purpose is plotting or graphing trace data, and from Message Complex, whose primary elements are messages connected geometrically rather than identities connected relationally.

Trace Network analysis pattern differs from Causal History, Causal Messages, and Causal Chains in both primitive elements and construction intent. Causal History is a message-level structure whose arrows represent possible causation; Causal Messages are those messages selected as causally relevant within that history; and Causal Chains are abstractions of causal relations into linked 1-chains, 2-chains, and higher n-chains. By contrast, Trace Network is a general constructed network whose vertices are typically diagnostic identities rather than messages, and whose edges are induced by an explicitly declared relation derived from trace evidence, such as causal linkage, adjoint correspondence, request/response coupling, shared-resource mediation, or correlation transfer. Accordingly, a Trace Network may encode causal structure as one special case, but it is not restricted to causality and does not by itself imply chain-complex abstraction.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes, we want to count the number of (Adjoint) Threads of Activity corresponding to a specified ATID:

We can view this as (adjoint) threads coming into or out of the specified ATID, similar to divergence, which gives the name Trace Divergence log analysis pattern. This analysis pattern differs from Cord of Activity, which is not a number, and the latter may not have a single, unvarying source or target ATID to which other A(TID)s correspond. It is also different from Trace Flux, where the number of threads is an external variable not related to traces and logs, and from Message Flow, which operates on the individual message level, temporal in nature, and counters are set in advance.

Typical examples include SYN floods in network traces (src and dst ATIDs), the number of threads corresponding to the specific PID, or the number of threads contending for the specified API.

Activity Divergence may look similar, but its surface is temporal, whereas Trace Divergence’s, surface is structural. There can be several Trace Divergencies in the same trace or log since they are per ATID.

Formally, Trace Divergence is a property of a constructed graph, for example, D_in​(a)=∣{x∈V∣x→a}∣; Activity Divergence is a property of a constructed signal, interpreted as dynamics, for example, D_in​(a,t).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In ARM64 Virtualized Process memory dumps, their Stack Trace Collections, and their Execution Residue we may see pointers that point to ISA-Specific Code. For example, in an x86 process thread stack we may see this x86 disassembly of code pointers:

0:001> u 7573e81c
kernel32!BaseThreadInitThunk+0x2c:
7573e81c 2808 sub byte ptr [eax],cl
7573e81e 0090083142b9 add byte ptr [eax-46BDCEF8h],dl
7573e824 e003 loopne kernel32!BaseThreadInitThunk+0x39 (7573e829)
7573e826 002a add byte ptr [edx],ch
7573e828 0001 add byte ptr [ecx],al
7573e82a 3f aas
7573e82b d6 ???
7573e82c 2808 sub byte ptr [eax],cl

0:001> u 76e12640
KERNELBASE!SetEvent:
76e12640 fd std
76e12641 7bbe jnp KERNELBASE!UnmapViewOfFile+0x11 (76e12601)
76e12643 29fd sub ebp,edi
76e12645 0300 add eax,dword ptr [eax]
76e12647 91 xchg eax,ecx
76e12648 6810009008 push 8900010h
76e1264d a5 movs dword ptr es:[edi],dword ptr [esi]
76e1264e 43 inc ebx

0:001> ub 76e0c11c
^ Unable to find valid previous instruction for 'ub 76e0c11c'

0:001> ub 5f82d9c9
ACE!ACEInitializeEx+0x65573:
5f82d9b7 c3 ret
5f82d9b8 56 push esi
5f82d9b9 57 push edi
5f82d9ba 8b3da8b0835f mov edi,dword ptr [ACE!ACEInitializeEx+0x72c64 (5f83b0a8)]
5f82d9c0 8bf1 mov esi,ecx
5f82d9c2 6aff push 0FFFFFFFFh
5f82d9c4 ff7610 push dword ptr [esi+10h]
5f82d9c7 ffd7 call edi

0:001> ub ntdll!NtWaitForSingleObject+0xc
ntdll!NtMapUserPhysicalPagesScatter:
779fd030 b803000a00 mov eax,0A0003h
779fd035 ba70a6a077 mov edx,offset ntdll!Wow64SystemServiceCall (77a0a670)
779fd03a ffd2 call edx
779fd03c c20c00 ret 0Ch
779fd03f 90 nop
ntdll!NtWaitForSingleObject:
779fd040 b804000d00 mov eax,0D0004h
779fd045 ba70a6a077 mov edx,offset ntdll!Wow64SystemServiceCall (77a0a670)
779fd04a ffd2 call edx

The first 3 look like Wild Code (or Coincidental Symbolic Information if we use function names). But if we switch to CHPE architecture, we get the inverse, the first 3 right and the last 2 invalid:

0:001> .effmach CHPE
Effective machine: CHPE on X86 (read only) (CHPE)

0:001:CHPE> u 7573e81c
kernel32!BaseThreadInitThunk+0x2c:
7573e81c 90000828 adrp x8,kernel32!_imp_#LdrQueryImageFileKeyOption (75842000)
7573e820 b9423108 ldr w8,[x8,#0x230]
7573e824 2a0003e0 mov w0,w0
7573e828 d63f0100 blr x8
7573e82c 90000828 adrp x8,kernel32!_imp_#LdrQueryImageFileKeyOption (75842000)
7573e830 b9429d08 ldr w8,[x8,#0x29C]
7573e834 d63f0100 blr x8
7573e838 36225700 tbz w0,#4,kernel32!#IsFusionFullySupported+0x50 (75743318)

0:001:CHPE> u 76e12640
KERNELBASE!SetEvent:
76e12640 29be7bfd stp wfp,wlr,[sp,#-0x10]!
76e12644 910003fd mov fp,sp
76e12648 90001068 adrp x8,KERNELBASE!__hybrid_auxiliary_iat (7701e000)
76e1264c b943a508 ldr w8,[x8,#0x3A4]
76e12650 2a0003e0 mov w0,w0
76e12654 52800001 mov w1,#0
76e12658 d63f0100 blr x8
76e1265c 37f887e0 tbnz w0,#0x1F,KERNELBASE!BasepCheckImageVersion+0xe8 (76e13758)

0:001:CHPE> ub 76e0c11c
KERNELBASE!#WaitForSingleObjectEx+0xdc:
76e0c0fc 110083a2 add w2,wfp,#0x20
76e0c100 b90017a2 str w2,[fp,#0x14]
76e0c104 53001e61 uxtb w1,w19
76e0c108 2a0203e2 mov w2,w2
76e0c10c 2a0003e0 mov w0,w0
76e0c110 d0001088 adrp x8,KERNELBASE!__hybrid_auxiliary_iat (7701e000)
76e0c114 b9440d08 ldr w8,[x8,#0x40C]
76e0c118 d63f0100 blr x8

0:001:CHPE> ub 5f82d9c9
ACE!ACEInitializeEx+0x65565:
5f82d9a9 000003e8 ???
^ Memory access error in 'ub 5f82d9c9'

0:001:CHPE> ub ntdll!NtWaitForSingleObject+0xc
ntdll!NtAcceptConnectPort+0xc:
779fd02c 900018c2 adrp x2,77d15000
ntdll!NtMapUserPhysicalPagesScatter:
779fd030 0a0003b8 and w24,wfp,w0
779fd034 a670ba00 ???
779fd038 d2ff77a0 mov x0,#-0x443000000000000
779fd03c 90000cc2 adrp x2,77b95000
ntdll!NtWaitForSingleObject:
779fd040 0d0004b8 st1 {v24.b}[1],[x5]
779fd044 a670ba00 ???
779fd048 d2ff77a0 mov x0,#-0x443000000000000

0:001:CHPE> .effmach x86
Effective machine: x86 compatible (x86)

The same is observable for the x64 process thread raw stack region pointers:

0:000> ub 00007ff7`83432ac9
pointers_c!invoke_main+0x16:
00007ff7`83432aa6 4889442430 mov qword ptr [rsp+30h],rax
00007ff7`83432aab e82ae8ffff call pointers_c!ILT+725(__p___argc) (00007ff7`834312da)
00007ff7`83432ab0 8b00 mov eax,dword ptr [rax]
00007ff7`83432ab2 89442420 mov dword ptr [rsp+20h],eax
00007ff7`83432ab6 4c8b442428 mov r8,qword ptr [rsp+28h]
00007ff7`83432abb 488b542430 mov rdx,qword ptr [rsp+30h]
00007ff7`83432ac0 8b4c2420 mov ecx,dword ptr [rsp+20h]
00007ff7`83432ac4 e8b7e7ffff call pointers_c!ILT+635(main) (00007ff7`83431280)

0:000> ub 00007ff8`046917ac
^ Unable to find valid previous instruction for 'ub 00007ff8`046917ac'

0:000> .effmach ARM64EC
Effective machine: ARM64EC (CHPEv2 on X64) (ARM64EC)

0:000:ARM64EC> ub 00007ff7`83432ac9
pointers_c!invoke_main+0x19:
00007ff7`83432aa9 2ae83024 ???
^ Memory access error in 'ub 00007ff7`83432ac9'

0:000:ARM64EC> ub 00007ff8`046917ac
kernel32!$iexit_thunk$cdecl$d$d+0x2c:
00007ff8`0469178c 00000000 ???
kernel32!$iexit_thunk$cdecl$i8$i8:
00007ff8`04691790 d503237f pacibsp
00007ff8`04691794 a9bf7bfd stp fp,lr,[sp,#-0x10]!
00007ff8`04691798 910003fd mov fp,sp
00007ff8`0469179c d10083ff sub sp,sp,#0x20
00007ff8`046917a0 b0000048 adrp x8,kernel32!_os_arm64x_dispatch_call_no_redirect (00007ff8`0469a000)
00007ff8`046917a4 f9400110 ldr xip0,[x8]
00007ff8`046917a8 d63f0200 blr xip0

0:000:ARM64EC> .effmach AMD64
Effective machine: x64 (AMD64)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Latent Structure pattern addresses situations where a memory region appears raw and untyped yet shows early, incomplete signs of structural organization. Signals, such as small or pointer-like values, alignment regularities, recurring byte sequences, partial strings, or fragments that resemble fields, suggest that a real structure might exist, but cannot yet be interpreted safely or confidently. Latent Structure represents the pre-suspect stage in structural diagnostics: the point where the analyst notices potential form but must resist premature interpretation. Acting too early risks misclassifying problems and misidentifying root causes. Several forces complicate this stage: partial overwrites, coincidental alignments, ABI or version mismatches, and cognitive biases that encourage overinterpretation. This analysis pattern, therefore, emphasizes careful, hypothesis-driven exploration using techniques such as tentative structure casting, pointer-chain heuristics, checks for internal semantic coherence, software internals, and domain knowledge, all without assuming the structure’s validity. When enough evidence accumulates, a Latent Structure transitions into a Suspect Structure (subject of the next analysis pattern), where it becomes testable.

For example, we may see these fragment in Execution Residue:

00000029`e7efeb00 00001f80`0010004b
00000029`e7efeb08 0053002b`002b0033
00000029`e7efeb10 00000242`002b002b

Finally, we write the formal pattern structure card.

Intent

Detect hidden or unclear structural organization in raw memory regions that exhibit early indicators of structure-like form but whose types are not yet known.

Context

Appears in:
Execution Residue, Pointer Cone, Memory Region, and Region Strata.

Problem

A memory dump shows a region of raw bytes without explicit type information that contains hints suggesting that a structure may be present. Prematurely interpreting such memory can lead to false positives, misclassification, incorrect casting, and a chain of misleading hypotheses.

Forces

Data:

• Memory may contain partial structures
• Overwrites blur structure boundaries
• Random-looking regions may hide structured subregions

Semantics:

• Pointer-like values may be real or coincidental
• Partial strings
• Field alignments may appear regular due to chance

Modules:

• Coincidental symbols
• ABI or version mismatches

Cognitive biases:

• Insufficient domain knowledge
• Premature suspicion

Symptoms

• Structural hints in bytes
• Pointer-like values
• Strings and identity hints
• Alignment and regularity
• Recurring patterns across multiple memory locations
• Partial structure validity
• Incomplete or corrupt-like structure

Resolution Strategies

• Structure casting
• Heuristic field and pointer chain analysis
• Verification of internal semantic coherence

Resulting Context

Structure becomes Suspect and testable for validity.

Related Patterns

Hidden Artifact Patterns, Corrupt Structure, Module Hint, Falsity and Coincidence Patterns, Shared Buffer Overwrite, Value References, Small Value, Design Value, Shared Structure, and Regular Data.

Formal Card

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When looking at Execution Residue in Windows ARM64 memory dumps, we may notice Encoded Pointers in the form of authenticated pointers (PAC, Pointer Authentication Code, see the Linux guide and Windows info). For example:

0:000> dps 00000053f62fc000 00000053f6300000
...
00000053`f62ffe28 e817fff7`6a0bc054 functions_c!__scrt_common_main+0x14
...

The return address isn’t possible to use directly (Invalid Pointer):

0:000> ub e817fff7`6a0bc054
e817fff7`6a0bc034 ?? ???
^ Memory access error in ‘ub e817fff7`6a0bc054′

However, the symbolic reference is ok:

0:000> ub functions_c!__scrt_common_main+0x14
functions_c!pre_cpp_initialization+0x7c:
00007ff7`6a0bc034 00000000 ???
00007ff7`6a0bc038 00000000 ???
00007ff7`6a0bc03c 00000000 ???
functions_c!__scrt_common_main:
00007ff7`6a0bc040 d503237f pacibsp
00007ff7`6a0bc044 a9bf7bfd stp fp,lr,[sp,#-0x10]!
00007ff7`6a0bc048 910003fd mov fp,sp
00007ff7`6a0bc04c 97ffd81f bl functions_c!ILT+4284(__security_init_cookie) (00007ff7`6a0b20c8)
00007ff7`6a0bc050 94000016 bl functions_c!__scrt_common_main_seh (00007ff7`6a0bc0a8)

Because of that, Rough Stack that uses the dpS WinDbg command instead, omits such valid symbolic references.

If you find such pointers, you can replace the higher 4-byte part with the higher part of the module start address, for example:

0:000> lm
start end module name
00007ff7`6a0a0000 00007ff7`6a0d0000 functions_c
…

0:000> ub 00007ff7`6a0bc054
functions_c!pre_cpp_initialization+0×7c:
00007ff7`6a0bc034 00000000 ???
00007ff7`6a0bc038 00000000 ???
00007ff7`6a0bc03c 00000000 ???
functions_c!__scrt_common_main:
00007ff7`6a0bc040 d503237f pacibsp
00007ff7`6a0bc044 a9bf7bfd stp fp,lr,[sp,#-0×10]!
00007ff7`6a0bc048 910003fd mov fp,sp
00007ff7`6a0bc04c 97ffd81f bl functions_c!ILT+4284(__security_init_cookie) (00007ff7`6a0b20c8)
00007ff7`6a0bc050 94000016 bl functions_c!__scrt_common_main_seh (00007ff7`6a0bc0a8)

Of course, this may not work for pointers, encoded by the Windows EncodePointer API.

Finally, we write the formal pattern structure card for Encoded Pointer.

Intent

To recognize situations where a pointer stored in memory is not directly usable: its value must be interpreted or transformed before it can be resolved to a valid code or data address.

Context

Appears in:
Stack Trace, Execution Residue, Context Pointer, Historical Information.

Common environments:

• Tagged pointers
• ARM64 pointer authentication (PAC)
• Top-Byte-Ignore tagging (AArch64)
• ASLR and relocations that have not yet been applied in the captured memory
• Managed space compressed and metadata-embedded GC pointers
• Objective-C and Swift tagged ISA pointers
• Sanitizers or checking runtimes that add metadata bits

Problem

A pointer in the dump visually appears to be an address, but fails to resolve using normal symbolic or spatial checks; dereferencing its raw value yields an incorrect memory address or a memory error.

Forces

• Performance/security constraints favor encoded pointer formats
• Debugger views often show raw stack/heap
• Encoding schemes vary by platform and compiler
• Hardware PAC may prevent guessing the correct pointer form without a proper decode context

Symptoms

• Pointer value not inside any loaded module or valid virtual address range
• Symbol resolution differs
• Adjacent stack slots look pointer-like, but this one does not
• Backwards disassembly shows an incorrect frame

Resolution Strategies

• Decode PAC
• Canonicalize upper bits
• Strip tags
• Expand bits
• Apply relocation deltas
• Mask metadata

Resulting Context

After correct interpretation, the pointer becomes:

• Resolvable to a target symbol
• Walkable for call-stack reconstruction
• Safe for dereferencing in analysis context
• Enables further analysis

Formal card

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The list of local variables displayed by the dv WinDbg command may contain False Local Addresses, especially if some non-standard alignment is used on ARM64 platforms. For example, we get this address that doesn’t look correct if we associate it with the source code:

* _Alignas(4096) long long ll = 1;

0:000> dv /V
0000000b`970fe260 @x27+0×1000 ll = 0n-3689348814741910324
0000000b`970fd490 @x27+0×0230 align = 8

It is not aligned on the page boundary, and the value is not the expected 1:

0:000> dq 0000000b`970fe260 L1
0000000b`970fe260 cccccccc`cccccccc

However, in the disassembly, we see the following sequence of instructions to initialize the variable:

00007ff7`d061afdc f9533f69 ldr x9,[x27,#0x2678]
00007ff7`d061afe0 d2800028 mov x8,#1
00007ff7`d061afe4 f9000128 str x8,[x9]

So, we can see that the local variable address is stored at x27+0×2678:

0:000> dp x27+0x2678 L1
0000000b`970ff8d8 0000000b`970fd000

and see the correct variable value:

0:000> dpp x27+0x2678 L1
0000000b`970ff8d8 0000000b`970fd000 00000000`00000001

This analysis pattern differs from False Effective Address analysis pattern in the correct value of the base register.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

On Windows 11 ARM64, it is possible to run x64 and x86 programs (ARM64EC and Compiled Hybrid Portable Executable, CHPE). When we capture memory dumps and examine the corresponding Stack Trace Collection, we see ARM64EC and CHPE frames. This is similar to our earlier Virtualized Process (WOW64) analysis pattern, although WinDbg can show us different architecture frames at the same time. Below are 2 examples of NULL Pointer (Data) analysis pattern.

* x64 process minidump

0:000> ~*kL

. 0 Id: 8030.677c Suspend: 0 Teb: 000000e0`5d015000 Unfrozen
# Arch Child-SP RetAddr Call Site
00 ARM64EC 000000e0`5d2fdf30 00007ff8`02901d6c ntdll!#NtWaitForMultipleObjects+0x14
01 ARM64EC 000000e0`5d2fdf40 00007ff8`046735e0 KERNELBASE!#WaitForMultipleObjectsEx+0xfc
02 ARM64EC 000000e0`5d2fe220 00007ff8`046730e0 kernel32!#WerpReportFaultInternal+0x4c0
03 ARM64EC 000000e0`5d2fe390 00007ff8`0463d3e4 kernel32!#WerpReportFault+0xe0
04 ARM64EC 000000e0`5d2fe3f0 00007ff8`02a047e8 kernel32!#BasepReportFault+0x24
05 ARM64EC 000000e0`5d2fe410 00007ff8`0754f7c4 KERNELBASE!#UnhandledExceptionFilter+0x308
06 ARM64EC 000000e0`5d2fe500 00007ff8`07547148 ntdll!RtlUserThreadStart$filt$0+0x64
07 ARM64EC 000000e0`5d2fe510 00007ff8`0749a304 ntdll!#__C_ExecuteExceptionFilter+0x38
08 ARM64EC 000000e0`5d2fe570 00007ff8`07547068 ntdll!#__C_specific_handler+0xf4
09 ARM64EC 000000e0`5d2fe5f0 00007ff8`07440820 ntdll!#RtlpExecuteHandlerForException+0x28
0a ARM64EC 000000e0`5d2fe610 00007ff8`07546e50 ntdll!#RtlDispatchException+0x298
0b ARM64EC 000000e0`5d2fed90 00007ff7`128d1ccc ntdll!KiUserExceptionDispatcher_DetourReturn+0x10
0c AMD64 000000e0`5d2ff8e0 00007ff7`128d2ac9 pointers_c!main+0x41c
0d AMD64 000000e0`5d2ffdb0 00007ff7`128d2972 pointers_c!invoke_main+0x39
0e AMD64 000000e0`5d2ffe00 00007ff7`128d282e pointers_c!__scrt_common_main_seh+0x132
0f AMD64 000000e0`5d2ffe70 00007ff7`128d2b5e pointers_c!__scrt_common_main+0xe
10 AMD64 000000e0`5d2ffea0 00007ff8`046917ac pointers_c!mainCRTStartup+0xe
11 ARM64EC 000000e0`5d2ffed0 00007ff8`046115e8 kernel32!$iexit_thunk$cdecl$i8$i8+0x1c
12 ARM64EC 000000e0`5d2fff00 00007ff8`0748c120 kernel32!#BaseThreadInitThunk+0x48
13 ARM64EC 000000e0`5d2fff10 00000000`00000000 ntdll!#RtlUserThreadStart+0x70

1 Id: 8030.7a64 Suspend: 0 Teb: 000000e0`5d017000 Unfrozen
# Arch Child-SP RetAddr Call Site
00 ARM64EC 000000e0`5d3ff820 00007ff8`07470084 ntdll!#NtWaitForWorkViaWorkerFactory+0x14
01 ARM64EC 000000e0`5d3ff830 00007ff8`046115e8 ntdll!#TppWorkerThread+0x5a4
02 ARM64EC 000000e0`5d3ffaf0 00007ff8`0748c120 kernel32!#BaseThreadInitThunk+0x48
03 ARM64EC 000000e0`5d3ffb00 00000000`00000000 ntdll!#RtlUserThreadStart+0x70

2 Id: 8030.119c Suspend: 0 Teb: 000000e0`5d019000 Unfrozen
# Arch Child-SP RetAddr Call Site
00 ARM64EC 000000e0`5d4ff980 00007ff8`07470084 ntdll!#NtWaitForWorkViaWorkerFactory+0x14
01 ARM64EC 000000e0`5d4ff990 00007ff8`046115e8 ntdll!#TppWorkerThread+0x5a4
02 ARM64EC 000000e0`5d4ffc50 00007ff8`0748c120 kernel32!#BaseThreadInitThunk+0x48
03 ARM64EC 000000e0`5d4ffc60 00000000`00000000 ntdll!#RtlUserThreadStart+0x70

3 Id: 8030.70f0 Suspend: 0 Teb: 000000e0`5d01b000 Unfrozen
# Arch Child-SP RetAddr Call Site
00 ARM64EC 000000e0`5d5ff810 00007ff8`07470084 ntdll!#NtWaitForWorkViaWorkerFactory+0x14
01 ARM64EC 000000e0`5d5ff820 00007ff8`046115e8 ntdll!#TppWorkerThread+0x5a4
02 ARM64EC 000000e0`5d5ffae0 00007ff8`0748c120 kernel32!#BaseThreadInitThunk+0x48
03 ARM64EC 000000e0`5d5ffaf0 00000000`00000000 ntdll!#RtlUserThreadStart+0x70

4 Id: 8030.4720 Suspend: 0 Teb: 000000e0`5d01d000 Unfrozen
# Arch Child-SP RetAddr Call Site
00 ARM64EC 000000e0`5d6ff740 00007ff8`0487ec00 ntdll!#NtWaitForSingleObject+0x14
01 ARM64EC 000000e0`5d6ff750 00007ff8`0487e2b0 xtajit64!BeginSimulation+0x12eb0
02 ARM64EC 000000e0`5d6ff7a0 00007ff8`0748c0f0 xtajit64!BeginSimulation+0x12560
03 ARM64EC 000000e0`5d6ff7d0 00000000`00000000 ntdll!#RtlUserThreadStart+0x40

0:000> .frame /c 4
04 000000e0`5d2fe3f0 00007ff8`02a047e8 kernel32!#BasepReportFault+0x24
x0=0000000000000003 x1=000000e05d2fe2e0 x2=0000000000000001 x3=0000000000000000
x4=0000000000000000 x5=0000000000000000 x6=0000000000000000 x7=0000000000000000
x8=000000000000012c x9=0000000000000000 x10=0000000000000000 x11=0000000000000000
x12=0000000000000000 x13=0000000000000000 x14=0000000000000000 x15=0000000000000000
x16=0000bbd3fe198401 x17=0000bbd3fe198401 x18=0000000000000000 x19=000000e05d2fe5a0
x20=0000000000000000 x21=000000e05d2fe5a0 x22=00007ff8045a0000 x23=0000000000000000
x24=0000000000000000 x25=0000000000000000 x26=000000e05d2fe410 x27=0000000000000001
x28=0000000000000000 fp=000000e05d2fe3f0 lr=00007ff80463d3e4 sp=000000e05d2fe3f0
pc=00007ff80463d3e4 psr=60000000 -ZC- EL0
kernel32!#BasepReportFault+0x24:
00007ff8`0463d3e4 14000002 b kernel32!#BasepReportFault+0x2c (00007ff8`0463d3ec)

0:000:ARM64EC> .frame /c c
0c 000000e0`5d2ff8e0 00007ff7`128d2ac9 pointers_c!main+0x41c [C:\ACPPWD\pointers_c\pointers_c.c @ 133]
rax=0000000000000004 rbx=0000000000000000 rcx=9ff2ebf5ac870000
rdx=00007ff7128dabc0 rsi=0000000000000000 rdi=000000e05d2ffc18
rip=00007ff7128d1ccc rsp=000000e05d2ff8e0 rbp=000000e05d2ff930
r8=00000000fffffffe r9=0000000000000000 r10=0000000000000001
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=3 nv up ei pl zr na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00003240
pointers_c!main+0x41c:
00007ff7`128d1ccc c70000000000 mov dword ptr [rax],0 ds:00000000`00000004=????????

0:000> .cxr
Resetting default scope

0:000:ARM64EC>

* x86 process full dump

0:000> ~*kL

. 0 Id: 1a68.8a54 Suspend: 0 Teb: 0295d000 Unfrozen
# Arch ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 x86 02afe454 779dd5dc 0x2730002
01 x86 02afe458 75eb2f10 ntdll!NtWaitForMultipleObjects+0xc
02 CHPE 02afe460 75eb2f10 KERNELBASE!$push_thunk$stdcall$u$uuuuu+0x60
03 CHPE 02afe4e0 75d30840 KERNELBASE!#WaitForMultipleObjectsEx+0x194
04 CHPE 02afe680 7712bc70 KERNELBASE!#WaitForMultipleObjects+0x20
05 CHPE 02afe690 7712b690 kernel32!#WerpReportFaultInternal+0x598
06 CHPE 02afe790 770e7fe4 kernel32!#WerpReportFault+0x118
07 CHPE 02afe800 75e90da8 kernel32!#BasepReportFault+0x24
08 CHPE 02afe820 779141b4 KERNELBASE!#UnhandledExceptionFilter+0x378
09 CHPE 02afe8f0 77910ef8 ntdll!strrchr+0x1eb4
0a CHPE 02afe910 778cf388 ntdll!#__C_ExecuteExceptionFilter+0x38
0b CHPE 02afe970 77861554 ntdll!#__C_specific_handler+0xf8
0c CHPE 02afe9e0 779b7154 ntdll!RtlpExecuteHandlerForExceptionCHPE+0x14
0d x86 02afeee0 779b7154 ntdll!RtlDispatchExceptionCHPE+0x2de
0e x86 02aff2bc 779e08d2 ntdll!RtlpProcessPushThunkForException+0x7b
0f x86 02aff354 779e0e5f ntdll!RtlDispatchException+0×1ee
10 x86 02aff360 02aff36c ntdll!KiUserExceptionDispatcher+0xf
11 x86 02aff88c 00712a03 0×2aff36c
12 x86 02aff8ac 0071284a pointers_c!invoke_main+0×33
13 x86 02aff908 007126dd pointers_c!__scrt_common_main_seh+0×15a
14 x86 02aff910 00712a88 pointers_c!__scrt_common_main+0xd
15 x86 02aff918 771487a8 pointers_c!mainCRTStartup+0×8
16 CHPE 02aff920 771487a8 kernel32!$push_thunk$cdecl$u$u+0×58
17 CHPE 02aff990 778bfc8c kernel32!BaseThreadInitThunk+0×2c
18 CHPE 02aff9a0 778bfbe8 ntdll!#__RtlUserThreadStart+0×3c
19 CHPE 02aff9f0 7799988c ntdll!#_RtlUserThreadStart+0×28

1 Id: 1a68.8194 Suspend: 0 Teb: 02961000 Unfrozen
# Arch ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 x86 086ff5a4 779dee8c 0x2730002
01 x86 086ff5a8 779ab648 ntdll!NtWaitForWorkViaWorkerFactory+0xc
02 CHPE 086ff5b0 779ab648 ntdll!#NtWaitForWorkViaWorkerFactory$push_thunk+0x68
03 CHPE 086ff630 7709e81c ntdll!#TppWorkerThread+0x238
04 CHPE 086ff810 778bfc8c kernel32!BaseThreadInitThunk+0x2c
05 CHPE 086ff820 778bfbe8 ntdll!#__RtlUserThreadStart+0x3c
06 CHPE 086ff870 7799988c ntdll!#_RtlUserThreadStart+0x28

2 Id: 1a68.499c Suspend: 0 Teb: 02965000 Unfrozen
# Arch ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 x86 087ffcc4 779dee8c 0x2730002
01 x86 087ffcc8 779ab648 ntdll!NtWaitForWorkViaWorkerFactory+0xc
02 CHPE 087ffcd0 779ab648 ntdll!#NtWaitForWorkViaWorkerFactory$push_thunk+0x68
03 CHPE 087ffd50 7709e81c ntdll!#TppWorkerThread+0x238
04 CHPE 087fff30 778bfc8c kernel32!BaseThreadInitThunk+0x2c
05 CHPE 087fff40 778bfbe8 ntdll!#__RtlUserThreadStart+0x3c
06 CHPE 087fff90 7799988c ntdll!#_RtlUserThreadStart+0x28

3 Id: 1a68.63f4 Suspend: 0 Teb: 02969000 Unfrozen
# Arch ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 x86 08b5f854 779dee8c 0x2730002
01 x86 08b5f858 779ab648 ntdll!NtWaitForWorkViaWorkerFactory+0xc
02 CHPE 08b5f860 779ab648 ntdll!#NtWaitForWorkViaWorkerFactory$push_thunk+0x68
03 CHPE 08b5f8e0 7709e81c ntdll!#TppWorkerThread+0x238
04 CHPE 08b5fac0 778bfc8c kernel32!BaseThreadInitThunk+0x2c
05 CHPE 08b5fad0 778bfbe8 ntdll!#__RtlUserThreadStart+0x3c
06 CHPE 08b5fb20 7799988c ntdll!#_RtlUserThreadStart+0x28

0:000> r
eax=001d005b ebx=00000180 ecx=00000003 edx=779ea670 esi=00000000 edi=00000003
eip=02730002 esp=02afe458 ebp=02afe480 iopl=0 nv up ei ng nz ac po cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0023 efl=00000293
02730002 c3 ret

0:000> .frame /c 7
07 02afe800 75e90da8 kernel32!#BasepReportFault+0x24
x0=0000000000000000 x1=0000000000000000 x2=0000000000000000 x3=0000000000000000
x4=0000000000000000 x5=0000000000000000 x6=0000000000000000 x7=0000000000000000
x8=0000000000000000 x9=0000000000000000 x10=0000000000000000 x11=0000000000000000
x12=0000000000000000 x13=0000000000000000 x14=0000000000000000 x15=0000000000000000
x16=0000000000000000 x17=0000000000000000 x18=0000000000000000 x19=0000000002afe990
x20=0000000002afe990 x21=0000000077090000 x22=0000000000000004 x23=0000000000000000
x24=0000000000000001 x25=0000000075f1e000 x26=0000000000000000 x27=0000000002afe830
x28=0000000002affa38 fp=0000000002afe800 lr=00000000770e7fe4 sp=0000000002afe800
pc=00000000770e7fe4 psr=00000000 ---- EL0
kernel32!#BasepReportFault+0x24:
770e7fe4 2a0003e0 mov w0,w0

0:000:CHPE> .cxr
Resetting default scope

0:000> dps 02aff354
02aff354 02aff88c
02aff358 779e0e5f ntdll!KiUserExceptionDispatcher+0xf
02aff35c 02aff36c
02aff360 02aff3bc
02aff364 02aff36c
02aff368 02aff3bc
02aff36c c0000005
02aff370 00000000
02aff374 00000000
02aff378 00711c6a pointers_c!main+0×3da
02aff37c 00000002
02aff380 00000001
02aff384 00000004
02aff388 00000000
02aff38c 00000000
02aff390 00000000
02aff394 00000000
02aff398 00000000
02aff39c 00000000
02aff3a0 00000000
02aff3a4 00000000
02aff3a8 00000000
02aff3ac 00000000
02aff3b0 00000000
02aff3b4 00000000
02aff3b8 00000000
02aff3bc 0001003f
02aff3c0 00000000
02aff3c4 00000000
02aff3c8 00000000
02aff3cc 00000000
02aff3d0 ffff0ff0

0:000> .cxr 02aff3bc
eax=00000004 ebx=0295a000 ecx=02aff4a0 edx=00000000 esi=02aff6a8 edi=02aff88c
eip=00711c6a esp=02aff6a8 ebp=02aff88c iopl=0 nv up ei pl nz ac po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0023 efl=00010212
pointers_c!main+0×3da:
00711c6a c70000000000 mov dword ptr [eax],0 ds:0023:00000004=????????

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
# Arch ChildEBP RetAddr
00 x86 02aff88c 00712a03 pointers_c!main+0x3da
01 x86 02aff8ac 0071284a pointers_c!invoke_main+0x33
02 x86 02aff908 007126dd pointers_c!__scrt_common_main_seh+0x15a
03 x86 02aff910 00712a88 pointers_c!__scrt_common_main+0xd
04 x86 02aff918 771487a8 pointers_c!mainCRTStartup+0x8
05 CHPE 02aff920 771487a8 kernel32!$push_thunk$cdecl$u$u+0x58
06 CHPE 02aff990 778bfc8c kernel32!BaseThreadInitThunk+0x2c
07 CHPE 02aff9a0 778bfbe8 ntdll!#__RtlUserThreadStart+0x3c
08 CHPE 02aff9f0 7799988c ntdll!#_RtlUserThreadStart+0x28

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We write software based on requirements and then see its execution. The same analogy can be applied to Declarative Traces, which are “executed.” Trace Plans serve the role of tracing and logging requirements. The following diagram illustrates trace engineering and the lifecycle of tracing and logging:

We look at a resulting trace or log and relate it to its Trace Plan to find anomalies and problems not only in software execution but also in traces and logs themselves and improve tracing source code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we get traces and logs, we are interested in Trace Context: an issue description, how its trace was collected, overall system information, related Adjoint Spaces, Trace Summary, and previous traces and logs and their analyses. This contextual information can be organized as a checklist to ensure situational awareness, diagnostic quality, and reduce the number of information request roundtrips.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we get memory dumps, we are interested in Dump Context: an issue description, how its memory dump was collected, overall system information, related Paratext, and previous memory dumps and their analyses. This contextual information can be organized as a checklist to ensure situational awareness, diagnostic quality, and reduce the number of information request roundtrips.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

DPC Stack Collection is yet another area to mine for Execution Residue and Rough Stack Traces. Some DPC Stacks may be visible in Stack Trace Collections such as from CPUs.

Each CPU has a DPC stack for execution of queued DPCs. We can get its base stack region address from the corresponding _KPRCB structure for each CPU. The stack region limit can be calculated from the KeKernelStackSize Module Variable:

0: kd> dd nt!KeKernelStackSize L1
fffff800`e27c4028 00007000

0: kd> !dpcs
CPU Type KDPC Function
0: Normal : 0xffffc9019b313400 0xfffff8008b6b31b0 igdkmd64

0: kd> !prcb 0
PRCB for Processor 0 at fffff8006ff97180:
Current IRQL — 0
Threads– Current ffffc901ad242040 Next 0000000000000000 Idle fffff800e27d1640
Processor Index 0 Number (0, 0) GroupSetMember 1
Interrupt Count — 06278469
Times — Dpc 0000b229 Interrupt 0000b897
Kernel 00d11420 User 000b6650

0: kd> dt nt!_KPRCB fffff8006ff97180 DpcStack
+0×38a0 DpcStack : 0xfffff800`745b1fb0 Void

0: kd> dpS 0xfffff800`745b1fb0-7000 L7000/8
fffff800`e1ba7e6c nt!RtlpHpLfhSlotAllocateSlow+0×484
fffff800`e26ee9c0 nt!ExPoolState+0×86940
fffff800`e1a3ecb4 nt!ExAllocateHeapPool+0×2134
fffff800`e1800000 nt!RtlCompressBufferProcs
fffff800`e236a196 nt!ExFreePoolWithTag+0×4c6
fffff800`e2369189 nt!ExAllocatePool2+0×99
fffff800`e1b9e0ba nt!AuthzBasepEvaluateExpression+0×3a
fffff800`e1b9c1a0 nt!AuthzBasepEvaluateAceCondition+0×2a0
fffff800`e1b9b649 nt!SepNormalAccessCheck+0×589
fffff800`e1b9a852 nt!SepAccessCheck+0×2c2
fffff800`e26ee9c0 nt!ExPoolState+0×86940
fffff800`e1a3ecb4 nt!ExAllocateHeapPool+0×2134
fffff800`7400e5d8 LXCORE!VfsFileGetPathString+0×114
fffff800`e1d022ee nt!qsort+0×3be
fffff800`e26ee9c0 nt!ExPoolState+0×86940
fffff800`e1a3ecb4 nt!ExAllocateHeapPool+0×2134
fffff800`e1800000 nt!RtlCompressBufferProcs
fffff800`e236a196 nt!ExFreePoolWithTag+0×4c6
fffff800`e2638a00 nt!MiSystemPartition
fffff800`e1b7166f nt!MmDeleteKernelStack+0×22f
fffff800`e1b7235b nt!KiExpandKernelStackAndCalloutOnStackSegment+0×31b
fffff800`75464450 NETIO!ArbitrateAndEnforceCallout
fffff800`e1aeef4c nt!KiExpandKernelStackAndCalloutSwitchStack+0×17c
fffff800`75464450 NETIO!ArbitrateAndEnforceCallout
fffff800`e1aeeca3 nt!KeExpandKernelStackAndCalloutInternal+0×33
fffff800`75464450 NETIO!ArbitrateAndEnforceCallout
fffff800`75282d7e ndis!NdisAcquireRWLockRead+0×2e
fffff800`e1aeec5d nt!KeExpandKernelStackAndCalloutEx+0×1d
fffff800`7544a27d NETIO!UpdateLayerClassifyStat+0×19d
fffff800`e2638a00 nt!MiSystemPartition
fffff800`e1b7166f nt!MmDeleteKernelStack+0×22f
fffff800`754a6000 NETIO!WPP_GLOBAL_Control
fffff800`e1b2db6f nt!KeSetEvent+0×10f
fffff800`786a3faa Ndu!NduUpdateProcessEnergyContext+0×6a
fffff800`786ab8c9 Ndu!NduUpdateInterfaceTimeStatsEntryList+0×149
fffff800`e1bf167c nt!ExFreeToLookasideListEx+0×4c
fffff800`786a1aee Ndu!NduUpdateInterfacePowerContext+0×1be
fffff800`786a38ff Ndu!NduDeleteNblContext+0×9f
fffff800`e1800000 nt!RtlCompressBufferProcs
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`75260642 ndis!NdisFSendNetBufferListsComplete+0×32
fffff800`75419b30 NETIO!WfpNblInfoDestroyIfUnused+0xf0
fffff800`8b732bc2 igdkmd64+0×392bc2
fffff800`752a7557 ndis!NdisFreeMemory+0×17
fffff800`8b73113c igdkmd64+0×39113c
fffff800`8b48f165 igdkmd64+0xef165
fffff800`8b57aeaa igdkmd64+0×1daeaa
fffff800`8b73113c igdkmd64+0×39113c
fffff800`8b5f3ad9 igdkmd64+0×253ad9
fffff800`8b48f165 igdkmd64+0xef165
fffff800`8b4778d3 igdkmd64+0xd78d3
fffff800`8b625db3 igdkmd64+0×285db3
fffff800`e1ae3a2f nt!KiSelectProcessorToPreempt+0xff
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`8b467e79 igdkmd64+0xc7e79
fffff800`8b7355ad igdkmd64+0×3955ad
fffff800`8b73a25a igdkmd64+0×39a25a
fffff800`8b732bc2 igdkmd64+0×392bc2
fffff800`8b732d0c igdkmd64+0×392d0c
fffff800`8b731209 igdkmd64+0×391209
fffff800`8b6c1081 igdkmd64+0×321081
fffff800`8b6a54ac igdkmd64+0×3054ac
fffff800`8b6cccec igdkmd64+0×32ccec
fffff800`8b6c0390 igdkmd64+0×320390
fffff800`8b6b3577 igdkmd64+0×313577
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`e1b3143c nt!KeAcquireSpinLockAtDpcLevel+0×1c
fffff800`786a406a Ndu!NduUpdateProcessEnergyContext+0×12a
fffff800`7525b85d ndis!ndisFreeToLookasideList+0×5d
fffff800`786ab8c9 Ndu!NduUpdateInterfaceTimeStatsEntryList+0×149
fffff800`7525b645 ndis!NdisFreeNetBufferList+0xa5
fffff800`786a1aee Ndu!NduUpdateInterfacePowerContext+0×1be
fffff800`75451260 NETIO!NetioFreeNetBufferAndNetBufferList+0×10
fffff800`e1b3143c nt!KeAcquireSpinLockAtDpcLevel+0×1c
fffff800`786a406a Ndu!NduUpdateProcessEnergyContext+0×12a
fffff800`7569b5dc tcpip!TcpTcbSendDatagramsComplete+0×9c
fffff800`7525b85d ndis!ndisFreeToLookasideList+0×5d
fffff800`7525b645 ndis!NdisFreeNetBufferList+0xa5
fffff800`75451260 NETIO!NetioFreeNetBufferAndNetBufferList+0×10
fffff800`75514e63 fwpkclnt!FwppNetBufferListAssociateContext+0×153
fffff800`75611fb1 tcpip!TcpSendDatagramsComplete+0xd1
fffff800`786a2f9f Ndu!NduHandleNblContextRemoved+0×1b3
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`75611ee0 tcpip!TcpSendDatagramsComplete
fffff800`75419dd4 NETIO!NetioDereferenceNetBufferListChain+0×174
fffff800`75512f51 fwpkclnt!FwppNetBufferListEventNotify+0×1a1
fffff800`7571ca2d tcpip!FlSendNetBufferListChainComplete+0×6d
fffff800`7527258b ndis!ndisMSendCompleteNetBufferListsInternal+0×25b
fffff800`7551342f fwpkclnt!FwpsNetBufferListRetrieveContext0+0×4f
fffff800`91054a30 bridge+0×4a30
fffff800`786a251e Ndu!NduFindOrAssociateNblContext+0×6e
fffff800`75287472 ndis!NdisMSendNetBufferListsComplete+0×5c2
fffff800`786a2b2d Ndu!NduOutboundMacClassifyProcessSingleNbl+0×5d
fffff800`786a2961 Ndu!NduOutboundMacClassify+0×181
fffff800`754a6000 NETIO!WPP_GLOBAL_Control
fffff800`75434553 NETIO!ProcessCallout2+0×163
fffff800`78016d00 nwifi!Dot11SendNBComplete+0×170
fffff800`75463246 NETIO!KfdClassify2+0xbb6
fffff800`7527258b ndis!ndisMSendCompleteNetBufferListsInternal+0×25b
fffff800`e1eaeb31 nt!HvcallpExtendedFastHypercall+0×51
fffff800`e1ae040b nt!HvcallFastExtended+0×13b
fffff800`75272330 ndis!ndisMSendCompleteNetBufferListsInternal
fffff800`e1ae0903 nt!HvlFlushRangeListTb+0×353
fffff800`8ad0b2cd wdiwifi!CPort::CompletePendingCancelSendsOrHaltJobs+0xdd
fffff800`8adea5f0 wdiwifi!WPP_de984c7e04793f3292dfaa0cae396821_Traceguids
fffff800`e1aee810 nt!EtwpReserveTraceBuffer+0×310
fffff800`e1aee810 nt!EtwpReserveTraceBuffer+0×310
fffff800`e1aed852 nt!EtwpTraceMessageVa+0×7f2
fffff800`8abae2d8 mrvlpcie8897+0×2e2d8
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8abae1af mrvlpcie8897+0×2e1af
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8abb7c74 mrvlpcie8897+0×37c74
fffff800`73ee2e1a WppRecorder!WppAutoLogTrace+0×31a
fffff800`e1c6b43e nt!WmiTraceMessage+0×1e
fffff800`e1800000 nt!RtlCompressBufferProcs
fffff800`e236a196 nt!ExFreePoolWithTag+0×4c6
fffff800`e1c6b43e nt!WmiTraceMessage+0×1e
fffff800`8ad05493 wdiwifi!WPP_RECORDER_SF_DDD+0xbf
fffff800`8ad184da wdiwifi!operator delete+0×1a
fffff800`73ee2e1a WppRecorder!WppAutoLogTrace+0×31a
fffff800`8acaf8f0 mrvlpcie8897+0×12f8f0
fffff800`8abad86d mrvlpcie8897+0×2d86d
fffff800`8acaf8f0 mrvlpcie8897+0×12f8f0
fffff800`8ab8facb mrvlpcie8897+0xfacb
fffff800`e1c3e85a nt!DbgPrint+0×5a
fffff800`8acaf8f0 mrvlpcie8897+0×12f8f0
fffff800`8abc4510 mrvlpcie8897+0×44510
fffff800`8acaf8f0 mrvlpcie8897+0×12f8f0
fffff800`8ab8fdad mrvlpcie8897+0xfdad
fffff800`e1eba502 nt! ?? ::FNODOBFM::`string’+0×2
fffff800`8abb000d mrvlpcie8897+0×3000d
fffff800`8abc4510 mrvlpcie8897+0×44510
fffff800`8acaf8f0 mrvlpcie8897+0×12f8f0
fffff800`8ab883d6 mrvlpcie8897+0×83d6
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8ab9311b mrvlpcie8897+0×1311b
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8abb7c40 mrvlpcie8897+0×37c40
fffff800`8abaf470 mrvlpcie8897+0×2f470
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8adea5e0 wdiwifi!WPP_fabfc031111e31c4b597567128b91120_Traceguids
fffff800`8ad0832b wdiwifi!CTxMgr::AddNBLToTxQueue+0×2bb
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`8ad06051 wdiwifi!CTxMgr::ServiceQueues+0×1c1
fffff800`75518cc8 fwpkclnt!FwpiGetValueFromClassifyContext+0×38
fffff800`8adfd040 wdiwifi!WPP_RECORDER_INITIALIZED
fffff800`8ad06599 wdiwifi!CPort::SendNetBufferLists+0×129
fffff800`786a2015 Ndu!NduInboundMacClassify+0×355
fffff800`8adea5f0 wdiwifi!WPP_de984c7e04793f3292dfaa0cae396821_Traceguids
fffff800`754a6000 NETIO!WPP_GLOBAL_Control
fffff800`75434553 NETIO!ProcessCallout2+0×163
fffff800`75463246 NETIO!KfdClassify2+0xbb6
fffff800`e26ee9c0 nt!ExPoolState+0×86940
fffff800`e1a3ecb4 nt!ExAllocateHeapPool+0×2134
fffff800`7527bfff ndis!ndisInvokeNextSendHandler+0×23f
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`e1a3cb12 nt!ExpAllocatePoolWithTagFromNode+0×52
fffff800`e2369189 nt!ExAllocatePool2+0×99
fffff800`e23690b4 nt!ExAllocatePoolWithTag+0xa4
fffff800`e1c4ea7d nt!ExAllocatePoolEx+0xd
fffff800`e2369189 nt!ExAllocatePool2+0×99
fffff800`e1bef072 nt!ExAllocateFromLookasideListEx+0×152
fffff800`91055984 bridge+0×5984
fffff800`9105dd94 bridge+0xdd94
fffff800`75451dc4 NETIO!PplpGenericAllocateFunction+0×14
fffff800`e1beef35 nt!ExAllocateFromLookasideListEx+0×15
fffff800`e1a8f495 nt!ObfReferenceObjectWithTag+0×25
fffff800`e23690b4 nt!ExAllocatePoolWithTag+0xa4
fffff800`75416008 NETIO!WfpNblInfoAlloc+0×58
fffff800`e1a8eace nt!ObfReferenceObject+0xe
fffff800`75514e63 fwpkclnt!FwppNetBufferListAssociateContext+0×153
fffff800`e1a3ecb4 nt!ExAllocateHeapPool+0×2134
fffff800`75514cc7 fwpkclnt!FwpsNetBufferListAssociateContext1+0×77
fffff800`786b42d8 Ndu!NduWfpCalloutProviderGuid
fffff800`786a1280 Ndu!NduNblNotifyCallback
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`786a2613 Ndu!NduFindOrAssociateNblContext+0×163
fffff800`75273918 ndis!NdisFIndicateReceiveNetBufferLists+0×68
fffff800`786a1280 Ndu!NduNblNotifyCallback
fffff800`754a6000 NETIO!WPP_GLOBAL_Control
fffff800`75988200 wfplwfs!L2NativeIsNetBufferListPermitted+0×2d0
fffff800`759897b3 wfplwfs!L2InspectNetBufferListsFast+0×183
fffff800`75463246 NETIO!KfdClassify2+0xbb6
fffff800`7609e7f3 afd!AFDETW_TRACESENDMSG+0×8f
fffff800`e2369189 nt!ExAllocatePool2+0×99
fffff800`75288882 ndis!NdisAcquireReadWriteLock+0×62
fffff800`91055a73 bridge+0×5a73
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`9105662a bridge+0×662a
fffff800`91057920 bridge+0×7920
fffff800`91057db5 bridge+0×7db5
fffff800`752888d3 ndis!NdisAcquireReadWriteLock+0xb3
fffff800`910543b0 bridge+0×43b0
fffff800`91054448 bridge+0×4448
fffff800`75988260 wfplwfs!LwfLowerRecvNetBufferLists
fffff800`75276dc1 ndis!NdisMIndicateReceiveNetBufferLists+0×1941
fffff800`7525b678 ndis!NdisFreeNetBufferList+0xd8
fffff800`786a38ff Ndu!NduDeleteNblContext+0×9f
fffff800`75419dd4 NETIO!NetioDereferenceNetBufferListChain+0×174
fffff800`756b95c0 tcpip!UdpSendMessagesDatagramsComplete
fffff800`786804d5 NdisImPlatform!implatUpdateInStatisticsCounters+0×235
fffff800`e1bf2146 nt!HalpApicRequestInterrupt+0×96
fffff800`e1b5141c nt!HalpInterruptSendIpi+0xac
fffff800`e1cf553c nt!KiSetProcessorIdle_LockFree+0×2b8
fffff800`e1ae22f8 nt!KiHeteroSelectIdleProcessorFromSubNode+0×308
fffff800`7867f6c7 NdisImPlatform!implatReceiveNetBufferLists+0×1f7
fffff800`e1ae9a0b nt!KiComputeThreadQos+0xfb
fffff800`e1bf2146 nt!HalpApicRequestInterrupt+0×96
fffff800`e1b5141c nt!HalpInterruptSendIpi+0xac
fffff800`e1cf553c nt!KiSetProcessorIdle_LockFree+0×2b8
fffff800`e1ae22f8 nt!KiHeteroSelectIdleProcessorFromSubNode+0×308
fffff800`e1ae9a0b nt!KiComputeThreadQos+0xfb
fffff800`e1dda677 nt!PpmEventTraceCoreParkingSelection+0×197
fffff800`e1b50cfc nt!KiExitDispatcher+0×4c
fffff800`e1cf277c nt!PpmParkComputeUnparkMask+0xa2c
fffff800`e1bb2241 nt!KiIntSteerCalculatePriorityDistribution+0×201
fffff800`e1bb361d nt!KiIntSteerLogMask+0×55
fffff800`e270f7b0 nt!KiIntTrackRootList
fffff800`e1bb3699 nt!KiIntSteerLogProc+0×5d
fffff800`e270f7b0 nt!KiIntTrackRootList
fffff800`e1bb37e3 nt!KiIntSteerCalculateDistribution+0×103
fffff800`e1bb32c3 nt!KeIntSteerPeriodic+0×17f
fffff800`e1bb2ed8 nt!PpmParkSteerInterrupts+0×5e8
fffff800`e1b335a4 nt!EtwpLogKernelEvent+0×2f4
fffff800`e270b0a8 nt!PpmPerfPolicyLock+0×8
fffff800`e1b2db6f nt!KeSetEvent+0×10f
fffff800`e270b0ac nt!PpmPerfPolicyLock+0xc
fffff800`e1be8f10 nt!PpmCheckMakeupSkippedChecks
fffff800`e1be9040 nt!PpmPerfReadFeedback
fffff800`e1be9177 nt!PpmReleaseLock+0×1b
fffff800`e1b32bc6 nt!KiExecuteAllDpcs+0xdc6
fffff800`e2709dc0 nt!PpmCheckDpc
fffff800`e1a0cfb9 nt!KiNormalPriorityReadyScan+0×2b9
fffff800`e1be8e00 nt!PpmCheckRun
fffff800`e1a0c228 nt!KiRetireDpcList+0×668
fffff800`e1bb4180 nt!PpmPerfAction
fffff800`e1be8e00 nt!PpmCheckRun
fffff800`e1eac3c5 nt!KxSwapStacksAndRetireDpcList+0×5

0: kd> !prcb 1
PRCB for Processor 1 at ffff84014911c180:
Current IRQL — 0
Threads– Current ffffc901a7d580c0 Next 0000000000000000 Idle ffffc9019375f040
Processor Index 1 Number (0, 1) GroupSetMember 2
Interrupt Count — 057d181c
Times — Dpc 00005ae1 Interrupt 000072cb
Kernel 00cf00e2 User 000d7983

0: kd> dt nt!_KPRCB ffff84014911c180 DpcStack
+0×38a0 DpcStack : 0xffffa206`68e47fb0 Void

0: kd> dpS 0xffffa206`68e47fb0-7000 L7000/8
fffff800`e1adfa8f nt!MiFlushTbList+0×35f
fffff800`e1b0b02c nt!MiGetPage+0×8dc
fffff800`e1b158c5 nt!MiFlushTbAsNeeded+0×265
fffff800`e1a636e0 nt!MiAssignNonPagedPoolPte+0×110
fffff800`e2638180 nt!MiState+0xb940
fffff800`e1a63fcb nt!MiReturnExcessPoolCommit+0×27
fffff800`e1a631c6 nt!MiCommitPoolMemory+0×1b6
fffff800`e1a62c6b nt!RtlpHpEnvAllocVA+0×22b
fffff800`e1e9fb70 nt!HvlEndSystemInterrupt
fffff800`e1c20b6a nt!HalPerformEndOfInterrupt+0×1a
fffff800`e1ea6feb nt!KiInterruptDispatchNoLockNoEtw+0×5b
fffff800`e2638180 nt!MiState+0xb940
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`e1bac123 nt!RtlpHpAllocVA+0xd7
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`e1e9fb70 nt!HvlEndSystemInterrupt
fffff800`8b732bc2 igdkmd64+0×392bc2
fffff800`e1ae3a2f nt!KiSelectProcessorToPreempt+0xff
fffff800`8b73113c igdkmd64+0×39113c
fffff800`8b48f165 igdkmd64+0xef165
fffff800`8b57aeaa igdkmd64+0×1daeaa
fffff800`e27cfbc0 nt!ExNode0
fffff800`e2615740 nt!KiInitialNodeStructures+0×40
fffff800`e1ae3a2f nt!KiSelectProcessorToPreempt+0xff
fffff800`e27cfbc0 nt!ExNode0
fffff800`e1bf2146 nt!HalpApicRequestInterrupt+0×96
fffff800`e27cfbc0 nt!ExNode0
fffff800`e1b5141c nt!HalpInterruptSendIpi+0xac
fffff800`e1cf553c nt!KiSetProcessorIdle_LockFree+0×2b8
fffff800`e1ae22f8 nt!KiHeteroSelectIdleProcessorFromSubNode+0×308
fffff800`e1ae5cff nt!KiUpdateSoftParkElectionStatisticsOnInsertion+0×16f
fffff800`e1ae9a0b nt!KiComputeThreadQos+0xfb
fffff800`e1b51124 nt!KiProcessThreadWaitList+0×224
fffff800`e1beef35 nt!ExAllocateFromLookasideListEx+0×15
fffff800`786a1aee Ndu!NduUpdateInterfacePowerContext+0×1be
fffff800`786aa0b4 Ndu!PplpGenericAllocateFunction+0×14
fffff800`e1b2db6f nt!KeSetEvent+0×10f
fffff800`e1b3143c nt!KeAcquireSpinLockAtDpcLevel+0×1c
fffff800`786a406a Ndu!NduUpdateProcessEnergyContext+0×12a
fffff800`7525b85d ndis!ndisFreeToLookasideList+0×5d
fffff800`7525b645 ndis!NdisFreeNetBufferList+0xa5
fffff800`75451260 NETIO!NetioFreeNetBufferAndNetBufferList+0×10
fffff800`75611fb1 tcpip!TcpSendDatagramsComplete+0xd1
fffff800`786a2f9f Ndu!NduHandleNblContextRemoved+0×1b3
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`75611ee0 tcpip!TcpSendDatagramsComplete
fffff800`75419dd4 NETIO!NetioDereferenceNetBufferListChain+0×174
fffff800`75512f51 fwpkclnt!FwppNetBufferListEventNotify+0×1a1
fffff800`7571ca2d tcpip!FlSendNetBufferListChainComplete+0×6d
fffff800`7527258b ndis!ndisMSendCompleteNetBufferListsInternal+0×25b
fffff800`91054a30 bridge+0×4a30
fffff800`75272330 ndis!ndisMSendCompleteNetBufferListsInternal
fffff800`75287472 ndis!NdisMSendNetBufferListsComplete+0×5c2
fffff800`75260993 ndis!NdisFSendNetBufferListsComplete+0×383
fffff800`7867fe70 NdisImPlatform!implatSendNetBufferListsComplete+0×1a0
fffff800`78016d00 nwifi!Dot11SendNBComplete+0×170
fffff800`7527258b ndis!ndisMSendCompleteNetBufferListsInternal+0×25b
fffff800`7615aff9 vwififlt!FilterSendNetBufferListsCompleteWDI+0xd9
fffff800`75272330 ndis!ndisMSendCompleteNetBufferListsInternal
fffff800`75287472 ndis!NdisMSendNetBufferListsComplete+0×5c2
fffff800`8b739bc0 igdkmd64+0×399bc0
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`8ad0b2cd wdiwifi!CPort::CompletePendingCancelSendsOrHaltJobs+0xdd
fffff800`8adea5f0 wdiwifi!WPP_de984c7e04793f3292dfaa0cae396821_Traceguids
fffff800`8ad09710 wdiwifi!CTxMgr::CompleteNdisNbl+0×250
fffff800`8adfd040 wdiwifi!WPP_RECORDER_INITIALIZED
fffff800`8b48f0c8 igdkmd64+0xef0c8
fffff800`8b48f0c8 igdkmd64+0xef0c8
fffff800`8adea5e0 wdiwifi!WPP_fabfc031111e31c4b597567128b91120_Traceguids
fffff800`8adfd040 wdiwifi!WPP_RECORDER_INITIALIZED
fffff800`8ad07570 wdiwifi!CTxMgr::TxTransferCompleteInd+0×2f0
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`73de6345 Wdf01000!imp_WdfSpinLockRelease+0×95 [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 180]
fffff800`73de6345 Wdf01000!imp_WdfSpinLockRelease+0×95 [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 180]
fffff800`8abab1d3 mrvlpcie8897+0×2b1d3
fffff800`73de62b0 Wdf01000!imp_WdfSpinLockRelease [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 159]
fffff800`8ab94257 mrvlpcie8897+0×14257
fffff800`73de62b0 Wdf01000!imp_WdfSpinLockRelease [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 159]
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8abac585 mrvlpcie8897+0×2c585
fffff800`786a2d6b Ndu!NduIsL2MediaTypeWan+0×3b
fffff800`8b625db3 igdkmd64+0×285db3
fffff800`8abb72a0 mrvlpcie8897+0×372a0
fffff800`8aba71db mrvlpcie8897+0×271db
fffff800`754a6000 NETIO!WPP_GLOBAL_Control
fffff800`8ab93f3e mrvlpcie8897+0×13f3e
fffff800`75463246 NETIO!KfdClassify2+0xbb6
fffff800`8b6ce960 igdkmd64+0×32e960
fffff800`8b6cfdf2 igdkmd64+0×32fdf2
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`8ab93e4a mrvlpcie8897+0×13e4a
fffff800`e1beef35 nt!ExAllocateFromLookasideListEx+0×15
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`73de6345 Wdf01000!imp_WdfSpinLockRelease+0×95 [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 180]
fffff800`8ab8c4f4 mrvlpcie8897+0xc4f4
fffff800`73de62b0 Wdf01000!imp_WdfSpinLockRelease [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 159]
fffff800`8adea5e0 wdiwifi!WPP_fabfc031111e31c4b597567128b91120_Traceguids
fffff800`73e18d40 Wdf01000!imp_WdfMemoryGetBuffer+0×60 [minkernel\wdf\framework\shared\core\fxmemorybufferapi.cpp @ 204]
fffff800`8ab8f1ca mrvlpcie8897+0xf1ca
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`73de6345 Wdf01000!imp_WdfSpinLockRelease+0×95 [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 180]
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`73de6345 Wdf01000!imp_WdfSpinLockRelease+0×95 [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 180]
fffff800`8ab92a47 mrvlpcie8897+0×12a47
fffff800`8ab87de2 mrvlpcie8897+0×7de2
fffff800`73de62b0 Wdf01000!imp_WdfSpinLockRelease [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 159]
fffff800`8ab9311b mrvlpcie8897+0×1311b
fffff800`73de62b0 Wdf01000!imp_WdfSpinLockRelease [minkernel\wdf\framework\shared\support\fxspinlockapi.cpp @ 159]
fffff800`8adea5e0 wdiwifi!WPP_fabfc031111e31c4b597567128b91120_Traceguids
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`8ad06051 wdiwifi!CTxMgr::ServiceQueues+0×1c1
fffff800`8adfd040 wdiwifi!WPP_RECORDER_INITIALIZED
fffff800`8ad06599 wdiwifi!CPort::SendNetBufferLists+0×129
fffff800`8adea5f0 wdiwifi!WPP_de984c7e04793f3292dfaa0cae396821_Traceguids
fffff800`7527eac1 ndis!ndisWdmSetBusyAsync+0×101
fffff800`8adfd040 wdiwifi!WPP_RECORDER_INITIALIZED
fffff800`8ad05c30 wdiwifi!MPWrapperSendNetBufferLists+0×160
fffff800`8ad05ad0 wdiwifi!MPWrapperSendNetBufferLists
fffff800`75280dc2 ndis!ndisMSendNBLToMiniportInternal+0×122
fffff800`8ad05ad0 wdiwifi!MPWrapperSendNetBufferLists
fffff800`76158ab3 vwififlt!FilterSendNetBufferListsWDI+0×1c3
fffff800`75280c80 ndis!ndisMSendNBLToMiniport
fffff800`75280c8e ndis!ndisMSendNBLToMiniport+0xe
fffff800`75988620 wfplwfs!LwfLowerSendNetBufferLists
fffff800`7527bf92 ndis!ndisInvokeNextSendHandler+0×1d2
fffff800`7525feed ndis!NdisFSendNetBufferLists+0×3bd
fffff800`759897b3 wfplwfs!L2InspectNetBufferListsFast+0×183
fffff800`75260564 ndis!FILTER_TEST_FLAG+0×14
fffff800`753325f2 ndis!NdisSendNetBufferLists+0xc1372
fffff800`78014750 nwifi!FilterSendNetBufferLists
fffff800`8b78c622 igdkmd64+0×3ec622
fffff800`759887e8 wfplwfs!LwfLowerSendNetBufferLists+0×1c8
fffff800`910581d1 bridge+0×81d1
fffff800`910559c5 bridge+0×59c5
fffff800`9105dd94 bridge+0xdd94
fffff800`e1adfa8f nt!MiFlushTbList+0×35f
fffff800`7867f394 NdisImPlatform!implatPrepareForSendNetBufferLists+0xec
fffff800`7867fc52 NdisImPlatform!implatSendNetBufferLists+0×182
fffff800`9105ce01 bridge+0xce01
fffff800`7867fad0 NdisImPlatform!implatSendNetBufferLists
fffff800`75280dc2 ndis!ndisMSendNBLToMiniportInternal+0×122
fffff800`9105d4ae bridge+0xd4ae
fffff800`e1b0b02c nt!MiGetPage+0×8dc
fffff800`7867fad0 NdisImPlatform!implatSendNetBufferLists
fffff800`e1b158c5 nt!MiFlushTbAsNeeded+0×265
fffff800`e1a636e0 nt!MiAssignNonPagedPoolPte+0×110
fffff800`e2638180 nt!MiState+0xb940
fffff800`e1a63fcb nt!MiReturnExcessPoolCommit+0×27
fffff800`e1a631c6 nt!MiCommitPoolMemory+0×1b6
fffff800`e1a62c6b nt!RtlpHpEnvAllocVA+0×22b
fffff800`e2638180 nt!MiState+0xb940
fffff800`e1bac123 nt!RtlpHpAllocVA+0xd7
fffff800`e1babdb7 nt!RtlpHpVaMgrCtxQuery+0×4b
fffff800`e1bab920 nt!RtlpHpSegMgrCommit+0×228
fffff800`e1ae3a2f nt!KiSelectProcessorToPreempt+0xff
fffff800`e1bf2146 nt!HalpApicRequestInterrupt+0×96
fffff800`e1ba7e6c nt!RtlpHpLfhSlotAllocateSlow+0×484
fffff800`e27cfbc0 nt!ExNode0
fffff800`e1b5141c nt!HalpInterruptSendIpi+0xac
fffff800`e1cf553c nt!KiSetProcessorIdle_LockFree+0×2b8
fffff800`e1ae22f8 nt!KiHeteroSelectIdleProcessorFromSubNode+0×308
fffff800`e1ae5cbc nt!KiUpdateSoftParkElectionStatisticsOnInsertion+0×12c
fffff800`e1ae9a0b nt!KiComputeThreadQos+0xfb
fffff800`e1dda677 nt!PpmEventTraceCoreParkingSelection+0×197
fffff800`e1b51124 nt!KiProcessThreadWaitList+0×224
fffff800`e1b50cfc nt!KiExitDispatcher+0×4c
fffff800`e1cf277c nt!PpmParkComputeUnparkMask+0xa2c
fffff800`e1b3149e nt!KeAcquireSpinLockRaiseToDpc+0×3e
fffff800`e1bb2241 nt!KiIntSteerCalculatePriorityDistribution+0×201
fffff800`e1bb361d nt!KiIntSteerLogMask+0×55
fffff800`e270f7b0 nt!KiIntTrackRootList
fffff800`e1bb3699 nt!KiIntSteerLogProc+0×5d
fffff800`e270f7b0 nt!KiIntTrackRootList
fffff800`e1bb37e3 nt!KiIntSteerCalculateDistribution+0×103
fffff800`e1bb32c3 nt!KeIntSteerPeriodic+0×17f
fffff800`e1bb2ed8 nt!PpmParkSteerInterrupts+0×5e8
fffff800`e1c594e9 nt!HvlUpdatePerformanceStateCountersForLp+0×79
fffff800`776b2781 intelppm!PerfHvReadFeedback+0×61
fffff800`e1cf553c nt!KiSetProcessorIdle_LockFree+0×2b8
fffff800`e1ae22f8 nt!KiHeteroSelectIdleProcessorFromSubNode+0×308
fffff800`e1a0afa5 nt!KiUpdateThreadQosGroupingSummaries+0×75
fffff800`e1a0a927 nt!KiCommitRescheduleContextEntry+0×1e7
fffff800`e27d1183 nt!KiInitialSharedReadyQueue+0×243
fffff800`e1ae9a0b nt!KiComputeThreadQos+0xfb
fffff800`e1b3673a nt!KiDeferredReadySingleThread+0×29fa
fffff800`e1a0f077 nt!PpmUpdatePerformanceFeedback+0×3b7
fffff800`e1b51006 nt!KiProcessThreadWaitList+0×106
fffff800`e1b335a4 nt!EtwpLogKernelEvent+0×2f4
fffff800`e1b2db6f nt!KeSetEvent+0×10f
fffff800`e1bb7182 nt!PopQueueTargetDpc+0xee
fffff800`e1b32bc6 nt!KiExecuteAllDpcs+0xdc6
fffff800`e1bb6680 nt!PopExecuteProcessorCallback
fffff800`e1bb6680 nt!PopExecuteProcessorCallback
fffff800`e1a0c228 nt!KiRetireDpcList+0×668
fffff800`e1bb6680 nt!PopExecuteProcessorCallback
fffff800`e1eac3c5 nt!KxSwapStacksAndRetireDpcList+0×5

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Message Embedding, as a representational technique in ML, are a variant of Trace Field. We can also consider the sequence of Message Embeddings as a trace itself with columns as latent features, forming separate latent Features of Activity. We can also treat these embeddings as sentence embeddings when interpreting traces and logs as Text Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We can view traces and logs as abstract polynomials that consists of abstract monomials. For example, if we have trace messages A,B,C, and D, the trace ABCACACACCD represents a single monomial. The multiplication operation in monomials represents message concatenation. But we can also split the trace as an abstract sum of several monomials, for example, ABC + AC + AC + AC + CD, or ABC + 3*AC + CD. The addition operation is a concatenation of traces even if concatenated traces consist of just one message. Note the distinction here between concatenation of messages and traces. By Trace Polynomial we mean a canonical abstract polynomial representation where we divide the trace by monomial when the next message in the message stream is already contained in the previous monomial, for example, ABC + 2AC + AC^2D.

Both addition and multiplication are non-commutative, and no distributivity between them. Mathematically speaking, we have the so-called a non‑distributive bi‑semigroup, or, in a category-theoretic sense, such abstract polynomials are objects in a free 2‑semigroupal category without interchange.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of collection-related analysis patterns is necessary, so I created this post:

• Stack Trace Collection (unmanaged space)
• Stack Trace Collection (managed space)
• Stack Trace Collection (predicate)
• Stack Trace Collection (I/O requests)
• Stack Trace Collection (CPUs)
• Last Error Collection
• Module Collection
• Module Collection (predicate)
• Structure Field Collection
• Rough Stack Trace Collection (unmanaged space)
• Exception Collection
• Interrupt Stack Collection
• DPC Stack Collection

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Interrupt Stack Collection is another area to mine for Execution Residue and Rough Stack Traces. Some Interrupt Stacks may be visible in Stack Trace Collections such as from CPUs. In addition to Stack Overflow double fault stack region, we also have debug, NMI, and machine check interrupt stack 6Kb regions:

6: kd> !idt

Dumping IDT: ffffbd014d6b1000

00: fffff806f53ad100 nt!KiDivideErrorFaultShadow
01: fffff806f53ad180 nt!KiDebugTrapOrFaultShadow Stack = 0xFFFFBD014D6B59D0
02: fffff806f53ad240 nt!KiNmiInterruptShadow Stack = 0xFFFFBD014D6B57D0
03: fffff806f53ad2c0 nt!KiBreakpointTrapShadow
04: fffff806f53ad340 nt!KiOverflowTrapShadow
05: fffff806f53ad3c0 nt!KiBoundFaultShadow
06: fffff806f53ad440 nt!KiInvalidOpcodeFaultShadow
07: fffff806f53ad4c0 nt!KiNpxNotAvailableFaultShadow
08: fffff806f53ad540 nt!KiDoubleFaultAbortShadow Stack = 0xFFFFBD014D6B53D0
09: fffff806f53ad5c0 nt!KiNpxSegmentOverrunAbortShadow
0a: fffff806f53ad640 nt!KiInvalidTssFaultShadow
0b: fffff806f53ad6c0 nt!KiSegmentNotPresentFaultShadow
0c: fffff806f53ad740 nt!KiStackFaultShadow
0d: fffff806f53ad7c0 nt!KiGeneralProtectionFaultShadow
0e: fffff806f53ad840 nt!KiPageFaultShadow
10: fffff806f53ad8c0 nt!KiFloatingErrorFaultShadow
11: fffff806f53ad940 nt!KiAlignmentFaultShadow
12: fffff806f53ad9c0 nt!KiMcheckAbortShadow Stack = 0xFFFFBD014D6B55D0
13: fffff806f53adac0 nt!KiXmmExceptionShadow
[…]

These stacks are different for each CPU. It is also possible to get these stack bases from TSS:

6: kd> ~0s

0: kd> !pcr
KPCR for Processor 0 at fffff80680079000:
Major 1 Minor 1
NtTib.ExceptionList: fffff8068743efb0
NtTib.StackBase: fffff8068743d000
NtTib.StackLimit: 0000000000000000
NtTib.SubSystemTib: fffff80680079000
NtTib.Version: 0000000080079180
NtTib.UserPointer: fffff80680079870
NtTib.SelfTib: 00000060414a8000

SelfPcr: 0000000000000000
Prcb: fffff80680079180
Irql: 0000000000000000
IRR: 0000000000000000
IDR: 0000000000000000
InterruptMode: 0000000000000000
IDT: 0000000000000000
GDT: 0000000000000000
TSS: 0000000000000000

CurrentThread: ffffa80b0c8240c0
NextThread: 0000000000000000
IdleThread: fffff806f57d0640

DpcQueue:

0: kd> dt nt!_KPCR fffff80680079000
nt!_KPCR
+0×000 NtTib : _NT_TIB
+0×000 GdtBase : 0xfffff806`8743efb0 _KGDTENTRY64
+0×008 TssBase : 0xfffff806`8743d000 _KTSS64
+0×010 UserRsp : 0
+0×018 Self : 0xfffff806`80079000 _KPCR
+0×020 CurrentPrcb : 0xfffff806`80079180 _KPRCB
+0×028 LockArray : 0xfffff806`80079870 _KSPIN_LOCK_QUEUE
+0×030 Used_Self : 0×00000060`414a8000 Void
+0×038 IdtBase : 0xfffff806`8743c000 _KIDTENTRY64
+0×040 Unused : [2] 0
+0×050 Irql : 0 ”
+0×051 SecondLevelCacheAssociativity : 0×10 ”
+0×052 ObsoleteNumber : 0 ”
+0×053 Fill0 : 0 ”
+0×054 Unused0 : [3] 0
+0×060 MajorVersion : 1
+0×062 MinorVersion : 1
+0×064 StallScaleFactor : 0×840
+0×068 Unused1 : [3] (null)
+0×080 KernelReserved : [15] 0
+0×0bc SecondLevelCacheSize : 0×800000
+0×0c0 HalReserved : [16] 0×7de29000
+0×100 Unused2 : 0
+0×108 KdVersionBlock : (null)
+0×110 Unused3 : (null)
+0×118 PcrAlign1 : [24] 0

0: kd> dt nt!_KTSS64 0xfffff806`8743d000
nt!_KTSS64
+0×000 Reserved0 : 0
+0×004 Rsp0 : 0xfffff806`87440200
+0×00c Rsp1 : 0
+0×014 Rsp2 : 0
+0×01c Ist : [8] 0
+0×05c Reserved1 : 0
+0×064 Reserved2 : 0
+0×066 IoMapBase : 0×68

0: kd> dps 0xfffff806`8743d000+1c L8
fffff806`8743d01c 00000000`00000000
fffff806`8743d024 fffff806`874403d0
fffff806`8743d02c fffff806`874405d0
fffff806`8743d034 fffff806`874407d0
fffff806`8743d03c fffff806`874409d0
fffff806`8743d044 00000000`00000000
fffff806`8743d04c 00000000`00000000
fffff806`8743d054 00000000`00000000

0: kd> !idt 2

Dumping IDT: fffff8068743c000

02: fffff806f53ad240 nt!KiNmiInterruptShadow Stack = 0xFFFFF806874407D0

These stack base values may be transition stack values. In such a case, a redirection is required:

0: kd> dps fffff806`874407d0 L4
fffff806`874407d0 fffff806`80079000
fffff806`874407d8 fffff806`87471fe0
fffff806`874407e0 fffff806`80079000
fffff806`874407e8 00000004`237bf002

0: kd> dpS fffff806`87471fe0+20-6000 L6000/8
fffff806`f4dcd566 nt!KiSaveProcessorState+0xb6
fffff806`f4dc588a nt!KiFreezeTargetExecution+0×1ba
fffff806`f4db72ea nt!KiCheckForFreezeExecution+0×2a
fffff806`f4dbb242 nt!KiProcessNMI+0×52
fffff806`f4eb0fc2 nt!KxNmiInterrupt+0×82
fffff806`f4dcd124 nt!KiMcheckFastForward+0×64

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Trace Precision and Recall analysis patterns are borrowed from information retrieval and machine learning. In essence, Trace Precision is the fraction of trace messages used for successful diagnostics among all selected trace messages, for example, among CoTrace messages to reduce the noise. Trace Recall is the fraction of trace messages used for successful diagnostics among all relevant messages (these can also be messages not included in the trace, see Trace Bias).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Traces and logs can be considered models of computation as was first proposed in Trace Bias analysis pattern. Borrowing metaphors from machine learning we can consider trace messages as Trace Parameters having associated values, for example, through Trace Field. These values can be numerical weights or linguistic values, for example, as proposed by Zadeh. Message values can be field values themselves, for example, Counter Values. As was noted in Trace Variance, different tracing environments with their set of “features” can be considered input data. if we use an artificial neural network metaphor, trace analysis itself can be considered as an activation function. CoTraces, as models of trace analysis, may form other “hidden” layers with “backpropagation” updating message weights. Tracing configuration parameters including Trace Schemas can be considered as Trace Hyperparameters. These concepts can be roughly illustrated by the following diagram:

To reduce the number of Trace Parameters we can use Quotient Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In some cases, when we have two trace messages we can infer the missing Implicit Message. This trace augmentation may help when we Sparse Traces and we want to train ML models on them. To illustrate, we asked ChatGPT:

We have a log file with two messages: 1. Reading file test.txt 2. Access denied. There is an implicit message in between.

The implicit message between the two log entries is:

“Attempting to open file ‘test.txt’.”

Reasoning:

1. “Reading file test.txt” → This suggests that the system or user has issued a command to read the file.
2. Implicit step (not logged explicitly) → The system attempts to open the file to read its contents.
3. “Access denied.” → The attempt fails due to permission restrictions or security policies.

Thus, the missing step is the system trying to open or access the file, which leads to the access denial.

Implicit Messages are different from Silent Messages which are padding messages to occupy the minimum resolution gaps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The fact that messages have types is mentioned in some trace and log analysis patterns but have never had its own analysis pattern. Trace and log messages may have the same Message Type if they have the same Adjoint Thread of Activity ATID (or Message Metadata) value, for example, “Warning”. Or they may have the same structure, for example, the same constants or message invariants, or belong to the same Trace Schema in Container Traces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Terminated threads are not listed in unmanaged space Stack Trace Collections. In kernel space, we may notice them if we expect N kernel threads but see less like Missing Threads in user space. If we see less kernel threads in a process context then, definitely, user space counterparts to Dual Stack Traces are missing (but we may still recover Hidden Stacks). Sometimes, using appropriate extensions, like SwishDbgExt, we can see terminated threads based on exit time:

0: kd> !ms_process /pid 0x250 /threads
[...]
| 0x0250 | 0x02a0 | 0x00007FFC858FE680 | winsrvext!TerminalServerRequestThread | 13/11/2021 22:14:28 | 00/00/ 0 00:00:00 |
| 0×0250 | 0×02a4 | 0×00007FFC858F2710 | winsrvext!GdiAddInitialFontsThread | 13/11/2021 22:14:28 | 13/11/2021 22:14:29 |
| 0×0250 | 0×02a8 | 0×00007FFC858F3430 | winsrvext!NotificationThread | 13/11/2021 22:14:28 | 00/00/ 0 00:00:00 |
[…]

If we get thread ids from some Paratext, we can directly check if the thread is terminated or not:

0: kd> !thread -t 2a4 3f
THREAD ffffc38c3040e080 Cid 0250.02a4 Teb: 0000000000000000 Win32Thread: 0000000000000000 TERMINATED
Not impersonating
DeviceMap ffffac8a0423d290
Owning Process ffffc38c30880140 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1282 Ticks: 10674 (0:00:02:46.781)
Context Switch Count 1192 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.078
Win32 Start Address winsrvext!GdiAddInitialFontsThread (0×00007ffc858f2710)
Stack Init 0000000000000000 Current ffffbe8295331670
Base ffffbe8295332000 Limit ffffbe829532c000 Call 0000000000000000
Priority 14 BasePriority 13 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffbe82`953316b0 fffffc57`1e5ba085 0×4
ffffbe82`953316b8 fffff806`6255f501 0xfffffc57`1e5ba085
ffffbe82`953316c0 000002ac`02048e80 nt!PspThreadFromTicket+0×51
ffffbe82`953316f0 ffffffff`ffffffff 0×000002ac`02048e80
ffffbe82`953316f8 ffffbe82`95331b60 0xffffffff`ffffffff
ffffbe82`95331700 ffffbe82`953319a0 0xffffbe82`95331b60
ffffbe82`95331708 fffff806`62136778 0xffffbe82`953319a0
ffffbe82`95331710 fffff806`62138fdc nt!IoRemoveIoCompletion+0×98
ffffbe82`95331830 fffff806`62227b75 nt!NtWaitForWorkViaWorkerFactory+0×39c
ffffbe82`95331a70 00000000`00000000 nt!KiSystemServiceCopyEnd+0×25

Please note that in case of Incorrect Stack Trace we can get Rough Stack Trace or try to reconstruct the one manually from Execution Residue:

0: kd> dpS ffffbe829532c000 ffffbe8295332000
fffff806`6210aeb4 nt!MiGetPerfectColorHeadPage+0×94
fffff806`624e9fa2 nt!PspGetContext+0×2e2
fffff806`62a54e00 nt!MiSystemPartition
fffff806`624e9aba nt!PspGetSetContextInternal+0×3aa
fffff806`624e9aba nt!PspGetSetContextInternal+0×3aa
fffff806`621090b1 nt!MiAddWorkingSetEntries+0×451
fffff806`62108965 nt!MiAllocateWsle+0×295
fffff806`62a54e00 nt!MiSystemPartition
fffff806`62107eac nt!MiCompletePrivateZeroFault+0×77c
fffff806`62a54e00 nt!MiSystemPartition
fffff806`62107315 nt!MiResolvePrivateZeroFault+0×1a5
fffff806`62105c28 nt!MiResolveDemandZeroFault+0×298
fffff806`62a54e00 nt!MiSystemPartition
fffff806`621290cc nt!MiDispatchFault+0×2ac
fffff806`6221db3d nt!PspGetSetContextSpecialApc+0×6d
fffff806`624ea5fd nt!PspSetContextThreadInternal+0×16d
fffff806`624e9083 nt!PspInitializeThunkContext+0×28f
00007ffc`884b6870 ntdll!TppWorkerThread
00007ffc`884a4830 ntdll!RtlUserThreadStart
fffff806`620d58e4 nt!EtwpEventWriteFull+0×3f4
fffff806`620d58e4 nt!EtwpEventWriteFull+0×3f4
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`6221d818 nt!SwapContext+0×4d8
fffff806`6221d056 nt!KiSwapContext+0×76
fffff806`62132457 nt!KiSwapThread+0×3a7
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`62134309 nt!KiCommitThreadWait+0×159
fffff806`62136d66 nt!KeRemoveQueueEx+0×2b6
fffff806`6255f501 nt!PspThreadFromTicket+0×51
fffff806`62136778 nt!IoRemoveIoCompletion+0×98
fffff806`6256d901 nt!ObpReferenceObjectByHandleWithTag+0×231
fffff806`6256d6be nt!ObReferenceObjectByHandle+0×2e
fffff806`62138fdc nt!NtWaitForWorkViaWorkerFactory+0×39c
fffff806`62227b75 nt!KiSystemServiceCopyEnd+0×25
fffff806`62227b75 nt!KiSystemServiceCopyEnd+0×25
00007ffc`88546f14 ntdll!NtWaitForWorkViaWorkerFactory+0×14

Such Historical Information may help in the reconstruction of past system behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -