Archive for September, 2024

Crash Dump Analysis Patterns (Part 298)

Saturday, September 14th, 2024

Terminated threads are not listed in unmanaged space Stack Trace Collections. In kernel space, we may notice them if we expect N kernel threads but see less like Missing Threads in user space. If we see less kernel threads in a process context then, definitely, user space counterparts to Dual Stack Traces are missing (but we may still recover Hidden Stacks). Sometimes, using appropriate extensions, like SwishDbgExt, we can see terminated threads based on exit time:

0: kd> !ms_process /pid 0x250 /threads
[...]
| 0x0250 | 0x02a0 | 0x00007FFC858FE680 | winsrvext!TerminalServerRequestThread | 13/11/2021 22:14:28 | 00/00/ 0 00:00:00 |
| 0×0250 | 0×02a4 | 0×00007FFC858F2710 | winsrvext!GdiAddInitialFontsThread | 13/11/2021 22:14:28 | 13/11/2021 22:14:29 |
| 0×0250 | 0×02a8 | 0×00007FFC858F3430 | winsrvext!NotificationThread | 13/11/2021 22:14:28 | 00/00/ 0 00:00:00 |
[…]

If we get thread ids from some Paratext, we can directly check if the thread is terminated or not:

0: kd> !thread -t 2a4 3f
THREAD ffffc38c3040e080 Cid 0250.02a4 Teb: 0000000000000000 Win32Thread: 0000000000000000 TERMINATED
Not impersonating
DeviceMap ffffac8a0423d290
Owning Process ffffc38c30880140 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1282 Ticks: 10674 (0:00:02:46.781)
Context Switch Count 1192 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.078
Win32 Start Address winsrvext!GdiAddInitialFontsThread (0×00007ffc858f2710)
Stack Init 0000000000000000 Current ffffbe8295331670
Base ffffbe8295332000 Limit ffffbe829532c000 Call 0000000000000000
Priority 14 BasePriority 13 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffbe82`953316b0 fffffc57`1e5ba085 0×4
ffffbe82`953316b8 fffff806`6255f501 0xfffffc57`1e5ba085
ffffbe82`953316c0 000002ac`02048e80 nt!PspThreadFromTicket+0×51
ffffbe82`953316f0 ffffffff`ffffffff 0×000002ac`02048e80
ffffbe82`953316f8 ffffbe82`95331b60 0xffffffff`ffffffff
ffffbe82`95331700 ffffbe82`953319a0 0xffffbe82`95331b60
ffffbe82`95331708 fffff806`62136778 0xffffbe82`953319a0
ffffbe82`95331710 fffff806`62138fdc nt!IoRemoveIoCompletion+0×98
ffffbe82`95331830 fffff806`62227b75 nt!NtWaitForWorkViaWorkerFactory+0×39c
ffffbe82`95331a70 00000000`00000000 nt!KiSystemServiceCopyEnd+0×25

Please note that in case of Incorrect Stack Trace we can get Rough Stack Trace or try to reconstruct the one manually from Execution Residue:

0: kd> dpS ffffbe829532c000 ffffbe8295332000
fffff806`6210aeb4 nt!MiGetPerfectColorHeadPage+0×94
fffff806`624e9fa2 nt!PspGetContext+0×2e2
fffff806`62a54e00 nt!MiSystemPartition
fffff806`624e9aba nt!PspGetSetContextInternal+0×3aa
fffff806`624e9aba nt!PspGetSetContextInternal+0×3aa
fffff806`621090b1 nt!MiAddWorkingSetEntries+0×451
fffff806`62108965 nt!MiAllocateWsle+0×295
fffff806`62a54e00 nt!MiSystemPartition
fffff806`62107eac nt!MiCompletePrivateZeroFault+0×77c
fffff806`62a54e00 nt!MiSystemPartition
fffff806`62107315 nt!MiResolvePrivateZeroFault+0×1a5
fffff806`62105c28 nt!MiResolveDemandZeroFault+0×298
fffff806`62a54e00 nt!MiSystemPartition
fffff806`621290cc nt!MiDispatchFault+0×2ac
fffff806`6221db3d nt!PspGetSetContextSpecialApc+0×6d
fffff806`624ea5fd nt!PspSetContextThreadInternal+0×16d
fffff806`624e9083 nt!PspInitializeThunkContext+0×28f
00007ffc`884b6870 ntdll!TppWorkerThread
00007ffc`884a4830 ntdll!RtlUserThreadStart
fffff806`620d58e4 nt!EtwpEventWriteFull+0×3f4
fffff806`620d58e4 nt!EtwpEventWriteFull+0×3f4
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`6221d818 nt!SwapContext+0×4d8
fffff806`6221d056 nt!KiSwapContext+0×76
fffff806`62132457 nt!KiSwapThread+0×3a7
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`61e0f808 nt!ThreadWorkOnBehalfUpdate
fffff806`62134309 nt!KiCommitThreadWait+0×159
fffff806`62136d66 nt!KeRemoveQueueEx+0×2b6
fffff806`6255f501 nt!PspThreadFromTicket+0×51
fffff806`62136778 nt!IoRemoveIoCompletion+0×98
fffff806`6256d901 nt!ObpReferenceObjectByHandleWithTag+0×231
fffff806`6256d6be nt!ObReferenceObjectByHandle+0×2e
fffff806`62138fdc nt!NtWaitForWorkViaWorkerFactory+0×39c
fffff806`62227b75 nt!KiSystemServiceCopyEnd+0×25
fffff806`62227b75 nt!KiSystemServiceCopyEnd+0×25
00007ffc`88546f14 ntdll!NtWaitForWorkViaWorkerFactory+0×14

Such Historical Information may help in the reconstruction of past system behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Memory Forensics, x64 Windows | Comments Off