Archive for February, 2020

Crash Dump Analysis Patterns (Part 264)

Thursday, February 27th, 2020

Interrupts can happen in either kernel or user mode. In the latter case, upon transition to kernel mode, a special memory region in is used for interrupt processing in kernel space, distinct from the thread’s kernel stack, that we call Interrupt Stack. It can also be used for mining Execution Residue.

2: kd> !thread -1 1f
THREAD fffffa801a9fa3e0  Cid 0f74.0804  Teb: 000007ffffdf8000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff88000007400
Owning Process            fffffa801a949c10       Image:         App.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      81642662       Ticks: 0
Context Switch Count      58671950       IdealProcessor: 4
UserTime                  01:33:39.702
KernelTime                00:01:11.401
Win32 Start Address 0x000007fef9b1050c
Stack Init fffffa6005af4db0 Current fffffa6005af4950
Base fffffa6005af5000 Limit fffffa6005aef000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`01793b98 fffff800`01a58eee nt!KeBugCheckEx
fffffa60`01793ba0 fffff800`01a57dcb nt!KiBugCheckDispatch+0×6e
fffffa60`01793ce0 fffffa60`00eb279b nt!KiPageFault+0×20b (TrapFrame @ fffffa60`01793ce0)
fffffa60`01793e70 fffffa60`00e62739 tcpip! ?? ::FNODOBFM::`string’+0×3883b
fffffa60`01794020 fffffa60`00e62194 tcpip!TcpMatchReceive+0×1b9
fffffa60`01794120 fffffa60`00e52ddd tcpip!TcpPreValidatedReceive+0×2e4
fffffa60`017941c0 fffffa60`00e52e89 tcpip!IppDeliverListToProtocol+0×4d
fffffa60`01794280 fffffa60`00e52463 tcpip!IppProcessDeliverList+0×59
fffffa60`017942f0 fffffa60`00e5176c tcpip!IppReceiveHeaderBatch+0×223
fffffa60`017943d0 fffffa60`00e50d54 tcpip!IpFlcReceivePackets+0×8dc
fffffa60`017945d0 fffffa60`00e61133 tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0×264
fffffa60`017946b0 fffffa60`009a40bc tcpip!FlReceiveNetBufferListChain+0xd3
fffffa60`01794700 fffffa60`0096c8c9 NDIS!ndisMIndicateNetBufferListsToOpen+0xac
fffffa60`01794750 fffffa60`008016f7 NDIS!ndisMDispatchReceiveNetBufferLists+0×1d9
fffffa60`01794bd0 fffffa60`02b4e2d3 NDIS!NdisMIndicateReceiveNetBufferLists+0×67
fffffa60`01794c10 fffffa60`02b3de0c Driver+0×152d3
fffffa60`01794de0 fffffa60`02b3df6b Driver+0×4e0c
fffffa60`01794e20 fffffa60`02b3e0b3 Driver+0×4f6b
fffffa60`01794e60 fffffa60`00801670 Driver+0×50b3
fffffa60`01794ec0 fffff800`01a5d367 NDIS!ndisInterruptDpc+0xc0
fffffa60`01794f40 fffff800`01a5bc35 nt!KiRetireDpcList+0×117
fffffa60`01794fb0 fffff800`01a5ba47 nt!KyRetireDpcList+0×5 (TrapFrame @ fffffa60`01794e70)
fffffa60`05af4bf0 fffff800`01aa1b28 nt!KiDispatchInterruptContinue
fffffa60`05af4c20 000007fe`f7e5c55a nt!KiDpcInterrupt+0xf8 (TrapFrame @ fffffa60`05af4c20)
00000000`4deae430 00000000`00000000 0×000007fe`f7e5c55a

2: kd> !address fffffa60`01794e60
Usage:
Base Address:           fffffa60`011ff000
End Address:            fffffa60`019dc000
Region Size:            00000000`007dd000

VA Type:                SystemDynamicSpace
VAD Address:            0×27676e69727473
Commit Charge:          0×244a0f51940
Protection:             0×244a0f51940 []
Memory Usage:           Private
No Change:              yes
More info:              !vad 0xfffffa60011ff000

2: kd> !address fffffa60`05af4c20
Usage:                  Stack
Base Address:           fffffa60`05aef000
End Address:            fffffa60`05af5000
Region Size:            00000000`00006000

VA Type:                SystemDynamicSpace

2: kd> dpS fffffa60`01793b98 fffffa60`01794fb0
[…]
fffffa60`05657c3f Driver2+0×4c3f
fffffa60`05656369 Driver2+0×3369

[…]
fffffa60`00801670 NDIS!ndisInterruptDpc+0xc0
fffff800`01a5d367 nt!KiRetireDpcList+0×117
fffff800`01a5bc35 nt!KyRetireDpcList+0×5
fffffa60`008015b0 NDIS!ndisInterruptDpc

2: kd> ub fffffa60`05657c3f
Driver2+0×4c25:
fffffa60`05657c25 8bf2            mov     esi,edx
fffffa60`05657c27 33d2            xor     edx,edx
fffffa60`05657c29 418be8          mov     ebp,r8d
fffffa60`05657c2c 488bd9          mov     rbx,rcx
fffffa60`05657c2f 448d4240        lea     r8d,[rdx+40h]
fffffa60`05657c33 488d48b8        lea     rcx,[rax-48h]
fffffa60`05657c37 418bf9          mov     edi,r9d
fffffa60`05657c3a e8010e0000      call    Driver2+0×5a40 (fffffa60`05658a40)

2: kd> ub fffffa60`05656369
Driver2+0×334d:
fffffa60`0565634d cc              int     3
fffffa60`0565634e cc              int     3
fffffa60`0565634f cc              int     3
fffffa60`05656350 4889542410      mov     qword ptr [rsp+10h],rdx
fffffa60`05656355 48894c2408      mov     qword ptr [rsp+8],rcx
fffffa60`0565635a 4883ec58        sub     rsp,58h
fffffa60`0565635e 488d4c2428      lea     rcx,[rsp+28h]
fffffa60`05656363 ff15972c0000    call    qword ptr [Driver2+0×6000 (fffffa60`05659000)]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Hidden Artifact Patterns

Sunday, February 23rd, 2020

A page to reference all different kinds of patterns related to uncovering hidden entities, artifacts, and actions is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 263)

Friday, February 21st, 2020

Sometimes, when we have One-Thread Process memory dumps, it is possible to get other stack regions indirectly through the analysis of virtual memory regions. Consider, for example, this dump that has only one process exit thread:

0:000> kL
# Child-SP          RetAddr           Call Site
00 000000f1`828ff848 00007ff9`d29aa9b8 ntdll!NtTerminateProcess+0x14
01 000000f1`828ff850 00007ff9`d113cd8a ntdll!RtlExitUserProcess+0xb8
02 000000f1`828ff880 00007ff7`fbb91231 kernel32!ExitProcessImplementation+0xa
03 000000f1`828ff8b0 00007ff7`fbb9125f HiddenStack!bar1+0x41
04 000000f1`828ffa80 00007ff7`fbb91cb5 HiddenStack!foo1+0x1f
05 000000f1`828ffc40 00007ff7`fbb91b1b HiddenStack!std::_Invoker_functor::_Call<void (__cdecl*)(void)>+0x15
06 000000f1`828ffc70 00007ff7`fbb917c4 HiddenStack!std::invoke<void (__cdecl*)(void)>+0x1b
07 000000f1`828ffca0 00007ff7`fbb99728 HiddenStack!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0x64
08 000000f1`828ffcf0 00007ff9`d1137bd4 HiddenStack!thread_start<unsigned int (__cdecl*)(void *),1>+0x50
09 000000f1`828ffd20 00007ff9`d29aced1 kernel32!BaseThreadInitThunk+0x14
0a 000000f1`828ffd50 00000000`00000000 ntdll!RtlUserThreadStart+0x21

There are no more thread stack traces:

0:000> ~
. 0 Id: 27d4.22a4 Suspend: -1 Teb: 000000f1`8266a000 Unfrozen

However, in addition to thread #0, we can find several regions having PAGE_GUARD protection:

0:000> !address
[...]
+       f1`82800000       f1`828fb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~0; 27d4.22a4]
f1`828fb000       f1`828fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          Stack      [~0; 27d4.22a4]
f1`828fe000       f1`82900000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~0; 27d4.22a4]
+       f1`82900000       f1`829fb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`829fb000       f1`829fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`829fe000       f1`82a00000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82a00000       f1`82afc000        0`000fc000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`82afc000       f1`82aff000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`82aff000       f1`82b00000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82b00000       f1`82bfb000        0`000fb000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
f1`82bfb000       f1`82bfe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE|PAGE_GUARD          <unknown>
f1`82bfe000       f1`82c00000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unknown>  […………….]
+       f1`82c00000      1fe`828f0000      10c`ffcf0000             MEM_FREE    PAGE_NOACCESS                      Free
[…]

We then can get Rough Stack Traces out of them:

0:000> .lines -d
Line number information will not be loaded

0:000> dpS f1`829fe000       f1`82a00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d0070000 KERNELBASE!UrlHashW <PERF> (KERNELBASE+0×0)
00007ff9`d007b4b1 KERNELBASE!SetTEBLangID+0×2d
00007ff9`d007ac70 KERNELBASE!_KernelBaseBaseDllInitialize+0×90
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d19e7890 msvcrt!CRTDLL_INIT
00007ff9`d29db5a3 ntdll!RTL_BINARY_ARRAY<RTLP_FLS_SLOT,8,4>::ChunkAllocate+0×67
00007ff9`d19e0000 msvcrt!`dynamic initializer for ‘__ExceptionPtr::m_badAllocExceptionPtr” <PERF> (msvcrt+0×0)
00007ff9`d29db65d ntdll!RTL_BINARY_ARRAY<RTLP_FLS_SLOT,8,4>::SetValue+0×39
00007ff9`d2964ef7 ntdll!RtlDeactivateActivationContextUnsafeFast+0xc7
00007ff9`d299439c ntdll!RtlFlsSetValue+0xec
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00000000`7ffe0301 SharedUserData+0×301
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00000000`7ffe0358 SharedUserData+0×358
00007ff7`fbb923ca HiddenStack!std::chrono::duration_cast<std::chrono::duration<double,
std::ratio<1,1000000000> >,__int64,std::ratio<1,1000000000>,void>+0×4a
00000000`7ffe0358 SharedUserData+0×358
00007ff9`d294bb47 ntdll!RtlGetSystemTimePrecise+0×57
00007ff9`d00b6931 KERNELBASE!SleepEx+0xa1
00007ff9`d00d3890 KERNELBASE!GetSystemTimePreciseAsFileTime+0×10
00007ff7`fbb931b4 HiddenStack!_Thrd_sleep+0×3c
00007ff7`fbb916c5 HiddenStack!std::this_thread::sleep_until<std::chrono::steady_clock,
std::chrono::duration<__int64,std::ratio<1,1000000000> > >+0×65
00007ff7`fbb91651 HiddenStack!std::chrono::operator+<std::chrono::steady_clock,
std::chrono::duration<__int64,std::ratio<1,1000000000> >,__int64,std::ratio<1,1> >+0×41
00007ff7`fbb913fd HiddenStack!std::this_thread::sleep_for<__int64,std::ratio<1,1> >+0×2d
00007ff7`fbb912a9 HiddenStack!bar2+0×39
00007ff7`fbb912df HiddenStack!foo2+0×1f
00007ff7`fbb91cb5 HiddenStack!std::_Invoker_functor::_Call<void (__cdecl*)(void)>+0×15
00007ff7`fbb91aec HiddenStack!std::unique_ptr<std::tuple<void (__cdecl*)(void)>,std::default_delete<std::tuple<void (__cdecl*)(void)> > >::unique_ptr<std::tuple<void (__cdecl*)(void)>,std::default_delete<std::tuple<void (__cdecl*)(void)> > ><std::default_delete<std::tuple<void (__cdecl*)(void)> >,0>+0×2c
00007ff7`fbb91b1b HiddenStack!std::invoke<void (__cdecl*)(void)>+0×1b
00007ff7`fbb917c4 HiddenStack!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0×64
00007ff7`fbb9c1d7 HiddenStack!__acrt_getptd+0xb3
00007ff7`fbb99728 HiddenStack!thread_start<unsigned int (__cdecl*)(void *),1>+0×50
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

0:000> dpS f1`82aff000       f1`82b00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d2962da8 ntdll!LdrpInitializeThread+0×40
00007ff9`d297562f ntdll!TppCallbackCheckThreadAfterCallback+0×9f
00007ff9`d29700e5 ntdll!RtlRegisterThreadWithCsrss+0×35
00007ff9`d29b18f5 ntdll!_LdrpInitialize+0×89
00007ff9`d2975394 ntdll!TppCallbackEpilog+0×144
00007ff9`d29701d6 ntdll!TppCritSetThread+0×7a
00007ff9`d2973155 ntdll!TppWorkCallbackPrologRelease+0×1c9
00007ff9`d296e2c3 ntdll!LdrpWorkCallback+0×63
00007ff9`d2aa52f0 ntdll!LdrpWorkQueue
00007ff9`d29708a2 ntdll!TppWorkpExecuteCallback+0xb2
00000000`7ffe0386 SharedUserData+0×386
00007ff9`d2974060 ntdll!TppWorkerThread+0×300
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

0:000> dpS f1`82bfe000       f1`82c00000
00007ff9`d2986139 ntdll!RtlpFindEntry+0×4d
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297dbea ntdll!RtlpAllocateHeap+0xcfa
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a463fa ntdll!RtlpValidateHeap+0×32
00007ff9`d2a44b25 ntdll!RtlDebugAllocateHeap+0×35d
00007ff9`d29f49d6 ntdll!RtlpAllocateHeap+0×77ae6
00007ff9`d2962da8 ntdll!LdrpInitializeThread+0×40
00007ff9`d29700e5 ntdll!RtlRegisterThreadWithCsrss+0×35
00007ff9`d29b18f5 ntdll!_LdrpInitialize+0×89
00007ff9`d297babb ntdll!RtlpAllocateHeapInternal+0×1cb
00007ff9`d29701d6 ntdll!TppCritSetThread+0×7a
00007ff9`d2970098 ntdll!TppPoolAddWorker+0×68
00007ff9`d2974060 ntdll!TppWorkerThread+0×300
00007ff9`d1137bd4 kernel32!BaseThreadInitThunk+0×14
00007ff9`d29aced1 ntdll!RtlUserThreadStart+0×21

We call such analysis pattern Hidden Stack as another way to get Historical Information from memory dumps.

The example memory dump, the application PDB file, and source code can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -