Software Diagnostics Library

Sometimes when a high number of interrupts is reported but there are no signs of an interrupt storm or pending DPCs in a memory dump file it is useful to search for Hardware Activity in running and / or suspected threads. This can be done by examining execution residue left on a thread raw stack. Although found driver activity might not be related to reported problems it can be a useful start for driver elimination procedure for the general recommendation to check drivers for any updates. Here is an example of a thread raw stack with a network card doing “Scatter-Gather” DMA (more extensive example is coming in a separate pattern cooperation case study):

1: kd> !thread
THREAD f7732090 Cid 0000.0000 Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 1
Not impersonating
Owning Process 8089db40 Image: Idle
Attached Process N/A Image: N/A
Wait Start TickCount 0 Ticks: 24437545 (4:10:03:56.640)
Context Switch Count 75624870
UserTime 00:00:00.000
KernelTime 4 Days 08:56:05.125
Stack Init f78b3000 Current f78b2d4c Base f78b3000 Limit f78b0000 Call 0
Priority 0 BasePriority 0 PriorityDecrement 0
ChildEBP RetAddr Args to Child
f3b30c5c 00000000 00000000 00000000 00000000 LiveKdD+0x1c07

1: kd> dds f78b0000 f78b3000
f78b0000 00000000
f78b0004 00000000
f78b0008 00000000
f78b000c 00000000
f78b0010 00000000
[...]
f78b2870 8b3de0d0
f78b2874 80887b75 nt!KiFlushTargetSingleTb+0xd
f78b2878 8b49032c
f78b287c 00000000
f78b2880 2d003202
f78b2884 00000000
f78b2888 00000000
f78b288c 2d003202
f78b2890 8b490302
f78b2894 f78b28a4
f78b2898 80a61456 hal!KfLowerIrql+0x62
f78b289c 2d00320a
f78b28a0 00000000
f78b28a4 8b3de0d0
f78b28a8 8b3e3730
f78b28ac 00341eb0
f78b28b0 f78b2918
f78b28b4 f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b28b8 8b3de0d0
f78b28bc 8b341eb0
f78b28c0 f78b28d4
f78b28c4 00000000
f78b28c8 80a5f3c0 hal!KfAcquireSpinLock
f78b28cc 00000000
f78b28d0 8b3de0d0
f78b28d4 00000000
f78b28d8 8b3de0d0
f78b28dc 8b3eb730
f78b28e0 005a7340
f78b28e4 f78b294c
f78b28e8 f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b28ec 8b3de0d0
f78b28f0 8a5a7340
f78b28f4 f78b2908
f78b28f8 00000000
f78b28fc 8b3de0d0
f78b2900 8b0f5158
f78b2904 001e2340
f78b2908 f78b2970
f78b290c f63fbf78 NetworkAdapterA!SendWithScatterGather+0×318
f78b2910 8b3de0d0
f78b2914 8b1e2340
f78b2918 f78b292c
f78b291c 00000000
f78b2920 80a5f3c0 hal!KfAcquireSpinLock
f78b2924 00000000
f78b2928 8b3de0d0
f78b292c 00000000
f78b2930 8b3eb700
f78b2934 00000000
f78b2938 00000000
f78b293c 00000000
f78b2940 00000000
f78b2944 00000000
f78b2948 00000000
f78b294c 0a446aa2
f78b2950 f78b29b8
f78b2954 8b0f5158
f78b2958 8b01ce10
f78b295c 00000001
f78b2960 8b3de0d0
f78b2964 80a5f302 hal!HalpPerfInterrupt+0×32
f78b2968 00000001
f78b296c 8b3de0d0
f78b2970 80a5f302 hal!HalpPerfInterrupt+0×32
f78b2974 8b3de302
f78b2978 f78b2988
f78b297c 80a61456 hal!KfLowerIrql+0×62
f78b2980 80a5f3c0 hal!KfAcquireSpinLock
f78b2984 8b3de302
f78b2988 f78b29a4
f78b298c 80a5f44b hal!KfReleaseSpinLock+0xb
f78b2990 f63fbbbf NetworkAdapterA!SendPackets+0×1b3
f78b2994 8a446a90
f78b2998 8b0e8ab0
f78b299c 00000000
f78b29a0 008b29d0
f78b29a4 f78b29bc
f78b29a8 f7163790 NDIS!ndisMProcessSGList+0×90
f78b29ac 8b3de388
f78b29b0 f78b29d0
f78b29b4 00000001
f78b29b8 00000000
f78b29bc f78b29e8
f78b29c0 80a60147 hal!HalBuildScatterGatherList+0×1c7
f78b29c4 8b0e89b0
f78b29c8 00000000
f78b29cc 8a44cde8
f78b29d0 8b1e2340
f78b29d4 8a446aa2
f78b29d8 8b026ca0
f78b29dc 8b1e2340
f78b29e0 8b0e8ab0
f78b29e4 8b0e8ab0
f78b29e8 f78b2a44
f78b29ec f716369f NDIS!ndisMAllocSGList+0xda
f78b29f0 8a44cde8
f78b29f4 8b0e89b0
f78b29f8 8a446a70
f78b29fc 00000000
f78b2a00 00000036
f78b2a04 f7163730 NDIS!ndisMProcessSGList
f78b2a08 8b1e2340
f78b2a0c 00000000
f78b2a10 8a44cde8
f78b2a14 00000218
f78b2a18 8b1e2308
f78b2a1c 00000103
f78b2a20 8b0e8ab0
f78b2a24 8a446a70
f78b2a28 8a44cde8
f78b2a2c 00000036
f78b2a30 8b0e8ab0
f78b2a34 00000036
f78b2a38 00000000
f78b2a3c 00000000
f78b2a40 029a9e02
f78b2a44 f78b2a60
f78b2a48 f71402ff NDIS!ndisMSendX+0×1dd
f78b2a4c 8b490310
f78b2a50 8b1e2340
f78b2a54 8a446a70
f78b2a58 8a9a9e02
f78b2a5c 8a9a9e02
f78b2a60 f78b2a88
f78b2a64 f546c923 tcpip!ARPSendData+0×1a9
f78b2a68 8b3e76c8
f78b2a6c 8b1e2340
f78b2a70 8a9a9ea8
f78b2a74 8b490310
f78b2a78 80888b00 nt!RtlBackoff+0×68
f78b2a7c 8a446a70
f78b2a80 8a446aa2
f78b2a84 8a446a70
f78b2a88 f78b2ab4
f78b2a8c f546ba5d tcpip!ARPTransmit+0×112
f78b2a90 8b490310
f78b2a94 8b1e2340
f78b2a98 8a9a9ea8
f78b2a9c 00000103
f78b2aa0 8a446a70
f78b2aa4 00000000
f78b2aa8 8b342398
f78b2aac 8a47e1f8
f78b2ab0 8b1e2340
f78b2ab4 f78b2bf0
f78b2ab8 f546c4fc tcpip!_IPTransmit+0×866
f78b2abc 8a9a9ebc
f78b2ac0 f78b2b02
f78b2ac4 00000001
[…]

We also do a sanity check for coincidental symbols:

1: kd> ub f63fbf78
NetworkAdapterA!SendWithScatterGather+0x304:
f63fbf64 push    eax
f63fbf65 push    edi
f63fbf66 push    esi
f63fbf67 mov     dword ptr [ebp-44h],ecx
f63fbf6a mov     dword ptr [ebp-3Ch],ecx
f63fbf6d mov     dword ptr [ebp-34h],ecx
f63fbf70 mov     dword ptr [ebp-2Ch],ecx
f63fbf73 call    NetworkAdapterA!PacketRetrieveNicActions (f63facd2)

1: kd> ub f63fbbbf
NetworkAdapterA!SendPackets+0x190:
f63fbb9c cmp     dword ptr [esi+0Ch],2
f63fbba0 jl      NetworkAdapterA!SendPackets+0x19e (f63fbbaa)
f63fbba2 mov     dword ptr [ecx+3818h],eax
f63fbba8 jmp     NetworkAdapterA!SendPackets+0x1a4 (f63fbbb0)
f63fbbaa mov     dword ptr [ecx+438h],eax
f63fbbb0 mov     dl,byte ptr [esi+2BCh]
f63fbbb6 mov     ecx,dword ptr [ebp+8]
f63fbbb9 call    dword ptr [NetworkAdapterA!_imp_KfReleaseSpinLock (f640ca18)]

1: kd> ub 80a60147
hal!HalBuildScatterGatherList+0x1b0:
80a60130 je      hal!HalBuildScatterGatherList+0x1b9 (80a60139)
80a60132 mov     dword ptr [eax+4],1
80a60139 push    dword ptr [ebp+20h]
80a6013c push    eax
80a6013d mov     eax,dword ptr [ebp+0Ch]
80a60140 push    dword ptr [eax+14h]
80a60143 push    eax
80a60144 call    dword ptr [ebp+1Ch]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -