Data Contents Locality is a comparative pattern that helps not only in identifying the class of the problem but increases our confidence and degree of belief in the specific hypothesis. Suppose we have a database of notes of previous problems. If we see the same or similar data accessed in the new memory dump we might suppose that the issue is similar. If Data Contents Locality is complemented by Code Path Locality (similar partial stack traces and code residues) it even greater boosts our confidence in suggesting specific troubleshooting steps, recommending fixes and service packs or routing the problem to the next support or development service supply chain (like escalating the issue).
Suppose we got a new kernel memory dump with IRQL_NOT_LESS_OR_EQUAL (A) bugcheck pointing to our module and we notice the write access to a structure in a nonpaged pool having specific pool tag:
3: kd> .trap 9ee8d9b0
ErrCode = 00000002
eax=85407650 ebx=858f6650 ecx=ffffffff edx=85407648 esi=858f65a8 edi=858f6620
eip=8083df4c esp=9ee8da24 ebp=9ee8da64 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
nt!KeWaitForSingleObject+0x24f:
8083df4c 8919 mov dword ptr [ecx],ebx ds:0023:ffffffff=????????
STACK_TEXT:
9ee8d9b0 8083df4c badb0d00 85407648 00000000 nt!KiTrap0E+0x2a7
9ee8da64 80853f3f 85407648 0000001d 00000000 nt!KeWaitForSingleObject+0×24f
9ee8da7c 8081d45f 865b18d8 854076b0 f4b9e53b nt!KiAcquireFastMutex+0×13
9ee8da88 f4b9e53b 00000004 86940110 85407638 nt!ExAcquireFastMutex+0×20
9ee8daa8 f4b9ed98 85407638 00000000 86940110 driver!Query+0×143
[…]
3: kd> !pool 85407648
Pool page 85407648 region is Nonpaged pool
85407000 size: 80 previous size: 0 (Allocated) Mdl
85407080 size: 30 previous size: 80 (Allocated) Even (Protected)
854070b0 size: 28 previous size: 30 (Allocated) Ntfn
854070d8 size: 28 previous size: 28 (Allocated) NtFs
85407100 size: 28 previous size: 28 (Allocated) Ntfn
[...]
85407570 size: 28 previous size: 70 (Allocated) Ntfn
85407598 size: 98 previous size: 28 (Allocated) File (Protected)
*85407630 size: b0 previous size: 98 (Free ) *DrvA
Dumping the memory address passed to KeWaitForSingleObject shows simple but peculiar pattern:
3: kd> dd 85407648
85407648 ffffffff ffffffff ffffffff ffffffff
85407658 ffffffff ffffffff ffffffff ffffffff
85407668 ffffffff ffffffff ffffffff ffffffff
85407678 ffffffff ffffffff ffffffff ffffffff
85407688 ffffffff ffffffff ffffffff ffffffff
85407698 ffffffff ffffffff ffffffff ffffffff
854076a8 ffffffff ffffffff ffffffff ffffffff
854076b8 ffffffff ffffffff ffffffff ffffffff
We find several similar cases in our database but with different overall call stacks except the topmost wait call. Then we notice that in previous cases there were mutants associated with thread structure and we have the same now:
0: kd> !thread
THREAD 858f65a8 Cid 474c.4530 Teb: 7ffdf000 Win32Thread: bc012410 RUNNING on processor 0
[…]
3: kd> dt /r _KTHREAD 858f65a8 MutantListHead
nt!_KTHREAD
+0×010 MutantListHead : _LIST_ENTRY [ 0×86773040 - 0×86773040 ]
3: kd> !pool 86773040
Pool page 86773040 region is Nonpaged pool
*86773000 size: 50 previous size: 0 (Allocated) *Muta (Protected)
Pooltag Muta : Mutant objects
[…]
This narrows the issue to only a few previous cases. In one previous case WaitBlockList associated with a thread structure had 0xffffffff in its pointers. Our block shows the same pattern:
0: kd> dt -r _KTHREAD 858f65a8 WaitBlockList
nt!_KTHREAD
+0×054 WaitBlockList : 0×858f6650 _KWAIT_BLOCK
0: kd> dt _KWAIT_BLOCK 0x858f6650
nt!_KWAIT_BLOCK
+0x000 WaitListEntry : _LIST_ENTRY [ 0x85407650 - 0xffffffff ]
+0×008 Thread : 0×858f65a8 _KTHREAD
+0×00c Object : 0×85407648
+0×010 NextWaitBlock : 0×858f6650 _KWAIT_BLOCK
+0×014 WaitKey : 0
+0×016 WaitType : 0×1 ”
+0×017 SpareByte : 0 ”
We have probably narrowed down the issue to a specific case. Although this doesn’t work always and mostly based on intuition there are spectacular cases where it really helps in troubleshooting. Here is another example where the contents of EDI register from exception context provided specific recommendation hints. When looking at the crash point we see an instance of Wild Code pattern:
0:000> kv
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
49ab5bba 00000000 00000000 00000000 00000000 0x60f1011a
0:000> r
eax=38084ff0 ebx=52303340 ecx=963f1416 edx=0000063d esi=baaff395 edi=678c5804
eip=60f1011a esp=5a9d0f48 ebp=49ab5bba iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00210206
60f1011a cd01 int 1
0:000> u
60f1011a cd01 int 1
60f1011c cc int 3
60f1011d 8d ???
60f1011e c0eb02 shr bl,2
60f10121 0f840f31cd01 je 62be3236
60f10127 8d ???
60f10128 c0cc0f ror ah,0Fh
60f1012b 0bce or ecx,esi
Looking at raw stack data we notice the presence of a specific component that is known to patch the process import table. Applying techniques outlined in Hooked Functions pattern we notice two different 3rd-party components that patched two different modules (kernel32 and user32):
0:000> !chkimg -lo 50 -d !kernel32 -v
Searching for module with expression: !kernel32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\kernel32.dll\4626487F102000\kernel32.dll
No range specified
Scanning section: .text
Size: 564709
Range to scan: 77e41000-77ecade5
77e41ae5-77e41ae9 5 bytes - kernel32!LoadLibraryExW
[ 6a 34 68 48 7b:e9 16 e5 f4 07 ]
77e44a8a-77e44a8e 5 bytes - kernel32!WaitNamedPipeW (+0×2fa5)
[ 8b ff 55 8b ec:e9 71 b5 f9 07 ]
77e5106a-77e5106e 5 bytes - kernel32!CreateProcessInternalW (+0xc5e0)
[…]
Total bytes compared: 564709(100%)
Number of errors: 49
49 errors : !kernel32 (77e41ae5-77e9aa16)
0:000> u 77e41ae5
kernel32!LoadLibraryExW:
77e41ae5 jmp 7fd90000
77e41aea out 77h,al
77e41aec call kernel32!_SEH_prolog (77e6b779)
77e41af1 xor edi,edi
77e41af3 mov dword ptr [ebp-28h],edi
77e41af6 mov dword ptr [ebp-2Ch],edi
77e41af9 mov dword ptr [ebp-20h],edi
77e41afc cmp dword ptr [ebp+8],edi
0:000> u 7fd90000
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ComponentA.dll -
7fd90000 jmp ComponentA!DllUnregisterServer+0×2700 (678c4280)
7fd90005 push 34h
7fd90007 push offset kernel32!`string’+0xc (77e67b48)
7fd9000c jmp kernel32!LoadLibraryExW+0×7 (77e41aec)
7fd90011 add byte ptr [eax],al
7fd90013 add byte ptr [eax],al
7fd90015 add byte ptr [eax],al
7fd90017 add byte ptr [eax],al
0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:\mss\user32.dll\45E7BFD692000\user32.dll
No range specified
Scanning section: .text
Size: 396943
Range to scan: 77381000-773e1e8f
77383f38-77383f3c 5 bytes - user32!EnumDisplayDevicesW
[ 8b ff 55 8b ec:e9 c3 c0 82 08 ]
77384406-7738440a 5 bytes - user32!EnumDisplaySettingsExW (+0×4ce)
[ 8b ff 55 8b ec:e9 f5 bb 7e 08 ]
773844d9-773844dd 5 bytes - user32!EnumDisplaySettingsW (+0xd3)
[ 8b ff 55 8b ec:e9 22 bb 80 08 ]
7738619b-7738619f 5 bytes - user32!EnumDisplayDevicesA (+0×1cc2)
[ 8b ff 55 8b ec:e9 60 9e 83 08 ]
7738e985-7738e989 5 bytes - user32!CreateWindowExA (+0×87ea)
[ 8b ff 55 8b ec:e9 76 16 8c 08 ]
[…]
Total bytes compared: 396943(100%)
Number of errors: 119
119 errors : !user32 (77383f38-773c960c)
0:000> u 77383f38
user32!EnumDisplayDevicesW:
77383f38 e9c3c08208 jmp 7fbb0000
77383f3d 81ec58030000 sub esp,358h
77383f43 a1ac243e77 mov eax,dword ptr [user32!__security_cookie (773e24ac)]
77383f48 8b5508 mov edx,dword ptr [ebp+8]
77383f4b 83a5acfcffff00 and dword ptr [ebp-354h],0
77383f52 53 push ebx
77383f53 56 push esi
77383f54 8b7510 mov esi,dword ptr [ebp+10h]
0:000> u 7fbb0000
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ComponentB.dll -
7fbb0000 e91b43d5e5 jmp ComponentB+0×4320 (65904320)
7fbb0005 8bff mov edi,edi
7fbb0007 55 push ebp
7fbb0008 8bec mov ebp,esp
7fbb000a e92e3f7df7 jmp user32!EnumDisplayDevicesW+0×5 (77383f3d)
7fbb000f 0000 add byte ptr [eax],al
7fbb0011 0000 add byte ptr [eax],al
7fbb0013 0000 add byte ptr [eax],al
Which one should we try to eliminate first to test our assumption that they somehow resulted in application faults? Looking at register context again we see that one specific register (EDI) has a value that lies in ComponentA address range:
0:000> r
eax=38084ff0 ebx=52303340 ecx=963f1416 edx=0000063d esi=baaff395 edi=678c5804
eip=60f1011a esp=5a9d0f48 ebp=49ab5bba iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00210206
60f1011a cd01 int 1
0:000> lm
start end module name
00400000 01901000 Application
[...]
678c0000 6791d000 ComponentA ComponentA.DLL
[…]
- Dmitry Vostokov @ DumpAnalysis.org -
