NULL data pointer, stack trace, inline function optimization and platformorphic fault: pattern cooperation

We have the following crash pointing to Driver.sys 

7: kd> KL
Child-SP          RetAddr           Call Site
fffffadd`7671a678 fffff800`0102e5f4 nt!KeBugCheckEx
fffffadd`7671a680 fffff800`0102d587 nt!KiBugCheckDispatch+0x74
fffffadd`7671a800 fffffadd`88e5dbf3 nt!KiPageFault+0x207
fffffadd`7671a998 fffffadd`88df63f5 Driver!memcpy+0×83
fffffadd`7671a9a0 fffffadd`88dfe97b Driver!ItemCopyTo+0×85

fffffadd`7671a9e0 fffffadd`88e45bd1 Driver!CallbackEx+0×3cb
fffffadd`7671aa80 fffffadd`88dfb130 Driver!Callback+0×131
fffffadd`7671ab90 fffffadd`88dfaef3 Driver!Reply+0×1a0
fffffadd`7671ac40 fffffadd`88de9e23 Driver!OnDataReceive+0×1a3
fffffadd`7671acc0 fffff800`0124e932 Driver!ReaderThread+0×553
fffffadd`7671ad70 fffff800`010202b6 nt!PspSystemThreadStartup+0×3e
fffffadd`7671add0 00000000`00000000 nt!KxStartSystemThread+0×16

7: kd> !analyze -v
[...]
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high.  This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000007, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
Arg4: fffffadd88e5dbf3, address which referenced memory
[...]
TRAP_FRAME:  fffffadd7671a800 -- (.trap 0xfffffadd7671a800)
[...]

7: kd> .trap 0xfffffadd7671a800
[...]
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect

7: kd> r
Last set context:
rax=0000000000001000 rbx=0000000000000000 rcx=0000000000000007
rdx=fffffadda17d001d rsi=0000000000000000 rdi=0000000000000000
rip=fffffadd88e5dbf3 rsp=fffffadd7671a998 rbp=0000000000000001
 r8=0000000000000001  r9=fffffadda254f2f0 r10=0000000000000000
r11=0000000000000007 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=0006 es=0000 fs=fadf gs=ffff efl=00010202
Driver!memcpy+0×83:
fffffadd`88e5dbf3 8801  mov byte ptr [rcx],al ds:0006:0007=??

.trap warning above perhaps explains why we have non-standard values in ds, fs and gs. Typical expected values in kernel mode are these (ds is ignored in 64-bit mode anyway):

cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202

7: kd> kL
Child-SP          RetAddr           Call Site
fffffadd`7671a998 fffffadd`88df63f5 Driver!memcpy+0×83
fffffadd`7671a9a0 fffffadd`88dfe97b Driver!ItemCopyTo+0×85
fffffadd`7671a9e0 fffffadd`88e45bd1 Driver!CallbackEx+0×3cb
fffffadd`7671aa80 fffffadd`88dfb130 Driver!Callback+0×131
fffffadd`7671ab90 fffffadd`88dfaef3 Driver!Reply+0×1a0
fffffadd`7671ac40 fffffadd`88de9e23 Driver!OnDataReceive+0×1a3
fffffadd`7671acc0 fffff800`0124e932 Driver!ReaderThread+0×553
fffffadd`7671ad70 fffff800`010202b6 nt!PspSystemThreadStartup+0×3e
fffffadd`7671add0 00000000`00000000 nt!KxStartSystemThread+0×16

We clearly have an instance of a NULL pointer data access. If we try to match this stack trace to known faults in database we would probably find many entries because memcpy is a generic function from C library. So we should try with ItemCopyTo. Indeed, we find a few matches but with slightly different stack traces:

b7535c7c b75931fa Driver!ItemCopyTo+0×6a
b7535ca4 b75c24c4 Driver!CallbackEx+0×23a
b7535d04 b7590c79 Driver!Callback+0xd4
b7535d44 b7590b41 Driver!Reply+0xe9
b7535d68 b7584b87 Driver!OnDataReceive+0×111
b7535dac 8094bea4 Driver!ReaderThread+0×397
b7535ddc 8088f61e nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

Offsets are different but the function names are the same. We also don’t see memcpy call but if we look at the faulted instruction we suspect it was inlined memcpy call:

eax=88642870 ebx=00000005 ecx=00000001 edx=00000005 esi=88f3f9a4 edi=00000029
eip=b758d3ca esp=b7535c6c ebp=b7535c7c iopl=0 nv up ei pl nz ac po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010212
Driver!ItemCopyTo+0×6a:
b758d3ca f3a5  rep movs dword ptr es:[edi],dword ptr [esi]

We also notice that the found stack trace is from x86 32-bit Windows but ours is from x64 Windows so we suspect the platformorphic fault here and check if we have a fix for x64 binaries.

- Dmitry Vostokov @ DumpAnalysis.org -

Leave a Reply