Archive for August 25th, 2007

Crash Dump Analysis Patterns (Part 23b)

Saturday, August 25th, 2007

In contrast to Double Free pattern in a user mode process heap double free in a kernel mode pool results in immediate bugcheck in order to identify the driver causing the problem (BAD_POOL_CALLER bugcheck with Arg1 == 7):

2: kd> !analyze -v
...
...
...
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000007, Attempt to free pool which was already freed
Arg2: 0000121a, (reserved)
Arg3: 02140001, Memory contents of the pool block
Arg4: 89ba74f0, Address of the block of pool being deallocated

If we look at the block being deallocated we would see that it was marked as “Free” block: 

2: kd> !pool 89ba74f0
Pool page 89ba74f0 region is Nonpaged pool
 89ba7000 size:  270 previous size:    0  (Allocated)  Thre (Protected)
 89ba7270 size:    8 previous size:  270  (Free)       ....
 89ba7278 size:   18 previous size:    8  (Allocated)  ReEv
 89ba7290 size:   80 previous size:   18  (Allocated)  Mdl
 89ba7310 size:   80 previous size:   80  (Allocated)  Mdl
 89ba7390 size:   30 previous size:   80  (Allocated)  Vad
 89ba73c0 size:   98 previous size:   30  (Allocated)  File (Protected)
 89ba7458 size:    8 previous size:   98  (Free)       Wait
 89ba7460 size:   28 previous size:    8  (Allocated)  FSfm
 89ba74a0 size:   40 previous size:   18  (Allocated)  Ntfr
 89ba74e0 size:    8 previous size:   40  (Free)       File
*89ba74e8 size:   a0 previous size:    8  (Free )     *ABCD
  Owning component : Unknown (update pooltag.txt)
 89ba7588 size:   38 previous size:   a0  (Allocated)  Sema (Protected)
 89ba75c0 size:   38 previous size:   38  (Allocated)  Sema (Protected)
 89ba75f8 size:   10 previous size:   38  (Free)       Nbtl
 89ba7608 size:   98 previous size:   10  (Allocated)  File (Protected)
 89ba76a0 size:   28 previous size:   98  (Allocated)  Ntfn
 89ba76c8 size:   40 previous size:   28  (Allocated)  Ntfr
 89ba7708 size:   28 previous size:   40  (Allocated)  NtFs
 89ba7730 size:   40 previous size:   28  (Allocated)  Ntfr
 89ba7770 size:   40 previous size:   40  (Allocated)  Ntfr
 89ba7a10 size:  270 previous size:  260  (Allocated)  Thre (Protected)
 89ba7c80 size:   20 previous size:  270  (Allocated)  VadS
 89ba7ca0 size:   40 previous size:   20  (Allocated)  Ntfr
 89ba7ce0 size:   b0 previous size:   40  (Allocated)  Ussy
 89ba7d90 size:  270 previous size:   b0  (Free)       Thre

The pool tag is a 4 byte character sequence used to associate drivers with pool blocks and is useful to identify a driver allocated or freed a block. In our case the pool tag is ABCD and it is associated with the driver which previously freed the block. All known pool tags corresponding to kernel components can be found in pooltag.txt located in triage subfolder where you installed WinDbg. However our ABCD tag is not listed there. We can try to find the driver corresponding to ABCD tag using findstr CMD command:

C:\Windows\System32\drivers>findstr /m /l hABCD *.sys

The results of the search will help us to identify the driver which freed the block first. The driver which double freed the same block can be found from the call stack and it might be the same driver or a different driver:

2: kd> k
ChildEBP RetAddr
f78be910 8089c8f4 nt!KeBugCheckEx+0x1b
f78be978 8089c622 nt!ExFreePoolWithTag+0x477
f78be988 f503968b nt!ExFreePool+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
f78be990 f5024a6e driver+0×1768b
f78be9a0 f50249e7 driver+0×2a6e
f78be9a4 84b430e0 driver+0×29e7

Because we don’t have symbol files for driver.sys WinDbg warns us that it was unable to identify the correct stack trace and driver.sys might not have called ExFreePool or ExFreePoolWithTag. To verify that driver.sys called ExFreePool indeed we disassemble backwards the return address of ExFreePool: 

2: kd> ub f503968b
driver+0×1767b:
f503967b 90              nop
f503967c 90              nop
f503967d 90              nop
f503967e 90              nop
f503967f 90              nop
f5039680 8b442404        mov     eax,dword ptr [esp+4]
f5039684 50              push    eax
f5039685 ff15202302f5    call    dword ptr [driver+0×320 (f5022320)]

Finally we can get some info from the driver: 

2: kd> lmv m driver
start    end        module name
f5022000 f503e400   driver   (no symbols)
    Loaded symbol image file: driver.sys
    Image path: \SystemRoot\System32\drivers\driver.sys
    Image name: driver.sys
    Timestamp:  Tue Aug 12 11:32:16 2007

If the company name developed the driver is absent we can try techniques outlined in Unknown Component pattern.

If we have symbols it is very easy to identify the code as can be seen from this 64-bit dump:

BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 0000000000000007, Attempt to free pool which was already freed
Arg2: 000000000000121a, (reserved)
Arg3: 0000000000000080, Memory contents of the pool block
Arg4: fffffade6d54e270, Address of the block of pool being deallocated

0: kd> kL
fffffade`45517b08 fffff800`011ad905 nt!KeBugCheckEx
fffffade`45517b10 fffffade`5f5991ac nt!ExFreePoolWithTag+0x401
fffffade`45517bd0 fffffade`5f59a0b0 driver64!ProcessDataItem+0×198
fffffade`45517c70 fffffade`5f5885a6 driver64!OnDataArrival+0×2b4
fffffade`45517cd0 fffff800`01299cae driver64!ReaderThread+0×15a
fffffade`45517d70 fffff800`0102bbe6 nt!PspSystemThreadStartup+0×3e
fffffade`45517dd0 00000000`00000000 nt!KiStartSystemThread+0×16

- Dmitry Vostokov @ DumpAnalysis.org -