Crash Dump Analysis Patterns (Part 23b)

In contrast to Double Free pattern in a user mode process heap double free in a kernel mode pool results in immediate bugcheck in order to identify the driver causing the problem (BAD_POOL_CALLER bugcheck with Arg1 == 7):

2: kd> !analyze -v
...
...
...
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000007, Attempt to free pool which was already freed
Arg2: 0000121a, (reserved)
Arg3: 02140001, Memory contents of the pool block
Arg4: 89ba74f0, Address of the block of pool being deallocated

If we look at the block being deallocated we would see that it was marked as “Free” block: 

2: kd> !pool 89ba74f0
Pool page 89ba74f0 region is Nonpaged pool
 89ba7000 size:  270 previous size:    0  (Allocated)  Thre (Protected)
 89ba7270 size:    8 previous size:  270  (Free)       ....
 89ba7278 size:   18 previous size:    8  (Allocated)  ReEv
 89ba7290 size:   80 previous size:   18  (Allocated)  Mdl
 89ba7310 size:   80 previous size:   80  (Allocated)  Mdl
 89ba7390 size:   30 previous size:   80  (Allocated)  Vad
 89ba73c0 size:   98 previous size:   30  (Allocated)  File (Protected)
 89ba7458 size:    8 previous size:   98  (Free)       Wait
 89ba7460 size:   28 previous size:    8  (Allocated)  FSfm
 89ba74a0 size:   40 previous size:   18  (Allocated)  Ntfr
 89ba74e0 size:    8 previous size:   40  (Free)       File
*89ba74e8 size:   a0 previous size:    8  (Free )     *ABCD
  Owning component : Unknown (update pooltag.txt)
 89ba7588 size:   38 previous size:   a0  (Allocated)  Sema (Protected)
 89ba75c0 size:   38 previous size:   38  (Allocated)  Sema (Protected)
 89ba75f8 size:   10 previous size:   38  (Free)       Nbtl
 89ba7608 size:   98 previous size:   10  (Allocated)  File (Protected)
 89ba76a0 size:   28 previous size:   98  (Allocated)  Ntfn
 89ba76c8 size:   40 previous size:   28  (Allocated)  Ntfr
 89ba7708 size:   28 previous size:   40  (Allocated)  NtFs
 89ba7730 size:   40 previous size:   28  (Allocated)  Ntfr
 89ba7770 size:   40 previous size:   40  (Allocated)  Ntfr
 89ba7a10 size:  270 previous size:  260  (Allocated)  Thre (Protected)
 89ba7c80 size:   20 previous size:  270  (Allocated)  VadS
 89ba7ca0 size:   40 previous size:   20  (Allocated)  Ntfr
 89ba7ce0 size:   b0 previous size:   40  (Allocated)  Ussy
 89ba7d90 size:  270 previous size:   b0  (Free)       Thre

The pool tag is a 4 byte character sequence used to associate drivers with pool blocks and is useful to identify a driver allocated or freed a block. In our case the pool tag is ABCD and it is associated with the driver which previously freed the block. All known pool tags corresponding to kernel components can be found in pooltag.txt located in triage subfolder where you installed WinDbg. However our ABCD tag is not listed there. We can try to find the driver corresponding to ABCD tag using findstr CMD command:

C:\Windows\System32\drivers>findstr /m /l hABCD *.sys

The results of the search will help us to identify the driver which freed the block first. The driver which double freed the same block can be found from the call stack and it might be the same driver or a different driver:

2: kd> k
ChildEBP RetAddr
f78be910 8089c8f4 nt!KeBugCheckEx+0x1b
f78be978 8089c622 nt!ExFreePoolWithTag+0x477
f78be988 f503968b nt!ExFreePool+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
f78be990 f5024a6e driver+0×1768b
f78be9a0 f50249e7 driver+0×2a6e
f78be9a4 84b430e0 driver+0×29e7

Because we don’t have symbol files for driver.sys WinDbg warns us that it was unable to identify the correct stack trace and driver.sys might not have called ExFreePool or ExFreePoolWithTag. To verify that driver.sys called ExFreePool indeed we disassemble backwards the return address of ExFreePool: 

2: kd> ub f503968b
driver+0×1767b:
f503967b 90              nop
f503967c 90              nop
f503967d 90              nop
f503967e 90              nop
f503967f 90              nop
f5039680 8b442404        mov     eax,dword ptr [esp+4]
f5039684 50              push    eax
f5039685 ff15202302f5    call    dword ptr [driver+0×320 (f5022320)]

Finally we can get some info from the driver: 

2: kd> lmv m driver
start    end        module name
f5022000 f503e400   driver   (no symbols)
    Loaded symbol image file: driver.sys
    Image path: \SystemRoot\System32\drivers\driver.sys
    Image name: driver.sys
    Timestamp:  Tue Aug 12 11:32:16 2007

If the company name developed the driver is absent we can try techniques outlined in Unknown Component pattern.

If we have symbols it is very easy to identify the code as can be seen from this 64-bit dump:

BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 0000000000000007, Attempt to free pool which was already freed
Arg2: 000000000000121a, (reserved)
Arg3: 0000000000000080, Memory contents of the pool block
Arg4: fffffade6d54e270, Address of the block of pool being deallocated

0: kd> kL
fffffade`45517b08 fffff800`011ad905 nt!KeBugCheckEx
fffffade`45517b10 fffffade`5f5991ac nt!ExFreePoolWithTag+0x401
fffffade`45517bd0 fffffade`5f59a0b0 driver64!ProcessDataItem+0×198
fffffade`45517c70 fffffade`5f5885a6 driver64!OnDataArrival+0×2b4
fffffade`45517cd0 fffff800`01299cae driver64!ReaderThread+0×15a
fffffade`45517d70 fffff800`0102bbe6 nt!PspSystemThreadStartup+0×3e
fffffade`45517dd0 00000000`00000000 nt!KiStartSystemThread+0×16

- Dmitry Vostokov @ DumpAnalysis.org -

8 Responses to “Crash Dump Analysis Patterns (Part 23b)”

  1. Bill Says:

    What happens if the block being deallocated can’t be analyzed:

    4: kd> !pool a4ef7920
    Pool page a4ef7920 region is Nonpaged pool
    a4ef7000 size: e0 previous size: 0 (Allocated) MmCi
    a4ef70e0 size: 68 previous size: e0 (Allocated) TCIZ
    a4ef7148 size: e0 previous size: 68 (Allocated) MmCi
    a4ef7228 size: 98 previous size: e0 (Allocated) File (Protected)
    a4ef72c0 size: 98 previous size: 98 (Allocated) File (Protected)
    a4ef7358 size: 100 previous size: 98 (Allocated) MmCi
    a4ef7458 size: 28 previous size: 100 (Allocated) NtFs
    a4ef7480 size: 40 previous size: 28 (Allocated) Ntfr
    a4ef74c0 size: 98 previous size: 40 (Allocated) File (Protected)
    a4ef7558 size: 8 previous size: 98 (Free) CcPL
    a4ef7560 size: 40 previous size: 8 (Allocated) SevE
    a4ef75a0 size: 40 previous size: 40 (Allocated) Ntfr
    a4ef75e0 size: 10 previous size: 40 (Free) TCI1
    a4ef75f0 size: 180 previous size: 10 (Allocated) MmCi
    a4ef7770 size: 98 previous size: 180 (Allocated) File (Protected)
    a4ef7808 size: 88 previous size: 98 (Allocated) Adap (Protected)
    a4ef7890 size: 88 previous size: 88 (Allocated) NEtd
    a4ef7918 is not a valid large pool allocation, checking large session pool…
    a4ef7918 is freed (or corrupt) pool
    Bad allocation size @a4ef7918, zero is invalid

    ***
    *** An error (or corruption) in the pool was detected;
    *** Attempting to diagnose the problem.
    ***
    *** Use !poolval a4ef7000 for more details.
    ***

    Pool page [ a4ef7000 ] is __inVALID.

    Analyzing linked list…
    [ a4ef7890 –> a4ef7a10 (size = 0×180 bytes)]: Corrupt region

    Scanning for single bit errors…

    None found

  2. Dmitry Vostokov Says:

    a4ef7890 size: 88 previous size: 88 (Allocated) NEtd
    a4ef7918 is not a valid large pool allocation, checking large session pool…

    a4ef7890+88 = a4ef7918

    I would try to see its contents, perhaps dds and dps would point to some symbolic data. Also search for this address in kernel space might point to some other blocks as well.

    If you suspect some driver you might want to enable driver verifier special pool

  3. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 42) Says:

    […] we introduce an icon for Double Free (kernel pool) […]

  4. Crash Dump Analysis » Blog Archive » Insufficient memory, handle leak, wait chain, deadlock, inconsistent dump and overaged system: pattern cooperation Says:

    […] However we see that drivers using AAAA and BBBB consumed almost 65Mb and we can search for them as described here. […]

  5. Sinni Says:

    In my case, special pool is enabled and it shows both the free happening through same thread, interestingly, both the stacks are exactly same.

    2: kd> .bugcheck
    Bugcheck code 000000C2
    Arguments 00000000`00000007 00000000`00001097 00000000`00210007 fffff8a0`04b98e00
    2: kd> !pool fffff8a0`04b98e00 2
    Pool page fffff8a004b98e00 region is Paged pool
    *fffff8a004b98df0 size: 210 previous size: 70 (Free) *MmSt
    Pooltag MmSt : Mm section object prototype ptes, Binary : nt!mm
    2: kd> !verifier 0×80 fffff8a0`04b98e00

    Log of recent kernel pool Allocate and Free operations:

    There are up to 0×10000 entries in the log.

    Parsing 0×0000000000010000 log entries, searching for address 0xfffff8a004b98e00.

    =================================
    Pool block fffff8a004b98df0, Size 0000000000000210, Thread fffffa80122674f0
    fffff80001b0bc9a nt!VfFreePoolNotification+0×4a
    fffff800017a367c nt!ExDeferredFreePool+0×126d
    fffff8000165b880 nt!MiDeleteSegmentPages+0×35c
    fffff8000195cf2f nt!MiSegmentDelete+0×7b
    fffff80001637e07 nt!MiCleanSection+0×2f7
    fffff80001676754 nt!ObfDereferenceObject+0xd4
    fffff80001661170 nt!CcDeleteSharedCacheMap+0×1bc
    fffff80001699880 nt!CcUninitializeCacheMap+0×2f0
    fffff880030ecfa6 picadm!OwCommonCleanup+0×4b6
    fffff880030ec840 picadm!FsdCleanup+0×2a8
    fffff880030ec994 picadm!OwFsdCleanup+0×38
    fffff80001b16750 nt!IovCallDriver+0xa0
    fffff800019824bf nt!IopCloseFile+0×11f
    =================================
    Pool block fffff8a004b98df0, Size 0000000000000210, Thread fffffa80122674f0
    fffff80001b0bc9a nt!VfFreePoolNotification+0×4a
    fffff800017a367c nt!ExDeferredFreePool+0×126d
    fffff8000165b880 nt!MiDeleteSegmentPages+0×35c
    fffff8000195cf2f nt!MiSegmentDelete+0×7b
    fffff80001637e07 nt!MiCleanSection+0×2f7
    fffff80001676754 nt!ObfDereferenceObject+0xd4
    fffff80001661170 nt!CcDeleteSharedCacheMap+0×1bc
    fffff80001699880 nt!CcUninitializeCacheMap+0×2f0
    fffff880030ecfa6 picadm!OwCommonCleanup+0×4b6
    fffff880030ec840 picadm!FsdCleanup+0×2a8
    fffff880030ec994 picadm!OwFsdCleanup+0×38
    fffff80001b16750 nt!IovCallDriver+0xa0
    fffff800019824bf nt!IopCloseFile+0×11f

    But current thread has no sign of this driver : picadm.sys. It is something like this:

    2: kd> k
    Child-SP RetAddr Call Site
    fffff880`02378b28 fffff800`017a360e nt!KeBugCheckEx
    fffff880`02378b30 fffff800`0178a53e nt!ExDeferredFreePool+0×11eb
    fffff880`02378be0 fffff800`01798a0a nt!MiDeleteCachedSubsection+0×10ae
    fffff880`02378c90 fffff800`01798b43 nt!MiRemoveUnusedSegments+0×8a
    fffff880`02378cc0 fffff800`01910726 nt!MiDereferenceSegmentThread+0×103
    fffff880`02378d40 fffff800`0164fae6 nt!PspSystemThreadStartup+0×5a
    fffff880`02378d80 00000000`00000000 nt!KiStartSystemThread+0×16

    Please suggest!

  6. Dmitry Vostokov Says:

    Recentlyh found that search needs to be done without h prefix:
    findstr /m /l ABCD

  7. Dmitry Vostokov Says:

    If you see some leaking pool tag you can find its entries using this command !poolfind ABCD and dump their memory

  8. Dmitry Vostokov Says:

    DRIVER_PAGE_FAULT_IN_FREED_SPECIAL_POOL (d5)
    Memory was referenced after it was freed.
    This cannot be protected by try-except.
    When possible, the guilty driver’s name (Unicode string) is printed on
    the bugcheck screen and saved in KiBugCheckDriver.

Leave a Reply