Software Diagnostics Library

Sometimes we get pool allocation failures and the driver’s tag is ‘Ddk’:

0: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:      851775 (   3407100 Kb)
 Page File: \??\C:\pagefile.sys
   Current:   4190208 Kb  Free Space:   4175708 Kb
   Minimum:   4190208 Kb  Maximum:      4190208 Kb
 Available Pages:      147274 (    589096 Kb)
 ResAvail Pages:       769287 (   3077148 Kb)
 Locked IO Pages:         118 (       472 Kb)
 Free System PTEs:     184910 (    739640 Kb)
 Free NP PTEs:            110 (       440 Kb)
 Free Special NP:           0 (         0 Kb)
 Modified Pages:          168 (       672 Kb)
 Modified PF Pages:       168 (       672 Kb)
 NonPagedPool Usage:    64445 (    257780 Kb)
 NonPagedPool Max:      64640 (    258560 Kb)
 ********** Excessive NonPaged Pool Usage *****
 PagedPool 0 Usage:     21912 (     87648 Kb)
 PagedPool 1 Usage:       691 (      2764 Kb)
 PagedPool 2 Usage:       706 (      2824 Kb)
 PagedPool 3 Usage:       704 (      2816 Kb)
 PagedPool 4 Usage:       708 (      2832 Kb)
 PagedPool Usage:       24721 (     98884 Kb)
 PagedPool Maximum:    134144 (    536576 Kb)

 ********** 429 pool allocations have failed **********

 Shared Commit:          5274 (     21096 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:         3958 (     15832 Kb)
 PagedPool Commit:      24785 (     99140 Kb)
 Driver Commit:         19289 (     77156 Kb)
 Committed pages:      646282 (   2585128 Kb)
 Commit limit:        1860990 (   7443960 Kb)

0: kd> !poolused 3
   Sorting by  NonPaged Pool Consumed

  Pool Used:
            NonPaged
 Tag    Allocs    Frees     Diff     Used
 Ddk   9074558  3859522  5215036 225708304 Default for driver allocated memory (user’s of ntddk.h)

How do we find which driver had caused this memory leak? We can search for drivers using the following command:

C:\>findstr /S /m /l hDdk *.sys  

or we can guess the driver using the fact that long time ago ExAllocatePool was defined as ExAllocatePoolWithTag(, … ‘Ddk ‘). Currently all DDK samples use their separate driver tags and ExAllocatePool uses ‘None’:

0: kd> .asm no_code_bytes
Assembly options: no_code_bytes

0: kd> uf ExAllocatePool
nt!ExAllocatePool:
80894d1f mov     edi,edi
80894d21 push    ebp
80894d22 mov     ebp,esp
80894d24 push    656E6F4Eh
80894d29 push    dword ptr [ebp+0Ch]
80894d2c push    dword ptr [ebp+8]
80894d2f call    nt!ExAllocatePoolWithTag (8089b93f)
80894d34 pop     ebp
80894d35 ret     8

0: kd> .formats 656E6F4Eh
Evaluate expression:
  Hex:     656e6f4e
  Decimal: 1701736270
  Octal:   14533467516
  Binary:  01100101 01101110 01101111 01001110
  Chars:   enoN
  Time:    Tue Dec 05 00:31:10 2023
  Float:   low 7.03735e+022 high 0
  Double:  8.40769e-315

Note: we push ‘None’ but see ‘enoN’ in memory because of little endian byte ordering.

Most of the recent drivers use their own tags and it is common not to encounter ‘None’ at all:

kd> !poolused
   Sorting by  Tag

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
...
...
...
 None        0        0         1     8192 call to ExAllocatePool
…
…

Therefore the driver must be old and if we see most drivers dated 2006-2007 and some dated 1998-2001 the chances are that 2001 drivers were responsible for our memory leak:

b9840000 b9842980   newdriver     Sat Feb 10 00:33:41 2007 (45CD12E5)
b8cfa000 b8d39e00   olddriver     Tue Aug 21 12:18:35 2001 (3B82438B)
f79e5000 f79e6400   veryolddriver Wed Sep 23 13:09:52 1998 (3608E510)

However veryolddriver.sys doesn’t use ExAllocatePoolWithTag so olddriver.sys is under suspicion:

0: kd> !dh f79e5000
...
...
...
     A00 [      33] address [size] of Export Directory
     C00 [      3C] address [size] of Import Directory
     E00 [     3A4] address [size] of Resource Directory
       0 [       0] address [size] of Exception Directory
       0 [       0] address [size] of Security Directory
    1200 [      34] address [size] of Base Relocation Directory
     440 [      54] address [size] of Debug Directory
       0 [       0] address [size] of Description Directory
       0 [       0] address [size] of Special Directory
       0 [       0] address [size] of Thread Storage Directory
       0 [       0] address [size] of Load Configuration Directory
       0 [       0] address [size] of Bound Import Directory
     400 [      34] address [size] of Import Address Table Directory
       0 [       0] address [size] of Delay Import Directory
       0 [       0] address [size] of COR20 Header Directory
       0 [       0] address [size] of Reserved Directory
…
…
…

0: kd> dds f79e5000+400
f79e5400  80a82264 hal!HalTranslateBusAddress
f79e5404  80a84358 hal!READ_PORT_BUFFER_UCHAR
f79e5408  00000000
f79e540c  80840bd9 nt!IofCompleteRequest
f79e5410  808e8f01 nt!IoCreateSymbolicLink
f79e5414  80838035 nt!RtlInitUnicodeString
f79e5418  808fbe85 nt!IoDeleteSymbolicLink
f79e541c  80816a6e nt!MmUnmapIoSpace
f79e5420  808ef1f1 nt!IoCreateDevice
f79e5424  80837e3a nt!READ_REGISTER_BUFFER_UCHAR
f79e5428  80815fc8 nt!IoDeleteDevice
f79e542c  80816814 nt!MmMapIoSpace
f79e5430  00000000
f79e5434  00000000

To confirm that olddriver.sys uses ‘Ddk ‘ tag we can search its address space for code that calls ExAllocatePoolWithTag:

b8cfa000 b8d39e00 olddriver Tue Aug 21 12:18:35 2001 (3B82438B)

0: kd> !dh b8cfa000
...
...
...
       0 [       0] address [size] of Export Directory
   3D330 [      50] address [size] of Import Directory
   3DE00 [     380] address [size] of Resource Directory
       0 [       0] address [size] of Exception Directory
   3FE00 [      88] address [size] of Security Directory
   3E180 [    1BE8] address [size] of Base Relocation Directory
   3B640 [      1C] address [size] of Debug Directory
       0 [       0] address [size] of Description Directory
       0 [       0] address [size] of Special Directory
       0 [       0] address [size] of Thread Storage Directory
       0 [       0] address [size] of Load Configuration Directory
       0 [       0] address [size] of Bound Import Directory
   3B480 [     1B4] address [size] of Import Address Table Directory
       0 [       0] address [size] of Delay Import Directory
       0 [       0] address [size] of COR20 Header Directory
       0 [       0] address [size] of Reserved Directory
…
…
…

0: kd> dds b8cfa000+3B480 b8cfa000+3B480+1B4
b8d35480  80a83dba hal!KeQueryPerformanceCounter
b8d35484  80a7e3c0 hal!KfAcquireSpinLock
b8d35488  80a7e440 hal!KfReleaseSpinLock
b8d3548c  00000000
...
...
...
b8d35544  80812b1a nt!IoWriteErrorLogEntry
b8d35548  8081287b nt!IoAllocateErrorLogEntry
b8d3554c  8082f12f nt!swprintf
b8d35550  8089b93f nt!ExAllocatePoolWithTag
b8d35554  8087c465 nt!KeBugCheckEx
b8d35558  80815407 nt!wcsncat
b8d3555c  8083bc54 nt!ZwQueryValueKey
b8d35560  8083affb nt!ZwClose
b8d35564  80841a14 nt!_wcsicmp
b8d35568  80928d30 nt!ObReferenceObjectByHandle
…
…
… 

0: kd> s-d b8cfa000 b8d39e00 b8d35550
b8d19f08  b8d35550 555425ff 25ffb8d3 b8d35480  PU…%TU…%.T..
b8d1a068  b8d35550 ff85f88b 75fc7d89 b85e5f0c  PU…….}.u._^.
b8d2c4e4  b8d35550 0375c085 89c35d5e 04c08330  PU….u.^]..0…

0: kd> u b8d19f08-2
olddriver!ExAllocatePoolWithTag:
b8d19f06 jmp     dword ptr [olddriver!_imp__ExAllocatePoolWithTag (b8d35550)]

0: kd> u b8d2c4e4-2
olddriver!malloc+0x12
b8d2c4e2 call    dword ptr [olddriver!_imp__ExAllocatePoolWithTag (b8d35550)]
b8d2c4e8 test    eax,eax
b8d2c4ea jne     olddriver!malloc+0x1f (b8d2c4ef)
b8d2c4ec pop     esi
b8d2c4ed pop     ebp
b8d2c4ee ret
b8d2c4ef mov     dword ptr [eax],esi
b8d2c4f1 add     eax,4

0: kd> ub b8d1a068-2
olddriver!TraceRoutine+0xc1
b8d1a051 mov     esp,ebp
b8d1a053 pop     ebp
b8d1a054 ret
b8d1a055 cmp     edi,8
b8d1a058 jbe     olddriver!TraceRoutine+0x157 (b8d1a0e7)
b8d1a05e push    206b6444h
b8d1a063 push    edx
b8d1a064 push    0

0: kd> .formats 206b6444
Evaluate expression:
  Hex:     206b6444
  Decimal: 543908932
  Octal:   04032662104
  Binary:  00100000 01101011 01100100 01000100
  Chars:    kdD
  Time:    Sat Mar 28 05:48:52 1987
  Float:   low 1.99384e-019 high 0
  Double:  2.68727e-315

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Monday, October 22nd, 2007 at 5:45 pm and is filed under Crash Dump Analysis, Crash Dump Patterns. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.