Invalid pointer, incorrect stack trace, multiple exceptions, insufficient memory and memory leak: pattern cooperation

Users cannot connect and ultimately the server bluescreens. The dump shows an invalid pointer access while copying a buffer:

Loading Dump File [MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

6: kd> !analyze -v
[...]
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fe2cc128, memory referenced.
Arg2: 00000001, value 0 = read operation, 1 = write operation.
Arg3: b574414c, If non-zero, the instruction address which referenced the bad memory
 address.
Arg4: 00000000, (reserved)
[...]

TRAP_FRAME:  aefed9d8 -- (.trap 0xffffffffaefed9d8)
ErrCode = 00000002
eax=fe2fb09c ebx=00000006 ecx=00000001 edx=00000006 esi=f5312bcd edi=fe2cc128
eip=b574414c esp=aefeda4c ebp=aefeda58 iopl=0 nv up ei pl nz na po cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010203
Driver+0×1614c:
b574414c f3a5            rep movs dword ptr es:[edi],dword ptr [esi]
Resetting default scope

STACK_TEXT: 
aefed948 8085ed19 00000050 fe2cc128 00000001 nt!KeBugCheckEx+0x1b
aefed9c0 8088c7c8 00000001 fe2cc128 00000000 nt!MmAccessFault+0xb25
aefed9c0 b574414c 00000001 fe2cc128 00000000 nt!KiTrap0E+0xdc
WARNING: Stack unwind information not available. Following frames may be wrong.
aefeda58 b574472d f5312bcd 00000006 fe2fb09c Driver+0×1614c
[…]
aefedb70 8081df85 fe2fb000 aefedbcc 88d0e010 Driver+0×2758a
aefedc50 808f5437 8c556c88 8a1fea78 8c556c18 nt!IofCallDriver+0×45
aefedc64 808f61bf 90510070 8c556c18 8a1fea78 nt!IopSynchronousServiceTail+0×10b
aefedd00 808eed08 00001250 00000000 00000000 nt!IopXxxControlFile+0×5e5
aefedd34 808897bc 00001250 00000000 00000000 nt!NtDeviceIoControlFile+0×2a
aefedd34 7c82860c 00001250 00000000 00000000 nt!KiFastCallEntry+0xfc
0620f428 00000000 00000000 00000000 00000000 0×7c82860c

6: kd> !pte edi
               VA fe2cc000
PDE at 00000000C0603F88    PTE at 00000000C07F1660
contains 0000000000ABE863  contains 0000000000000000
pfn abe        —DA–KWEV

The warning about frames suggests that stack trace could be incorrect but backward and forward disassembly and preceding frames show that it is correct indeed and the warning results from the lack of symbol files:

6: kd> ub b574414c
*** ERROR: Module load completed but symbols could not be loaded for Driver.SYS
Driver+0×16139:
b5744139 8d341a          lea     esi,[edx+ebx]
b574413c 03fa            add     edi,edx
b574413e 3bf1            cmp     esi,ecx
b5744140 8b7508          mov     esi,dword ptr [ebp+8]
b5744143 771f            ja      Driver+0×16164 (b5744164)
b5744145 8bcb            mov     ecx,ebx
b5744147 8bd1            mov     edx,ecx
b5744149 c1e902          shr     ecx,2

6: kd> u b574414c
Driver+0×1614c:
b574414c f3a5            rep movs dword ptr es:[edi],dword ptr [esi]
b574414e 8bca            mov     ecx,edx
b5744150 83e103          and     ecx,3
b5744153 f3a4            rep movs byte ptr es:[edi],byte ptr [esi]
b5744155 8b4810          mov     ecx,dword ptr [eax+10h]
b5744158 5f              pop     edi
b5744159 03cb            add     ecx,ebx
b574415b 5e              pop     esi

At the same time we see a second fault on another processor but it seems to be a demand zero page access:

6: kd> ~7s;kv
ChildEBP RetAddr  Args to Child             
ae331854 8095dddf badb0d00 0000000f 808a623c nt!KiTrap0E+0xbc (FPO: [0,0] TrapFrame @ ae331854)
ae3318cc 809731b2 de019000 00000001 3b9aca07 nt!RtlCreateSecurityDescriptorRelative+0×13 (FPO: [2,0,4])
ae331934 808d3229 ae3319e8 de019000 ae3319ec nt!SeQuerySecurityDescriptorInfo+0×198 (FPO: [SEH])
ae33194c 808d443c dc0d9330 ae3319e8 de019000 nt!CmpQuerySecurityDescriptorInfo+0×23 (FPO: [5,0,0])
ae3319b0 809383d3 e09e33f0 00000001 ae3319e8 nt!CmpSecurityMethod+0×212 (FPO: [SEH])
ae3319f0 8093866e e09e33f0 ae331a1c e09e33ec nt!ObpGetObjectSecurity+0×99 (FPO: [4,2,0])
ae331a20 808d8ecc e09e33f0 8ae88740 00000001 nt!ObCheckObjectAccess+0×2c (FPO: [5,4,0])
ae331a7c 808da58b dbef6008 009fd5e0 00000000 nt!CmpDoOpen+0×3a0 (FPO: [SEH])
ae331b90 809374b1 f0d9d348 00000000 8ae88740 nt!CmpParseKey+0×547 (FPO: [10,50,0])
ae331c10 80933a76 0000004c ae331c50 00000040 nt!ObpLookupObjectName+0×11f (FPO: [11,17,4])
ae331c64 808bb471 00000000 908fb868 80934601 nt!ObOpenObjectByName+0xea (FPO: [7,5,4])
ae331d50 808897bc 0013f7bc 00020019 0013f720 nt!NtOpenKey+0×1ad (FPO: [SEH])
ae331d50 7c82860c 0013f7bc 00020019 0013f720 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ ae331d64)

7: kd> .trap ae331854
ErrCode = 00000002
eax=00000000 ebx=ae3319e8 ecx=de019000 edx=0000000f esi=de019000 edi=de019000
eip=8095dddf esp=ae3318c8 ebp=ae3318cc iopl=0  nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
nt!RtlCreateSecurityDescriptorRelative+0×13:
8095dddf ab              stos    dword ptr es:[edi]   es:0023:de019000=????????

7: kd> !pte edi
               VA de019000
PDE at 00000000C0603780    PTE at 00000000C06F00C8
contains 00000003D2FF9863  contains 00000000000000C0
pfn 3d2ff9     —DA–KWEV    not valid
                       DemandZero
                       Protect: 6 - ReadWriteExecute

We suspect that users cannot connect because of the lack of kernel pool resources and see the shortage of paged pool along with session pool shortage for one session:

7: kd> !vm 4

*** Virtual Memory Usage ***
Physical Memory:     4193705 (  16774820 Kb)
Page File: \??\C:\pagefile.sys
  Current:  25160704 Kb  Free Space:  18350628 Kb
  Minimum:  25160704 Kb  Maximum:     25160704 Kb
Available Pages:     2556764 (  10227056 Kb)
ResAvail Pages:      3996322 (  15985288 Kb)
Locked IO Pages:         292 (      1168 Kb)
Free System PTEs:      22827 (     91308 Kb)

******* 4 system PTE allocations have failed ******

Free NP PTEs:          32534 (    130136 Kb)
Free Special NP:           0 (         0 Kb)
Modified Pages:           86 (       344 Kb)
Modified PF Pages:        85 (       340 Kb)
NonPagedPool Usage:    32376 (    129504 Kb)
NonPagedPool Max:      65215 (    260860 Kb)
PagedPool 0 Usage:     84341 (    337364 Kb)
PagedPool 1 Usage:      7945 (     31780 Kb)
PagedPool 2 Usage:      8073 (     32292 Kb)
PagedPool 3 Usage:      7979 (     31916 Kb)
PagedPool 4 Usage:      8048 (     32192 Kb)
PagedPool Usage:      116386 (    465544 Kb)
PagedPool Maximum:    134144 (    536576 Kb)

********** 841 pool allocations have failed **********

Shared Commit:         95929 (    383716 Kb)
Special Pool:              0 (         0 Kb)
Shared Process:       126106 (    504424 Kb)
PagedPool Commit:     116435 (    465740 Kb)
Driver Commit:          2605 (     10420 Kb)
Committed pages:     3454252 (  13817008 Kb)
Commit limit:       10419631 (  41678524 Kb)

Total Private:       2961488 (  11845952 Kb)
721c Application.exe  334248 (   1336992 Kb)
4b4c iexplore.exe      36048 (    144192 Kb)
[…]

 Terminal Server Memory Usage By Session:

 Session Paged Pool Maximum is 32768K
 Session View Space Maximum is 20480K

 Session ID 0 @ ba1c5000:
 Paged Pool Usage:        2288K
 Commit Usage:            3344K

[...]

 Session ID 1b @ ba1e3000:
 Paged Pool Usage:       12020K

 *** 622 Pool Allocation Failures ***

 Commit Usage:           13176K

[...]

We also see that the process with PID 721c (Application.exe) consumed 1.3Gb of memory and it belongs to the problem session 0×1b (0n27):

7: kd> !sprocess 1b
Dumping Session 1b

_MM_SESSION_SPACE ba1e3000
_MMSESSION        ba1e3d80
PROCESS 8a4fd020  SessionId: 27  Cid: 1f24    Peb: 7ffde000  ParentCid: 01e4
    DirBase: cfe3f820  ObjectTable: ec731d08  HandleCount: 281.
    Image: csrss.exe

PROCESS 8a4d6b18  SessionId: 27  Cid: 2160    Peb: 7ffde000  ParentCid: 01e4
    DirBase: cfe3f800  ObjectTable: ee932bf0  HandleCount: 436.
    Image: winlogon.exe

[...]

PROCESS 8a467020  SessionId: 27  Cid: 721c    Peb: 7ffdf000  ParentCid: 5728
    DirBase: cfe3f8a0  ObjectTable: f0397350  HandleCount: 380.
    Image: Application.exe

[...]

Unfortunately the dump file is only kernel dump and we cannot inspect user space of that process. We can only say that it was not a handle leak because the number of handles is low (380). We also inspect paged pool stats and find a 3rd-party driver with AAAA tag that used almost 80Mb of paged pool:

7: kd> !poolused 4
   Sorting by  Paged Pool Consumed

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 MmSt        0        0     57216 91753456 Mm section object prototype ptes , Binary: nt!mm
 AAAA        0        0       613 79465472 UNKNOWN pooltag ‘AAAA’, please update pooltag.txt
 CM35        0        0       529 40398848 Internal Configuration manager allocations , Binary: nt!cm
 Ntff        5     1040     35955 29339280 FCB_DATA , Binary: ntfs.sys
 Toke        0        0      3334 12631056 Token objects , Binary: nt!se
 IoNm        0        0     43494  9611392 Io parsing names , Binary: nt!io
 CMAl        0        0      1981  8114176 internal registry memory allocator pool tag , Binary: nt!cm
 Obtb        0        0      2569  6687344 object tables via EX handle.c , Binary: nt!ob
 CM16        0        0      1429  6299648 Internal Configuration manager allocations , Binary: nt!cm
[…]

We can find this driver like explained here. We recommend to contact the vendor of that driver and also monitor instances of Application.exe and get 2-3 subsequent process memory dumps, when it grows, to inspect its virtual memory using differential memory analysis techniques.

- Dmitry Vostokov @ DumpAnalysis.org -

Leave a Reply

You must be logged in to post a comment.