Invalid pointer, incorrect stack trace, multiple exceptions, insufficient memory and memory leak: pattern cooperation
Users cannot connect and ultimately the server bluescreens. The dump shows an invalid pointer access while copying a buffer:
Loading Dump File [MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
6: kd> !analyze -v
[...]
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fe2cc128, memory referenced.
Arg2: 00000001, value 0 = read operation, 1 = write operation.
Arg3: b574414c, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 00000000, (reserved)
[...]
TRAP_FRAME: aefed9d8 -- (.trap 0xffffffffaefed9d8)
ErrCode = 00000002
eax=fe2fb09c ebx=00000006 ecx=00000001 edx=00000006 esi=f5312bcd edi=fe2cc128
eip=b574414c esp=aefeda4c ebp=aefeda58 iopl=0 nv up ei pl nz na po cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010203
Driver+0×1614c:
b574414c f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
Resetting default scope
STACK_TEXT:
aefed948 8085ed19 00000050 fe2cc128 00000001 nt!KeBugCheckEx+0x1b
aefed9c0 8088c7c8 00000001 fe2cc128 00000000 nt!MmAccessFault+0xb25
aefed9c0 b574414c 00000001 fe2cc128 00000000 nt!KiTrap0E+0xdc
WARNING: Stack unwind information not available. Following frames may be wrong.
aefeda58 b574472d f5312bcd 00000006 fe2fb09c Driver+0×1614c
[…]
aefedb70 8081df85 fe2fb000 aefedbcc 88d0e010 Driver+0×2758a
aefedc50 808f5437 8c556c88 8a1fea78 8c556c18 nt!IofCallDriver+0×45
aefedc64 808f61bf 90510070 8c556c18 8a1fea78 nt!IopSynchronousServiceTail+0×10b
aefedd00 808eed08 00001250 00000000 00000000 nt!IopXxxControlFile+0×5e5
aefedd34 808897bc 00001250 00000000 00000000 nt!NtDeviceIoControlFile+0×2a
aefedd34 7c82860c 00001250 00000000 00000000 nt!KiFastCallEntry+0xfc
0620f428 00000000 00000000 00000000 00000000 0×7c82860c
6: kd> !pte edi
VA fe2cc000
PDE at 00000000C0603F88 PTE at 00000000C07F1660
contains 0000000000ABE863 contains 0000000000000000
pfn abe —DA–KWEV
The warning about frames suggests that stack trace could be incorrect but backward and forward disassembly and preceding frames show that it is correct indeed and the warning results from the lack of symbol files:
6: kd> ub b574414c
*** ERROR: Module load completed but symbols could not be loaded for Driver.SYS
Driver+0×16139:
b5744139 8d341a lea esi,[edx+ebx]
b574413c 03fa add edi,edx
b574413e 3bf1 cmp esi,ecx
b5744140 8b7508 mov esi,dword ptr [ebp+8]
b5744143 771f ja Driver+0×16164 (b5744164)
b5744145 8bcb mov ecx,ebx
b5744147 8bd1 mov edx,ecx
b5744149 c1e902 shr ecx,2
6: kd> u b574414c
Driver+0×1614c:
b574414c f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
b574414e 8bca mov ecx,edx
b5744150 83e103 and ecx,3
b5744153 f3a4 rep movs byte ptr es:[edi],byte ptr [esi]
b5744155 8b4810 mov ecx,dword ptr [eax+10h]
b5744158 5f pop edi
b5744159 03cb add ecx,ebx
b574415b 5e pop esi
At the same time we see a second fault on another processor but it seems to be a demand zero page access:
6: kd> ~7s;kv
ChildEBP RetAddr Args to Child
ae331854 8095dddf badb0d00 0000000f 808a623c nt!KiTrap0E+0xbc (FPO: [0,0] TrapFrame @ ae331854)
ae3318cc 809731b2 de019000 00000001 3b9aca07 nt!RtlCreateSecurityDescriptorRelative+0×13 (FPO: [2,0,4])
ae331934 808d3229 ae3319e8 de019000 ae3319ec nt!SeQuerySecurityDescriptorInfo+0×198 (FPO: [SEH])
ae33194c 808d443c dc0d9330 ae3319e8 de019000 nt!CmpQuerySecurityDescriptorInfo+0×23 (FPO: [5,0,0])
ae3319b0 809383d3 e09e33f0 00000001 ae3319e8 nt!CmpSecurityMethod+0×212 (FPO: [SEH])
ae3319f0 8093866e e09e33f0 ae331a1c e09e33ec nt!ObpGetObjectSecurity+0×99 (FPO: [4,2,0])
ae331a20 808d8ecc e09e33f0 8ae88740 00000001 nt!ObCheckObjectAccess+0×2c (FPO: [5,4,0])
ae331a7c 808da58b dbef6008 009fd5e0 00000000 nt!CmpDoOpen+0×3a0 (FPO: [SEH])
ae331b90 809374b1 f0d9d348 00000000 8ae88740 nt!CmpParseKey+0×547 (FPO: [10,50,0])
ae331c10 80933a76 0000004c ae331c50 00000040 nt!ObpLookupObjectName+0×11f (FPO: [11,17,4])
ae331c64 808bb471 00000000 908fb868 80934601 nt!ObOpenObjectByName+0xea (FPO: [7,5,4])
ae331d50 808897bc 0013f7bc 00020019 0013f720 nt!NtOpenKey+0×1ad (FPO: [SEH])
ae331d50 7c82860c 0013f7bc 00020019 0013f720 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ ae331d64)
7: kd> .trap ae331854
ErrCode = 00000002
eax=00000000 ebx=ae3319e8 ecx=de019000 edx=0000000f esi=de019000 edi=de019000
eip=8095dddf esp=ae3318c8 ebp=ae3318cc iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
nt!RtlCreateSecurityDescriptorRelative+0×13:
8095dddf ab stos dword ptr es:[edi] es:0023:de019000=????????
7: kd> !pte edi
VA de019000
PDE at 00000000C0603780 PTE at 00000000C06F00C8
contains 00000003D2FF9863 contains 00000000000000C0
pfn 3d2ff9 —DA–KWEV not valid
DemandZero
Protect: 6 - ReadWriteExecute
We suspect that users cannot connect because of the lack of kernel pool resources and see the shortage of paged pool along with session pool shortage for one session:
7: kd> !vm 4
*** Virtual Memory Usage ***
Physical Memory: 4193705 ( 16774820 Kb)
Page File: \??\C:\pagefile.sys
Current: 25160704 Kb Free Space: 18350628 Kb
Minimum: 25160704 Kb Maximum: 25160704 Kb
Available Pages: 2556764 ( 10227056 Kb)
ResAvail Pages: 3996322 ( 15985288 Kb)
Locked IO Pages: 292 ( 1168 Kb)
Free System PTEs: 22827 ( 91308 Kb)
******* 4 system PTE allocations have failed ******
Free NP PTEs: 32534 ( 130136 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 86 ( 344 Kb)
Modified PF Pages: 85 ( 340 Kb)
NonPagedPool Usage: 32376 ( 129504 Kb)
NonPagedPool Max: 65215 ( 260860 Kb)
PagedPool 0 Usage: 84341 ( 337364 Kb)
PagedPool 1 Usage: 7945 ( 31780 Kb)
PagedPool 2 Usage: 8073 ( 32292 Kb)
PagedPool 3 Usage: 7979 ( 31916 Kb)
PagedPool 4 Usage: 8048 ( 32192 Kb)
PagedPool Usage: 116386 ( 465544 Kb)
PagedPool Maximum: 134144 ( 536576 Kb)
********** 841 pool allocations have failed **********
Shared Commit: 95929 ( 383716 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 126106 ( 504424 Kb)
PagedPool Commit: 116435 ( 465740 Kb)
Driver Commit: 2605 ( 10420 Kb)
Committed pages: 3454252 ( 13817008 Kb)
Commit limit: 10419631 ( 41678524 Kb)
Total Private: 2961488 ( 11845952 Kb)
721c Application.exe 334248 ( 1336992 Kb)
4b4c iexplore.exe 36048 ( 144192 Kb)
[…]
Terminal Server Memory Usage By Session:
Session Paged Pool Maximum is 32768K
Session View Space Maximum is 20480K
Session ID 0 @ ba1c5000:
Paged Pool Usage: 2288K
Commit Usage: 3344K
[...]
Session ID 1b @ ba1e3000:
Paged Pool Usage: 12020K
*** 622 Pool Allocation Failures ***
Commit Usage: 13176K
[...]
We also see that the process with PID 721c (Application.exe) consumed 1.3Gb of memory and it belongs to the problem session 0×1b (0n27):
7: kd> !sprocess 1b
Dumping Session 1b
_MM_SESSION_SPACE ba1e3000
_MMSESSION ba1e3d80
PROCESS 8a4fd020 SessionId: 27 Cid: 1f24 Peb: 7ffde000 ParentCid: 01e4
DirBase: cfe3f820 ObjectTable: ec731d08 HandleCount: 281.
Image: csrss.exe
PROCESS 8a4d6b18 SessionId: 27 Cid: 2160 Peb: 7ffde000 ParentCid: 01e4
DirBase: cfe3f800 ObjectTable: ee932bf0 HandleCount: 436.
Image: winlogon.exe
[...]
PROCESS 8a467020 SessionId: 27 Cid: 721c Peb: 7ffdf000 ParentCid: 5728
DirBase: cfe3f8a0 ObjectTable: f0397350 HandleCount: 380.
Image: Application.exe
[...]
Unfortunately the dump file is only kernel dump and we cannot inspect user space of that process. We can only say that it was not a handle leak because the number of handles is low (380). We also inspect paged pool stats and find a 3rd-party driver with AAAA tag that used almost 80Mb of paged pool:
7: kd> !poolused 4
Sorting by Paged Pool Consumed
Pool Used:
NonPaged Paged
Tag Allocs Used Allocs Used
MmSt 0 0 57216 91753456 Mm section object prototype ptes , Binary: nt!mm
AAAA 0 0 613 79465472 UNKNOWN pooltag ‘AAAA’, please update pooltag.txt
CM35 0 0 529 40398848 Internal Configuration manager allocations , Binary: nt!cm
Ntff 5 1040 35955 29339280 FCB_DATA , Binary: ntfs.sys
Toke 0 0 3334 12631056 Token objects , Binary: nt!se
IoNm 0 0 43494 9611392 Io parsing names , Binary: nt!io
CMAl 0 0 1981 8114176 internal registry memory allocator pool tag , Binary: nt!cm
Obtb 0 0 2569 6687344 object tables via EX handle.c , Binary: nt!ob
CM16 0 0 1429 6299648 Internal Configuration manager allocations , Binary: nt!cm
[…]
We can find this driver like explained here. We recommend to contact the vendor of that driver and also monitor instances of Application.exe and get 2-3 subsequent process memory dumps, when it grows, to inspect its virtual memory using differential memory analysis techniques.
- Dmitry Vostokov @ DumpAnalysis.org -