This is a synthesized dump analysis of many similar print spooler crashes in multi-user terminal service environments where old printer drivers are used that were tested only in single-user environments or insufficiently tested in multi-threaded environments. Many such crashes result from dynamic memory corruption of a process heap:
(40dc.4278): Access violation - code c0000005 (!!! second chance !!!)
eax=00000079 ebx=00000001 ecx=0008bff8 edx=00107898 esi=07d7522a edi=00107890
eip=7c8199b2 esp=0155fc14 ebp=0155fc44 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202
ntdll!RtlpLowFragHeapFree+0×30:
7c8199b2 8b4604 mov eax,dword ptr [esi+4] ds:0023:07d7522e=????????
0:017> kL
ChildEBP RetAddr
0155fc44 7c819770 ntdll!RtlpLowFragHeapFree+0×30
0155fd1c 77c87a2b ntdll!RtlFreeHeap+0×5c
0155fd30 77c87a02 RPCRT4!FreeWrapper+0×1e
0155fd3c 77c821c2 RPCRT4!operator delete+0xd
0155fd50 77c8047b RPCRT4!LRPC_SCALL::FreeBuffer+0×77
0155fd9c 77c80353 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×192
0155fdc0 77c811dc RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3
0155fdfc 77c812f0 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0×42c
0155fe20 77c88678 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
0155ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
0155ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
0155ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
0155ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
0155ffec 00000000 kernel32!BaseThreadStart+0×34
Although any module could corrupt the heap and either Gflags or Application Verifier is recommended to enable full page heap, sometimes we need to point to some print drivers to eliminate or upgrade them in the meantime. When there are many of them we can point to the oldest one:
0:017> lmt
start end module name
[...]
010d0000 010d8000 PrintDriver1 2007
01260000 01272000 PrintDriver2 1999
01290000 012da000 PrintDriver3 2009
012f0000 01302000 PrintDriver4 2003
01310000 01320000 PrintDriver5 2004
01320000 01332000 PrintDriver6 2004
01340000 01353000 PrintDriver7 2005
01360000 0139e000 PrintDriver8 2007
013b0000 013c3000 PrintDriver9 2004
013d0000 013e0000 PrintDriver10 2005
013e0000 013f3000 PrintDriver11 2005
01400000 01413000 PrintDriver12 2006
01420000 0146b000 PrintDriver13 2007
01480000 01488000 PrintDriver14 2003
017f0000 0181e000 PrintDriver15 2004
01920000 0192d000 PrintDriver16 2008
01930000 01936000 PrintDriver17 2008
01950000 01959000 PrintDriver18 2008
01960000 01969000 PrintDriver19 2008
01f80000 021e8000 PrintDriver20 2004
032f0000 03514000 PrintDriver21 2003
03cd0000 03cd6000 PrintDriver22 2008
32100000 32148000 PrintDriver23 2008
3ea40000 3ea46000 PrintDriver24 2007
3f000000 3f03d000 PrintDriver25 2009
3f100000 3f133000 PrintDriver26 2009
[…]
The age distribution among 121 modules can be visualized on a CAD diagram:
Alternatively we can look at the execution residue on a raw thread stack:
0:017> !teb
TEB at 7ffa9000
ExceptionList: 0155fd0c
StackBase: 01560000
StackLimit: 01550000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffa9000
EnvironmentPointer: 00000000
ClientId: 000040dc . 00004278
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffd8000
LastErrorValue: 0
LastStatusValue: 8000001a
Count Owned Locks: 0
HardErrorMode: 0
0:017> dds 01550000 01560000
01550000 00000000
01550004 00000000
[...]
01554e78 00000000
01554e7c 00000000
01554e80 01554ecc
01554e84 7c82d1bb ntdll!RtlFindActivationContextSectionString+0xe1
01554e88 01554ea4
01554e8c 01554efc
01554e90 00000000
01554e94 020a0018 PrintDriver20!Callback+0×5c88
01554e98 7ffa9c00
01554e9c 00000000
01554ea0 01554ed4
01554ea4 7c82dd6c ntdll!RtlEncodeSystemPointer+0×45b
01554ea8 00020000
01554eac 01554ec8
01554eb0 01554ec8
01554eb4 01554ec8
[…]
01555eb4 0040003e
01555eb8 01556854
01555ebc 00000000
01555ec0 0000003e
01555ec4 0208003e PrintDriver20!GetValue+0xb37fe
01555ec8 00000000
01555ecc 43000000
01555ed0 0000003e
01555ed4 01555ffa
01555ed8 01555fbc
01555edc 01555fa8
01555ee0 001190d8
01555ee4 7c81990d ntdll!RtlpLowFragHeapAlloc+0×210
01555ee8 7c819962 ntdll!RtlpLowFragHeapAlloc+0xc6a
01555eec 0008bff8
01555ef0 00000000
01555ef4 00080000
[…]
The first address 020a0018 seems to be coincidental because its disassembled code is not good:
0:017> ub 020a0018
^ Unable to find valid previous instruction for 'ub 020a0018'
0:017> u 020a0018
PrintDriver20!Callback+0x5c88:
020a0018 048b add al,8Bh
020a001a c7 ???
020a001b ebe8 jmp PrintDriver20!Callback+0×5c75 (020a0005)
020a001d 8d4e28 lea ecx,[esi+28h]
020a0020 e8d9960100 call PrintDriver20!DlgProc+0×86ee (020b96fe)
020a0025 8d4e28 lea ecx,[esi+28h]
020a0028 e87b930100 call PrintDriver20!DlgProc+0×8398 (020b93a8)
020a002d 8b4618 mov eax,dword ptr [esi+18h]
However the second address code 0208003e seems sound: cmp is followed by jne:
0:017> ub 0208003e
PrintDriver20!GetValue+0xb37e5:
02080025 8d442414 lea eax,[esp+14h]
02080029 8b4b2c mov ecx,dword ptr [ebx+2Ch]
0208002c 50 push eax
0208002d e8ce3a0000 call PrintDriver20!GetValue+0xb72c0 (02083b00)
02080032 8b38 mov edi,dword ptr [eax]
02080034 8b4c2410 mov ecx,dword ptr [esp+10h]
02080038 8b41fc mov eax,dword ptr [ecx-4]
0208003b 3947fc cmp dword ptr [edi-4],eax
0:017> u 0208003e
PrintDriver20!GetValue+0xb37fe:
0208003e 751b jne PrintDriver20!GetValue+0xb381b (0208005b)
02080040 8bc8 mov ecx,eax
02080042 8b742410 mov esi,dword ptr [esp+10h]
02080046 c1e902 shr ecx,2
02080049 f3a7 repe cmps dword ptr [esi],dword ptr es:[edi]
0208004b 750e jne PrintDriver20!GetValue+0xb381b (0208005b)
0208004d 8bc8 mov ecx,eax
0208004f 83e103 and ecx,3
But this is not a function call resulted in saved return address so we can still consider it as a coincidence. However, on the raw stack we also see a large chunk of ASCII data pointing to the same driver in a textual form:
[...]
0155d360 6f742064
0155d364 696e6920
0155d368 6c616974
0155d36c 20657a69
0155d370 61636562
0155d374 20657375
0155d378 75732061
0155d37c 62617469
[...]
0:017> da 0155d360 0155d734
0155d360 "d to initialize because a suitab"
0155d380 "le PrinterDriver20 inf file was ”
[…]
This reinforces our belief in PrinterDriver20. Finally, when looking at critical section list we see corruption signs pointing to the same driver addresses:
0:017> !cs -l -o -s
DebugInfo = 0x0014bc60
Critical section = 0x020f7140 (PrintDriver20!DlgProc+0×46130)
LOCKED
LockCount = 0xFF85EA7F
WaiterWoken = Yes
OwningThread = 0×8b0c244c
RecursionCount = 0×8BFFFBB4
LockSemaphore = 0×83182444
SpinCount = 0×088908c4
WARNING: critical section DebugInfo = 0x00000008 doesn't point back
to the DebugInfo found in the active critical sections list = 0x0014bc60.
The critical section was probably reused without calling DeleteCriticalSection.
Cannot read structure field value at 0x0000000a, error 0
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
DebugInfo = 0x0014bc88
Critical section = 0x020f7110 (PrintDriver20!DlgProc+0×46100)
LOCKED
LockCount = 0×1E7245FF
WaiterWoken = No
OwningThread = 0xccccc304
RecursionCount = 0xC483FFFD
LockSemaphore = 0×158638B9
SpinCount = 0xff96e902
WARNING: critical section DebugInfo = 0x0f712068 doesn't point back
to the DebugInfo found in the active critical sections list = 0x0014bc88.
The critical section was probably reused without calling DeleteCriticalSection.
Cannot read structure field value at 0x0f71206a, error 0
[...]
- Dmitry Vostokov @ DumpAnalysis.org -
