10 Common Mistakes in Memory Analysis (Part 3)
In part 1 we discussed the common mistake of not looking at full stack traces. In this part we discuss the common mistake of not looking at all stack traces. This is important when the dump is partially truncated or inconsistent. For example, in one complete memory dump from one hang system WinDbg !locks command is not able to traverse them at all due to truncated dump:
3: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.......Error 1 in reading nt!_ERESOURCE.SystemResourcesList.Flink @ f71612a0
The common response, especially from beginners, would be to dismiss this dump and request the new one after increasing page file size. However, dumping all thread stacks reveals the resource contention around ERESOURCE objects similar to what was discussed in a mixed object deadlock example in kernel space:
3: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[85973590 csrss.exe]
4138.0051e0 85961db0 00cb222 Blocked driverA+0xec08
4138.0048c8 85d1d240 000006d Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
4138.0054cc 85c8a840 00c0d50 Blocked driverA+0xec08
4138.00227c 859be330 00c0d53 Blocked driverA+0xec08
4138.0053d8 8590f458 00000df Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
4138.003bb4 85b61020 00000e1 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
4138.002a08 85d1edb0 00000e1 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
4138.005e6c 85943020 00cc9cc Blocked driverA+0xec08
4138.00575c 858eeb40 00c0d4e Blocked driverA+0xec08
4138.003880 858ee5f8 00c0d51 Blocked driverA+0xec08
[85bb9b18 winlogon.exe]
50e0.0054d4 85a8cb30 00c0d53 Blocked driverA+0xec08
50e0.004b90 85b6c7b8 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.0032cc 85a1f850 0000084 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.005450 85c43db0 0000014 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.005648 85a1f5e0 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.004a80 85a7abd8 000001b Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.0036d8 85d886a8 000001b Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.0055b0 85d88438 0000014 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.004380 85962020 00c0d53 Blocked driverA+0xec08
50e0.005744 85a22db0 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.005dd4 8584c7a0 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.005e30 858902f0 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
50e0.005ce8 857bbdb0 00c0d53 Blocked driverA+0xec08
[85914868 explorer.exe]
5fd8.005fdc 85911020 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5fd8.005fec 8579d020 00bc253 Blocked driverA+0xec08
5fd8.005ff8 857ce020 0000014 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5fd8.003678 857ce8d0 00bc253 Blocked driverA+0xec08
5fd8.00556c 857ce3f0 00b85d9 Blocked driverA+0xec08
5fd8.005564 857e4db0 00bc253 Blocked driverA+0xec08
5fd8.005548 86529380 00bc253 Blocked driverA+0xec08
5fd8.006fd8 856095c8 00bc253 Blocked driverA+0xec08
5fd8.001844 85d50020 00bc253 Blocked driverA+0xec08
5fd8.0069cc 85ab8db0 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5fd8.0057c4 85fea2b0 00bc253 Blocked driverA+0xec08
5fd8.00394c 85a475b8 00bc253 Blocked driverA+0xec08
5fd8.004a8c 86090020 00bc253 Blocked driverA+0xec08
5fd8.00583c 85990db0 00bc253 Blocked driverA+0xec08
[858634a0 ApplicationA.EXE]
5b7c.005ad8 8597ddb0 0078325 Blocked driverA+0xec08
5b7c.0058b4 85735020 00b6852 Blocked driverA+0xec08
5b7c.00598c 8597db40 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.0059dc 85746a18 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.005b3c 85733ae8 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.005934 85733878 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.002b68 85bb8a40 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.0016dc 85747438 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.003fc0 8577ea60 00b6852 Blocked driverA+0xec08
5b7c.0066a4 8595c2f8 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.006b50 893d5660 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.0066f4 8605f530 00b6852 Blocked driverA+0xec08
5b7c.001554 85930cf0 00b6852 Blocked driverA+0xec08
5b7c.006f28 86132db0 00b6852 Blocked driverA+0xec08
5b7c.004448 85aa6890 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5b7c.000fa8 859073c8 00b6852 Blocked driverA+0xec08
[8595c928 ApplicationB.exe]
5990.0059a0 857c5508 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.005950 85ce7548 00b3b52 Blocked driverA+0xec08
5990.005c10 856dc910 00b3b52 Blocked driverA+0xec08
5990.005bd4 85767b40 00b3b52 Blocked driverA+0xec08
5990.005e38 859b6a18 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.005f14 85a747a0 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.005e68 85989020 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.005f10 859f42d8 0000015 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.005f0c 856ec5e8 00b3b52 Blocked driverA+0xec08
5990.0045d0 856ec9a8 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.004584 85728020 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.004754 8572d818 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.004b94 856cf020 00b3b52 Blocked driverA+0xec08
5990.003374 85722db0 0000016 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5990.000b1c 8647ddb0 00b3b52 Blocked driverA+0xec08
5990.003bdc 85f812f0 00b3b52 Blocked driverA+0xec08
[859bd598 dllhost.exe]
5e3c.00591c 8593e2f0 000001a Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5e3c.005e60 85777db0 000006e Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5e3c.005e64 85978b40 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
5e3c.0055c8 85903358 0000018 Blocked nt!ExEnterCriticalRegionAndAcquireResourceExclusive+0×19
[...]
Threads Processed: 1500
Different methods to list all thread stacks are listed in Stack Trace Collection pattern.
- Dmitry Vostokov @ DumpAnalysis.org -
December 8th, 2009 at 11:13 am
[…] stack trace is empty here but look at all threads (we list full traces in order not to miss any module) we find one that shows a dialog box […]