Memory leak, spiking threads, wait chain, high critical section contention and module variety: pattern cooperation
I noticed yesterday that my home Vista computer suddenly became slower than usual so I brought Task Manager, sorted processes by CPU usage and discovered an instance of IE7 with 50% - 60% of CPU consumption. Dumping processes in Vista is easier than ever, so I did the right click on that process and selected Create Dump File menu option. The dump was saved and I killed the process. The size of the dump file was 1.2Gb and that definitely indicated a memory leak. Examining process heap showed large heap segments amounting to 800Mb and therefore pointing to the possible heap leak:
0:000> !heap 0 0
Index Address Name Debugging options enabled
1: 00370000
Segment at 00370000 to 00470000 (00100000 bytes committed)
Segment at 04990000 to 04a90000 (00100000 bytes committed)
Segment at 063e0000 to 065e0000 (00200000 bytes committed)
Segment at 08440000 to 08840000 (00400000 bytes committed)
Segment at 0ce80000 to 0d680000 (00800000 bytes committed)
Segment at 160b0000 to 17080000 (00fd0000 bytes committed)
Segment at 19b00000 to 1aad0000 (00fd0000 bytes committed)
Segment at 1c8c0000 to 1d890000 (00fd0000 bytes committed)
Segment at 27870000 to 28840000 (00fd0000 bytes committed)
Segment at 29870000 to 2a840000 (00fd0000 bytes committed)
Segment at 2d1f0000 to 2e1c0000 (00fd0000 bytes committed)
Segment at 31fb0000 to 32f80000 (00fd0000 bytes committed)
Segment at 384c0000 to 39490000 (00fd0000 bytes committed)
Segment at 3c040000 to 3d010000 (00fd0000 bytes committed)
Segment at 41cf0000 to 42cc0000 (00fd0000 bytes committed)
Segment at 43c90000 to 44c60000 (00fd0000 bytes committed)
Segment at 44c60000 to 45c30000 (00fd0000 bytes committed)
Segment at 473f0000 to 483c0000 (00fd0000 bytes committed)
Segment at 4a390000 to 4b360000 (00fd0000 bytes committed)
Segment at 4b360000 to 4c330000 (00fd0000 bytes committed)
Segment at 4d300000 to 4e2d0000 (00fd0000 bytes committed)
Segment at 4e2d0000 to 4f2a0000 (00fd0000 bytes committed)
Segment at 50480000 to 51450000 (00fd0000 bytes committed)
Segment at 51450000 to 52420000 (00fd0000 bytes committed)
Segment at 533f0000 to 543c0000 (00fd0000 bytes committed)
Segment at 54810000 to 557e0000 (00fd0000 bytes committed)
Segment at 567b0000 to 57780000 (00fd0000 bytes committed)
Segment at 57c80000 to 58c50000 (00fc1000 bytes committed)
Segment at 59c20000 to 5abf0000 (00fc6000 bytes committed)
Segment at 5b0f0000 to 5c0c0000 (00fc1000 bytes committed)
Segment at 5c0c0000 to 5d090000 (00fc1000 bytes committed)
Segment at 5d090000 to 5e060000 (00fc1000 bytes committed)
Segment at 5f030000 to 60000000 (00fc1000 bytes committed)
Segment at 60000000 to 60fd0000 (00fc1000 bytes committed)
Segment at 60fd0000 to 61fa0000 (00fd0000 bytes committed)
Segment at 61fa0000 to 62f70000 (00e26000 bytes committed)
2: 00010000
Segment at 00010000 to 00020000 (00003000 bytes committed)
3: 00d80000
Segment at 00d80000 to 00d90000 (00010000 bytes committed)
Segment at 00050000 to 00150000 (00014000 bytes committed)
4: 00190000
Segment at 00190000 to 001a0000 (00010000 bytes committed)
Segment at 00d90000 to 00e90000 (00100000 bytes committed)
Segment at 0a430000 to 0a630000 (00200000 bytes committed)
Segment at 0d8d0000 to 0dcd0000 (00400000 bytes committed)
Segment at 0ecc0000 to 0f4c0000 (00800000 bytes committed)
Segment at 18690000 to 19660000 (00fd0000 bytes committed)
Segment at 24fe0000 to 25fb0000 (00fd0000 bytes committed)
Segment at 2bf40000 to 2cf10000 (00fd0000 bytes committed)
Segment at 303b0000 to 31380000 (00fd0000 bytes committed)
Segment at 33370000 to 34340000 (00fd0000 bytes committed)
Segment at 39490000 to 3a460000 (00fd0000 bytes committed)
Segment at 40d20000 to 41cf0000 (00fd0000 bytes committed)
Segment at 483c0000 to 49390000 (00fd0000 bytes committed)
Segment at 557e0000 to 567b0000 (00452000 bytes committed)
5: 00330000
Segment at 00330000 to 00340000 (00010000 bytes committed)
Segment at 00c10000 to 00d10000 (00100000 bytes committed)
Segment at 0c910000 to 0cb10000 (00200000 bytes committed)
Segment at 18280000 to 18680000 (00400000 bytes committed)
Segment at 2ec20000 to 2f420000 (00800000 bytes committed)
Segment at 42cc0000 to 43c90000 (00fc7000 bytes committed)
Segment at 4c330000 to 4d300000 (00d45000 bytes committed)
Segment at 52420000 to 533f0000 (00d39000 bytes committed)
Segment at 58c50000 to 59c20000 (00ddc000 bytes committed)
Segment at 5e060000 to 5f030000 (00dd1000 bytes committed)
6: 00e90000
Segment at 00e90000 to 00ea0000 (00010000 bytes committed)
Segment at 06780000 to 06880000 (00026000 bytes committed)
7: 00170000
Segment at 00170000 to 00180000 (00010000 bytes committed)
Segment at 06880000 to 06980000 (00026000 bytes committed)
8: 01bf0000
Segment at 01bf0000 to 01c00000 (00010000 bytes committed)
Segment at 03bb0000 to 03cb0000 (00100000 bytes committed)
Segment at 0e610000 to 0e810000 (00200000 bytes committed)
9: 00bf0000
Segment at 00bf0000 to 00c00000 (00001000 bytes committed)
10: 00b70000
Segment at 00b70000 to 00b80000 (00003000 bytes committed)
11: 01b60000
Segment at 01b60000 to 01ba0000 (00040000 bytes committed)
12: 03650000
Segment at 03650000 to 03690000 (00009000 bytes committed)
13: 039c0000
Segment at 039c0000 to 039d0000 (00008000 bytes committed)
Segment at 07e30000 to 07f30000 (00012000 bytes committed)
14: 00b20000
Segment at 00b20000 to 00b30000 (00003000 bytes committed)
15: 01b00000
Segment at 01b00000 to 01b40000 (00040000 bytes committed)
Segment at 22b80000 to 22c80000 (00032000 bytes committed)
16: 00b30000
Segment at 00b30000 to 00b70000 (00040000 bytes committed)
Segment at 08f00000 to 09000000 (00100000 bytes committed)
Segment at 376f0000 to 378f0000 (000e3000 bytes committed)
17: 03700000
Segment at 03700000 to 03740000 (00040000 bytes committed)
18: 03a70000
Segment at 03a70000 to 03ab0000 (00040000 bytes committed)
19: 00be0000
Segment at 00be0000 to 00bf0000 (00010000 bytes committed)
Segment at 0a630000 to 0a730000 (000a8000 bytes committed)
20: 04df0000
Segment at 04df0000 to 04ef0000 (00100000 bytes committed)
21: 044d0000
Segment at 044d0000 to 044e0000 (00010000 bytes committed)
Segment at 04390000 to 04490000 (00028000 bytes committed)
22: 04730000
Segment at 04730000 to 04740000 (00010000 bytes committed)
Segment at 04620000 to 04720000 (00100000 bytes committed)
Segment at 23fb0000 to 241b0000 (001f6000 bytes committed)
23: 055e0000
Segment at 055e0000 to 056e0000 (00100000 bytes committed)
24: 05ce0000
Segment at 05ce0000 to 05cf0000 (00010000 bytes committed)
Segment at 06bb0000 to 06cb0000 (00012000 bytes committed)
25: 05e20000
Segment at 05e20000 to 05e60000 (00020000 bytes committed)
26: 04860000
Segment at 04860000 to 04870000 (00010000 bytes committed)
Segment at 0df60000 to 0e060000 (00024000 bytes committed)
27: 04dc0000
Segment at 04dc0000 to 04dd0000 (00010000 bytes committed)
Segment at 062e0000 to 063e0000 (00100000 bytes committed)
Segment at 26d70000 to 26f70000 (001eb000 bytes committed)
28: 06aa0000
Segment at 06aa0000 to 06ab0000 (00010000 bytes committed)
Segment at 06980000 to 06a80000 (00100000 bytes committed)
Segment at 1ede0000 to 1efe0000 (00200000 bytes committed)
Segment at 1efe0000 to 1f3e0000 (00322000 bytes committed)
Segment at 1f3e0000 to 1fbe0000 (00800000 bytes committed)
Segment at 205e0000 to 215b0000 (001c7000 bytes committed)
29: 05420000
Segment at 05420000 to 05430000 (00010000 bytes committed)
Segment at 06ab0000 to 06bb0000 (00053000 bytes committed)
30: 05980000
Segment at 05980000 to 05990000 (00010000 bytes committed)
Segment at 17d90000 to 17e90000 (00012000 bytes committed)
31: 07c20000
Segment at 07c20000 to 07c60000 (00040000 bytes committed)
Segment at 08cc0000 to 08dc0000 (00100000 bytes committed)
Segment at 1fbe0000 to 1fde0000 (001fd000 bytes committed)
Segment at 241b0000 to 245b0000 (003fa000 bytes committed)
Segment at 2a840000 to 2b040000 (0007c000 bytes committed)
32: 07be0000
Segment at 07be0000 to 07c20000 (0003a000 bytes committed)
Segment at 17900000 to 17a00000 (000fd000 bytes committed)
Segment at 3b2b0000 to 3b4b0000 (001fe000 bytes committed)
Segment at 45c30000 to 46030000 (00289000 bytes committed)
33: 07df0000
Segment at 07df0000 to 07e30000 (0003a000 bytes committed)
Segment at 22810000 to 22910000 (0001c000 bytes committed)
34: 08000000
Segment at 08000000 to 08040000 (00001000 bytes committed)
35: 07da0000
Segment at 07da0000 to 07de0000 (00001000 bytes committed)
36: 04b60000
Segment at 04b60000 to 04b70000 (00002000 bytes committed)
37: 08990000
Segment at 08990000 to 089a0000 (00010000 bytes committed)
Segment at 06cb0000 to 06db0000 (00024000 bytes committed)
38: 051f0000
Segment at 051f0000 to 05200000 (00010000 bytes committed)
Segment at 050c0000 to 051c0000 (00100000 bytes committed)
Segment at 0c110000 to 0c310000 (00200000 bytes committed)
Segment at 0c310000 to 0c710000 (003f6000 bytes committed)
Segment at 1bd00000 to 1c500000 (00529000 bytes committed)
Segment at 216c0000 to 22690000 (00376000 bytes committed)
39: 0ac10000
Segment at 0ac10000 to 0ac20000 (00010000 bytes committed)
Segment at 0aa80000 to 0ab80000 (000c4000 bytes committed)
40: 12ed0000
Segment at 12ed0000 to 12ee0000 (00010000 bytes committed)
Segment at 199e0000 to 19ae0000 (00022000 bytes committed)
41: 15450000
Segment at 15450000 to 15490000 (00001000 bytes committed)
42: 17ad0000
Segment at 17ad0000 to 17b10000 (00001000 bytes committed)
43: 1b2f0000
Segment at 1b2f0000 to 1b300000 (00010000 bytes committed)
Segment at 1ad30000 to 1ae30000 (0002c000 bytes committed)
44: 232b0000
Segment at 232b0000 to 232f0000 (00015000 bytes committed)
45: 21680000
Segment at 21680000 to 216c0000 (00001000 bytes committed)
46: 23490000
Segment at 23490000 to 234d0000 (00001000 bytes committed)
47: 23670000
Segment at 23670000 to 236b0000 (00001000 bytes committed)
48: 17ed0000
Segment at 17ed0000 to 17f10000 (00001000 bytes committed)
49: 247f0000
Segment at 247f0000 to 24830000 (00040000 bytes committed)
50: 28c40000
Segment at 28c40000 to 28c80000 (00040000 bytes committed)
51: 2ffd0000
Segment at 2ffd0000 to 2ffe0000 (00006000 bytes committed)
52: 376b0000
Segment at 376b0000 to 376f0000 (00040000 bytes committed)
53: 2ff90000
Segment at 2ff90000 to 2ffd0000 (00040000 bytes committed)
54: 26260000
Segment at 26260000 to 262a0000 (00040000 bytes committed)
55: 3a530000
Segment at 3a530000 to 3a570000 (00040000 bytes committed)
However I concentrated on CPU spike and !runaway WinDbg command showed the following distribution of thread user mode times:
0:000> !runaway
User Mode Time
Thread Time
117:10a0 0 days 3:09:13.643
13:ca4 0 days 2:18:41.311
61:16c4 0 days 0:25:46.515
33:1690 0 days 0:25:25.954
4:fb0 0 days 0:22:20.797
29:840 0 days 0:21:25.385
23:1614 0 days 0:21:08.194
77:3e0 0 days 0:18:57.434
45:11f4 0 days 0:17:13.647
71:1314 0 days 0:17:10.667
31:1198 0 days 0:16:48.374
39:156c 0 days 0:16:40.980
59:d1c 0 days 0:16:37.610
115:3e8 0 days 0:16:32.384
57:170c 0 days 0:16:30.746
47:1364 0 days 0:16:18.360
84:12a8 0 days 0:15:56.145
112:a10 0 days 0:15:52.089
106:1374 0 days 0:15:51.652
89:b58 0 days 0:15:47.768
125:115c 0 days 0:15:41.122
101:1100 0 days 0:15:30.748
104:1294 0 days 0:15:16.147
99:d00 0 days 0:15:15.008
96:9b4 0 days 0:15:13.604
123:1624 0 days 0:15:12.247
86:1444 0 days 0:15:11.654
131:1728 0 days 0:14:35.914
135:100c 0 days 0:14:16.414
133:1530 0 days 0:14:04.963
137:a30 0 days 0:13:41.360
139:dd8 0 days 0:13:40.674
142:1098 0 days 0:12:51.284
0:efc 0 days 0:02:43.005
1:f44 0 days 0:01:34.536
19:8d0 0 days 0:00:42.557
98:54c 0 days 0:00:28.282
114:138c 0 days 0:00:26.598
83:1060 0 days 0:00:22.354
88:17ec 0 days 0:00:22.027
103:da8 0 days 0:00:20.404
141:15c8 0 days 0:00:19.843
10:b14 0 days 0:00:12.526
8:5b8 0 days 0:00:02.246
21:cfc 0 days 0:00:00.795
12:10c 0 days 0:00:00.561
11:8d4 0 days 0:00:00.312
65:b0c 0 days 0:00:00.202
22:ae8 0 days 0:00:00.187
17:744 0 days 0:00:00.124
28:168c 0 days 0:00:00.093
6:5a8 0 days 0:00:00.046
2:f90 0 days 0:00:00.031
130:fa4 0 days 0:00:00.015
113:17c4 0 days 0:00:00.015
76:1a4 0 days 0:00:00.015
70:10a8 0 days 0:00:00.015
32:df0 0 days 0:00:00.015
18:ee0 0 days 0:00:00.015
7:3f4 0 days 0:00:00.015
148:11cc 0 days 0:00:00.000
147:132c 0 days 0:00:00.000
146:1458 0 days 0:00:00.000
145:133c 0 days 0:00:00.000
144:1268 0 days 0:00:00.000
143:838 0 days 0:00:00.000
140:1168 0 days 0:00:00.000
138:f48 0 days 0:00:00.000
136:1f0 0 days 0:00:00.000
134:17ac 0 days 0:00:00.000
132:119c 0 days 0:00:00.000
129:fc4 0 days 0:00:00.000
128:bd8 0 days 0:00:00.000
127:1528 0 days 0:00:00.000
126:1058 0 days 0:00:00.000
124:16a4 0 days 0:00:00.000
122:1518 0 days 0:00:00.000
121:7c 0 days 0:00:00.000
120:103c 0 days 0:00:00.000
119:a2c 0 days 0:00:00.000
118:1524 0 days 0:00:00.000
116:1240 0 days 0:00:00.000
111:1248 0 days 0:00:00.000
110:de8 0 days 0:00:00.000
109:dc8 0 days 0:00:00.000
108:17e8 0 days 0:00:00.000
107:994 0 days 0:00:00.000
105:162c 0 days 0:00:00.000
102:112c 0 days 0:00:00.000
100:1764 0 days 0:00:00.000
97:1548 0 days 0:00:00.000
95:1334 0 days 0:00:00.000
94:1024 0 days 0:00:00.000
93:1170 0 days 0:00:00.000
92:12f0 0 days 0:00:00.000
91:12d4 0 days 0:00:00.000
90:1264 0 days 0:00:00.000
87:12d8 0 days 0:00:00.000
85:153c 0 days 0:00:00.000
82:14c4 0 days 0:00:00.000
81:834 0 days 0:00:00.000
80:17f4 0 days 0:00:00.000
79:1784 0 days 0:00:00.000
78:530 0 days 0:00:00.000
75:1320 0 days 0:00:00.000
74:15fc 0 days 0:00:00.000
73:16e4 0 days 0:00:00.000
72:17b0 0 days 0:00:00.000
69:af0 0 days 0:00:00.000
68:83c 0 days 0:00:00.000
67:b78 0 days 0:00:00.000
66:cc4 0 days 0:00:00.000
64:14fc 0 days 0:00:00.000
63:14dc 0 days 0:00:00.000
62:16b0 0 days 0:00:00.000
60:1130 0 days 0:00:00.000
58:1504 0 days 0:00:00.000
56:1160 0 days 0:00:00.000
55:16c0 0 days 0:00:00.000
54:bfc 0 days 0:00:00.000
53:f70 0 days 0:00:00.000
52:1178 0 days 0:00:00.000
51:1448 0 days 0:00:00.000
50:15e8 0 days 0:00:00.000
49:1410 0 days 0:00:00.000
48:10c0 0 days 0:00:00.000
46:14e4 0 days 0:00:00.000
44:1150 0 days 0:00:00.000
43:1454 0 days 0:00:00.000
42:131c 0 days 0:00:00.000
41:8cc 0 days 0:00:00.000
40:17bc 0 days 0:00:00.000
38:17c0 0 days 0:00:00.000
37:15a4 0 days 0:00:00.000
36:1048 0 days 0:00:00.000
35:143c 0 days 0:00:00.000
34:1384 0 days 0:00:00.000
30:fa0 0 days 0:00:00.000
27:1688 0 days 0:00:00.000
26:1684 0 days 0:00:00.000
25:1680 0 days 0:00:00.000
24:161c 0 days 0:00:00.000
20:500 0 days 0:00:00.000
16:1a0 0 days 0:00:00.000
15:a18 0 days 0:00:00.000
14:c44 0 days 0:00:00.000
9:6c4 0 days 0:00:00.000
5:ec8 0 days 0:00:00.000
3:fa8 0 days 0:00:00.000
Threads 117 and 13 were waiting for a critical section 6e1876c4:
0:000> ~117kv
ChildEBP RetAddr Args to Child
35f0e468 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet
35f0e46c 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
35f0e4d0 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155
35f0e4f8 6e16ac32 6e1876c4 00071370 35f0e59c ntdll!RtlEnterCriticalSection+0x152
35f0e510 6e16b4cc 6e16e2f1 00000000 35f0e59c AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22
35f0e524 6e16bb90 00071370 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3
35f0e544 6e16bbce 00071370 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a
35f0e574 6e16e327 00071370 00000002 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26
*** ERROR: Symbol file could not be found. Defaulted to export symbols for PDFCreator_Toolbar.dll -
35f0e620 05729772 00071370 00000002 35f0e690 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
35f0e6a4 76b60528 c02193db 00000128 00000000 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
35f0e6dc 73207be1 000319f8 00000128 00030001 user32!DefWindowProcW+0×86
76b60528 90909090 fffffffe 00000000 ffffffd0 comctl32!ToolbarWndProc+0×14f7
76b60528 00000000 fffffffe 00000000 ffffffd0 0×90909090
0:000> ~13kv
ChildEBP RetAddr Args to Child
0c90e5ec 77009254 76ff33b4 00000520 00000000 ntdll!KiFastSystemCallRet
0c90e5f0 76ff33b4 00000520 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
0c90e654 76ff323c 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x155
0c90e67c 6e16ac32 6e1876c4 00071348 0c90e720 ntdll!RtlEnterCriticalSection+0x152
0c90e694 6e16b4cc 6e16e2f1 00000000 0c90e720 AcRedir!NS_RedirectRegistry::RedirectorRegistry::LookupKOECache+0×22
0c90e6a8 6e16bb90 00071348 00000000 00000000 AcRedir!NS_RedirectRegistry::RedirectorRegistry::PreChecks+0xd3
0c90e6c8 6e16bbce 00071348 00000000 00000008 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeMergeW+0×1a
0c90e6f8 6e16e327 00071348 0000000c 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×26
0c90e7a4 05729772 00071348 0000000c 0c90e814 AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
0c90e858 76b60528 73207be1 000205e2 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
0c90e8d4 76b5f8d2 626f6441 44502065 00200046 user32!DefWindowProcW+0×86
0c90e978 76b60817 0041fecc 73207ae0 000205e2 user32!InternalCallWinProc+0×23
00030ad4 0031002e 00300038 00350036 006e005f user32!DispatchClientMessage+0xda
00030ad4 00000000 00300038 00350036 006e005f 0×31002e
Examining critical section locks showed this section to be the only one locked and having high contention:
0:000> !locks
CritSec AcRedir!NS_RedirectRegistry::RedirectorRegistry::ClassLock+0 at 6e1876c4
WaiterWoken No
LockCount 32
RecursionCount 1
OwningThread d1c
EntryCount 0
ContentionCount c74ad4
*** Locked
Scanned 22054 critical sections
There were 32 threads waiting on it. Examining its owning thread d1c showed similar stack trace pattern:
0:000> ~~[d1c]kv
ChildEBP RetAddr Args to Child
269ae72c 6e16f1da 269ae808 31f4a7e8 269ae75c AcRedir!NS_RedirectRegistry::OwnedRegistryKeyPair::Match+0×14
269ae73c 6e16f40c 269ae7ec 269ae808 269ae808 AcRedir!NS_RedirectRegistry::MergedRegistryKey::Match+0×22
269ae75c 6e16bc11 269ae7ec 269ae808 269ae784 AcRedir!NS_RedirectRegistry::MergedRegistryKeyList::FindItem+0×25
269ae790 6e16e327 00c211b0 00000008 00000002 AcRedir!NS_RedirectRegistry::RedirectorRegistry::InitializeEnumeration+0×69
269ae83c 05729772 000714a4 00000008 269ae8ac AcRedir!NS_RedirectRegistry::APIHook_RegEnumValueA+0×36
WARNING: Stack unwind information not available. Following frames may be wrong.
269ae8f0 76b60528 73207be1 00050cf8 00000128 PDFCreator_Toolbar!DllUnregisterServer+0×3b7ce
269ae96c 76b5f8d2 00000001 00070598 00040582 user32!DefWindowProcW+0×86
269aea10 76b60817 0041fecc 73207ae0 00050cf8 user32!InternalCallWinProc+0×23
00030ad4 0031002e 00300038 00350036 006e005f user32!DispatchClientMessage+0xda
00030ad4 00000000 00300038 00350036 006e005f 0×31002e
Two components immediately came to suspicion, AcRedir.dll and PDFCreator_Toolbar.dll:
0:000> lmv m AcRedir
start end module name
6e150000 6e18e000 AcRedir (pdb symbols) c:\mss\AcRedir.pdb\923AF38F594246C99580DC1CFB4B3AE02\AcRedir.pdb
Loaded symbol image file: AcRedir.dll
Image path: C:\Windows\AppPatch\AcRedir.dll
Image name: AcRedir.dll
Timestamp: Sat Jan 19 07:26:39 2008 (4791A62F)
CheckSum: 0003F278
ImageSize: 0003E000
File version: 6.0.6001.18000
Product version: 6.0.6001.18000
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: Microsoft® Windows® Operating System
OriginalFilename: Microsoft® Windows® Operating System
ProductVersion: 6.0.6001.18000
FileVersion: 6.0.6001.18000 (longhorn_rtm.080118-1840)
FileDescription: Windows Compatibility DLL
LegalCopyright: © Microsoft Corporation. All rights reserved.
0:000> lmv m PDFCreator_Toolbar
start end module name
056e0000 057bb000 PDFCreator_Toolbar (export symbols) PDFCreator_Toolbar.dll
Loaded symbol image file: PDFCreator_Toolbar.dll
Image path: C:\Program Files\PDFCreator Toolbar\v3.3.0.1\PDFCreator_Toolbar.dll
Image name: PDFCreator_Toolbar.dll
Timestamp: Sat Aug 09 08:53:38 2008 (489D4D02)
CheckSum: 000AA334
ImageSize: 000DB000
File version: 3.3.0.1
Product version: 3.3.0.1
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
ProductName: PDFCreator Toolbar
InternalName: PDFCreator Toolbar
OriginalFilename: Toolbar.dll
ProductVersion: 3,3,0,1
FileVersion: 3,3,0,1
FileDescription: PDFCreator Toolbar
LegalCopyright: Copyright 2006
Then I decided to examine some heap blocks from leaked segments and found the prolifiration of UNICODE string fragments containing “PDFCreator Toolbar”:
0:000> dc 567b0000 l3000
[...]
567b21a0 00000001 00000008 00000040 00650054 ............T.e.
567b21b0 0070006d 00610044 00610074 00730000 m.p.D.a.t.a...s.
567b21c0 00740069 006f0069 0000006e 00000072 i.t.i.o.n...r...
567b21d0 00000068 005c0067 00440050 00430046 h…g.\.P.D.F.C.
567b21e0 00650072 00740061 0072006f 00540020 r.e.a.t.o.r. .T.
567b21f0 006f006f 0062006c 00720061 002d0000 o.o.l.b.a.r…-.
567b2200 00300031 00300030 00300000 00420025 1.0.0.0…0.%.B.
567b2210 00250030 00310044 00380025 00250031 0.%.D.1.%.8.1.%.
567b2220 00310044 00380025 00000031 00000000 D.1.%.8.1…….
567b2230 52332e04 88000000 00000001 00000013 ..3R…………
567b2240 00000040 00690044 00450064 0061006e @…D.i.d.E.n.a.
567b2250 006c0062 00410065 00740075 0053006f b.l.e.A.u.t.o.S.
567b2260 00610065 00630072 00000068 005c0067 e.a.r.c.h…g.\.
567b2270 00440050 00430046 00650072 00740061 P.D.F.C.r.e.a.t.
567b2280 0072006f 00540020 006f006f 0062006c o.r. .T.o.o.l.b.
567b2290 00720061 002d0000 00300031 00300030 a.r…-.1.0.0.0.
567b22a0 00300000 00420025 00250030 00310044 ..0.%.B.0.%.D.1.
567b22b0 00380025 00250031 00310044 00380025 %.8.1.%.D.1.%.8.
567b22c0 00000031 00000000 52332e1b 88000000 1………3R….
567b22d0 00000001 00000005 00000040 004c0053 …………S.L.
567b22e0 00730069 00000074 00450052 0070005c i.s.t…R.E.\.p.
567b22f0 00660064 006f0066 00670072 002e0065 d.f.f.o.r.g.e…
567b2300 0072006f 005c0067 00440050 00430046 o.r.g.\.P.D.F.C.
567b2310 00650072 00740061 0072006f 00540020 r.e.a.t.o.r. .T.
567b2320 006f006f 0062006c 00720061 00300000 o.o.l.b.a.r…0.
567b2330 00420025 00250042 00300044 00420025 %.B.B.%.D.0.%.B.
567b2340 00250030 00310044 00380025 00250031 0.%.D.1.%.8.1.%.
567b2350 00310044 00380025 00000031 00000000 D.1.%.8.1…….
567b2360 52332e2e 88000000 00000001 00000005 ..3R…………
Because AcRedir code was called from PDFCreator Toolbar component the final decision was to uninstall PDFCreator Toolbar. Before I quit the debugger I dumped the list of modules and was astonished at the module variety:
0:000> lm
start end module name
00850000 008eb000 iexplore (pdb symbols)
05430000 054ab000 ssv (deferred)
056e0000 057bb000 PDFCreator_Toolbar (export symbols)
0a7f0000 0aa73000 igdumd32 (deferred)
10000000 100a4000 swg (deferred)
16080000 160a5000 mdnsNSP (deferred)
28f90000 28f9a000 icalogon (deferred)
29330000 29337000 PScript (deferred)
29440000 29462000 ctxmui (deferred)
29470000 29476000 TcpPServ (deferred)
29480000 29492000 CgpCore (deferred)
295a0000 295b0000 confmgr (deferred)
295b0000 295b7000 logging (deferred)
296c0000 296c7000 icafile (deferred)
296d0000 296d6000 cgpcfg (deferred)
296e0000 296e5000 ctxmuiUI (deferred)
2bd20000 2bd8e000 Wfica (deferred)
30000000 303ae000 Flash9e (export symbols)
63f00000 63f0c000 mscorie (deferred)
655e0000 65639000 rpbrowserrecordplugin (deferred)
692a0000 69a66000 wmploc (deferred)
69a70000 6a4a0000 wmp (deferred)
6b220000 6b2e3000 VGX (deferred)
6b2f0000 6b3bc000 d3dim700 (deferred)
6b450000 6b4d3000 AdobeUpdater (deferred)
6b4e0000 6b7de000 agcore (deferred)
6b7e0000 6ba63000 fastsearch_219B3E1547538286 (deferred)
6ba70000 6be17000 GoogleToolbarDynamic_F423308312A7B033 (export symbols)
6be20000 6be89000 vbscript (deferred)
6bf90000 6c302000 mshtml (export symbols)
6c320000 6c36a000 ntshrui (deferred)
6c3d0000 6c447000 mshtmled (deferred)
6c4d0000 6c527000 dxtmsft (deferred)
6c590000 6c60d000 jscript (pdb symbols)
6c610000 6c649000 dxtrans (pdb symbols)
6c770000 6c7e0000 dsound (deferred)
6c810000 6c839000 msls31 (deferred)
6ca50000 6ca6b000 cryptnet (deferred)
6ca90000 6cada000 rasapi32 (deferred)
6cca0000 6ccb4000 rasman (deferred)
6cd40000 6cd71000 tapi32 (deferred)
6d0b0000 6d0fc000 Wpc (deferred)
6d350000 6d410000 npctrl (deferred)
6d450000 6d482000 iepeers (deferred)
6d4d0000 6d530000 ieapfltr (deferred)
6d5b0000 6d603000 AcroIEFavClient (deferred)
6d6b0000 6d795000 ddraw (deferred)
6d7a0000 6d818000 AcSpecfc (deferred)
6d820000 6d82e000 pngfilt (deferred)
6d830000 6d892000 mscms (deferred)
6dbc0000 6dc5b000 msvcr80 (deferred)
6dc60000 6dce7000 msvcp80 (deferred)
6dd70000 6ddf8000 AcLayers (deferred)
6de00000 6de0a000 ddrawex (deferred)
6de60000 6de83000 msvfw32 (deferred)
6dfb0000 6dfc1000 AcroIEHelperShim (deferred)
6dff0000 6e036000 GoogleToolbar (deferred)
6e060000 6e086000 dssenh (deferred)
6e090000 6e0f0000 tiptsf (deferred)
6e0f0000 6e11f000 ieui (pdb symbols)
6e130000 6e140000 AcroIEHelper (deferred)
6e150000 6e18e000 AcRedir (pdb symbols)
6e570000 6e57b000 msimtf (deferred)
6e580000 6e58f000 davclnt (deferred)
6e590000 6e5a3000 ntlanman (deferred)
6e610000 6e618000 drprov (deferred)
6e620000 6e630000 iebrshim (deferred)
6e650000 6e680000 mlang (deferred)
6f7b0000 6f7b8000 dispex (deferred)
6f8a0000 6f8ab000 cscapi (deferred)
6fa70000 6fb4c000 dbghelp (deferred)
6fe40000 6fe73000 msrating (deferred)
6ff00000 6ff3a000 sqlite (deferred)
70530000 70afe000 ieframe (pdb symbols)
71260000 71462000 msi (deferred)
717c0000 717d2000 pnrpnsp (deferred)
71870000 71877000 wsock32 (deferred)
718a0000 718a7000 msiltcfg (pdb symbols)
71920000 71973000 actxprxy (deferred)
71980000 7198c000 wshbth (deferred)
71990000 71998000 winrnr (deferred)
719e0000 71b06000 msxml3 (deferred)
71b10000 71b1f000 NapiNSP (deferred)
71b20000 71b29000 linkinfo (deferred)
71c70000 71c76000 SensApi (deferred)
71d10000 71e56000 browseui (deferred)
71ee0000 71fe7000 shdocvw (deferred)
72100000 72109000 snmpapi (deferred)
72580000 725c2000 winspool (deferred)
725d0000 725d6000 rasadhlp (deferred)
72610000 72615000 sfc (deferred)
72620000 7262c000 dwmapi (deferred)
72640000 72676000 mfplat (deferred)
72850000 72857000 midimap (deferred)
72860000 72874000 msacm32_72860000 (deferred)
72880000 72933000 WindowsCodecs (deferred)
72940000 729a6000 AudioEng (deferred)
729b0000 729d1000 AudioSes (deferred)
729e0000 72a0f000 wdmaud (pdb symbols)
72a50000 72a59000 msacm32 (deferred)
72a60000 72a64000 ksuser (deferred)
72a70000 72aa2000 winmm (pdb symbols)
72b60000 72b6c000 imgutil (deferred)
72b80000 72b8d000 sfc_os (deferred)
72b90000 72bae000 shimeng (deferred)
72bb0000 72bb6000 dciman32 (deferred)
72c60000 72c8f000 xmllite (deferred)
72c90000 72c9c000 rtutils (deferred)
72ed0000 72f66000 FWPUCLNT (deferred)
73080000 731ca000 msxml6 (deferred)
731d0000 731d5000 msimg32 (deferred)
73200000 73285000 comctl32 (pdb symbols)
73290000 732ef000 winhttp (deferred)
73380000 733b9000 oleacc (deferred)
733c0000 733ff000 uxtheme (deferred)
73400000 73430000 duser (deferred)
73430000 735db000 GdiPlus (deferred)
738d0000 7398b000 propsys (deferred)
74460000 74474000 atl (deferred)
74580000 7471e000 comctl32_74580000 (pdb symbols)
74890000 748b7000 MMDevAPI (deferred)
74960000 74975000 cabinet (deferred)
74980000 749ad000 wintrust (deferred)
74a40000 74a4f000 nlaapi (deferred)
74a50000 74a5a000 wtsapi32 (deferred)
74b10000 74b15000 WSHTCPIP (deferred)
74b20000 74b27000 avrt (deferred)
74b30000 74b4a000 powrprof (deferred)
74b50000 74b71000 ntmarta (deferred)
74bb0000 74beb000 rsaenh (deferred)
74c20000 74c64000 schannel (deferred)
74dd0000 74de5000 gpapi (deferred)
74ed0000 74f0b000 mswsock (pdb symbols)
74f10000 74f55000 bcrypt (deferred)
74f60000 74f95000 ncrypt (deferred)
74fb0000 74fd1000 dhcpcsvc6 (deferred)
74fe0000 74fe7000 winnsi (deferred)
74ff0000 75025000 dhcpcsvc (deferred)
75030000 75049000 IPHLPAPI (deferred)
75050000 75090000 wevtapi (deferred)
75090000 750ca000 SLC (deferred)
750d0000 751c1000 crypt32 (deferred)
75200000 75214000 mpr (deferred)
75260000 75265000 wship6 (deferred)
75270000 75278000 version (deferred)
75280000 75287000 credssp (deferred)
752c0000 752d2000 msasn1 (deferred)
752e0000 752f1000 samlib (deferred)
75300000 7532c000 dnsapi (deferred)
75360000 753d5000 netapi32 (deferred)
755a0000 755ff000 sxs (deferred)
75660000 7568c000 apphelp (deferred)
756c0000 756d4000 secur32 (deferred)
756e0000 756fe000 userenv (deferred)
75820000 75865000 iertutil (deferred)
75870000 76380000 shell32 (deferred)
76380000 763ca000 Wldap32 (deferred)
763d0000 76428000 shlwapi (deferred)
76430000 764b4000 clbcatq (deferred)
764c0000 76588000 msctf (deferred)
76590000 765b9000 imagehlp (deferred)
765c0000 76682000 rpcrt4 (pdb symbols)
76690000 76760000 wininet (pdb symbols)
76760000 767ab000 gdi32 (deferred)
767b0000 767dd000 ws2_32 (pdb symbols)
767e0000 76924000 ole32 (pdb symbols)
76930000 76aba000 setupapi (deferred)
76ac0000 76b33000 comdlg32 (deferred)
76b40000 76bdd000 user32 (pdb symbols)
76be0000 76cbb000 kernel32 (pdb symbols)
76cc0000 76d3d000 usp10 (deferred)
76d40000 76dea000 msvcrt (pdb symbols)
76df0000 76f19000 urlmon (deferred)
76f20000 76fad000 oleaut32 (deferred)
76fb0000 770d7000 ntdll (pdb symbols)
770e0000 770e7000 psapi (deferred)
770f0000 770f9000 lpk (deferred)
77100000 7711e000 imm32 (deferred)
77120000 77126000 nsi (deferred)
77130000 77133000 normaliz (deferred)
77140000 77206000 advapi32 (deferred)
79000000 79046000 mscoree (deferred)
7c340000 7c396000 msvcr71 (deferred)
7c3a0000 7c41b000 msvcp71 (deferred)
Actually, before I quit the debugger, I saved a secured stripped version of the dump file using this command:
0:000> .dump /mrRFt c:\UserDumps\ie7_pattern_cooperation.dmp
The dump file is available on ftp:
ftp://dumpanalysis.org/pub/ie7_pattern_cooperation.zip
Thread times and stack traces are available in it together with module information. However heap data and critical section list was not included in it.
- Dmitry Vostokov @ DumpAnalysis.org -