Consider an application randomly crashing at different addresses or hanging sometimes. One day we are lucky to get this process postmortem memory dump:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(f34.c6c): Access violation - code c0000005 (first/second chance not available)
eax=73726946 ebx=00403378 ecx=656c2070 edx=656c2074 esi=00403374 edi=00000004
eip=7d64d233 esp=0012ff24 ebp=0012ff4c iopl=0 nv up ei pl nz ac pe cy
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010217
ntdll!RtlpWaitOnCriticalSection+0xdf:
7d64d233 ff4014 inc dword ptr [eax+14h] ds:002b:7372695a=????????
Aha! It involves critical sections! Let’s see whether we have an instance of Critical Section Corruption pattern. The first disappointment comes when !locks command takes ages to finish so we break it:
0:000> !locks
Stopped scanning because of control-C
Scanned 154686373 critical sections
Next we try to list all of them but without any success:
0:000> !locks -v
CritSec at 00000000 could not be read
Perhaps the critical section was a global variable in a dll that was unloaded?
CritSec at 00000000 could not be read
Perhaps the critical section was a global variable in a dll that was unloaded?
CritSec at 00000000 could not be read
Perhaps the critical section was a global variable in a dll that was unloaded?
CritSec at 00000000 could not be read
Perhaps the critical section was a global variable in a dll that was unloaded?
[...]
Next we look at stack trace to find critical section address:
0:000> kv
ChildEBP RetAddr Args to Child
0012ff4c 7d628576 64726f77 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0xdf
0012ff6c 00401074 00403374 00403394 00000001 ntdll!RtlEnterCriticalSection+0xa8
0012ff7c 004011e9 00000001 004d2fc0 004d3030 application!wmain+0×74
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 application!__tmainCRTStartup+0×10f
0012fff0 00000000 00401332 00000000 00000000 kernel32!BaseProcessStart+0×28
0:000> dt CRITICAL_SECTION 00403374
application!CRITICAL_SECTION
+0×000 DebugInfo : 0×73726946 _RTL_CRITICAL_SECTION_DEBUG
+0×004 LockCount : 1701585008
+0×008 RecursionCount : 1919251571
+0×00c OwningThread : 0×20666f20
+0×010 LockSemaphore : 0×64726f77
+0×014 SpinCount : 0×73
It looks corrupt indeed so let’s see if it has ASCII fragments:
0:000> db 00403374
00403374 46 69 72 73 70 20 6c 65-73 74 65 72 20 6f 66 20 Firsp lester of
00403384 77 6f 72 64 73 00 00 00-00 00 00 00 02 00 00 00 words………..
[…]
0:000> da 00403374
00403374 “Firsp lester of words”
Looks like garbled sentence “First letter of words”. Who wrote this? Sherlock would say: “Elementary, my dear Watson”, take the first letters, literally: “First letter of words”. Flow component or a component with similar name causes corruption at random addresses! We can’t believe this, run lm WinDbg command and to our astonishment we see Flows module:
0:000> lm
start end module name
00400000 00405000 application
00410000 004ab000 advapi32
71c20000 71c32000 tsappcmp
75490000 754f5000 usp10
77ba0000 77bfa000 msvcrt
78130000 781cb000 msvcr80
7d4c0000 7d5f0000 kernel32
7d600000 7d6f0000 ntdll
7d800000 7d890000 gdi32
7d8d0000 7d920000 secur32
7d930000 7da00000 user32
7da20000 7db00000 rpcrt4
7dbc0000 7dbc9000 Flows
7dee0000 7df40000 imm32
Unloaded modules:
77b90000 77b98000 VERSION.dll
76920000 769e2000 USERENV.dll
71c40000 71c97000 NETAPI32.dll
771f0000 77201000 WINSTA.dll
770e0000 771e8000 SETUPAPI.dll
004e0000 00532000 SHLWAPI.dll
69500000 69517000 faultrep.dll
Checking the module information we see that it is the part of some unstable 3rd-party hookware and removing it solves the problem of elusive crashes. The problem solving power of Mind! The example is a bit contrived but my point here is that there are problems computers would never debug and troubleshoot. Answering the question of Dreyfus’ book “What computers still can’t do”: they still can’t debug…
- Dmitry Vostokov @ DumpAnalysis.org -
If you come across this post and wonder ”Why Physics?” here is the background:
