Crash Dump Analysis Patterns (Part 24)
Raw stack dumps can be useful for finding any suspicious modules that might have caused the problem. For example, it is common for some programs to install hooks to monitor GUI changes, intercept window messages to provide value added services on top of the existing applications. These hooks are implemented as DLLs. Another use would be to examine raw stack data for printer drivers that caused problems before. The fact that these modules had been loaded doesn’t mean that they were used. If we find references to their code it would mean that they might have been used.
However when looking at raw stack dump with symbol information we should be aware of Coincidental Symbolic Information pattern. Here is the first example. Loading the crash dump and displaying the problem thread stack shows the following reference:
...
...
...
00b1ed00 0063006f
00b1ed04 006d0075
00b1ed08 006e0065
00b1ed0c 00200074
00b1ed10 006f004c
00b1ed14 00640061
00b1ed18 00720065
00b1ed1c 005b0020
00b1ed20 00500055
00b1ed24 003a0044
00b1ed28 00430050 Application!Array::operator=+0x2f035
00b1ed2c 0035004c
00b1ed30 005d0063
00b1ed34 00630000
00b1ed38 0000005d
...
...
...
Applying symbols gives us more meaningful name:
...
...
...
00b1ed00 0063006f
00b1ed04 006d0075
00b1ed08 006e0065
00b1ed0c 00200074
00b1ed10 006f004c
00b1ed14 00640061
00b1ed18 00720065
00b1ed1c 005b0020
00b1ed20 00500055
00b1ed24 003a0044
00b1ed28 00430050 Application!Print::DocumentLoad+0x5f
00b1ed2c 0035004c
00b1ed30 005d0063
00b1ed34 00630000
...
...
...
However this is the pure coincidence. The data pattern 00NN00NN clearly belongs to a Unicode string:
0:020> du 00b1ed00
00b1ed00 "ocument Loader [UPD:PCL5c]"
It just happens that 00430050 value can be interpreted as an address that falls into Application module address range and its code section:
0:020> lm
start end module name
00400000 0044d000 Application
In the second example, the crash dump is from some 3rd-party application called AppSql for which we don’t have PDB files. Also we know that myhook.dll is installed as a system wide hook and had some problems in the past. It is loaded into any address space but is not necessarily used. We want to see if there are traces of it on the problem thread stack. Dumping stack contents shows us the only one reference:
...
...
...
00118cb0 37302f38
00118cb4 00000000
00118cb8 10008e00 myhook!notify_me+0×22c
00118cbc 01400000
00118cc0 00118abc
00118cc4 06a129f0
00118cc8 00118d04
00118ccc 02bc57d0
00118cd0 04ba5d74
00118cd4 00118d30
00118cd8 0000001c
00118cdc 00000010
00118ce0 075922bc
00118ce4 04a732e0
00118ce8 075922bc
00118cec 04a732e0
00118cf0 0066a831 AppSql+0×26a831
00118cf4 04a732d0
00118cf8 02c43190
00118cfc 00000001
00118d00 0000001c
00118d04 00118d14
00118d08 0049e180 AppSql+0×9e180
00118d0c 02c43190
00118d10 0000001c
00118d14 00118d34
…
…
…
0:020> lm
start end module name
00400000 00ba8000 AppSql
...
...
...
10000000 100e0000 myhook
The address 10008e00 looks very “round” and it might be the set of bit flags and also, if we disassemble the code at this address backwards, we don’t see the usual call instruction that saved that address on the stack:
0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000 add ecx,80h
10008deb 899578ffffff mov dword ptr [ebp-88h],edx
10008df1 89458c mov dword ptr [ebp-74h],eax
10008df4 894d98 mov dword ptr [ebp-68h],ecx
10008df7 6a01 push 1
10008df9 8d45ec lea eax,[ebp-14h]
10008dfc 50 push eax
10008dfd ff75e0 push dword ptr [ebp-20h]
In contrast, the other two addresses are return addresses saved on the stack:
0:000> ub 0066a831
AppSql+0x26a81e:
0066a81e 8bfb mov edi,ebx
0066a820 f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
0066a822 8bca mov ecx,edx
0066a824 83e103 and ecx,3
0066a827 f3a4 rep movs byte ptr es:[edi],byte ptr [esi]
0066a829 8b00 mov eax,dword ptr [eax]
0066a82b 50 push eax
0066a82c e8affeffff call AppSql+0×26a6e0 (0066a6e0)
0:000> ub 0049e180
AppSql+0x9e16f:
0049e16f cc int 3
0049e170 55 push ebp
0049e171 8bec mov ebp,esp
0049e173 8b4510 mov eax,dword ptr [ebp+10h]
0049e176 8b4d0c mov ecx,dword ptr [ebp+0Ch]
0049e179 50 push eax
0049e17a 51 push ecx
0049e17b e840c61c00 call AppSql+0×26a7c0 (0066a7c0)
Therefore the appearance of myhook!notify_me+0×22c could be a coincidence unless it was a pointer to a function. However, if it was the function pointer address then it wouldn’t have pointed to the middle of the function call sequence that pushes arguments:
0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000 add ecx,80h
10008deb 899578ffffff mov dword ptr [ebp-88h],edx
10008df1 89458c mov dword ptr [ebp-74h],eax
10008df4 894d98 mov dword ptr [ebp-68h],ecx
10008df7 6a01 push 1
10008df9 8d45ec lea eax,[ebp-14h]
10008dfc 50 push eax
10008dfd ff75e0 push dword ptr [ebp-20h]
0:000> u 10008e00
myhook!notify_me+0×22c
10008e00 e82ff1ffff call myhook!copy_data (10007f34)
10008e05 8b8578ffffff mov eax,dword ptr [ebp-88h]
10008e0b 3945ac cmp dword ptr [ebp-54h],eax
10008e0e 731f jae myhook!notify_me+0×25b (10008e2f)
10008e10 8b4598 mov eax,dword ptr [ebp-68h]
10008e13 0fbf00 movsx eax,word ptr [eax]
10008e16 8945a8 mov dword ptr [ebp-58h],eax
10008e19 8b45e0 mov eax,dword ptr [ebp-20h]
Also, because we have source code and private symbols, we know that if it was a function pointer then it would have been myhook!notify_me address and not notify_me+0×22c address.
All this evidence supports the hypothesis that myhook occurrence on the problem stack is just the coincidence and should be ignored.
- Dmitry Vostokov @ DumpAnalysis.org -
September 10th, 2007 at 1:57 pm
The most coincidental symbolic information I have found so far in one crash dump is accidental correspondence between exported _DebuggerHookData and the location of the postmortem debugger NTSD:
002dd434 003a0043
002dd438 0057005c
002dd43c 004e0049 LegacyApp!_DebuggerHookData+0xc4a5
002dd440 004f0044 LegacyApp!_DebuggerHookData+0×1c4a0
002dd444 00530057
002dd448 0073005c
002dd44c 00730079
002dd450 00650074
002dd454 0033006d
002dd458 005c0032
002dd45c 0074006e
002dd460 00640073
002dd464 0065002e
002dd468 00650078
0:000> du 002dd434
002dd434 “C:\WINDOWS\system32\ntsd.exe”
October 15th, 2008 at 3:57 pm
[…] bugtation is quite wise and dedicated to beginners learning WinDbg (see Common Mistakes and Coincidental Symbolic Information for some […]
February 5th, 2009 at 10:53 am
[…] Execution residue from hookA module was also found on the problem thread raw stack and it looks like real code (not a coincidental symbolic information): […]
November 30th, 2009 at 9:04 pm
[…] component we look at execution residue left on their raw stack data. Indeed, we see lots of non-coincidental symbolic references to 3rdPartyExtension […]
February 8th, 2010 at 10:21 pm
[…] Another common mistake I observe is relying on what debuggers report without double-checking. Present day debuggers, like WinDbg or GDB, are symbol-driven, they do not possess much of that semantic knowledge that a human debugger has. Also, it is better to report more than less: what is irrelevant can be skipped over by a skilled memory analyst but what looks suspicious to the problem at hand shall be double-checked to see if it is not coincidental. One example we consider here is Coincidental Symbolic Information. […]
February 21st, 2010 at 11:54 pm
[…] certain stack traces we should always be aware of Coninsidental Frames similar to Coincidental Symbolic Information pattern for raw stack data. Such frames can lead to a wrong analysis conclusion. Consider this […]
May 8th, 2010 at 11:20 am
[…] We also do a sanity check for coincidental symbols: […]
June 9th, 2010 at 9:31 pm
[…] But they seem to be coincidental: […]
July 7th, 2010 at 4:33 pm
[…] find a few references to DllBHooks module and initially 11201000 (DllBHooks+0×1000) looks like coincidental symbolic information and it is not a meaningful code […]
October 14th, 2010 at 9:52 pm
[…] see and also double check from disassembly by using u/ub WinDbg command that function names are coincidental. It just happened that ApplicationA module spans the address range including 00bf00be and 00cb00ca […]
August 4th, 2018 at 9:01 pm
Due to possible Disassembly Ambiguity we should try to specify the different number of instructions to disassembly, for example, ub L1, ub L2, …