Crash Dump Analysis Patterns (Part 24)

Raw stack dumps can be useful for finding any suspicious modules that might have caused the problem. For example, it is common for some programs to install hooks to monitor GUI changes, intercept window messages to provide value added services on top of the existing applications. These hooks are implemented as DLLs. Another use would be to examine raw stack data for printer drivers that caused problems before. The fact that these modules had been loaded doesn’t mean that they were used. If we find references to their code it would mean that they might have been used.

However when looking at raw stack dump with symbol information we should be aware of Coincidental Symbolic Information pattern. Here is the first example. Loading the crash dump and displaying the problem thread stack shows the following reference:

...
...
...
00b1ed00  0063006f
00b1ed04  006d0075
00b1ed08  006e0065
00b1ed0c  00200074
00b1ed10  006f004c
00b1ed14  00640061
00b1ed18  00720065
00b1ed1c  005b0020
00b1ed20  00500055
00b1ed24  003a0044
00b1ed28  00430050 Application!Array::operator=+0x2f035
00b1ed2c  0035004c
00b1ed30  005d0063
00b1ed34  00630000
00b1ed38  0000005d
...
...
...

Applying symbols gives us more meaningful name:

...
...
...
00b1ed00  0063006f
00b1ed04  006d0075
00b1ed08  006e0065
00b1ed0c  00200074
00b1ed10  006f004c
00b1ed14  00640061
00b1ed18  00720065
00b1ed1c  005b0020
00b1ed20  00500055
00b1ed24  003a0044
00b1ed28  00430050 Application!Print::DocumentLoad+0x5f
00b1ed2c  0035004c
00b1ed30  005d0063
00b1ed34  00630000
...
...
...

However this is the pure coincidence. The data pattern 00NN00NN clearly belongs to a Unicode string:

0:020> du 00b1ed00
00b1ed00  "ocument Loader [UPD:PCL5c]"

It just happens that 00430050 value can be interpreted as an address that falls into Application module address range and its code section:

0:020> lm
start    end        module name
00400000 0044d000   Application

In the second example, the crash dump is from some 3rd-party application called AppSql for which we don’t have PDB files. Also we know that myhook.dll is installed as a system wide hook and had some problems in the past. It is loaded into any address space but is not necessarily used. We want to see if there are traces of it on the problem thread stack. Dumping stack contents shows us the only one reference:

...
...
...
00118cb0  37302f38
00118cb4  00000000
00118cb8  10008e00 myhook!notify_me+0×22c
00118cbc  01400000
00118cc0  00118abc
00118cc4  06a129f0
00118cc8  00118d04
00118ccc  02bc57d0
00118cd0  04ba5d74
00118cd4  00118d30
00118cd8  0000001c
00118cdc  00000010
00118ce0  075922bc
00118ce4  04a732e0
00118ce8  075922bc
00118cec  04a732e0
00118cf0  0066a831 AppSql+0×26a831
00118cf4  04a732d0
00118cf8  02c43190
00118cfc  00000001
00118d00  0000001c
00118d04  00118d14
00118d08  0049e180 AppSql+0×9e180
00118d0c  02c43190
00118d10  0000001c
00118d14  00118d34


0:020> lm
start    end        module name
00400000 00ba8000   AppSql
...
...
...
10000000 100e0000   myhook

The address 10008e00 looks very “round” and it might be the set of bit flags and also, if we disassemble the code at this address backwards, we don’t see the usual call instruction that saved that address on the stack:

0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000    add     ecx,80h
10008deb 899578ffffff    mov     dword ptr [ebp-88h],edx
10008df1 89458c          mov     dword ptr [ebp-74h],eax
10008df4 894d98          mov     dword ptr [ebp-68h],ecx
10008df7 6a01            push    1
10008df9 8d45ec          lea     eax,[ebp-14h]
10008dfc 50              push    eax
10008dfd ff75e0          push    dword ptr [ebp-20h]

In contrast, the other two addresses are return addresses saved on the stack:

0:000> ub 0066a831
AppSql+0x26a81e:
0066a81e 8bfb            mov     edi,ebx
0066a820 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]
0066a822 8bca            mov     ecx,edx
0066a824 83e103          and     ecx,3
0066a827 f3a4            rep movs byte ptr es:[edi],byte ptr [esi]
0066a829 8b00            mov     eax,dword ptr [eax]
0066a82b 50              push    eax
0066a82c e8affeffff      call    AppSql+0×26a6e0 (0066a6e0)

0:000> ub 0049e180
AppSql+0x9e16f:
0049e16f cc              int     3
0049e170 55              push    ebp
0049e171 8bec            mov     ebp,esp
0049e173 8b4510          mov     eax,dword ptr [ebp+10h]
0049e176 8b4d0c          mov     ecx,dword ptr [ebp+0Ch]
0049e179 50              push    eax
0049e17a 51              push    ecx
0049e17b e840c61c00      call    AppSql+0×26a7c0 (0066a7c0)

Therefore the appearance of myhook!notify_me+0×22c could be a coincidence unless it was a pointer to a function. However, if it was the function pointer address then it wouldn’t have pointed to the middle of the function call sequence that pushes arguments:

0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000    add     ecx,80h
10008deb 899578ffffff    mov     dword ptr [ebp-88h],edx
10008df1 89458c          mov     dword ptr [ebp-74h],eax
10008df4 894d98          mov     dword ptr [ebp-68h],ecx
10008df7 6a01            push    1
10008df9 8d45ec          lea     eax,[ebp-14h]
10008dfc 50              push    eax
10008dfd ff75e0          push    dword ptr [ebp-20h]
0:000> u 10008e00
myhook!notify_me+0×22c
10008e00 e82ff1ffff      call    myhook!copy_data (10007f34)
10008e05 8b8578ffffff    mov     eax,dword ptr [ebp-88h]
10008e0b 3945ac          cmp     dword ptr [ebp-54h],eax
10008e0e 731f            jae     myhook!notify_me+0×25b (10008e2f)
10008e10 8b4598          mov     eax,dword ptr [ebp-68h]
10008e13 0fbf00          movsx   eax,word ptr [eax]
10008e16 8945a8          mov     dword ptr [ebp-58h],eax
10008e19 8b45e0          mov     eax,dword ptr [ebp-20h]

Also, because we have source code and private symbols, we know that if it was a function pointer then it would have been myhook!notify_me address and not notify_me+0×22c address.

All this evidence supports the hypothesis that myhook occurrence on the problem stack is just the coincidence and should be ignored.

- Dmitry Vostokov @ DumpAnalysis.org -

11 Responses to “Crash Dump Analysis Patterns (Part 24)”

  1. Dmitry Vostokov Says:

    The most coincidental symbolic information I have found so far in one crash dump is accidental correspondence between exported _DebuggerHookData and the location of the postmortem debugger NTSD:

    002dd434 003a0043
    002dd438 0057005c
    002dd43c 004e0049 LegacyApp!_DebuggerHookData+0xc4a5
    002dd440 004f0044 LegacyApp!_DebuggerHookData+0×1c4a0
    002dd444 00530057
    002dd448 0073005c
    002dd44c 00730079
    002dd450 00650074
    002dd454 0033006d
    002dd458 005c0032
    002dd45c 0074006e
    002dd460 00640073
    002dd464 0065002e
    002dd468 00650078

    0:000> du 002dd434
    002dd434 “C:\WINDOWS\system32\ntsd.exe”

  2. Crash Dump Analysis » Blog Archive » Bugtation No.51 Says:

    […] bugtation is quite wise and dedicated to beginners learning WinDbg (see Common Mistakes and Coincidental Symbolic Information for some […]

  3. Crash Dump Analysis » Blog Archive » NULL code pointer, changed environment, hooked functions and execution residue: pattern cooperation Says:

    […] Execution residue from hookA module was also found on the problem thread raw stack and it looks like real code (not a coincidental symbolic information): […]

  4. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 94a) Says:

    […] component we look at execution residue left on their raw stack data. Indeed, we see lots of non-coincidental symbolic references to 3rdPartyExtension […]

  5. Crash Dump Analysis » Blog Archive » 10 Common Mistakes in Memory Analysis (Part 7) Says:

    […] Another common mistake I observe is relying on what debuggers report without double-checking. Present day debuggers, like WinDbg or GDB, are symbol-driven, they do not possess much of that semantic knowledge that a human debugger has. Also, it is better to report more than less: what is irrelevant can be skipped over by a skilled memory analyst but what looks suspicious to the problem at hand shall be double-checked to see if it is not coincidental. One example we consider here is Coincidental Symbolic Information.  […]

  6. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 96) Says:

    […] certain stack traces we should always be aware of Coninsidental Frames similar to Coincidental Symbolic Information pattern for raw stack data. Such frames can lead to a wrong analysis conclusion. Consider this […]

  7. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 98) Says:

    […] We also do a sanity check for coincidental symbols: […]

  8. Crash Dump Analysis » Blog Archive » IRP distribution anomaly, inconsistent dump, execution residue, hardware activity, coincidental symbolic information, not my version, virtualized system: pattern cooperation Says:

    […] But they seem to be coincidental: […]

  9. Crash Dump Analysis » Blog Archive » Spiking thread, main thread, message hooks, hooked functions, semantic split, coincidental symbolic information and not my version: pattern cooperation Says:

    […] find a few references to DllBHooks module and initially 11201000 (DllBHooks+0×1000) looks like coincidental  symbolic information and it is not a meaningful code […]

  10. Crash Dump Analysis » Blog Archive » 10 Common Mistakes in Memory Analysis (Part 9) Says:

    […] see and also double check from disassembly by using u/ub WinDbg command that function names are coincidental. It just happened that ApplicationA module spans the address range including 00bf00be and 00cb00ca […]

  11. Dmitry Vostokov Says:

    Due to possible Disassembly Ambiguity we should try to specify the different number of instructions to disassembly, for example, ub L1, ub L2, …

Leave a Reply