Software Diagnostics Library

Raw stack dumps can be useful for finding any suspicious modules that might have caused the problem. For example, it is common for some programs to install hooks to monitor GUI changes, intercept window messages to provide value added services on top of the existing applications. These hooks are implemented as DLLs. Another use would be to examine raw stack data for printer drivers that caused problems before. The fact that these modules had been loaded doesn’t mean that they were used. If we find references to their code it would mean that they might have been used.

However when looking at raw stack dump with symbol information we should be aware of Coincidental Symbolic Information pattern. Here is the first example. Loading the crash dump and displaying the problem thread stack shows the following reference:

...
...
...
00b1ed00  0063006f
00b1ed04  006d0075
00b1ed08  006e0065
00b1ed0c  00200074
00b1ed10  006f004c
00b1ed14  00640061
00b1ed18  00720065
00b1ed1c  005b0020
00b1ed20  00500055
00b1ed24  003a0044
00b1ed28  00430050 Application!Array::operator=+0x2f035
00b1ed2c  0035004c
00b1ed30  005d0063
00b1ed34  00630000
00b1ed38  0000005d
...
...
...

Applying symbols gives us more meaningful name:

...
...
...
00b1ed00  0063006f
00b1ed04  006d0075
00b1ed08  006e0065
00b1ed0c  00200074
00b1ed10  006f004c
00b1ed14  00640061
00b1ed18  00720065
00b1ed1c  005b0020
00b1ed20  00500055
00b1ed24  003a0044
00b1ed28  00430050 Application!Print::DocumentLoad+0x5f
00b1ed2c  0035004c
00b1ed30  005d0063
00b1ed34  00630000
...
...
...

However this is the pure coincidence. The data pattern 00NN00NN clearly belongs to a Unicode string:

0:020> du 00b1ed00
00b1ed00  "ocument Loader [UPD:PCL5c]"

It just happens that 00430050 value can be interpreted as an address that falls into Application module address range and its code section:

0:020> lm
start    end        module name
00400000 0044d000   Application

In the second example, the crash dump is from some 3rd-party application called AppSql for which we don’t have PDB files. Also we know that myhook.dll is installed as a system wide hook and had some problems in the past. It is loaded into any address space but is not necessarily used. We want to see if there are traces of it on the problem thread stack. Dumping stack contents shows us the only one reference:

...
...
...
00118cb0  37302f38
00118cb4  00000000
00118cb8  10008e00 myhook!notify_me+0×22c
00118cbc  01400000
00118cc0  00118abc
00118cc4  06a129f0
00118cc8  00118d04
00118ccc  02bc57d0
00118cd0  04ba5d74
00118cd4  00118d30
00118cd8  0000001c
00118cdc  00000010
00118ce0  075922bc
00118ce4  04a732e0
00118ce8  075922bc
00118cec  04a732e0
00118cf0  0066a831 AppSql+0×26a831
00118cf4  04a732d0
00118cf8  02c43190
00118cfc  00000001
00118d00  0000001c
00118d04  00118d14
00118d08  0049e180 AppSql+0×9e180
00118d0c  02c43190
00118d10  0000001c
00118d14  00118d34
…
…
…

0:020> lm
start    end        module name
00400000 00ba8000   AppSql
...
...
...
10000000 100e0000   myhook

The address 10008e00 looks very “round” and it might be the set of bit flags and also, if we disassemble the code at this address backwards, we don’t see the usual call instruction that saved that address on the stack:

0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000    add     ecx,80h
10008deb 899578ffffff    mov     dword ptr [ebp-88h],edx
10008df1 89458c          mov     dword ptr [ebp-74h],eax
10008df4 894d98          mov     dword ptr [ebp-68h],ecx
10008df7 6a01            push    1
10008df9 8d45ec          lea     eax,[ebp-14h]
10008dfc 50              push    eax
10008dfd ff75e0          push    dword ptr [ebp-20h]

In contrast, the other two addresses are return addresses saved on the stack:

0:000> ub 0066a831
AppSql+0x26a81e:
0066a81e 8bfb            mov     edi,ebx
0066a820 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]
0066a822 8bca            mov     ecx,edx
0066a824 83e103          and     ecx,3
0066a827 f3a4            rep movs byte ptr es:[edi],byte ptr [esi]
0066a829 8b00            mov     eax,dword ptr [eax]
0066a82b 50              push    eax
0066a82c e8affeffff      call    AppSql+0×26a6e0 (0066a6e0)

0:000> ub 0049e180
AppSql+0x9e16f:
0049e16f cc              int     3
0049e170 55              push    ebp
0049e171 8bec            mov     ebp,esp
0049e173 8b4510          mov     eax,dword ptr [ebp+10h]
0049e176 8b4d0c          mov     ecx,dword ptr [ebp+0Ch]
0049e179 50              push    eax
0049e17a 51              push    ecx
0049e17b e840c61c00      call    AppSql+0×26a7c0 (0066a7c0)

Therefore the appearance of myhook!notify_me+0×22c could be a coincidence unless it was a pointer to a function. However, if it was the function pointer address then it wouldn’t have pointed to the middle of the function call sequence that pushes arguments:

0:000> ub 10008e00
myhook!notify_me+0x211
10008de5 81c180000000    add     ecx,80h
10008deb 899578ffffff    mov     dword ptr [ebp-88h],edx
10008df1 89458c          mov     dword ptr [ebp-74h],eax
10008df4 894d98          mov     dword ptr [ebp-68h],ecx
10008df7 6a01            push    1
10008df9 8d45ec          lea     eax,[ebp-14h]
10008dfc 50              push    eax
10008dfd ff75e0          push    dword ptr [ebp-20h]
0:000> u 10008e00
myhook!notify_me+0×22c
10008e00 e82ff1ffff      call    myhook!copy_data (10007f34)
10008e05 8b8578ffffff    mov     eax,dword ptr [ebp-88h]
10008e0b 3945ac          cmp     dword ptr [ebp-54h],eax
10008e0e 731f            jae     myhook!notify_me+0×25b (10008e2f)
10008e10 8b4598          mov     eax,dword ptr [ebp-68h]
10008e13 0fbf00          movsx   eax,word ptr [eax]
10008e16 8945a8          mov     dword ptr [ebp-58h],eax
10008e19 8b45e0          mov     eax,dword ptr [ebp-20h]

Also, because we have source code and private symbols, we know that if it was a function pointer then it would have been myhook!notify_me address and not notify_me+0×22c address.

All this evidence supports the hypothesis that myhook occurrence on the problem stack is just the coincidence and should be ignored.

- Dmitry Vostokov @ DumpAnalysis.org -

This entry was posted on Thursday, August 30th, 2007 at 2:58 pm and is filed under Crash Dump Analysis, Crash Dump Patterns. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.