10 Common Mistakes in Memory Analysis (Part 2)

Mistake #2 - Not seeing semantic and pragmatic inconsistencies

Why would FreeHeap need a file name? See Incorrect Stack Trace pattern case study for semantic inconsistency. Why is this function on the stack trace

dll!exit+0x10,834

67,636 bytes long (0×10,834 in decimal)?  

The latter is an example of pragmatic inconsistency and the answer is that we don’t have symbols and the name appears from the DLL export table. The code on the stack has nothing to do with exit action when proper symbols are applied.

Another example. The memory dump of a hanging process has only one thread and it is waiting for an event. Is this the problem in ThreadProc and application logic or in the fact that _endthreadex was called when the thread was created?

STACK_TEXT: 
0379fa50 7642dcea ntdll!NtWaitForMultipleObjects+0x15
0379faec 75e08f76 kernel32!WaitForMultipleObjectsEx+0x11d
0379fb40 75e08fbf user32!RealMsgWaitForMultipleObjectsEx+0x14d
0379fb5c 00f6b45d user32!MsgWaitForMultipleObjects+0x1f
0379fba8 752e29bb application!ThreadProc+0xad
0379fbe0 752e2a47 msvcr80!_endthreadex+0×3b
0379fbe8 7649e3f3 msvcr80!_endthreadex+0xc7
0379fbf4 7773cfed kernel32!BaseThreadInitThunk+0xe
0379fc34 7773d1ff ntdll!__RtlUserThreadStart+0×23
0379fc4c 00000000 ntdll!_RtlUserThreadStart+0×1b

The latter assumption is wrong. The presence of _endthreadex stems from the fact that its address was pushed to let a user thread procedure to automatically call it upon the normal function return:  

0:000> u 752e29bb
msvcr80!_endthreadex+0x3b:
752e29bb 50              push    eax
752e29bc e8bfffffff      call    msvcr80!_endthreadex (752e2980)
752e29c1 8b45ec          mov     eax,dword ptr [ebp-14h]
752e29c4 8b08            mov     ecx,dword ptr [eax]
752e29c6 8b09            mov     ecx,dword ptr [ecx]
752e29c8 894de4          mov     dword ptr [ebp-1Ch],ecx
752e29cb 50              push    eax
752e29cc 51              push    ecx

A thread procedure passed to thread creation API call can be any C function. How would a C/C++ compiler understand that it needs to generate a call to thread exit API especially if ThreadProc is named FooBar and resides in a different compilation unit or a library? It seems logical that the runtime environment provides such an automatic return address dynamically. Also why and how _endthreadex knows about our custom ThreadProc to call it? Looks like inconsistency. The ability to see and reason about them is very important skill in memory dump analysis and debugging. The lack of sufficient unmanaged code programming experience might partly explain many analysis mistakes.

- Dmitry Vostokov @ DumpAnalysis.org -

2 Responses to “10 Common Mistakes in Memory Analysis (Part 2)”

  1. Crash Dump Analysis » Blog Archive » The Importance of Symbols Says:

    […] string comparison function could be called from Application code it has a semantic inconsistency: FreeHeap surely wouldn’t call Application code. So why it appears on the stack trace? To […]

  2. Dmitry Vostokov Says:

    Another example are very large numbers that may result from inconsistent dumps:

    0: kd> !poolused 2
    Sorting by NonPaged Pool Consumed
    Pool Used:
    NonPaged
    RxBm 9 4294967264 0 0 buffering manager
    Nbtm -1 4294967248 0 0 NetBT allocations
    Nb22 -3 4294967152 0 0 NetBT allocations
    NbL1 -1 4294967112 0 0 NetBT allocations
    PsEx -12 4294966624 0 0 Process exit APC , Binary: nt!ps
    IoEr -17 4294966088 0 0 Io error log packets , Binary: nt!io
    SmSr 24 4294966056 -56 4294963712 Server (special build only)
    LBwi -58 4294965904 0 0 Work item
    SmMt -46 4294964872 0 0 mailslot buffer (special build only)
    Icp -65 4294964696 0 0 I/O completion packets queue on a completion ports
    93DC -352 4294958848 0 0 UNKNOWN pooltag '93DC', please update pooltag.txt
    AfdL -235 4294957896 0 0 Afd local address buffer , Binary: afd.sys
    ObSt -43 4294957664 0 0 Object Manager temporary storage , Binary: nt!ob
    tmt. -110 4294956872 0 0 UNKNOWN pooltag ' tmt', please update pooltag.txt
    TCPa -363 4294955680 0 0 TCP/IP network protocol , Binary: TCP
    CM 23 4294955576 -2 11240 Configuration Manager (registry) , Binary: nt!cm
    CcWk -493 4294955464 0 0 Kernel Cache Manager lookaside list , Binary: nt!cc
    TMsc -327 4294954216 0 0 UNKNOWN pooltag 'TMsc', please update pooltag.txt
    NbL2 -123 4294944664 0 0 NetBT allocations
    Psap -405 4294941376 0 0 Block used to hold a user mode APC while its queued to a thread , Binary: nt!ps
    TCiA -300 4294940896 0 0 TCP/IP network protocol , Binary: TCP
    NbL3 -146 4294940432 0 0 NetBT allocations
    LBma -316 4294939488 0 0 Master announce context
    Nbtl -269 4294928560 0 0 NetBT allocations
    AfdE -214 4294907376 0 0 Afd endpoint structure , Binary: afd.sys
    LBpd -581 4294903064 0 0 LM Datagram receiver allocations
    Ntfi -462 4294841632 0 0 IRP_CONTEXT , Binary: ntfs.sys
    TCPA -371 4294830768 0 0 TCP/IP network protocol , Binary: TCP
    TCiV -3100 4294818496 0 0 TCP/IP network protocol , Binary: TCP
    NpFR -280 4294699712 0 0 UNKNOWN pooltag 'NpFR', please update pooltag.txt
    LvKd -139 4294397952 0 0 UNKNOWN pooltag 'LvKd', please update pooltag.txt
    TCPB -510 4294194704 0 0 TCP/IP network protocol , Binary: TCP

Leave a Reply

You must be logged in to post a comment.