Software Diagnostics Library

Sometimes there are similar crashes in multiplatform products where only some potion of Crash Signature is similar. We call such a pattern Crash Signature Invariant, for example:

x86: cmp dword ptr [eax], 1
x64: cmp dword ptr [r10]. 1

One crash dump had the following condensed stack trace: 

0: kd> kc
DriverA
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServiceCopyEnd

with the following faulting instruction:

DriverA+0x1234:
cmp     dword ptr [r11],1 ds:002b:00000000`00000000=????????

A search for DriverA led to this x86 crash analysed some time ago:

0: kd> kc
DriverA
nt!IopfCallDriver
win32k!GreDeviceIoControl
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!xxxWrapRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServicePostCall

0: kd> r
DtiverA+0x1423:
cmp     dword ptr [ecx],1    ds:0023:00000000=????????

We see common function names on both stack traces and overall flow is the same (only 3 functions are omitted in x64 trace); we see the same NULL pointer dereference for the same comparison instruction with the same comparison operand, #1.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you have found module related patterns in a complete memory dump and suspect a particular module it may be worth looking at Module Product Process if it exists especially if this module (component, DLL) has product information or some related hint (lmv or !lmi commands). In complex environments such modules may be loaded not only by hooking mechanisms but also as plugins. If you are not sure if there is any such process the best ways is to get module collection and find a process module that has the same vendor as the module in question. Then such process should also be analysed for anomalies.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a kernel space counterpart of Ubiquitous Component pattern. Such a component especially when it is Top Module can be a sign of Wait Chain(s) and Blocking Module and if it is present in the same process names - a sign of Distributed Wait Chain.

0: kd> !stacks 0 ModuleA
Proc.Thread  .Thread  Ticks   ThreadState Blocker

                            [fffffa800e673b30 svchost.exe]
 534.006240  fffffa801388f5f0 fffd41d9 Blocked    ModuleA+0x12468

                            [fffffa800e705b30 svchost.exe]
 630.000e14  fffffa800edacb50 fffdcf7a Blocked    ModuleA+0x12468
 630.000f04  fffffa8012c2fb50 fffdcf49 Blocked    ModuleA+0x12468
 630.006610  fffffa80134f5b50 fffdcf46 Blocked    ModuleA+0x12468
 630.001cfc  fffffa800f55a2d0 fffdcf44 Blocked    ModuleA+0x12468
 630.003db8  fffffa80121f1540 fffdcf43 Blocked    ModuleA+0x12468
 630.000b9c  fffffa80133d1780 fffdcf3c Blocked    ModuleA+0x12468
 630.0041c4  fffffa8013c77b50 fffdcf43 Blocked    ModuleA+0x12468
 630.00641c  fffffa8012476b50 fffdcf43 Blocked    ModuleA+0x12468
 630.006424  fffffa8013207b50 fffdcf40 Blocked    ModuleA+0x12468
 630.002fcc  fffffa80128f9060 fffdcf3e Blocked    ModuleA+0x12468
 630.003de8  fffffa80139edb50 fffdcf3d Blocked    ModuleA+0x12468
 630.0062c4  fffffa800f5ff2d0 fffdcf3c Blocked    ModuleA+0x12468
 630.0065e8  fffffa80139dcb50 fffdcf3b Blocked    ModuleA+0x12468
 630.004524  fffffa8011e51b50 fffdcf3a Blocked    ModuleA+0x12468
 630.004570  fffffa801346b060 fffdcf39 Blocked    ModuleA+0x12468
 630.00173c  fffffa8010b99b50 fffdcf39 Blocked    ModuleA+0x12468

                            [fffffa800f63db30 iexplore.exe]
24c4.0024c8  fffffa800fe854e0 fffcb6cf Blocked    ModuleA+0x12468

                            [fffffa8010b9ab30 explorer.exe]
2b64.0043d0  fffffa8012e8ab00 fffd9095 Blocked    ModuleA+0x12468

                            [fffffa800fe55060 explorer.exe]
2c80.002e58  fffffa8012e75060 fffba7af Blocked    ModuleA+0x12468

                            [fffffa8010c54b30 iexplore.exe]
2e3c.002e98  fffffa8010c75620 fffcbb7f Blocked    ModuleA+0x12468

                            [fffffa80111c3720 iexplore.exe]
32d8.003230  fffffa80111b1b00 fffd41d9 Blocked    ModuleA+0x12468

                            [fffffa80110cb690 iexplore.exe]
2e74.002854  fffffa8011121b00 fffbe8a4 Blocked    ModuleA+0x12468

                            [fffffa801146cb30 OUTLOOK.EXE]
35cc.0035e8  fffffa8013831b00 fffaf33a Blocked    ModuleA+0x12468

                            [fffffa80105a5640 OUTLOOK.EXE]
3858.00385c  fffffa801133ab00 fffd3691 Blocked    ModuleA+0x12468

                            [fffffa8011998060 explorer.exe]
3d70.004a0c  fffffa80139ddb00 fffd0482 Blocked    ModuleA+0x12468

                            [fffffa8010ff5850 OUTLOOK.EXE]
3540.000458  fffffa8011052b00 fffbd007 Blocked    ModuleA+0x12468

                            [fffffa8011d3d060 OUTLOOK.EXE]
49f8.0049fc  fffffa8011c78060 fffdbbf9 Blocked    ModuleA+0x12468

                            [fffffa801241b060 OUTLOOK.EXE]
4888.005af0  fffffa8012e8eab0 fffae442 Blocked    ModuleA+0x12468
4888.003d24  fffffa800eca7b00 fffae443 Blocked    ModuleA+0x12468

                            [fffffa8012687b30 explorer.exe]
5048.0051fc  fffffa801129cb00 fffca8bf Blocked    ModuleA+0x12468

                            [fffffa8011c1e060 OUTLOOK.EXE]
52c4.00117c  fffffa80130f8710 fffaa157 Blocked    ModuleA+0x12468
52c4.0045fc  fffffa801374f060 fffaa15e Blocked    ModuleA+0x12468

                            [fffffa8011c42b30 explorer.exe]
5898.0001ec  fffffa80137a1b00 fffd8da0 Blocked    ModuleA+0x12468

                            [fffffa8012e04b30 OUTLOOK.EXE]
5a74.004954  fffffa8012e05060 fffa9ff8 Blocked    ModuleA+0x12468

                            [fffffa8010908b30 spoolsv.exe]
2724.004190  fffffa8011ea1060 fffdcafb Blocked    ModuleA+0x12468

                            [fffffa801206eb30 WerFault.exe]
3e50.005424  fffffa8013c5eb00 fffdcf39 Blocked    ModuleA+0x12468

                            [fffffa800f8cf2a0 WerFault.exe]
 9f4.00570c  fffffa8013c8ab00 fffdca9f Blocked    ModuleA+0x12468

                            [fffffa8013af1060 WerFault.exe]
3c74.002b80  fffffa8013c5c060 fffd9dc8 Blocked    ModuleA+0x12468

                            [fffffa800f8053a0 WINWORD.EXE]
3dd0.0066a8  fffffa800ce618c0 fffd7c02 Blocked    ModuleA+0x12468

                            [fffffa8010b66b30 WINWORD.EXE]
62a4.001934  fffffa801368c430 fffd7ce7 Blocked    ModuleA+0x12468

                            [fffffa80141dc060 WerFault.exe]
17d0.0052e4  fffffa801347a060 fffd57b8 Blocked    ModuleA+0x12468

                            [fffffa8012629760 WerFault.exe]
621c.005b64  fffffa8011e395d0 fffc8dc2 Blocked    ModuleA+0x12468

                            [fffffa80131a75d0 explorer.exe]
4884.002b34  fffffa8013dc3b00 fffd67bc Blocked    ModuleA+0x12468

[...]

Threads Processed: 5948

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Processes with one thread like Notepad are rare. Such a process is always suspicious especially if it is a service or belongs to a complex product. We call such a pattern One-Thread Process. Usually this happens when all other threads terminated and the remaining thread is blocked in some wait chain. For example, this process has a thread which is blocked in an ALPC request to itself (the same process):

0: kd> !process fffffa8013ed9b30 ff
PROCESS fffffa8013ed9b30
    SessionId: 0  Cid: 44b4    Peb: 7fffffd8000  ParentCid: 0114
    DirBase: 2da448000  ObjectTable: fffff8a01948c670  HandleCount: 660.
    Image: ServiceA.exe
    VadRoot fffffa801356dd10 Vads 398 Clone 0 Private 5795. Modified 204253. Locked 0.
    DeviceMap fffff8a000008340
    Token                             fffff8a01b546060
    ElapsedTime                       01:32:37.622
    UserTime                          00:00:01.421
    KernelTime                        00:00:01.578
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (1525, 50, 345) (6100KB, 200KB, 1380KB)
    PeakWorkingSetSize                7607
    VirtualSize                       178 Mb
    PeakVirtualSize                   182 Mb
    PageFaultCount                    752709
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      8043

        THREAD fffffa8012caab50  Cid 44b4.4f70  Teb: 000007fffff5a000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) KernelMode Non-Alertable
            fffffa8012caaf18  Semaphore Limit 0x1
        Waiting for reply to ALPC Message fffff8a0194d4780 : queued at port fffffa8012911c80 : owned by process fffffa8013ed9b30
        IRP List:
            fffffa8013923300: (0006,0118) Flags: 00060000  Mdl: 00000000
        Not impersonating
        DeviceMap                 fffff8a000008340
        Owning Process            fffffa8013ed9b30       Image:         ServiceA.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      139828         Ticks: 347372 (0:01:30:27.687)
        Context Switch Count      7380            
        UserTime                  00:00:00.031
        KernelTime                00:00:04.890
        Win32 Start Address ServiceA (0×00000001401156e0)
        Stack Init fffff88014c9ddb0 Current fffff88014c9c6b0
        Base fffff88014c9e000 Limit fffff88014c98000 Call 0
        Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        fffff880`14c9c6f0 fffff800`01873652 nt!KiSwapContext+0×7a
        fffff880`14c9c830 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2
        fffff880`14c9c8c0 fffff800`0189f04f nt!KeWaitForSingleObject+0×19f
        fffff880`14c9c960 fffff800`01b919f6 nt!AlpcpSignalAndWait+0×8f
        fffff880`14c9ca10 fffff800`01b910f0 nt!AlpcpReceiveSynchronousReply+0×46
        fffff880`14c9ca70 fffff800`01b9519d nt!AlpcpProcessSynchronousRequest+0×33d
        fffff880`14c9cbb0 fffff800`01b95276 nt!LpcpRequestWaitReplyPort+0×9c
        fffff880`14c9cc10 fffff800`0187ced3 nt!NtRequestWaitReplyPort+0×76
        fffff880`14c9cc60 fffff800`01879490 nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9cc60)
        fffff880`14c9cdf8 fffff880`05c31050 nt!KiServiceLinkage
        fffff880`14c9ce70 fffff880`045ce005 ModuleA+0×12468
        […]
        fffff880`14c9da10 fffff800`01b9d3b6 nt!IopXxxControlFile+0×607
        fffff880`14c9db40 fffff800`0187ced3 nt!NtDeviceIoControlFile+0×56
        fffff880`14c9dbb0 00000000`76d8138a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9dc20)
        00000000`082af028 000007fe`fd366cf6 ntdll!NtDeviceIoControlFile+0xa
        00000000`082af030 00000000`76c2683f KERNELBASE!TlsGetValue+0×1a36
        00000000`082af0a0 00000001`4019d38c kernel32!DeviceIoControlImplementation+0×7f
        […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

All previous wait chain patterns were about single wait chains. However, it is often a case when there are many different wait chains in a memory dump especially in terminal services environments. There can be ALPC and critical section wait chains at the same time. The can be related or completely disjoint. Here we call a special case of several wait chains having the same structure (and possibly pointing to one direction) Distributed Wait Chain. One such example we put below. In a stack trace collection from a complete memory dump from a hanging system we found several explorer.exe processes with critical section wait chains having the same structure and endpoint of a top and blocking ModuleA:

THREAD fffffa80137cf060  Cid 4884.4f9c  Teb: 000007fffffaa000 Win32Thread: fffff900c0fb98b0 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8013570dc0  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      274752         Ticks: 212448 (0:00:55:19.500)
Context Switch Count      9889                 LargeStack
UserTime                  00:00:00.093
KernelTime                00:00:00.171
Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefdafc608)
Stack Init fffff88013c25db0 Current fffff88013c25900
Base fffff88013c26000 Limit fffff88013c1b000 Call 0
Priority 11 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`13c25940 fffff800`01873652 nt!KiSwapContext+0×7a
fffff880`13c25a80 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2
fffff880`13c25b10 fffff800`01b7768e nt!KeWaitForSingleObject+0×19f
fffff880`13c25bb0 fffff800`0187ced3 nt!NtWaitForSingleObject+0xde
fffff880`13c25c20 00000000`76d8135a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`13c25c20)
00000000`0489e518 00000000`76d7e4e8 ntdll!ZwWaitForSingleObject+0xa
00000000`0489e520 00000000`76d7e3db ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0489e5d0 000007fe`fdf8ff50 ntdll!RtlEnterCriticalSection+0xd1
00000000`0489e600 000007fe`fdf8fbd3 SHELL32!CFSFolder::GetIconOf+0×24b
00000000`0489f3a0 000007fe`fdf903d3 SHELL32!SHGetIconIndexFromPIDL+0×3f
00000000`0489f3d0 00000000`ff900328 SHELL32!SHMapIDListToSystemImageListIndexAsync+0×73
00000000`0489f470 00000000`ff8fff4b Explorer!SFTBarHost::AddImageForItem+0×9c
00000000`0489f4d0 00000000`ff8fd2f1 Explorer!SFTBarHost::_InternalRepopulateList+0×4ad
00000000`0489f5d0 00000000`ff8fd0b4 Explorer!SFTBarHost::_RepopulateList+0×1f3
00000000`0489f600 00000000`ff8fcccd Explorer!SFTBarHost::_OnBackgroundEnumDone+0xc1
00000000`0489f630 00000000`ff8fc9e2 Explorer!SFTBarHost::_WndProc+0×451
00000000`0489f680 00000000`76669bd1 Explorer!SFTBarHost::_WndProc_ProgramsMFU+0×1b
00000000`0489f6b0 00000000`766698da USER32!UserCallWinProcCheckWow+0×1ad
00000000`0489f770 00000000`ff8f1177 USER32!DispatchMessageWorker+0×3b5
00000000`0489f7f0 00000000`ff9130e9 Explorer!CTray::_MessageLoop+0×446
00000000`0489f880 000007fe`fdafc71e Explorer!CTray::MainThreadProc+0×8a
00000000`0489f8b0 00000000`76c2652d SHLWAPI!WrapperThreadProc+0×19b
00000000`0489f9b0 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0489f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .process /r /p fffffa80131a75d0
Implicit process is now fffffa80`131a75d0
Loading User Symbols

0: kd> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x0000000000499d90
Critical section   = 0x000007fefe3d5900 (SHELL32!g_csIconCache+0x0)
LOCKED
LockCount          = 0×2
WaiterWoken        = No
OwningThread       = 0×0000000000002b34
RecursionCount     = 0×1
LockSemaphore      = 0×7F8
SpinCount          = 0×0000000000000000
OwningThread       = .thread fffffa8013dc3b00

THREAD fffffa8013dc3b00  Cid 4884.2b34  Teb: 000007fffffac000 Win32Thread: fffff900c2bc1010 WAIT: (Executive) KernelMode Non-Alertable
    fffff88011c03600  SynchronizationEvent
IRP List:
    fffffa800f8fc790: (0006,0430) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      170052         Ticks: 317148 (0:01:22:35.437)
Context Switch Count      2                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address SHELL32!ShutdownThreadProc (0x000007fefe13ef54)
Stack Init fffff88011c03db0 Current fffff88011c03320
Base fffff88011c04000 Limit fffff88011bfd000 Call 0
Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`11c03360 fffff800`01873652 nt!KiSwapContext+0x7a
fffff880`11c034a0 fffff800`01884a9f nt!KiCommitThreadWait+0x1d2
fffff880`11c03530 fffff880`05c12383 nt!KeWaitForSingleObject+0x19f
fffff880`11c035d0 fffff880`012b9288 ModuleA+0×12468
fffff880`11c03750 fffff880`012b7d1b fltmgr!FltpPerformPostCallbacks+0×368
fffff880`11c03820 fffff880`012b66df fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×39b
fffff880`11c038b0 fffff880`01b895ff fltmgr!FltpDispatch+0xcf
fffff880`11c03a30 fffff800`01b783b4 nt!IopCloseFile+0×11f
fffff880`11c03ac0 fffff800`01b78171 nt!ObpDecrementHandleCount+0xb4
fffff880`11c03b40 fffff800`01b78734 nt!ObpCloseHandleTableEntry+0xb1
fffff880`11c03bd0 fffff800`0187ced3 nt!ObpCloseHandle+0×94
fffff880`11c03c20 00000000`76d8140a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`11c03c20)
00000000`0754f348 000007fe`fd341873 ntdll!NtClose+0xa
00000000`0754f350 00000000`76c32f51 KERNELBASE!CloseHandle+0×13
00000000`0754f380 000007fe`fdaf9690 kernel32!CloseHandleImplementation+0×3d
00000000`0754f490 000007fe`fe191d7f SHLWAPI!CFileStream::Release+0×84
00000000`0754f4c0 000007fe`fe13ed57 SHELL32!IconCacheSave+0×2b7
00000000`0754f780 000007fe`fe13f0c6 SHELL32!CommonRestart+0×2f
00000000`0754f7f0 00000000`76c2652d SHELL32!ShutdownThreadProc+0×172
00000000`0754f820 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0754f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Another best practice that is directly related to productivity is a parallel processing of the same memory dump especially in the case of complete memory dumps. Here an analysis might start with running time consuming scripts that dump all process and threads in the variety of formats such as x64 and x86 thread stack traces. However, if the nature of the problem is such that it is possible to start with some pattern and continue unfolding its analysis then we can do that in parallel. One of examples may be a discovered Incomplete Session with an ALPC Wait Chain. Here we can follow such a wait chain while another WinDbg instance dumps all threads for further pattern search later.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org - 

We continue with best practices, the previous was SCP. The second best practice is to check the system for additional patterns after the main pattern was found (similar to avoiding Common Mistake 8). For example, in the case of a bug check resulted from NULL pointer dereference or any other exception in some 3rd-party driver code don’t stop but look at all CPUs, processes and threads to find any other patterns such as Spiking Threads, Busy System, and Contention. Inspection of associated thread stack traces might reveal the same module and/or give additional clues to system behaviour prior to the fault.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In the past we viewed software traces and logs as temporarily ordered event sequences. Since events are just memory data we have a map

T  -> M

as can be seen in the definition of a software trace. Here we generalize the domain to any arbitrary set, for example, it can be a list of indexes or pointers or even memory itself. The latter map can give us narrative chains such as

M -> M -> M -> M

and even give us a grand unification of memory and log analysis and the possibility to apply software narratology to memory dump analysis as well. We talk about it soon and provide some generalized software narrative examples.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Because we finished the list of 10 common mistakes some time ago we continue with “more” series. A year ago we discovered the need to pay attention to differences between 32-bit and 64-bit versions of critical section structures and the need for explicit symbol qualification in x86 mode to avoid mistakes. This post was in draft since then and we now publish it. Suppose we see the address of a critical section on a 32-bit stack trace:

0:000:x86> kv
ChildEBP RetAddr  Args to Child
0044f40c 774e8dd4 00000a94 00000000 00000000 ntdll_774b0000!ZwWaitForSingleObject+0x15
0044f470 774e8cb8 00000000 00000000 041f4b78 ntdll_774b0000!RtlpWaitOnCriticalSection+0x13e
0044f498 0123f70c 010d97c0 8c62ec9c 010cc5fc ntdll_774b0000!RtlEnterCriticalSection+0×150

0:000:x86> dt _RTL_CRITICAL_SECTION 010d97c0
ntdll!_RTL_CRITICAL_SECTION
+0x000 DebugInfo        : 0x00862680 _RTL_CRITICAL_SECTION_DEBUG
+0x008 LockCount        : 0n1
+0x00c RecursionCount   : 0n103356
+0×010 OwningThread     : 0×00000a94 Void
+0×018 LockSemaphore    : 0×0817d72d Void
+0×020 SpinCount        : 0×6130910c`010d9840

Its owner thread has a94 TID but we don’t see it in the thread list:

0:000:x86> ~
.  0  Id: 19508.17944 Suspend: 0 Teb: 7efdb000 Unfrozen
1  Id: 19508.1922c Suspend: 0 Teb: 7efd8000 Unfrozen
2  Id: 19508.195d4 Suspend: 0 Teb: 7efd5000 Unfrozen
3  Id: 19508.19a80 Suspend: 0 Teb: 7efa7000 Unfrozen
4  Id: 19508.19544 Suspend: 0 Teb: 7efa4000 Unfrozen
5  Id: 19508.1925c Suspend: 0 Teb: 7efa1000 Unfrozen
6  Id: 19508.193d4 Suspend: 0 Teb: 7ef9d000 Unfrozen
7  Id: 19508.19b18 Suspend: 0 Teb: 7ef9a000 Unfrozen
8  Id: 19508.19bfc Suspend: 0 Teb: 7ef97000 Unfrozen
9  Id: 19508.19bc4 Suspend: 0 Teb: 7ef94000 Unfrozen
10  Id: 19508.19a90 Suspend: 0 Teb: 7ef91000 Unfrozen
11  Id: 19508.189c0 Suspend: 0 Teb: 7ef8d000 Unfrozen
12  Id: 19508.193bc Suspend: 0 Teb: 7ef8a000 Unfrozen
13  Id: 19508.18f3c Suspend: 0 Teb: 7ef87000 Unfrozen
14  Id: 19508.18834 Suspend: 0 Teb: 7ef84000 Unfrozen
15  Id: 19508.19aec Suspend: 0 Teb: 7ef81000 Unfrozen
16  Id: 19508.180f4 Suspend: 0 Teb: 7ef7d000 Unfrozen
17  Id: 19508.19a3c Suspend: 0 Teb: 7ef7a000 Unfrozen
18  Id: 19508.1916c Suspend: 0 Teb: 7ef77000 Unfrozen
19  Id: 19508.19324 Suspend: 0 Teb: 7ef74000 Unfrozen
20  Id: 19508.19a78 Suspend: 0 Teb: 7ef71000 Unfrozen
21  Id: 19508.19ad4 Suspend: 0 Teb: 7ef6d000 Unfrozen
22  Id: 19508.19834 Suspend: 0 Teb: 7ef6a000 Unfrozen
23  Id: 19508.19754 Suspend: 0 Teb: 7ef67000 Unfrozen
24  Id: 19508.19aa0 Suspend: 0 Teb: 7ef64000 Unfrozen
25  Id: 19508.19bd0 Suspend: 0 Teb: 7ef61000 Unfrozen
26  Id: 19508.19384 Suspend: 0 Teb: 7ef5d000 Unfrozen
27  Id: 19508.1734c Suspend: 0 Teb: 7ef5a000 Unfrozen
28  Id: 19508.19148 Suspend: 0 Teb: 7ef57000 Unfrozen
29  Id: 19508.19b74 Suspend: 0 Teb: 7ef54000 Unfrozen
30  Id: 19508.18290 Suspend: 0 Teb: 7ef51000 Unfrozen
31  Id: 19508.19a4c Suspend: 0 Teb: 7ef4d000 Unfrozen
32  Id: 19508.19bc0 Suspend: 0 Teb: 7ef4a000 Unfrozen
33  Id: 19508.18bf0 Suspend: 0 Teb: 7ef47000 Unfrozen
34  Id: 19508.1895c Suspend: 0 Teb: 7ef44000 Unfrozen
35  Id: 19508.19314 Suspend: 0 Teb: 7ef41000 Unfrozen
36  Id: 19508.19934 Suspend: 0 Teb: 7ef3a000 Unfrozen
37  Id: 19508.197b0 Suspend: 0 Teb: 7ef31000 Unfrozen
38  Id: 19508.1962c Suspend: 0 Teb: 7ef2d000 Unfrozen
39  Id: 19508.191e0 Suspend: 0 Teb: 7ef2a000 Unfrozen
40  Id: 19508.19438 Suspend: 0 Teb: 7ef27000 Unfrozen
41  Id: 19508.197e8 Suspend: 0 Teb: 7ef24000 Unfrozen
42  Id: 19508.18c38 Suspend: 0 Teb: 7ef21000 Unfrozen
43  Id: 19508.197b4 Suspend: 0 Teb: 7ef1d000 Unfrozen
44  Id: 19508.1978c Suspend: 0 Teb: 7ef1a000 Unfrozen
45  Id: 19508.19b84 Suspend: 0 Teb: 7ef17000 Unfrozen
46  Id: 19508.197a8 Suspend: 0 Teb: 7ef14000 Unfrozen
47  Id: 19508.19660 Suspend: 0 Teb: 7ef3d000 Unfrozen
48  Id: 19508.18574 Suspend: 0 Teb: 7efad000 Unfrozen
49  Id: 19508.17a04 Suspend: 0 Teb: 7efaa000 Unfrozen

We see a correct result if we specify a different structure:

0:000:x86> dt CRITICAL_SECTION 010d97c0
ModuleA!CRITICAL_SECTION
+0x000 DebugInfo        : 0x00862680 _RTL_CRITICAL_SECTION_DEBUG
+0x004 LockCount        : 0n-6
+0x008 RecursionCount   : 0n1
+0×00c OwningThread     : 0×000193bc Void
+0×010 LockSemaphore    : 0×00000a94 Void
+0×014 SpinCount        : 0

This is because the structure definition is from a 32-bit module:

0:000:x86> dt ModuleA!CRITICAL_SECTION
ModuleA!CRITICAL_SECTION
+0x000 DebugInfo        : Ptr32 _RTL_CRITICAL_SECTION_DEBUG
+0x004 LockCount        : Int4B
+0x008 RecursionCount   : Int4B
+0×00c OwningThread     : Ptr32 Void
+0×010 LockSemaphore    : Ptr32 Void
+0×014 SpinCount        : Uint4B

However, the structure we used first is from 64-bit module and has a different offset and size for OwningThread field:

0:000:x86> dt ntdll!_RTL_CRITICAL_SECTION
+0x000 DebugInfo        : Ptr64 _RTL_CRITICAL_SECTION_DEBUG
+0x008 LockCount        : Int4B
+0x00c RecursionCount   : Int4B
+0×010 OwningThread     : Ptr64 Void
+0×018 LockSemaphore    : Ptr64 Void
+0×020 SpinCount        : Uint8B

Because a different 32-bit ntdll module is also loaded we can use it for explicit symbol qualification:

0:000:x86> dt ntdll_774b0000!_RTL_CRITICAL_SECTION
+0×000 DebugInfo        : Ptr32 _RTL_CRITICAL_SECTION_DEBUG
+0×004 LockCount        : Int4B
+0×008 RecursionCount   : Int4B
+0×00c OwningThread     : Ptr32 Void
+0×010 LockSemaphore    : Ptr32 Void
+0×014 SpinCount        : Uint4B

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There is an extension shortcut to the usual WinDbg command .effmach for 64-bit memory dumps of 32-bit processes:

0:000> .load wow64exts

0:000> !sw

Switched to 32bit mode

0:000:x86> !sw

Switched to 64bit mode

Also !k command will display both thread stacks (32-bit and 64-bit):

0:000> !k
Walking 64bit Stack...
Child-SP          RetAddr           Call Site
00000000`0016e018 00000000`74f9aea8 wow64win!NtUserGetMessage+0xa
00000000`0016e020 00000000`74fecf87 wow64win!whNtUserGetMessage+0x30
00000000`0016e080 00000000`74f72776 wow64!Wow64SystemServiceEx+0xd7
00000000`0016e940 00000000`74fed07e wow64cpu!ServiceNoTurbo+0x2d
00000000`0016ea00 00000000`74fec549 wow64!RunCpuSimulation+0xa
00000000`0016ea50 00000000`77c54956 wow64!Wow64LdrpInitialize+0x429
00000000`0016efa0 00000000`77c51a17 ntdll!LdrpInitializeProcess+0x17e4
00000000`0016f490 00000000`77c3c32e ntdll! ?? ::FNODOBFM::`string'+0x29220
00000000`0016f500 00000000`00000000 ntdll!LdrInitializeThunk+0xe
Walking 32bit Stack...
ChildEBP RetAddr
002cf6a0 76ba790d user32!NtUserGetMessage+0x15
002cf6bc 0048148a user32!GetMessageW+0x33
002cf6fc 004816ec notepad!WinMain+0xe6
002cf78c 755533aa notepad!_initterm_e+0x1a1
002cf798 77e29ef2 kernel32!BaseThreadInitThunk+0xe
002cf7d8 77e29ec5 ntdll_77df0000!__RtlUserThreadStart+0x70
002cf7f0 00000000 ntdll_77df0000!_RtlUserThreadStart+0x1b

However, I don’t recommend its usage in iterative scripts because if something goes wrong at one iteration then all subsequent !sw commands will trigger the wrong machine mode but explicit .effmach will set the correct one.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Injected Symbols pattern can be used to add missing symbols when we have Reduced Symbol Information like it was done previously in this old case study. For example, TestWER module was compiled with static MFC and CRT libraries and its private PDB file contains all necessary symbols including MSG structure. We can load that module into notepad.exe process space and apply symbols:

0:000:x86> lm
start             end                 module name
00fc0000 00ff0000   notepad    (pdb symbols)          c:\mss\notepad.pdb\E325F5195AE94FAEB58D25C9DF8C0CFD2\notepad.pdb
10000000 10039000   WinCRT     (deferred)
727f0000 7298e000   comctl32   (deferred)
72aa0000 72af1000   winspool   (deferred)
72b10000 72b19000   version    (deferred)
72e40000 72e48000   wow64cpu   (deferred)
72e50000 72eac000   wow64win   (pdb symbols)          c:\mss\wow64win.pdb\B2D08CC152D64E71B79167DC0A0A53E91\wow64win.pdb
72eb0000 72eef000   wow64      (deferred)
733d0000 733e3000   dwmapi     (deferred)
735b0000 73606000   uxtheme    (deferred)
746f0000 746fc000   CRYPTBASE   (deferred)
74700000 74760000   sspicli    (deferred)
747c0000 74817000   shlwapi    (deferred)
74830000 7547a000   shell32    (deferred)
755d0000 7564b000   comdlg32   (deferred)
75650000 7567e000   imm32      (deferred)
75770000 75810000   advapi32   (deferred)
75810000 75920000   kernel32   (pdb symbols)          c:\mss\wkernel32.pdb\1C690A8592304467BB15A09CEA7180FA2\wkernel32.pdb
75920000 759b0000   gdi32      (deferred)
759b0000 759f7000   KERNELBASE   (deferred)
75a00000 75b00000   user32     (pdb symbols)          c:\mss\wuser32.pdb\0FCE9CC301ED4567A819705B2718E1D62\wuser32.pdb
75b00000 75b8f000   oleaut32   (deferred)
75be0000 75c7d000   usp10      (deferred)
75ff0000 76009000   sechost    (deferred)
76010000 76100000   rpcrt4     (deferred)
76230000 762dc000   msvcrt     (deferred)
76470000 7647a000   lpk        (deferred)
76480000 7654c000   msctf      (deferred)
76550000 766ac000   ole32      (deferred)
766d0000 76753000   clbcatq    (deferred)
76e40000 76fe9000   ntdll      (deferred)
77020000 771a0000   ntdll_77020000   (pdb symbols)          c:\mss\wntdll.pdb\D74F79EB1F8D4A45ABCD2F476CCABACC2\wntdll.pdb

0:000:x86> .sympath+ C:\DebuggingTV\TestWER\x86
Symbol search path is: srv*;C:\DebuggingTV\TestWER\x86
Expanded Symbol search path is: SRV*c:\mss*http://msdl.microsoft.com/download/symbols;c:\debuggingtv\testwer\x86

0:000:x86> .reload /f /i C:\DebuggingTV\TestWER\x86\TestWER.exe=10000000

0:000:x86> lm
start             end                 module name
00fc0000 00ff0000   notepad    (pdb symbols)          c:\mss\notepad.pdb\E325F5195AE94FAEB58D25C9DF8C0CFD2\notepad.pdb
10000000 10039000   TestWER    (private pdb symbols)  c:\debuggingtv\testwer\x86\TestWER.pdb
727f0000 7298e000   comctl32   (deferred)
72aa0000 72af1000   winspool   (deferred)
72b10000 72b19000   version    (deferred)
72e40000 72e48000   wow64cpu   (deferred)
72e50000 72eac000   wow64win   (pdb symbols)          c:\mss\wow64win.pdb\B2D08CC152D64E71B79167DC0A0A53E91\wow64win.pdb
72eb0000 72eef000   wow64      (deferred)
733d0000 733e3000   dwmapi     (deferred)
735b0000 73606000   uxtheme    (deferred)
746f0000 746fc000   CRYPTBASE   (deferred)
74700000 74760000   sspicli    (deferred)
747c0000 74817000   shlwapi    (deferred)
74830000 7547a000   shell32    (deferred)
755d0000 7564b000   comdlg32   (deferred)
75650000 7567e000   imm32      (deferred)
75770000 75810000   advapi32   (deferred)
75810000 75920000   kernel32   (pdb symbols)          c:\mss\wkernel32.pdb\1C690A8592304467BB15A09CEA7180FA2\wkernel32.pdb
75920000 759b0000   gdi32      (deferred)
759b0000 759f7000   KERNELBASE   (deferred)
75a00000 75b00000   user32     (pdb symbols)          c:\mss\wuser32.pdb\0FCE9CC301ED4567A819705B2718E1D62\wuser32.pdb
75b00000 75b8f000   oleaut32   (deferred)
75be0000 75c7d000   usp10      (deferred)
75ff0000 76009000   sechost    (deferred)
76010000 76100000   rpcrt4     (deferred)
76230000 762dc000   msvcrt     (deferred)
76470000 7647a000   lpk        (deferred)
76480000 7654c000   msctf      (deferred)
76550000 766ac000   ole32      (deferred)
766d0000 76753000   clbcatq    (deferred)
76e40000 76fe9000   ntdll      (deferred)
77020000 771a0000   ntdll_77020000   (pdb symbols)          c:\mss\wntdll.pdb\D74F79EB1F8D4A45ABCD2F476CCABACC2\wntdll.pdb

0:000:x86> kv
ChildEBP RetAddr  Args to Child
0013fe34 75a1790d 0013fe74 00000000 00000000 user32!NtUserGetMessage+0x15
0013fe50 00fc148a 0013fe74 00000000 00000000 user32!GetMessageW+0×33
0013fe90 00fc16ec 00fc0000 00000000 00354082 notepad!WinMain+0xe6
0013ff20 758233aa 7efde000 0013ff6c 77059ef2 notepad!_initterm_e+0×1a1
0013ff2c 77059ef2 7efde000 57785ae5 00000000 kernel32!BaseThreadInitThunk+0xe
0013ff6c 77059ec5 00fc3689 7efde000 00000000 ntdll_77020000!__RtlUserThreadStart+0×70
0013ff84 00000000 00fc3689 7efde000 00000000 ntdll_77020000!_RtlUserThreadStart+0×1b

0:000:x86> dt -r MSG 0013fe74
TestWER!MSG
  +0x000 hwnd             : 0x0007149c HWND__
    +0x000 unused           : ??
  +0×004 message          : 0×113
  +0×008 wParam           : 0×38a508
  +0×00c lParam           : 0n1921500630
  +0×010 time             : 0×2079a177
  +0×014 pt               : tagPOINT
    +0×000 x                : 0n1337
    +0×004 y                : 0n448

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have Reduced Symbolic Information for modules which can range from stripped or public symbol files to exported only function names. In such cases we can use API function prototypes, structure definitions and possible String Parameters to make sense of function arguments:

0:000:x86> kv
ChildEBP RetAddr  Args to Child
0013fe34 75a1790d 0013fe74 00000000 00000000 user32!NtUserGetMessage+0x15
0013fe50 00fc148a 0013fe74 00000000 00000000 user32!GetMessageW+0×33
0013fe90 00fc16ec 00fc0000 00000000 00354082 notepad!WinMain+0xe6
0013ff20 758233aa 7efde000 0013ff6c 77059ef2 notepad!_initterm_e+0×1a1
0013ff2c 77059ef2 7efde000 57785ae5 00000000 kernel32!BaseThreadInitThunk+0xe
0013ff6c 77059ec5 00fc3689 7efde000 00000000 ntdll_77020000!__RtlUserThreadStart+0×70
0013ff84 00000000 00fc3689 7efde000 00000000 ntdll_77020000!_RtlUserThreadStart+0×1b

The first parameter of GetMessage API is a pointer to MSG structure:

0:000:x86> dt MSG 0013fe74
Symbol MSG not found.

From MSDN we find this structure definition:

typedef struct tagMSG {
    HWND   hwnd;
    UINT   message;
    WPARAM wParam;
    LPARAM lParam;
    DWORD  time;
    POINT  pt;
} MSG, *PMSG, *LPMSG;

0:000:x86> dc 0013fe74 L7
0013fe74  0007149c 00000113 0038a508 7287c5d6  ……….8….r
0013fe84  2079a177 00000539 000001c0           w.y 9…….

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have a Truncated Stack Trace and need to perform manual stack trace reconstruction of the missing part to get approximate full stack trace. Often we are only able to reconstruct some parts and glue them together perhaps with some missing intermediate frames:

For example, we have this truncated stack trace due to the lack of symbols:

1: kd> k
ChildEBP RetAddr
97543b6c 85adf579 nt!KiTrap0E+0x2ac
WARNING: Stack unwind information not available. Following frames may be wrong.
97543be8 85adf770 myfault+0x579
97543bf4 85adf7fc myfault+0x770
97543c2c 81827ecf myfault+0x7fc
97543c44 81988f65 nt!IofCallDriver+0x63
97543c64 81989f25 nt!IopSynchronousServiceTail+0x1e0
97543d00 8198ee8d nt!IopXxxControlFile+0x6b7
97543d34 8188c96a nt!NtDeviceIoControlFile+0x2a
97543d34 77510f34 nt!KiFastCallEntry+0x12a
0012f9a0 7750f850 ntdll!KiFastSystemCallRet
0012f9a4 77417c92 ntdll!NtDeviceIoControlFile+0xc
0012fa04 00401a5b kernel32!DeviceIoControl+0x14a
0012fa94 7700becf NotMyfault+0x1a5b
0012facc 00000000 USER32!xxxDrawButton+0xc1

Manual stack reconstruction brings this fragment:

1: kd> k L=0012fb94 0012fb94 0012fb94
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012fb94 77001ae8 0x12fb94
0012fc0c 7700286a USER32!UserCallWinProcCheckWow+0x14b
0012fc4c 77002bba USER32!SendMessageWorker+0x4b7
0012fc6c 7700c6b4 USER32!SendMessageW+0x7c
0012fc84 7700c7c9 USER32!xxxButtonNotifyParent+0x41
0012fca0 7700c7e8 USER32!xxxBNReleaseCapture+0xf7
0012fd24 7701632e USER32!ButtonWndProcWorker+0x910
0012fd44 77001a10 USER32!ButtonWndProcA+0x4c
0012fd70 77001ae8 USER32!InternalCallWinProc+0x23
0012fde8 77002a47 USER32!UserCallWinProcCheckWow+0x14b
0012fe4c 77002a98 USER32!DispatchMessageWorker+0x322
0012fe5c 76ff11fc USER32!DispatchMessageW+0xf
0012fe80 76fe98d2 USER32!IsDialogMessageW+0x586
0012fea0 00401cc9 USER32!IsDialogMessageA+0xff
0012ff10 004022ec NotMyfault+0x1cc9
00000000 00000000 NotMyfault+0x22ec

And finally we get the 3rd usual thread start fragment:

1: kd> k L=0012ffa0 0012ffa0 0012ffa0
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012ffa0 77413833 0x12ffa0
0012ffac 774ea9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

Gluing them together we get this approx. stack trace:

97543b6c 85adf579 nt!KiTrap0E+0x2ac
WARNING: Stack unwind information not available. Following frames may be wrong.
97543be8 85adf770 myfault+0x579
97543bf4 85adf7fc myfault+0x770
97543c2c 81827ecf myfault+0x7fc
97543c44 81988f65 nt!IofCallDriver+0x63
97543c64 81989f25 nt!IopSynchronousServiceTail+0x1e0
97543d00 8198ee8d nt!IopXxxControlFile+0x6b7
97543d34 8188c96a nt!NtDeviceIoControlFile+0x2a
97543d34 77510f34 nt!KiFastCallEntry+0x12a
0012f9a0 7750f850 ntdll!KiFastSystemCallRet
0012f9a4 77417c92 ntdll!NtDeviceIoControlFile+0xc
0012fa04 00401a5b kernel32!DeviceIoControl+0x14a
0012fa94 7700becf NotMyfault+0x1a5b
0012fc0c 7700286a USER32!UserCallWinProcCheckWow+0x14b
0012fc4c 77002bba USER32!SendMessageWorker+0x4b7
0012fc6c 7700c6b4 USER32!SendMessageW+0x7c
0012fc84 7700c7c9 USER32!xxxButtonNotifyParent+0x41
0012fca0 7700c7e8 USER32!xxxBNReleaseCapture+0xf7
0012fd24 7701632e USER32!ButtonWndProcWorker+0x910
0012fd44 77001a10 USER32!ButtonWndProcA+0x4c
0012fd70 77001ae8 USER32!InternalCallWinProc+0x23
0012fde8 77002a47 USER32!UserCallWinProcCheckWow+0x14b
0012fe4c 77002a98 USER32!DispatchMessageWorker+0x322
0012fe5c 76ff11fc USER32!DispatchMessageW+0xf
0012fe80 76fe98d2 USER32!IsDialogMessageW+0x586
0012fea0 00401cc9 USER32!IsDialogMessageA+0xff
0012ff10 004022ec NotMyfault+0x1cc9
0012ffac 774ea9bd kernel32!BaseThreadInitThunk+0xe
0012ffec 00000000 ntdll!_RtlUserThreadStart+0x23

We call this pattern Glued Stack Trace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Raw Pointer pattern is about pointers without matching symbol files. They may be in the expected module range or in some other known module range in the form of module+offset or can be completely out of range of any module from the loaded module list and therefore just a number. For example, usually we have certain structures or arrays (tables) where we expect pointers with matching symbols such as IAT, IDT and 32-bit SSDT where an occurrence of a raw pointer immediately triggers a suspicion such as in this Import Address Table from ProcessA:

[...]
00000001`3f8a9048 00000000`76e282d0 ntdll!RtlSizeHeap
00000001`3f8a9050 00000000`76bf9070 kernel32!GetStringTypeWStub
00000001`3f8a9058 00000000`76c03580 kernel32!WideCharToMultiByteStub
00000001`3f8a9060 00000000`76e33f20 ntdll!RtlReAllocateHeap
00000001`3f8a9068 00000000`76e533a0 ntdll!RtlAllocateHeap
00000001`3f8a9070 00000000`76bfc420 kernel32!GetCommandLineWStub
00000001`3f8a9078 00000001`3f8a1638 ProcessA+0×10ac
00000001`3f8a9080 00000000`76c2cc50 kernel32!IsProcessorFeaturePresent
00000001`3f8a9088 00000000`76c02d60 kernel32!GetLastErrorStub
00000001`3f8a9090 00000000`76c02d80 kernel32!SetLastError
00000001`3f8a9098 00000000`76bf3ee0 kernel32!GetCurrentThreadIdStub
[…]

Note that structures are not limited to the above and can me any OS or even application specific structure where we have symbol files. Raw pointers that are outside of expected module range are covered in the next pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Hooksware pattern originally came from memory dump analysis pattern catalog and is too general for malware analysis pattern catalog. So we decided to factor out 3 separate patterns. The first one is called Patched Code and includes cases such as in-place patching:

0:004> u ntdll!ZwQueryDirectoryFile
ntdll!ZwQueryDirectoryFile:
77814db4 b8da000000      mov     eax,0DAh
77814db9 bae8af0500      mov     edx,5AFE8h
77814dbe ff12            call    dword ptr [edx]
77814dc0 c22c00          ret     2Ch
77814dc3 90              nop
ntdll!NtQueryDirectoryObject:
77814dc4 b8db000000      mov     eax,0DBh
77814dc9 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
77814dce ff12            call    dword ptr [edx]

and detour patching:

0:004> u wininet!InternetReadFile
wininet!InternetReadFile:
7758654b e98044ac88      jmp     0004a9d0
77586550 83ec24          sub     esp,24h
77586553 53              push    ebx
77586554 56              push    esi
77586555 57              push    edi
77586556 33ff            xor     edi,edi
77586558 393db8116277    cmp     dword ptr [wininet!GlobalDataInitialized (776211b8)],edi
7758655e 897df4          mov     dword ptr [ebp-0Ch],edi

In case of WinDbg such pattern is usually detected on the crash spot such as from RIP Stack Trace or from !chkimg command output.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As usual a new pattern arises with the need to communicate analysis findings. Most often when analyzing malware we don’t have symbol files (No Component Symbols) for an Unknown Module. By looking at IAT (if any present) we can guess module purpose. Sometimes a module itself is not malicious but is used in the larger malicious context such as screen grabbing:

[...]
10002000  76376101 gdi32!CreateCompatibleDC
10002004  763793d6 gdi32!StretchBlt
10002008  76377461 gdi32!CreateDIBSection
1000200c  763762a0 gdi32!SelectObject
10002010  00000000
10002024  77429ced user32!ReleaseDC
10002028  77423ba7 user32!NtUserGetWindowDC
1000202c  77430e21 user32!GetWindowRect
10002030  00000000
10002034  744a75e9 GdiPlus!GdiplusStartup
10002038  744976dd GdiPlus!GdipSaveImageToStream
1000203c  744cdd38 GdiPlus!GdipGetImageEncodersSize
10002040  744971cf GdiPlus!GdipDisposeImage
10002044  744a8591 GdiPlus!GdipCreateBitmapFromHBITMAP
10002048  744cdbae GdiPlus!GdipGetImageEncoders
[...]

There are also cases where these API names are not in IAT but found as String Hint in raw data such LoadLibrary / GetProcAddress and even a group of modules themselves as a collective API:

[...]
00058e20  "kernel32.dll"
00058e3c  "user32.dll"
00058e54  "ws2_32.dll"
00058e6c  "ntdll.dll"
00058e80  "wininet.dll"
00058e98  "nspr4.dll"
00058eac  "ssl3.dll"
[...]

We name this pattern Namespace.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

It is a well-known fact that page heap is implemented by placing allocations at the end of pages with the next non-accessible page to catch buffer overruns leading to heap corruption. The best way to see it is to use !address command that dumps all such allocations:

0:004> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pages

0:004> !address
[...]
20b10000 20b11000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b11000 20b12000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b12000 20b13000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b13000 20b14000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b14000 20b15000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b15000 20b1a000     5000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1a000 20b1b000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1b000 20b1c000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1c000 20b1d000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1d000 20b1e000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1e000 20b1f000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b1f000 20b20000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
+ 20b20000 20b21000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b21000 20b26000     5000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b26000 20b27000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b27000 20b28000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b28000 20b29000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b29000 20b2a000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b2a000 20b2b000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b2b000 20b2f000     4000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b2f000 20b30000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b30000 20b3f000     f000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b3f000 20b40000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b40000 20b41000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b41000 20b42000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b42000 20b45000     3000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b45000 20b46000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b46000 20b4b000     5000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b4b000 20b4c000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b4c000 20b4d000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b4d000 20b4e000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b4e000 20b4f000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b4f000 20b50000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b50000 20b51000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b51000 20b52000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b52000 20b57000     5000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b57000 20b58000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b58000 20b5d000     5000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b5d000 20b5e000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b5e000 20b5f000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b5f000 20b60000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b60000 20b61000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b61000 20b62000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b62000 20b6b000     9000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b6b000 20b6f000     4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b6f000 20b71000     2000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b71000 20b72000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b72000 20b73000     1000 MEM_PRIVATE MEM_RESERVE                                    PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
20b73000 20b74000     1000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PageHeap   [PageHeap: 1f241000; NormalHeap: 1f410000]
[…]

0:004> dc 20b26000 20b27000
20b26000  00000000 00000000 00000000 00000000  …………….
20b26010  00000000 00000000 00000000 00000000  …………….
20b26020  00000000 00000000 00000000 00000000  …………….
20b26030  00000000 00000000 00000000 00000000  …………….
20b26040  00000000 00000000 00000000 00000000  …………….
20b26050  00000000 00000000 00000000 00000000  …………….
20b26060  00000000 00000000 00000000 00000000  …………….
20b26070  00000000 00000000 00000000 00000000  …………….
20b26080  00000000 00000000 00000000 00000000  …………….
20b26090  00000000 00000000 00000000 00000000  …………….
20b260a0  00000000 00000000 00000000 00000000  …………….
20b260b0  00000000 00000000 00000000 00000000  …………….
20b260c0  00000000 00000000 00000000 00000000  …………….
20b260d0  00000000 00000000 00000000 00000000  …………….
20b260e0  00000000 00000000 00000000 00000000  …………….
20b260f0  00000000 00000000 00000000 00000000  …………….
20b26100  00000000 00000000 00000000 00000000  …………….
20b26110  00000000 00000000 00000000 00000000  …………….
20b26120  00000000 00000000 00000000 00000000  …………….
20b26130  00000000 00000000 00000000 00000000  …………….
20b26140  00000000 00000000 00000000 00000000  …………….
20b26150  00000000 00000000 00000000 00000000  …………….
20b26160  00000000 00000000 00000000 00000000  …………….
20b26170  00000000 00000000 00000000 00000000  …………….
20b26180  00000000 00000000 00000000 00000000  …………….
20b26190  00000000 00000000 00000000 00000000  …………….
20b261a0  00000000 00000000 00000000 00000000  …………….
20b261b0  00000000 00000000 00000000 00000000  …………….
20b261c0  00000000 00000000 00000000 00000000  …………….
20b261d0  00000000 00000000 00000000 00000000  …………….
20b261e0  00000000 00000000 00000000 00000000  …………….
20b261f0  00000000 00000000 00000000 00000000  …………….
20b26200  00000000 00000000 00000000 00000000  …………….
20b26210  00000000 00000000 00000000 00000000  …………….
20b26220  00000000 00000000 00000000 00000000  …………….
20b26230  00000000 00000000 00000000 00000000  …………….
20b26240  00000000 00000000 00000000 00000000  …………….
20b26250  00000000 00000000 00000000 00000000  …………….
20b26260  00000000 00000000 00000000 00000000  …………….
20b26270  00000000 00000000 00000000 00000000  …………….
20b26280  00000000 00000000 00000000 00000000  …………….
20b26290  00000000 00000000 00000000 00000000  …………….
20b262a0  00000000 00000000 00000000 00000000  …………….
20b262b0  00000000 00000000 00000000 00000000  …………….
20b262c0  00000000 00000000 00000000 00000000  …………….
20b262d0  00000000 00000000 00000000 00000000  …………….
20b262e0  00000000 00000000 00000000 00000000  …………….
20b262f0  00000000 00000000 00000000 00000000  …………….
20b26300  00000000 00000000 00000000 00000000  …………….
20b26310  00000000 00000000 00000000 00000000  …………….
20b26320  00000000 00000000 00000000 00000000  …………….
20b26330  00000000 00000000 00000000 00000000  …………….
20b26340  00000000 00000000 00000000 00000000  …………….
20b26350  00000000 00000000 00000000 00000000  …………….
20b26360  00000000 00000000 00000000 00000000  …………….
20b26370  00000000 00000000 00000000 00000000  …………….
20b26380  00000000 00000000 00000000 00000000  …………….
20b26390  00000000 00000000 00000000 00000000  …………….
20b263a0  00000000 00000000 00000000 00000000  …………….
20b263b0  00000000 00000000 00000000 00000000  …………….
20b263c0  00000000 00000000 00000000 00000000  …………….
20b263d0  00000000 00000000 00000000 00000000  …………….
20b263e0  00000000 00000000 00000000 00000000  …………….
20b263f0  00000000 00000000 00000000 00000000  …………….
20b26400  00000000 00000000 00000000 00000000  …………….
20b26410  00000000 00000000 00000000 00000000  …………….
20b26420  00000000 00000000 00000000 00000000  …………….
20b26430  00000000 00000000 00000000 00000000  …………….
20b26440  00000000 00000000 00000000 00000000  …………….
20b26450  00000000 00000000 00000000 00000000  …………….
20b26460  00000000 00000000 00000000 00000000  …………….
20b26470  00000000 00000000 00000000 00000000  …………….
20b26480  00000000 00000000 00000000 00000000  …………….
20b26490  00000000 00000000 00000000 00000000  …………….
20b264a0  00000000 00000000 00000000 00000000  …………….
20b264b0  00000000 00000000 00000000 00000000  …………….
20b264c0  00000000 00000000 00000000 00000000  …………….
20b264d0  00000000 00000000 00000000 00000000  …………….
20b264e0  00000000 00000000 00000000 00000000  …………….
20b264f0  00000000 00000000 00000000 00000000  …………….
20b26500  00000000 00000000 00000000 00000000  …………….
20b26510  00000000 00000000 00000000 00000000  …………….
20b26520  00000000 00000000 00000000 00000000  …………….
20b26530  00000000 00000000 00000000 00000000  …………….
20b26540  00000000 00000000 00000000 00000000  …………….
20b26550  00000000 00000000 00000000 00000000  …………….
20b26560  00000000 00000000 00000000 00000000  …………….
20b26570  00000000 00000000 00000000 00000000  …………….
20b26580  00000000 00000000 00000000 00000000  …………….
20b26590  00000000 00000000 00000000 00000000  …………….
20b265a0  00000000 00000000 00000000 00000000  …………….
20b265b0  00000000 00000000 00000000 00000000  …………….
20b265c0  00000000 00000000 00000000 00000000  …………….
20b265d0  00000000 00000000 00000000 00000000  …………….
20b265e0  00000000 00000000 00000000 00000000  …………….
20b265f0  00000000 00000000 00000000 00000000  …………….
20b26600  00000000 00000000 00000000 00000000  …………….
20b26610  00000000 00000000 00000000 00000000  …………….
20b26620  00000000 00000000 00000000 00000000  …………….
20b26630  00000000 00000000 00000000 00000000  …………….
20b26640  00000000 00000000 00000000 00000000  …………….
20b26650  00000000 00000000 00000000 00000000  …………….
20b26660  00000000 00000000 00000000 00000000  …………….
20b26670  00000000 00000000 00000000 00000000  …………….
20b26680  00000000 00000000 00000000 00000000  …………….
20b26690  00000000 00000000 00000000 00000000  …………….
20b266a0  00000000 00000000 00000000 00000000  …………….
20b266b0  00000000 00000000 00000000 00000000  …………….
20b266c0  00000000 00000000 00000000 00000000  …………….
20b266d0  00000000 00000000 00000000 00000000  …………….
20b266e0  00000000 00000000 00000000 00000000  …………….
20b266f0  00000000 00000000 00000000 00000000  …………….
20b26700  00000000 00000000 00000000 00000000  …………….
20b26710  00000000 00000000 00000000 00000000  …………….
20b26720  00000000 00000000 00000000 00000000  …………….
20b26730  00000000 00000000 00000000 00000000  …………….
20b26740  00000000 00000000 00000000 00000000  …………….
20b26750  00000000 00000000 00000000 00000000  …………….
20b26760  00000000 00000000 00000000 00000000  …………….
20b26770  00000000 00000000 00000000 00000000  …………….
20b26780  00000000 00000000 00000000 00000000  …………….
20b26790  00000000 00000000 00000000 00000000  …………….
20b267a0  00000000 00000000 00000000 00000000  …………….
20b267b0  00000000 00000000 00000000 00000000  …………….
20b267c0  00000000 00000000 00000000 00000000  …………….
20b267d0  00000000 00000000 00000000 00000000  …………….
20b267e0  00000000 00000000 00000000 00000000  …………….
20b267f0  00000000 00000000 00000000 00000000  …………….
20b26800  00000000 00000000 00000000 00000000  …………….
20b26810  00000000 00000000 00000000 00000000  …………….
20b26820  00000000 00000000 00000000 00000000  …………….
20b26830  00000000 00000000 00000000 00000000  …………….
20b26840  00000000 00000000 00000000 00000000  …………….
20b26850  00000000 00000000 00000000 00000000  …………….
20b26860  00000000 00000000 00000000 00000000  …………….
20b26870  00000000 00000000 00000000 00000000  …………….
20b26880  00000000 00000000 00000000 00000000  …………….
20b26890  00000000 00000000 00000000 00000000  …………….
20b268a0  00000000 00000000 00000000 00000000  …………….
20b268b0  00000000 00000000 00000000 00000000  …………….
20b268c0  00000000 00000000 00000000 00000000  …………….
20b268d0  00000000 00000000 00000000 00000000  …………….
20b268e0  00000000 00000000 00000000 00000000  …………….
20b268f0  00000000 00000000 00000000 00000000  …………….
20b26900  00000000 00000000 00000000 00000000  …………….
20b26910  00000000 00000000 00000000 00000000  …………….
20b26920  00000000 00000000 00000000 00000000  …………….
20b26930  00000000 00000000 00000000 00000000  …………….
20b26940  00000000 00000000 00000000 00000000  …………….
20b26950  00000000 00000000 00000000 00000000  …………….
20b26960  00000000 00000000 00000000 00000000  …………….
20b26970  00000000 00000000 00000000 00000000  …………….
20b26980  00000000 00000000 00000000 00000000  …………….
20b26990  00000000 00000000 00000000 00000000  …………….
20b269a0  00000000 00000000 00000000 00000000  …………….
20b269b0  00000000 00000000 00000000 00000000  …………….
20b269c0  00000000 00000000 00000000 00000000  …………….
20b269d0  00000000 00000000 00000000 00000000  …………….
20b269e0  00000000 00000000 00000000 00000000  …………….
20b269f0  00000000 00000000 00000000 00000000  …………….
20b26a00  00000000 00000000 00000000 00000000  …………….
20b26a10  00000000 00000000 00000000 00000000  …………….
20b26a20  00000000 00000000 00000000 00000000  …………….
20b26a30  00000000 00000000 00000000 00000000  …………….
20b26a40  00000000 00000000 00000000 00000000  …………….
20b26a50  00000000 00000000 00000000 00000000  …………….
20b26a60  00000000 00000000 00000000 00000000  …………….
20b26a70  00000000 00000000 00000000 00000000  …………….
20b26a80  00000000 00000000 00000000 00000000  …………….
20b26a90  00000000 00000000 00000000 00000000  …………….
20b26aa0  00000000 00000000 00000000 00000000  …………….
20b26ab0  00000000 00000000 00000000 00000000  …………….
20b26ac0  00000000 00000000 00000000 00000000  …………….
20b26ad0  00000000 00000000 00000000 00000000  …………….
20b26ae0  00000000 00000000 00000000 00000000  …………….
20b26af0  00000000 00000000 00000000 00000000  …………….
20b26b00  00000000 00000000 00000000 00000000  …………….
20b26b10  00000000 00000000 00000000 00000000  …………….
20b26b20  00000000 00000000 00000000 00000000  …………….
20b26b30  00000000 00000000 00000000 00000000  …………….
20b26b40  00000000 00000000 00000000 00000000  …………….
20b26b50  00000000 00000000 00000000 00000000  …………….
20b26b60  00000000 00000000 00000000 00000000  …………….
20b26b70  00000000 00000000 00000000 00000000  …………….
20b26b80  00000000 00000000 00000000 00000000  …………….
20b26b90  00000000 00000000 00000000 00000000  …………….
20b26ba0  00000000 00000000 00000000 00000000  …………….
20b26bb0  00000000 00000000 00000000 00000000  …………….
20b26bc0  00000000 00000000 00000000 00000000  …………….
20b26bd0  00000000 00000000 00000000 00000000  …………….
20b26be0  00000000 00000000 00000000 00000000  …………….
20b26bf0  00000000 00000000 00000000 00000000  …………….
20b26c00  00000000 00000000 00000000 00000000  …………….
20b26c10  00000000 00000000 00000000 00000000  …………….
20b26c20  00000000 00000000 00000000 00000000  …………….
20b26c30  00000000 00000000 00000000 00000000  …………….
20b26c40  00000000 00000000 00000000 00000000  …………….
20b26c50  00000000 00000000 00000000 00000000  …………….
20b26c60  00000000 00000000 00000000 00000000  …………….
20b26c70  00000000 00000000 00000000 00000000  …………….
20b26c80  00000000 00000000 00000000 00000000  …………….
20b26c90  00000000 00000000 00000000 00000000  …………….
20b26ca0  00000000 00000000 00000000 00000000  …………….
20b26cb0  00000000 00000000 00000000 00000000  …………….
20b26cc0  00000000 00000000 00000000 00000000  …………….
20b26cd0  00000000 00000000 00000000 00000000  …………….
20b26ce0  00000000 00000000 00000000 00000000  …………….
20b26cf0  00000000 00000000 00000000 00000000  …………….
20b26d00  00000000 00000000 00000000 00000000  …………….
20b26d10  00000000 00000000 00000000 00000000  …………….
20b26d20  00000000 00000000 00000000 00000000  …………….
20b26d30  00000000 00000000 00000000 00000000  …………….
20b26d40  00000000 00000000 00000000 00000000  …………….
20b26d50  00000000 00000000 00000000 00000000  …………….
20b26d60  00000000 00000000 00000000 00000000  …………….
20b26d70  00000000 00000000 00000000 00000000  …………….
20b26d80  00000000 00000000 00000000 00000000  …………….
20b26d90  00000000 00000000 00000000 00000000  …………….
20b26da0  00000000 00000000 00000000 00000000  …………….
20b26db0  00000000 00000000 00000000 00000000  …………….
20b26dc0  00000000 00000000 00000000 00000000  …………….
20b26dd0  00000000 00000000 00000000 00000000  …………….
20b26de0  00000000 00000000 00000000 00000000  …………….
20b26df0  00000000 00000000 00000000 00000000  …………….
20b26e00  00000000 00000000 00000000 00000000  …………….
20b26e10  00000000 00000000 00000000 00000000  …………….
20b26e20  00000000 00000000 00000000 00000000  …………….
20b26e30  00000000 00000000 00000000 00000000  …………….
20b26e40  00000000 00000000 00000000 00000000  …………….
20b26e50  00000000 00000000 00000000 00000000  …………….
20b26e60  00000000 00000000 00000000 00000000  …………….
20b26e70  00000000 00000000 00000000 00000000  …………….
20b26e80  00000000 00000000 00000000 00000000  …………….
20b26e90  00000000 00000000 00000000 00000000  …………….
20b26ea0  00000000 00000000 00000000 00000000  …………….
20b26eb0  00000000 00000000 00000000 00000000  …………….
20b26ec0  00000000 00000000 00000000 00000000  …………….
20b26ed0  00000000 00000000 abcdbbbb 1f241000  …………..$.
20b26ee0  00000108 00000108 00000000 00000000  …………….
20b26ef0  011c6b10 dcbabbbb 1f1bc8b4 00000002  .k…………..
20b26f00  20b79fd0 20b85fd0 20b28fe8 20b2ffe0  … ._. … …
20b26f10  20b3ffe0 20b4bfe8 20b51fe8 20b57fe8  … … … …
20b26f20  00000000 00000000 20b5dfa8 00000000  ……….. ….
20b26f30  00000000 00000000 1f1bcbf0 00000000  …………….
20b26f40  20b71ff8 00000010 1f1bcbf0 00000000  … …………
20b26f50  20b73ff8 00000010 1f1bcbf0 00000000  .?. …………
20b26f60  20b75ff8 00000010 1f1bcbf0 00000000  ._. …………
20b26f70  20b77ff8 00000010 00000000 00000000  … …………
20b26f80  c0c0c001 00000000 c0c00000 00000002  …………….
20b26f90  01000000 00000101 00000000 00000000  …………….
20b26fa0  00000000 c0c0c000 00000000 00000001  …………….
20b26fb0  00000000 00000000 00000000 00000000  …………….
20b26fc0  00000000 00000000 00000000 00000000  …………….
20b26fd0  00000000 00000000 00000000 00000000  …………….
20b26fe0  00000000 00000000 00000000 00000000  …………….
20b26ff0  00000000 00000000 00000000 c0c0c000  …………….
20b27000 ???????? ????

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This pattern (we call it String Hint) covers traces of ASCII and UNICODE strings that look suspicious such as website, password and HTTP forms or strange names that intuitively shouldn’t be present according to the purpose of a module or its container process (example is taken from Victimware presentation case study):

0:005> s-sa 00040000 L1d000
0004004d  "!This program cannot be run in D"
0004006d  "OS mode."
00040081  "3y@"
000400b8  "Rich"
000401d0  ".text"
000401f7  "`.rdata"
0004021f  "@.data"
00040248  ".reloc"
[...]
00054018  "GET /stat?uptime=%d&downlink=%d&"
00054038  "uplink=%d&id=%s&statpass=%s&comm"
00054058  "ent=%s HTTP/1.0"
000540ac  "%s%s%s"
000540d8  "ftp://%s:%s@%s:%d"
000540fc  "Accept-Encoding:"
00054118  "Accept-Encoding:"
00054130  "0123456789ABCDEF"
00054144  "://"
00054160  "POST %s HTTP/1.0"
00054172  "Host: %s"
0005417c  "User-Agent: %s"
0005418c  "Accept: text/html"
0005419f  "Connection: Close"
000541b2  "Content-Type: application/x-www-"
000541d2  "form-urlencoded"
000541e3  "Content-Length: %d"
000541fc  "id="
00054208  "POST %s HTTP/1.1"
0005421a  "Host: %s"
00054224  "User-Agent: %s"
00054234  "Accept: text/html"
00054247  "Connection: Close"
0005425a  "Content-Type: application/x-www-"
0005427a  "form-urlencoded"
0005428b  "Content-Length: %d"
000542a4  "id=%s&base="
000542b8  "id=%s&brw=%d&type=%d&data="
000542d8  "POST %s HTTP/1.1"
000542ea  "Host: %s"
000542f4  "User-Agent: %s"
00054304  "Accept: text/html"
00054317  "Connection: Close"
0005432a  "Content-Type: application/x-www-"
0005434a  "form-urlencoded"
0005435b  "Content-Length: %d"
00054378  "id=%s&os=%s&plist="
00054390  "POST %s HTTP/1.1"
000543a2  "Host: %s"
000543ac  "User-Agent: %s"
000543bc  "Accept: text/html"
000543cf  "Connection: Close"
000543e2  "Content-Type: application/x-www-"
00054402  "form-urlencoded"
00054413  "Content-Length: %d"
00054430  "id=%s&data=%s"
00054440  "POST %s HTTP/1.1"
00054452  "Host: %s"
0005445c  "User-Agent: %s"
0005446c  "Accept: text/html"
0005447f  "Connection: Close"
00054492  "Content-Type: application/x-www-"
000544b2  "form-urlencoded"
000544c3  "Content-Length: %d"
000544e0  "GET %s HTTP/1.0"
000544f1  "Host: %s"
000544fb  "User-Agent: %s"
0005450b  "Connection: close"
00054528  "POST /get/scr.html HTTP/1.0"
00054545  "Host: %s"
0005454f  "User-Agent: %s"
0005455f  "Connection: close"
00054572  "Content-Length: %d"
00054586  "Content-Type: multipart/form-dat"
000545a6  "a; boundary=--------------------"
000545c6  "-------%d"
000545d4  "-----------------------------%d"
000545f8  "%sContent-Disposition: form-data"
00054618  "; name="id""
00054630  "%sContent-Disposition: form-data"
00054650  "; name="screen"; filename="%d""
00054670  "Content-Type: application/octet-"
00054690  "stream"
000546a0  "%s(%d) : %s"
000546ac  "%s failed with error %d: %s"
000546c8  "%02X"
000546d8  "BlackwoodPRO"
000546e8  "FinamDirect"
000546f4  "GrayBox"
000546fc  "MbtPRO"
00054704  "Laser"
0005470c  "LightSpeed"
00054718  "LTGroup"
00054720  "Mbt"
00054724  "ScotTrader"
00054730  "SaxoTrader"
00054740  "Program:   %s"
0005474f  "Username:  %s"
0005475e  "Password:  %s"
0005476d  "AccountNO: %s"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Injected code address may not be in the address ranges of loaded modules. In such cases, in the execution call history we would see plain EIP and RIP return addresses on stack traces. We call this pattern RIP Stack Trace partly because we have seen these addresses after something had gone wrong and a process crashed:

0:005> k
ChildEBP RetAddr
02aec974 77655620 ntdll!KiFastSystemCallRet
02aec978 77683c62 ntdll!NtWaitForSingleObject+0xc
02aec9fc 77683d4b ntdll!RtlReportExceptionEx+0x14b
02aeca3c 7769fa87 ntdll!RtlReportException+0x3c
02aeca50 7769fb0d ntdll!RtlpTerminateFailureFilter+0x14
02aeca5c 775f9bdc ntdll!RtlReportCriticalFailure+0x6b
02aeca70 775f4067 ntdll!_EH4_CallFilterFunc+0x12
02aeca98 77655f79 ntdll!_except_handler4+0x8e
02aecabc 77655f4b ntdll!ExecuteHandler2+0x26
02aecb6c 77655dd7 ntdll!ExecuteHandler+0x24
02aecb6c 7769faf8 ntdll!KiUserExceptionDispatcher+0xf
02aecee0 776a0704 ntdll!RtlReportCriticalFailure+0x5b
02aecef0 776a07f2 ntdll!RtlpReportHeapFailure+0x21
02aecf24 7766b1a5 ntdll!RtlpLogHeapFailure+0xa1
02aecf6c 7765730a ntdll!RtlpCoalesceFreeBlocks+0x4b9
02aed064 77657545 ntdll!RtlpFreeHeap+0x1e2
02aed080 75e47e4b ntdll!RtlFreeHeap+0x14e
02aed0c8 77037277 kernel32!GlobalFree+0x47
02aed0dc 774b4a1f ole32!ReleaseStgMedium+0x124
02aed0f0 77517feb urlmon!ReleaseBindInfo+0x4c
02aed100 774d9a87 urlmon!CINet::ReleaseCNetObjects+0x3d
02aed118 774d93f0 urlmon!CINetHttp::OnWininetRequestHandleClosing+0x60
02aed12c 76432078 urlmon!CINet::CINetCallback+0x2de
02aed274 76438f5d wininet!InternetIndicateStatus+0xfc
02aed2a4 7643937a wininet!HANDLE_OBJECT::~HANDLE_OBJECT+0xc9
02aed2c0 7643916b wininet!INTERNET_CONNECT_HANDLE_OBJECT::~INTERNET_CONNECT_HANDLE_OBJECT+0x209
02aed2cc 76438d5e wininet!HTTP_REQUEST_HANDLE_OBJECT::`vector deleting destructor'+0xd
02aed2dc 76434e72 wininet!HANDLE_OBJECT::Dereference+0x22
02aed2e8 76439419 wininet!DereferenceObject+0x21
02aed310 76439114 wininet!_InternetCloseHandle+0x9d
02aed330 0004aaaf wininet!InternetCloseHandle+0x11e
WARNING: Frame IP not in any known module. Following frames may be wrong.
02aed33c 774c5d25 0×4aaaf
02aed358 774c5d95 urlmon!CINet::TerminateRequest+0×82
02aed364 774c5d7c urlmon!CINet::MyUnlockRequest+0×10
02aed370 774c5d63 urlmon!CINetProtImpl::UnlockRequest+0×10
02aed37c 774c5d49 urlmon!CINetEmbdFilter::UnlockRequest+0×11
02aed388 774b743d urlmon!CINet::UnlockRequest+0×13
02aed394 774b73e1 urlmon!COInetProt::UnlockRequest+0×11
02aed3a8 774b7530 urlmon!CTransaction::UnlockRequest+0×36
02aed3b4 774b74e0 urlmon!CTransData::~CTransData+0×3a
02aed3c0 774b74c9 urlmon!CTransData::`scalar deleting destructor’+0xd
02aed3d8 774e221f urlmon!CTransData::Release+0×25
02aed3e0 774b6d0a urlmon!CReadOnlyStreamDirect::~CReadOnlyStreamDirect+0×1a
02aed3ec 774b7319 urlmon!CReadOnlyStreamDirect::`vector deleting destructor’+0xd
02aed404 774b72be urlmon!CReadOnlyStreamDirect::Release+0×25
02aed410 774b71f4 urlmon!CBinding::~CBinding+0xb9
02aed41c 774b71dd urlmon!CBinding::`scalar deleting destructor’+0xd
02aed434 6b20b0e8 urlmon!CBinding::Release+0×25
02aed448 6b20b0ba mshtml!ATL::AtlComPtrAssign+0×2b
02aed458 6b20b8de mshtml!ATL::CComPtr<IBindCallbackInternal>::operator=+0×15
02aed464 6b20b8aa mshtml!CBindingXSSFilter::TearDown+0×2b
02aed46c 6b20b887 mshtml!BindingXSSFilter_TearDown+0×19
02aed478 6b0da61a mshtml!CStreamProxy::Passivate+0×12
02aed484 6b0ddf3a mshtml!CBaseFT::Release+0×1d
02aed4ac 6b0e0b70 mshtml!CDwnBindData::TerminateBind+0×11d
02aed4b8 6b11a2a9 mshtml!CDwnBindData::TerminateOnApt+0×14
02aed4ec 6b105066 mshtml!GlobalWndOnMethodCall+0xfb
02aed50c 7742fd72 mshtml!GlobalWndProc+0×183
02aed538 7742fe4a user32!InternalCallWinProc+0×23
02aed5b0 7743018d user32!UserCallWinProcCheckWow+0×14b
02aed614 7743022b user32!DispatchMessageWorker+0×322
02aed624 6ecac1d5 user32!DispatchMessageW+0xf
02aef72c 6ec5337e ieframe!CTabWindow::_TabWindowThreadProc+0×54c
02aef7e4 760f426d ieframe!LCIETab_ThreadProc+0×2c1
02aef7f4 75e4d0e9 iertutil!CIsoScope::RegisterThread+0xab
02aef800 776319bb kernel32!BaseThreadInitThunk+0xe
02aef840 7763198e ntdll!__RtlUserThreadStart+0×23
02aef858 00000000 ntdll!_RtlUserThreadStart+0×1b

However, such addresses need to be checked whether they belong to .NET CLR JIT code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We skip parts 5 (Module Collection), 6 (No Component Symbols, for looking at import tables), 7 (Stack Trace Collection, for listing active processes, threads and their stack traces), 8 (Hidden Module), and 9 (Hidden Process). The new pattern here is called Driver Device Collection and can be used to compare the current list of device and driver objects with some saved reference list to find out any changes. This listing can be done by using !object command:

0: kd> !object \Driver
[...]

0: kd> !object \FileSystem
[...]

0: kd> !object \Device
[...]

Note that the collection is called Driver Device and not Device Driver.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -