Software Diagnostics Library

This case study centers on 3 process dump files (two first chance exception and one second chance exception). To recall the difference between them please read first chance exceptions explained series. When we get first and second chance exception dumps together we usually open a second chance exception dump first. However, in this case, the second chance exception dump had an incorrect stack trace:

(f54.b80): Access violation - code c0000005 (!!! second chance !!!)
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

0:000> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

The default analysis command detected stack overflow pattern: 

0:000> !analyze -v

[...]

FAULTING_IP:
ntdll!RtlDispatchException+8
7c92a978 56              push    esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c92a978 (ntdll!RtlDispatchException+0x00000008)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00032fc0

DEFAULT_BUCKET_ID:  STACK_OVERFLOW

ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.

[...]

Indeed ESP was outside the stack region and that happened during unhandled exception processing:

0:000> r esp
esp=00030e4c

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        000310c4
    StackBase:            00130000
    StackLimit:           00031000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             00000f54 . 00000b80
    RpcHandle:            00000000
    Tls Storage:          001537a8
    PEB Address:          7ffdb000
    LastErrorValue:       2
    LastStatusValue:      c000000f
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dps esp l100
00030e4c  ????????
00030e50  ????????
[...]
00030ff8  ????????
00030ffc  ????????
00031000  00000000
00031004  00000000
00031008  00000000
0003100c  00000000
00031010  00000000
00031014  00000000
00031018  00000000
0003101c  00000000
00031020  00000000
00031024  7c910323 ntdll!RtlpImageNtHeader+0x56
00031028  004054e8 Application+0x54e8
0003102c  00400000 Application
00031030  00400000 Application
00031034  00400100 Application+0x100
00031038  00031028
0003103c  7e390000 USER32!_imp__GetClipRgn <PERF> (USER32+0x0)
00031040  00031060
00031044  7c910385 ntdll!RtlImageDirectoryEntryToData+0x57
00031048  00400000 Application
0003104c  00000001
00031050  0000000e
00031054  00031084
00031058  7c910323 ntdll!RtlpImageNtHeader+0x56
0003105c  004054e8 Application+0x54e8
00031060  7c900000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
00031064  0012ff00
00031068  7c9000d0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xd0)
0003106c  0003105c
00031070  0000000e
00031074  00114e88
00031078  7c90e920 ntdll!_except_handler3
0003107c  7c910328 ntdll!`string'+0x4
00031080  ffffffff
00031084  7c910323 ntdll!RtlpImageNtHeader+0x56
00031088  7c935f1c ntdll!RtlLookupFunctionTable+0xc5
0003108c  7c92ab3a ntdll!RtlLookupFunctionTable+0xf2
00031090  7c97e178 ntdll!LdrpLoaderLock
00031094  000310c4
00031098  7c809ad8 kernel32!_except_handler3
0003109c  7c833fd9 kernel32!UnhandledExceptionFilter+0xf
000310a0  7c834aa8 kernel32!`string’+0×1c
000310a4  000310d0
000310a8  0040550c Application+0×550c
000310ac  000310b4
000310b0  7c9032a8 ntdll!ExecuteHandler2+0×26
000310b4  00031198
000310b8  0012ffb4
000310bc  000311ac 

Before we tried to reconstruct the stack trace we opened the earlier first-chance exception dump file:

0:000> .opendump 1stchance.dmp

Loading Dump File [1stchance.dmp]
User Dump File: Only application data is available

Opened '1stchance.dmp'

||0:0:000> g

(f54.b80): Stack overflow - code c00000fd (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0003332c ebx=00033040 ecx=00033054 edx=7c90e514 esi=000333a8 edi=00000000
eip=7c92a978 esp=00032fc4 ebp=00033028 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
ntdll!RtlDispatchException+0×8:
7c92a978 56              push    esi

Here we were able to get stack trace from the saved nested exception: 

||1:1:020> kL 1000
ChildEBP RetAddr 
00033028 7c90e48a ntdll!RtlDispatchException+0x8
00033028 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033390 7c90e48a ntdll!RtlDispatchException+0x133
00033390 7c95019e ntdll!KiUserExceptionDispatcher+0xe
000336f8 7c90e48a ntdll!RtlDispatchException+0x133
000336f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033a60 7c90e48a ntdll!RtlDispatchException+0x133
00033a60 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033dc8 7c90e48a ntdll!RtlDispatchException+0x133
00033dc8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034130 7c90e48a ntdll!RtlDispatchException+0x133
00034130 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034498 7c90e48a ntdll!RtlDispatchException+0x133
00034498 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034800 7c90e48a ntdll!RtlDispatchException+0x133
00034800 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034b68 7c90e48a ntdll!RtlDispatchException+0x133
00034b68 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034ed0 7c90e48a ntdll!RtlDispatchException+0x133
00034ed0 7c95019e ntdll!KiUserExceptionDispatcher+0xe
[...]
001143f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114760 7c90e48a ntdll!RtlDispatchException+0x133
00114760 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114ac8 7c90e48a ntdll!RtlDispatchException+0x133
00114ac8 7c7e2afb ntdll!KiUserExceptionDispatcher+0xe
00114e30 0057ad17 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
00114e54 0098ff95 Application+0x17ad17
[...]
00121fd8 7e398734 Application+0x313be
00122004 7e398816 USER32!InternalCallWinProc+0x28
0012206c 7e3a8ea0 USER32!UserCallWinProcCheckWow+0x150
001220c0 7e3aacd1 USER32!DispatchClientMessage+0xa3
001220f0 7c90e473 USER32!__fnINSTRING+0x37
0012212c 7e3993e9 ntdll!KiUserCallbackDispatcher+0x13
00122158 7e3aa43b USER32!NtUserPeekMessage+0xc
00122184 004794d9 USER32!PeekMessageA+0xeb
00122234 00461667 Application+0x794d9
[...]
0012ffc0 7c7e7077 Application+0x60610b
0012fff0 00000000 kernel32!BaseProcessStart+0x23

This all pointed to a problem exception handler:

||1:1:020> !analyze -v

[...]

CONTEXT:  00114b10 -- (.cxr 0x114b10)
eax=00114de0 ebx=0eedfade ecx=00000000 edx=001537a8 esi=00114e88 edi=00000007
eip=7c7e2afb esp=00114ddc ebp=00114e30 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
kernel32!RaiseException+0x53:
7c7e2afb 5e              pop     esi
Resetting default scope

[...]

||1:1:020> .cxr 0x114b10

||1:1:020> kv 1
ChildEBP RetAddr  Args to Child             
00114e30 0057ad17 0eedfade 00000001 00000007 kernel32!RaiseException+0×53

Being curious we also opened the second first chance exception dump and it pointed to the expected crash point (the same as seen in the second chance exception crash dump)

||1:1:020> .opendump 1stchance2.dmp

Loading Dump File [1stchance2.dmp]
User Dump File: Only application data is available

Opened '1stchance2.dmp'

||1:1:020> g

[...]

(f54.b80): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

||2:2:040> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

We found the similar past issue for a different process name but our main process module information included the same vendor name so it was easy to contact the corresponding vendor.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you develop and debug user space applications (and/or doing crash dump analysis in user space) or specialize in user space security and you want to understand Windows kernel dumps and device drivers better (and probably start writing your own kernel tools) or understand malware rootkits better here is the reading list I found the most effective over the last 7 years:

0.0. Read and re-read Windows Internals book in parallel while reading all other books. I read all editions by the way. It will show you the big picture and useful WinDbg commands and techniques but you need to read device driver books to fill the gaps and be confident in kernel space:

0.1. Start with The Windows 2000 Device Driver Book: A Guide for Programmers. This short book will show you the basics and you can start writing your drivers and kernel tools immediately.

0.2. Next read Windows NT Device Driver Development book to consolidate your knowledge. This book has been reprinted by OSR (I own the original New Riders Press edition):

0.3. Don’t stop here. Read Developing Windows NT Device Drivers: A Programmer’s Handbook. This is the very good book explaining everything in great detail and good pictures. You will finally understand various buffering methods.

0.4. Continue with WDM drivers and modern presentation: Programming the Microsoft Windows Driver Model. Must read even if your drivers are not WDM.

0.5. Finally read Developing Drivers with the Windows Driver Foundation book. It also covers ETW (event tracing for Windows), WinDbg extensions, PREfast and static driver verifier.

0.6. There is a forthcoming book Windows 7 Device Driver at the time of this writing that also covers WDF so you might want to start with #0.6 and continue with #0.5 as a reference:

Additional reading (not including DDK Help which you will use anyway) can be done in parallel after finishing “Windows NT Device Driver Development” book:

1.1. OSR NT Insider articles. I have their full printed collection 1996 - 2006 plus all the latest issues (looks like print editions are discontinued and the new ones are only digital):

http://www.osronline.com/

1.2. Windows NT File System Internals reprinted by OSR (I have the original O’Reilly edition):

1.3. Windows NT/2000 Native API Reference is fun to browse occasionally and indispensable if you don’t have access to Windows source code:

1.4. Rootkits: Subverting the Windows Kernel book will show you Windows kernel from the hacker perspective. In addition you will find the overview of kernel areas not covered in other books.

1.5. The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System is another excellent book that is up to date and explains kernel staff from ab initio. I’m reading it at the time of this writing and recommend it to read first or in parallel to all other books:

Of course, you must know C language and its idioms really well. Really know it down to assembly language level! I’ll publish other reading lists soon including reverse engineering classics. Stay tuned.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have very similar abnormal software behaviour dispositions (like crashes with similar stack traces) for different applications or services. In such cases, we should also check application or service vendor and copyright in the output of lmv command. Similar to Template Module Same Vendor pattern can be useful to relate such different incidents. Usually, in the same company, code and people reuse tends to distribute code fragments and code construction styles across different product lines, and software defects might surface in different images. For example:

0:000> lmv m ApplicationA
start    end        module name
00400000 00d99000   ApplicationA   (deferred)
[...]
Image name: ApplicationA.exe
Timestamp:        [...]
CheckSum:         00000000
[...]
CompanyName:      CompanyA
ProductName:      CompanyA Application
LegalCopyright:   Copyright (c) CompanyA
[…]

0:000> lmv m ApplicationB
start    end        module name
00400000 019d0000   ApplicationB  C (no symbols)
Image name: ApplicationB.exe
[...]
CompanyName:      CompanyA
ProductName:      ApplicationB
LegalCopyright:   Copyright (c) CompanyA
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having spent 16 years in software engineering I ventured into software support in 2003 (with 8th year started at the time of this writing). Now it is time for the next gradual shift into software security (the domain I previously had exposure to but not as a primary focus):

The title of this post is borrowed from the book I read from cover to cover long time ago and recently put on my desk again:

Journey of the Software Professional: The Sociology of Software Development

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Previously announced DSS has got its logo and now affiliated with DA+TA Facebook group:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Motivated by the existence of London School of Economics (LSE) I just founded DSS. The program to be communicated soon and includes general memory dump and software trace analysis as a foundation for security. I like the name very much because of its additional meaning:

DUmps Binary Logs INternals

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Deadlock (self) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a variation of a deadlock pattern where a thread that owns a resource (either in shared or exclusive mode) attempts to acquire it exclusively again. This results in a self-deadlock:

Resource @ 0x85d9c018    Shared 1 owning threads
    Contention Count = 2
    NumberOfExclusiveWaiters = 2
     Threads: 85db0030-02<*>
     Threads Waiting On Exclusive Access:
              85f07d78       85db0030

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When I started getting several process memory dumps with very similar crash dispositions I decided to factor them into a separate pattern called Problem Exception Handler. This usually happens with custom exception handlers not written according to prescribed rules (for example, a handler for a non-continuable exception) or have other defects common to normal code.

In the example below we have a different stack trace epilogue for a similar issue that shows a relationship with a custom exception handler:

0:000> kv 1000
ChildEBP RetAddr  Args to Child
0003300c 77af9904 77b8929c 792ea99b 00000000 ntdll!RtlAcquireSRWLockShared+0x1a
00033058 77af9867 00406ef8 00033098 000330a0 ntdll!RtlLookupFunctionTable+0×2a
000330a8 77af97f9 00406ef8 00000000 00000000 ntdll!RtlIsValidHandler+0×26
00033128 77b25dd7 01033140 00033154 00033140 ntdll!RtlDispatchException+0×10b
00033128 77b40726 01033140 00033154 00033140 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 00033154)
00033490 77b25dd7 010334a8 000334bc 000334a8 ntdll!RtlDispatchException+0×18a
00033490 77b40726 010334a8 000334bc 000334a8 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 000334bc)
000337f8 77b25dd7 01033810 00033824 00033810 ntdll!RtlDispatchException+0×18a
[…]
0012f228 77b40726 0112f240 0012f254 0012f240 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0012f254)
0012f590 77b25dd7 0112f5a8 0012f5d8 0012f5a8 ntdll!RtlDispatchException+0×18a
0012f590 768bfbae 0112f5a8 0012f5d8 0012f5a8 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0012f5d8)
0012f8f4 0059ecad 0eedfade 00000001 00000007 kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f918 00473599 0eedfade 00000001 00000007 Application+0×19ecad
[…]
0012ff88 768cd0e9 7ffdf000 0012ffd4 77b019bb Application+0×70f8
0012ff94 77b019bb 7ffdf000 793f6617 00000000 kernel32!BaseThreadInitThunk+0xe
0012ffd4 77b0198e 011263c0 7ffdf000 ffffffff ntdll!__RtlUserThreadStart+0×23
0012ffec 00000000 011263c0 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> !exchain
00033048: ntdll!_except_handler4+0 (77ac99fa)
0012ff78: Application+6ef8 (00406ef8)
0012ffc4: ntdll!_except_handler4+0 (77ac99fa)
0012ffe4: ntdll!FinalExceptionHandler+0 (77b66f9b)
Invalid exception stack at ffffffff

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes when looking at a module list (lmv WinDbg command) we see the presence of the whole Pervasive System. It is not just a module that does function and/or message hooking but the whole system of modules from a single vendor that is context-aware (for example, reads its configuration from registry) and consists of several components that communicate with other processes. The penetrating system is supposed to add some additional value or to coexist peacefully in a larger environment. The system thus becomes coupled strongly and/or weakly with other processes it was never intended to work with as opposed to intended module variety. At one extreme modules from pervasive system can be ubiquitous and at the other end hidden. In such cases troubleshooting consists of the total removal of pervasive modules and if the problem disappears their exclusion one by one to find the problem component.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here is the description of a high-level architecture of the previously announced project CARE (Crash Analysis Report Environment). To remind, the main idea of the project is to process memory dumps on a client to save debugger logs. They can be sent to a server for pattern-driven analysis of software behaviour. Textual logs can also be inspected by a client security team before sending. Certain sensitive information can be excluded or modified to have generic meaning according to the built-in processing rules like renaming (for example, server names and folders). Before processing, verified secured logs are converted to abstract debugger logs. Abstracting platform-specific debugger log format allows reuse of the same architecture for different computer platforms. We call it CIA (Computer Independent Architecture). Do not confuse it with ICA (Independent Computer Architecture) and CIA acronym is more appropriate for memory analysis (like similar MAFIA acronym, Memory Analysis Forensics and Intelligence Architecture). These abstract logs are checked for various patterns (in abstracted form) using abstract debugger commands and an abstract report is generated according to various checklists. Abstract reports are then converted to structured reports for the required audience level. Abstract memory analysis pattern descriptions are prepared from platform-specific pattern descriptions. In certain architectural component deployment configurations both client and server parts can reside on the same machine. Here’s the simple diagram depicting the flow of processing:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I detour for MARS expedition. You may also call it Memory Analysis Report System as malware analysis is always exploration of memory (in general). Why is this sudden change of course? After reading Gilles Deleuze I want to broaden the concept of “malware” and give it new orientation and direction of thinking. Beside that I also want new challenges after many years of research in pattern-driven memory dump and software trace analysis of abnormal software behaviour.

You may have also noticed small restructuring (rebranding) of this blog and DumpAnalysis.org headers.

See you there

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for False Function Parameters pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As a practical example of applying behavioral and structural pattern analysis of computer memory and traces OpenTask plans to publish the following title next year:

• Title: Malware Patterns: Structure and Behavior of Computer Adware, Crimeware, Rootkits, Scareware, Spyware, Trojans, Viruses, Victimware and Worms
• Author: Dmitry Vostokov
• Paperback: 1200 pages
• Publisher: OpenTask (October 2011)
• ISBN-13: 978-1-908043-01-6

The inclusion of victimware is necessary because of the effects of defective malware.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The next pattern with the name borrowed from historiography is Layered Periodization. This periodization of software trace messages includes individual messages, then aggregated messages from threads, then processes as wholes and finally individual computers (in client-server or similar sense). This is best illustrated graphically.

Message layer:

Thread layer (different colors correspond to different TID):

Process layer (different colors correspond to different PID):

Please note that it is also possible to have a periodization based on modules, functions and individual messages but it might be complicated because different threads can enter the same module or function and here other patterns more appropriate like Activity Region, Characteristic Message Block, and Background and Foreground Components.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of my home computers got infected. I confess that I don’t have an antivirus because I’m conscious while browsing Internet (the last infected machine I had was an MSDOS one) so perhaps one of my family members was less careful. I paid attention to the possible infection when IE started crashing when I was pushing a login button on one of online banking websites. However I didn’t pay enough attention because it was a heap corruption (see my previous case study) and simply switched to another non-crashing browser vendor such as Apple Safari. Since then IE was crashing periodically when I was pushing various admin buttons in WordPress but I didn’t pay much attention too because it was still heap corruption and I was thinking it was a script processing defect, waiting for a new IE update. Until one day explorer.exe crashed as well when I was entering a password for an ftp account. Here’s the stack trace that I got after opening a crash dump in WinDbg:

0:030> kL 100
ChildEBP RetAddr
0663e9c4 76f05610 ntdll!KiFastSystemCallRet
0663e9c8 7706a5d7 ntdll!NtWaitForMultipleObjects+0xc
0663ea64 7706a6f0 kernel32!WaitForMultipleObjectsEx+0×11d
0663ea80 770de2a5 kernel32!WaitForMultipleObjects+0×18
0663eaec 770de4d1 kernel32!WerpReportFaultInternal+0×16d
0663eb00 770bff4d kernel32!WerpReportFault+0×70
0663eb8c 76f17fc1 kernel32!UnhandledExceptionFilter+0×1b5
0663eb94 76ea9bdc ntdll!__RtlUserThreadStart+0×6f
0663eba8 76ea4067 ntdll!_EH4_CallFilterFunc+0×12
0663ebd0 76f05f79 ntdll!_except_handler4+0×8e
0663ebf4 76f05f4b ntdll!ExecuteHandler2+0×26
0663eca4 76f05dd7 ntdll!ExecuteHandler+0×24
0663eca4 93181a08 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0663efa0 0321aaaf 0×93181a08
0663efac 6b887974 0×321aaaf
0663efbc 6b8973ad msieftp!InternetCloseHandleWrap+0×10
0663f810 6b897fbf msieftp!CFtpSite::_QueryServerFeatures+0×57
0663fa50 6b8981ae msieftp!CFtpSite::_LoginToTheServer+0×235
0663fa94 6b88b39e msieftp!CFtpSite::GetHint+0xe8
0663fab4 6b88b412 msieftp!CFtpDir::GetHint+0×1f
0663fae4 6b88ed38 msieftp!CFtpDir::WithHint+0×49
0663fb10 6b88eda4 msieftp!CFtpEidl::_Init+0×6e
0663fb2c 7584ecb4 msieftp!CFtpEidl::Next+0×41
0663fb64 7584f63b shell32!CEnumThread::_EnumFolder+0×65
0663fb80 7584f5ba shell32!CEnumThread::_RunEnum+0×6f
0663fb8c 7645c2c9 shell32!CEnumThread::s_EnumThreadProc+0×14
0663fc10 7706d0e9 shlwapi!WrapperThreadProc+0×11c
0663fc1c 76ee19bb kernel32!BaseThreadInitThunk+0xe
0663fc5c 76ee198e ntdll!__RtlUserThreadStart+0×23
0663fc74 00000000 ntdll!_RtlUserThreadStart+0×1b

Notice 0×321aaaf address. We see that wininet function was hooked by a code running in 0×0321XXXX range:

0:030> ub 6b887974
msieftp!InternetOpenWrap+0×46:
6b887963 cc              int     3
msieftp!InternetCloseHandleWrap:
6b887964 8bff            mov     edi,edi
6b887966 55              push    ebp
6b887967 8bec            mov     ebp,esp
6b887969 56              push    esi
6b88796a ff7508          push    dword ptr [ebp+8]
6b88796d 33f6            xor     esi,esi
6b88796f e82e610100      call    msieftp!InternetCloseHandle (6b89daa2)

0:030> u 6b89daa2
msieftp!InternetCloseHandle:
6b89daa2 ff2500278a6b    jmp     dword ptr [msieftp!_imp__InternetCloseHandle (6b8a2700)]
msieftp!_imp_load__InternetConnectW:
6b89daa8 b834278a6b      mov     eax,offset msieftp!_imp__InternetConnectW (6b8a2734)
6b89daad e9b4feffff      jmp     msieftp!_tailMerge_WININET_dll (6b89d966)
6b89dab2 cc              int     3
6b89dab3 cc              int     3
6b89dab4 cc              int     3
6b89dab5 cc              int     3
6b89dab6 cc              int     3

0:030> dp 6b8a2700 l1
6b8a2700  76dc9088

0:030> u 76dc9088
wininet!InternetCloseHandle:
76dc9088 e9031a458c      jmp     0321aa90
76dc908d 51              push    ecx
76dc908e 51              push    ecx
76dc908f 53              push    ebx
76dc9090 56              push    esi
76dc9091 57              push    edi
76dc9092 33db            xor     ebx,ebx
76dc9094 33ff            xor     edi,edi

0:030> u 0321aa90
0321aa90 55              push    ebp
0321aa91 8bec            mov     ebp,esp
0321aa93 837d0800        cmp     dword ptr [ebp+8],0
0321aa97 740c            je      0321aaa5
0321aa99 8b4508          mov     eax,dword ptr [ebp+8]
0321aa9c 50              push    eax
0321aa9d e82eedffff      call    032197d0
0321aaa2 83c404          add     esp,4

This address range was not on a loaded module list so I used image scanning command to detect Hidden Module:

0:030> .imgscan
MZ at 00080000, prot 00000002, type 01000000 - size 2cd000
Name: explorer.exe
MZ at 003d0000, prot 00000002, type 00040000 - size 2000
MZ at 018a0000, prot 00000008, type 00040000 - size 7000
MZ at 031c0000, prot 00000008, type 00040000 - size 3000
MZ at 031d0000, prot 00000002, type 01000000 - size c000
Name: DLAAPI_W.DLL
MZ at 03210000, prot 00000040, type 00020000 - size 1d000
[…]

!dh command was not showing any useful hints so I dumped the whole address range of that Unknown Component and found strange strings inside:

0:030> db 03210000 03210000+1d000
[...]
032246d0  2a 00 00 00 2a 00 00 00-42 6c 61 63 6b 77 6f 6f  *...*...Blackwoo
032246e0  64 50 52 4f 00 00 00 00-46 69 6e 61 6d 44 69 72  dPRO....FinamDir
032246f0  65 63 74 00 47 72 61 79-42 6f 78 00 4d 62 74 50  ect.GrayBox.MbtP
03224700  52 4f 00 00 4c 61 73 65-72 00 00 00 4c 69 67 68  RO..Laser...Ligh
03224710  74 53 70 65 65 64 00 00-4c 54 47 72 6f 75 70 00  tSpeed..LTGroup.
03224720  4d 62 74 00 53 63 6f 74-54 72 61 64 65 72 00 00  Mbt.ScotTrader..
03224730  53 61 78 6f 54 72 61 64-65 72 00 00 00 00 00 00  SaxoTrader......
03224740  50 72 6f 67 72 61 6d 3a-20 20 20 25 73 0d 0a 55  Program:   %s..U
03224750  73 65 72 6e 61 6d 65 3a-20 20 25 73 0d 0a 50 61  sername:  %s..Pa
03224760  73 73 77 6f 72 64 3a 20-20 25 73 0d 0a 41 63 63  ssword:  %s..Acc
03224770  6f 75 6e 74 4e 4f 3a 20-25 73 0d 0a 53 65 72 76  ountNO: %s..Serv
03224780  65 72 3a 20 20 20 20 25-73 0d 0a 00 5c 00 00 00  er:    %s...\...
03224790  25 73 20 25 73 00 00 00-25 73 00 00 50 52 4f 43  %s %s...%s..PROC
032247a0  45 53 53 4f 52 5f 49 44-45 4e 54 49 46 49 45 52  ESSOR_IDENTIFIER
032247b0  00 00 00 00 25 64 00 00-25 30 32 58 00 00 00 00  ....%d..%02X....
032247c0  30 00 00 00 2c 00 00 00-25 30 32 58 00 00 00 00  0...,...%02X....
[...]
03225000  01 01 00 00 5c 00 63 00-68 00 6b 00 6e 00 74 00  ....\.c.h.k.n.t.
03225010  66 00 73 00 2e 00 65 00-78 00 65 00 00 00 00 00  f.s...e.x.e.....
03225020  5c 00 63 00 68 00 6b 00-6e 00 74 00 66 00 73 00  \.c.h.k.n.t.f.s.
03225030  2e 00 64 00 61 00 74 00-00 00 00 00 a6 b7 04 5e  ..d.a.t........^
[...]

I didn’t pay attention to chkntfs.exe but did a search for SaxoTrader string in all files using findstr command and found chkntfs.exe as a system file in Start Menu \ Programs \ Startup folder in roaming user AppData. I couldn’t remove it so I had to boot in command line mode to do that. The crashes were gone since that. I double checked various iexplore.exe crash dumps saved previously and found the same module loaded, for example:

0:005> .imgscan
MZ at 00040000, prot 00000040, type 00020000 - size 1d000
MZ at 00340000, prot 00000002, type 01000000 - size 9c000
Name: iexplore.exe
[…]

Here we consider IE and Explorer as victimware of malware.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In order to start the analysis of a structured memory snapshot a debugger engine needs Anchor Region that describes memory layout and where to start unfolding of analysis. For example, it can be a list of modules (another forthcoming structural pattern). We can observe the importance of such regions when we try to open corrupt or severely truncated memory dumps:

[...]
KdDebuggerDataBlock is not present or unreadable.
[...]
Unable to read PsLoadedModuleList
[...]

For certain types of memory snapshots (like software traces) an anchor region coinsides with its structure description (message trace format for structured snapshots) and a trace file header (if any) for aggregate snapshots.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Shared Buffer Overwrite differs from Local Buffer Overflow and heap / pool memory corruption patterns in not writing over control structures situated at dynamically allocated memory or procedure frame (local call stack) boundaries. Its effect is visible when the buffer data contains pointers that become wild after the overwrite and are later dereferenced resulting in a crash. For example, when the overwriting data contains UNICODE and /or ASCII characters we see them in a pointer data:

1: kd> !analyze -v

[...]

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8086c949, The address that the exception occurred at
Arg3: f78eec54, Exception Record Address
Arg4: f78ee950, Context Record Address

[...]

EXCEPTION_RECORD:  f78eec54 -- (.exr 0xfffffffff78eec54)
ExceptionAddress: 8086c949 (nt!ObfDereferenceObject+0x00000023)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
   Parameter[1]: 006f0058
Attempt to write to address 006f0058

CONTEXT:  f78ee950 -- (.cxr 0xfffffffff78ee950)
eax=f78e0001 ebx=ffffffff ecx=006f0070 edx=00000000 esi=006f0058 edi=8087cdae
eip=8086c949 esp=f78eed1c ebp=f78eed2c iopl=0  nv up ei ng nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000     efl=00010286
nt!ObfDereferenceObject+0×23:
8086c949 f00fc11e        lock xadd dword ptr [esi],ebx ds:0023:006f0058=????????

[...]

STACK_TEXT:
f78eed2c f707212e 886e6530 f78eed80 f706e04e nt!ObfDereferenceObject+0x23
f78eed38 f706e04e e47b1258 8b2fcb40 808ae5c0 DriverA!CloseConnection+0x16
f78eed80 80880475 8835f248 00000000 8b2fcb40 DriverA!Resume+0x9f
f78eedac 80949c5a 8835f248 00000000 00000000 nt!ExpWorkerThread+0xeb
f78eeddc 8088e0c2 8088038a 00000000 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

1: kd> ub f707212e
DriverA!CloseConnection+0x2:
f707211a push    ebp
f707211b mov     ebp,esp
f707211d push    esi
f707211e mov     esi,dword ptr [ebp+8]
f7072121 mov     ecx,dword ptr [esi+14h]
f7072124 test    ecx,ecx
f7072126 je      DriverA!CloseConnection+0x1a (f7072132)
f7072128 call    dword ptr [DriverA!_imp_ObfDereferenceObject (f70610f4)]

1: kd> db e47b1258 L20
e47b1258  61 67 65 20 57 72 69 74-65 72 00 05 77 00 69 00  age Writer..w.i.
e47b1268  6e 00 73 00 70 00 6f 00-6f 00 6c 00 2c 00 4e 00  n.s.p.o.o.l.,.N.

1: kd> !pool e47b1258
Pool page e47b1258 region is Paged pool
e47b1000 size:  108 previous size:    0  (Allocated)  CM39
e47b1108 size:   38 previous size:  108  (Free)       CMVa
e47b1140 size:   28 previous size:   38  (Allocated)  NtFs
e47b1168 size:    8 previous size:   28  (Free)       CMDa
e47b1170 size:   80 previous size:    8  (Allocated)  FSim
e47b11f0 size:   28 previous size:   80  (Allocated)  CMNb (Protected)
*e47b1218 size:   70 previous size:   28  (Allocated) *CMDa
Pooltag CMDa : value data cache pool tag, Binary : nt!cm
e47b1288 size:   58 previous size:   70  (Allocated)  Sect (Protected)
e47b12e0 size:   18 previous size:   58  (Allocated)  Ntf0
e47b12f8 size:   28 previous size:   18  (Allocated)  NtFs
e47b1320 size:   20 previous size:   28  (Allocated)  CMNb (Protected)
e47b1340 size:   48 previous size:   20  (Allocated)  Ntfc
e47b1388 size:   68 previous size:   48  (Allocated)  Sect (Protected)
e47b13f0 size:   30 previous size:   68  (Allocated)  CMVa
e47b1420 size:   38 previous size:   30  (Allocated)  CMVa
e47b1458 size:    8 previous size:   38  (Free)       CMVa
e47b1460 size:   48 previous size:    8  (Allocated)  CMVa
e47b14a8 size:   d0 previous size:   48  (Allocated)  Ntfo
e47b1578 size:  330 previous size:   d0  (Allocated)  Ntff
e47b18a8 size:   10 previous size:  330  (Free)       NtfE
e47b18b8 size:   e0 previous size:   10  (Allocated)  Ntfo
e47b1998 size:   40 previous size:   e0  (Allocated)  MmSm
e47b19d8 size:    8 previous size:   40  (Free)       Ica
e47b19e0 size:   18 previous size:    8  (Allocated)  Ntf0
e47b19f8 size:   68 previous size:   18  (Allocated)  CMDa
e47b1a60 size:   28 previous size:   68  (Allocated)  ObNm
e47b1a88 size:   b8 previous size:   28  (Allocated)  Port (Protected)
e47b1b40 size:   58 previous size:   b8  (Allocated)  Sect (Protected)
e47b1b98 size:   30 previous size:   58  (Allocated)  CMVa
e47b1bc8 size:    8 previous size:   30  (Free)       NtFA
e47b1bd0 size:  100 previous size:    8  (Allocated)  IoNm
e47b1cd0 size:   18 previous size:  100  (Allocated)  ObDi
e47b1ce8 size:   38 previous size:   18  (Allocated)  CMnb Process: 88469928
e47b1d20 size:   78 previous size:   38  (Free )  NtFI
e47b1d98 size:   68 previous size:   78  (Allocated)  CMDa
e47b1e00 size:   18 previous size:   68  (Allocated)  PsIm (Protected)
e47b1e18 size:   e8 previous size:   18  (Free )  TunP
e47b1f00 size:  100 previous size:   e8  (Allocated)  IoNm

Another example:

0: kd> !analyze -v

[...]

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8083e4d6, The address that the exception occurred at
Arg3: f78cec54, Exception Record Address
Arg4: f78ce950, Context Record Address

[...]

EXCEPTION_RECORD:  f78cec54 -- (.exr 0xfffffffff78cec54)
ExceptionAddress: 8083e4d6 (nt!ObfDereferenceObject+0x00000023)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
   Parameter[1]: 65696c2b
Attempt to write to address 65696c2b

CONTEXT:  f78ce950 -- (.cxr 0xfffffffff78ce950)
eax=f78c0001 ebx=ffffffff ecx=65696c43 edx=00000000 esi=65696c2b edi=8083e407
eip=8083e4d6 esp=f78ced1c ebp=f78ced2c iopl=0  nv up ei ng nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000     efl=00010286
nt!ObfDereferenceObject+0×23:
8083e4d6 f00fc11e        lock xadd dword ptr [esi],ebx ds:0023:65696c2b=????????
Resetting default scope

[...]

STACK_TEXT:
f78ced2c f71bd12e 87216470 f78ced80 f71b904e nt!ObfDereferenceObject+0x23
f78ced38 f71b904e e49afb90 8a38eb40 808b70e0 DriverA!CloseConnection+0x16
f78ced80 8082db10 868989e0 00000000 8a38eb40 DriverA!Resume+0x9f
f78cedac 809208bb 868989e0 00000000 00000000 nt!ExpWorkerThread+0xeb
f78ceddc 8083fe9f 8082da53 00000000 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

[...]

0: kd> .formats 65696c2b
Evaluate expression:
Hex:     65696c2b
Decimal: 1701407787
Octal:   14532266053
Binary:  01100101 01101001 01101100 00101011
  Chars:   eil+
Time:    Fri Dec 01 05:16:27 2023
Float:   low 6.88942e+022 high 0
Double:  8.40607e-315

0: kd> db e49afb90 L20
e49afb90  41 41 22 00 1e 00 00 00-00 5f 07 00 01 00 00 00  AA”……_……
e49afba0  01 00 00 00 43 6c 69 65-6e 74 41 2f 41 41 41 41  ….ClientA/AAAA

0: kd> !pool e49afb90
Pool page e49afb90 region is Paged pool
e49af000 size:  330 previous size:    0  (Allocated)  Ntff
e49af330 size:  2c0 previous size:  330  (Allocated)  Toke (Protected)
e49af5f0 size:   78 previous size:  2c0  (Allocated)  NtFU
e49af668 size:   10 previous size:   78  (Free)       CMVI
e49af678 size:   a8 previous size:   10  (Allocated)  Ntfo
e49af720 size:   80 previous size:   a8  (Allocated)  NtFU
e49af7a0 size:   78 previous size:   80  (Allocated)  NtFU
e49af818 size:   18 previous size:   78  (Allocated)  Ntf0
e49af830 size:   20 previous size:   18  (Allocated)  ObHd
e49af850 size:   38 previous size:   20  (Allocated)  MmSm
e49af888 size:   78 previous size:   38  (Allocated)  NtFU
e49af900 size:   28 previous size:   78  (Allocated)  NtFs
e49af928 size:   48 previous size:   28  (Allocated)  Ntfc
e49af970 size:   40 previous size:   48  (Allocated)  CMNb (Protected)
e49af9b0 size:   28 previous size:   40  (Allocated)  NtFs
e49af9d8 size:   30 previous size:   28  (Allocated)  AtmA
e49afa08 size:  108 previous size:   30  (Allocated)  CM39
e49afb10 size:   18 previous size:  108  (Allocated)  Ntf0
e49afb28 size:   30 previous size:   18  (Allocated)  CMVw (Protected)
e49afb58 size:   28 previous size:   30  (Allocated)  MPXC
*e49afb80 size:   70 previous size:   28  (Free) *CMDa
Pooltag CMDa : value data cache pool tag, Binary : nt!cm
e49afbf0 size:   b8 previous size:   70  (Allocated)  Port (Protected)
e49afca8 size:   28 previous size:   b8  (Allocated)  CMNb (Protected)
e49afcd0 size:  330 previous size:   28  (Allocated)  Ntff

Notice that in the latter example the pointer references a freed pool element. If a pointer points to an overwritten buffer the result is similar to a dangling pointer pointing to a reallocated freed buffer. If an object was located in a shared buffer and its data becomes overwritten we can also observe Random Object pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Frame Pointer Omission pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the common mistakes that especially happens during a rush to provide an analysis results is overlooking UNICODE or ASCII fragments on thread stacks and mistakenly assuming that found symbolic references have some significance: 

0:001> dds 0bc9e5a8 0bc9e5d4
0bc9e5a8  00b900b8
0bc9e5ac  00bb00ba
0bc9e5b0  00bd00bc
0bc9e5b4  00bf00be ApplicationA!stdext::unchecked_uninitialized_fill_n<std::map<std:: basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::less<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > >,std::allocator<std::pair<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > const ,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > > > > * *,unsigned int,std::map<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::less<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > >,std::allocator<std::pair<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > const ,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > > > > *,std::allocator<std::map<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> >,std::less<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > >,std::allocator<std::pair<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > const ,std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short> > > > > *> >+0×1e
0bc9e5b8  00c100c0 ApplicationA!EnumData+0×670
0bc9e5bc  00c300c2 ApplicationA!CloneData+0xe2
0bc9e5c0  00c500c4 ApplicationA!LoadData+0×134
0bc9e5c4  00c700c6 ApplicationA!decompress+0×1ca6
0bc9e5c8  00c900c8 ApplicationA!_TT??_R2?AVout_of_rangestd+0×10c
0bc9e5cc  00cb00ca ApplicationA!AppHandle <PERF> (ApplicationA+0xd00ca)
0bc9e5d0  00cd00cc
0bc9e5d4  00cf00ce

0:001> du 0bc9e5a8
0bc9e5a8  "¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×"
0bc9e5e8  "ØÙÚÛÜÝÞßÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ÷"
0bc9e628  "ØÙÚÛÜÝÞŸĀ"

We can see and also double check from disassembly by using u/ub WinDbg command that function names are coincidental. It just happened that ApplicationA module spans the address range including 00bf00be and 00cb00ca UNICODE fragment values (having the pattern 00xx00xx):

0:001> lm m ApplicationA
start    end      module name
00be0000 00cb8000 ApplicationA (export symbols) ApplicationA.exe

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -