Software Diagnostics Library

Today we introduce an icon for Special Stack Trace pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I was inspecting the list of timers in a complete memory dump and found that in x64 W2K8 R2 (it may be also in other versions) there is a timer set to fire for the end of our century 2001 - 2100:

1: kd> !timer
Dump system timers
[...]
    fffff80001c31b80   f690c4d5 0064364d [12/31/2099 23:00:00.510]  nt!ExpCenturyDpcRoutine (DPC @ fffff80001c31bc0)
[…]

Its disassembly shows at that time the following work item will be executed:

1: kd> uf nt!ExpCenturyDpcRoutine
[...] 
fffff800`01ae81ab 488d0dae991400  lea     rcx,[nt!ExpCenturyWorkItem (fffff800`01c31b60)]
[...]

I don’t want to dig myself far into the details here May be someone from MS provides some comments what we should expect at the turn of the century? What should I tell my children to expect if they still run the Windows OS?

PS. Long live the Windows OS! (my the most sincere comment; no pun intended).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The presentation materials from the webinar (18th and 23rd of August, 2010) are available for download:

http://www.dumpanalysis.com/FCMDA-materials

Thanks to everyone who registered and attended!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The online version of Debugged! MZ/PE) underwent the total redesign:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Custom Exception Handler (user space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Do to the attendee limit not all registered for the first session were able to attend. The second session is available:

Date: 23rd of August 2010
Time: 19:00 (BST) 14:00 (Eastern) 11:00 (Pacific)
Duration: 90 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www1.gotomeeting.com/register/823155577

After the second session webinar slides will be published and later a Q&A page will be compiled.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having encountered several crash dumps with the code running on heap and the following similar stack traces

1: kd> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
02cdfbfc 0056511a 0x634648
02cdfc24 005651a1 ModuleA!ClassA::~ClassA+0x5a
02cdfc30 00562563 ModuleA!ClassA::`scalar deleting destructor'+0x11
[...]
02cdffec 00000000 kernel32!BaseThreadStart+0x37

I decided to model this situation. The idea was to corrupt a class member by overriding its vtable pointer with a heap entry address. Because the virtual destructor address is a first virtual method table entry in our class memory layout I made sure that it points to the same heap address by making vtable pointer a dereference fixpoint. Here is a source code based on how Visual C++ compiler implements objects in memory:

class Member {
public:
    virtual ~Member() { data = 1; };
public:
    int data;
};

class Compound {
public:
    Compound(): pm(NULL) { pm = new Member(); }
    virtual ~Compound() { delete pm; }
    void Corrupt() {
        unsigned int * pbuf = new unsigned int[0x10];
        *pbuf = reinterpret_cast<unsigned int>(pbuf); // to ensure that the code would run through pbuf pointer
        *reinterpret_cast<unsigned int *>(pm) = reinterpret_cast<unsigned int>(pbuf);
    }
    Member *pm;
};

int _tmain(int argc, _TCHAR* argv[])
{
    Compound *pc = new Compound();
    pc->Corrupt();
    delete pc;
    return 0;
}

In a crash dump we therefore see the similar stack trace:

0:000> .ecxr
eax=001f4c28 ebx=7efde000 ecx=001f4c18 edx=001f4c28 esi=00000000 edi=00000000
eip=001f4c28 esp=003cf7d0 ebp=003cf7e8 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
001f4c28 284c1f00        sub     byte ptr [edi+ebx],cl      ds:002b:7efde000=00

0:000> k
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
003cf7cc 011d10e5 0×1f4c28
003cf7e8 011d114f Destructors!Compound::~Compound+0×35
003cf7f4 011d121e Destructors!Compound::`scalar deleting destructor’+0xf
003cf82c 011d1498 Destructors!wmain+0×8e
003cf874 77043677 Destructors!__tmainCRTStartup+0xfa
003cf880 77719d72 kernel32!BaseThreadInitThunk+0xe
003cf8c0 77719d45 ntdll!__RtlUserThreadStart+0×70
003cf8d8 00000000 ntdll!_RtlUserThreadStart+0×1b

We now check the correctness of the stack trace by examining the return addresses:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> ub 011d10e5
Destructors!Compound::~Compound+0×21:
011d10d1 cmp  dword ptr [ebp-4],0
011d10d5 je   Destructors!Compound::~Compound+0×3a (011d10ea)
011d10d7 push 1
011d10d9 mov  ecx,dword ptr [ebp-4]
011d10dc mov  edx,dword ptr [ecx]
011d10de mov  ecx,dword ptr [ebp-4]
011d10e1 mov  eax,dword ptr [edx]
011d10e3 call eax

0:000> ub 011d114f
Destructors!Compound::Corrupt+0×3e:
011d113e int  3
011d113f int  3
Destructors!Compound::`scalar deleting destructor’:
011d1140 push ebp
011d1141 mov  ebp,esp
011d1143 push ecx
011d1144 mov  dword ptr [ebp-4],ecx
011d1147 mov  ecx,dword ptr [ebp-4]
011d114a call Destructors!Compound::~Compound (011d10b0)

We now examine the crash address:

0:000> u 001f4c28
001f4c28 sub  byte ptr [edi+ebx],cl
001f4c2c les  eax,fword ptr [eax]
001f4c2e pop  ds
001f4c2f add  byte ptr [eax],al
001f4c31 add  byte ptr [eax],al
001f4c33 add  byte ptr [eax],al
001f4c35 add  byte ptr [eax],al
001f4c37 add  byte ptr [eax],al

Then we check that it resides in a heap segment:

0:000> dt _PEB 7efde000
Destructors!_PEB
[...]
  +0x088 NumberOfHeaps : 2
  +0x08c MaximumNumberOfHeaps : 0x10
  +0x090 ProcessHeaps : 0x777e4740 -> 0x004b0000 Void
[...]

0:000> dd 0x777e4740 l2
777e4740 004b0000 001f0000

0:000> !heap 001f0000
Index Address Name Debugging options enabled
  2: 001f0000
  Segment at 001f0000 to 00200000 (00005000 bytes committed)

Now we check vtable to see that it was normal for Compound object but corrupt for Member object:

0:000> .frame 1
01 003cf7e8 011d114f Destructors!Compound::~Compound+0x35

0:000> dv /i /V
prv local  003cf7dc @ebp-0x0c            this = 0x001f4c08
0:000> dt Destructors!Compound 0x001f4c08
   +0x000 __VFN_table : 0x011daa0c
   +0x004 pm               : 0x001f4c18 Member

0:000> dps 0x001f4c08 l1
001f4c08  011daa0c Destructors!Compound::`vftable'

0:000> dps 0x001f4c18 l1
001f4c18  001f4c28

0:000> dps 001f4c28 l1
001f4c28  001f4c28

The application, its source code and PDB file are available for download here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In a complete memory dump we could see ALPC wait chains leading to ServiceA.exe process with a queue of 372 messages. Additionally we could also see ServiceB.exe process waiting for ServiceC.exe with the latter having a queue of 201 messages. Threads that were supposed to process some messages didn’t exist. ServiceC process had a thread that was waiting for ServiceA.exe as well. But there was no any indication for a thread-2-thread deadlock. We could also see that threads waiting for ServiceA.exe sometimes had the greater waiting time than threads waiting for ServiceC so it could be the case that the problem initially started with ServiceA.exe. However, after more thorough analysis we could also see several terminating ApplicationD.exe processes with just one thread waiting in ModuleE with the waiting time exceeding the waiting time of blocked threads waiting for ServiceA and ServiceC. Because of semantic process coupling between ServiceA and ApplicationD it was decided that ModuleE was responsible and its vendor was contacted for updates.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Madness, the absence of memory dumps.

Michel Foucault

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today I celebrated 4 years of blogging that resulted in more than 2,000 posts across 8 blogs. I would like to thank everyone for their continuing support!

The updated timeline

This blog post belongs to the 5th year of blogging.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The whole code is a series of balanced defects. Fix one and the program crashes.

Ralph Waldo Emerson

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump is a double buzzword.

Dmitry Vostokov

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Coupled Processes (semantics) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The one obvious pattern that is shown in many case studies on this blog is Exception Stack Trace (or Exception Thread). This is a stack trace that has exception processing functions, for example:

   9  Id: 1df4.a08 Suspend: -1 Teb: 7fff4000 Unfrozen
ChildEBP RetAddr 
1022f5a8 7c90df4a ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 kernel32!UnhandledExceptionFilter+0×8b9
1022f908 7c839b39 kernel32!BaseThreadStart+0×4d
1022f930 7c9032a8 kernel32!_except_handler3+0×61
1022f954 7c90327a ntdll!ExecuteHandler2+0×26
1022fa04 7c90e48a ntdll!ExecuteHandler+0×24
1022fa04 7c812afb ntdll!KiUserExceptionDispatcher+0xe
1022fd5c 0b82e680 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
1022fd94 0b82d2f2 DllA+0×21e640
1022fde8 7753004f DllA+0×21d4f2
1022fdfc 7753032f ole32!CClassCache::CDllPathEntry::CanUnload_rl+0×3b
1022ff3c 7753028b ole32!CClassCache::FreeUnused+0×70
1022ff4c 775300b5 ole32!CoFreeUnusedLibrariesEx+0×36
1022ff58 77596af5 ole32!CoFreeUnusedLibraries+0×9
1022ff6c 77566ff9 ole32!CDllHost::MTAWorkerLoop+0×25
1022ff8c 7752687c ole32!CDllHost::WorkerThread+0xc1
1022ff94 774fe3ee ole32!DLLHostThreadEntry+0xd
1022ffa8 774fe456 ole32!CRpcThread::WorkerLoop+0×1e
1022ffb4 7c80b729 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0×1b
1022ffec 00000000 kernel32!BaseThreadStart+0×37

Such exceptions can be detected by the default analysis command (for example, !analyze -v WinDbg command) or by inspecting a stack trace collection. However if we didn’t see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack data. 

In our case we can get the exception information by looking at parameters to in unhandled exception filter:

0:009> kv 3
ChildEBP RetAddr Args to Child
1022f5a8 7c90df4a 7c8648a2 00000002 1022f730 ntdll!KiFastSystemCallRet
1022f5ac 7c8648a2 00000002 1022f730 00000001 ntdll!ZwWaitForMultipleObjects+0xc
1022f900 7c83ab50 1022f928 7c839b39 1022f930 kernel32!UnhandledExceptionFilter+0×8b9

0:009> .exptr 1022f928

----- Exception record at 1022fa1c:
ExceptionAddress: 7c812afb (kernel32!RaiseException+0x00000053)
  ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
  Parameter[0]: 19930520
  Parameter[1]: 1022fda4
  Parameter[2]: 0b985074
  pExceptionObject: 1022fda4
  _s_ThrowInfo : 0b985074

----- Context record at 1022fa3c:
eax=1022fd0c ebx=00000001 ecx=00000000 edx=1022fda4 esi=1022fd94 edi=77606068
eip=7c812afb esp=1022fd08 ebp=1022fd5c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
kernel32!RaiseException+0x53:
7c812afb 5e pop esi

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we get memory dumps that are difficult to analyze in full because some if not most of information was omitted during saving. These are usually small memory dumps (contrasted with kernel and complete) and user process minidumps. We can easily recognize that when we open a dump:

User Mini Dump File: Only registers, stack and portions of memory are available

or

Mini Kernel Dump File: Only registers and stack trace are available

The same also applies to user dumps where thread times information is omitted so it is not possible to use !runaway WinDbg command or to a dump saved with various options of .dump command (including privacy-aware) instead of /ma or deprecated /f option. On the contrary, manually erased data in crash dumps looks more like an example of another pattern called Lateral Damage.

The similar cases of abridged dumps are discussed in Wrong Dump and Missing Space antipatterns.

Anyway, we shouldn’t dismiss such dumps and should try to analyze them. For example, some approaches (including using image binaries) are listed in kernel minidump analysis series. We can even see portions of raw stack data in search of execution residue:

0: kd> !thread
GetPointerFromAddress: unable to read from 81d315b0
THREAD 82f49020  Cid 0004.0034  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    Unable to read nt!_IRP @ 8391e008
Not impersonating
GetUlongFromAddress: unable to read from 81d0ad90
Owning Process            82f00ab0       Image:         System
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      4000214     
Context Switch Count      21886            
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0x81c78ea3)
Stack Init 85be0000 Current 85bdf7c0 Base 85be0000 Limit 85bdd000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps 85bdd000 85be0000
85bdd000  ????????
85bdd004  ????????
85bdd008  ????????
85bdd00c  ????????
85bdd010  ????????
85bdd014  ????????
85bdd018  ????????
85bdd01c  ????????
85bdd020  ????????
85bdd024  ????????
85bdd028  ????????
[...]
85bdf8c4  ????????
85bdf8c8  ????????
85bdf8cc  ????????
85bdf8d0  0000000a
85bdf8d4  a112883e
85bdf8d8  0000001b
85bdf8dc  00000000
85bdf8e0  81c28750 nt!KeSetEvent+0x4d
85bdf8e4  85bdf8e8
85bdf8e8  85bdf970
85bdf8ec  81c28750 nt!KeSetEvent+0x4d
85bdf8f0  badb0d00
85bdf8f4  00000000
85bdf8f8  00000000
85bdf8fc  81cf4820 nt!KiInitialPCR+0x120
85bdf900  00000000
85bdf904  85bdf938
85bdf908  81cf4820 nt!KiInitialPCR+0x120
85bdf90c  00000000
85bdf910  81d32300 nt!IopTimerLock
85bdf914  00000000
85bdf918  81fa0000 nt!_NULL_IMPORT_DESCRIPTOR <PERF> (nt+0x3a0000)
85bdf91c  85bd0023
85bdf920  00000023
85bdf924  00000000
85bdf928  81d323c0 nt!KiDispatcherLock
85bdf92c  a1128828
85bdf930  85bdf9b4
85bdf934  85bdfdb0
85bdf938  00000030
85bdf93c  84ca6f40
85bdf940  84ca6f38
85bdf944  00000001
85bdf948  85bdf970
85bdf94c  00000000
85bdf950  81c28750 nt!KeSetEvent+0x4d
85bdf954  00000008
85bdf958  00010246
85bdf95c  00000000
85bdf960  84ca68a0
[...]
85bdfd2c  82f49020
85bdfd30  835ca4d0
85bdfd34  a6684538
85bdfd38  81cfde7c nt!ExWorkerQueue+0x3c
85bdfd3c  00000001
85bdfd40  00000000
85bdfd44  85bdfd7c
85bdfd48  81c78fa0 nt!ExpWorkerThread+0xfd
85bdfd4c  835ca4d0
85bdfd50  00000000
85bdfd54  82f49020
85bdfd58  00000000
85bdfd5c  00000000
85bdfd60  0069000b
85bdfd64  00000000
85bdfd68  00000001
85bdfd6c  00000000
85bdfd70  835ca4d0
85bdfd74  81da9542 nt!PnpDeviceEventWorker
85bdfd78  00000000
85bdfd7c  85bdfdc0
85bdfd80  81e254e0 nt!PspSystemThreadStartup+0x9d
85bdfd84  835ca4d0
85bdfd88  85bd4680
85bdfd8c  00000000
85bdfd90  00000000
85bdfd94  00000000
85bdfd98  00000002
85bdfd9c  00000000
85bdfda0  00000000
85bdfda4  00000001
85bdfda8  85bdfd88
85bdfdac  85bdfdbc
85bdfdb0  ffffffff
85bdfdb4  81c8aad5 nt!_except_handler4
85bdfdb8  81c9ddb8 nt!`string'+0x4
85bdfdbc  00000000
85bdfdc0  00000000
85bdfdc4  81c9159e nt!KiThreadStartup+0x16
85bdfdc8  81c78ea3 nt!ExpWorkerThread
85bdfdcc  00000001
85bdfdd0  00000000
85bdfdd4  00000000
85bdfdd8  002e0069
85bdfddc  006c0064
85bdfde0  004c006c
85bdfde4  00000000
85bdfde8  000007f0
85bdfdec  00010000
85bdfdf0  0000027f
85bdfdf4  00000000
85bdfdf8  00000000
85bdfdfc  00000000
85bdfe00  00000000
85bdfe04  00000000
85bdfe08  00001f80
85bdfe0c  0000ffff
85bdfe10  00000000
85bdfe14  00000000
85bdfe18  00000000
[...]
85bdffe4  00000000
85bdffe8  00000000
85bdffec  00000000
85bdfff0  00000000
85bdfff4  00000000
85bdfff8  00000000
85bdfffc  00000000
85be0000  ????????

User minidumps are similar here:

0:001> k
ChildEBP RetAddr 
099bfe147c90daaa ntdll!KiFastSystemCallRet
099bfe18 77e765e3 ntdll!NtReplyWaitReceivePortEx+0xc
099bff80 77e76caf rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×12a
099bff88 77e76ad1 rpcrt4!RecvLotsaCallsWrapper+0xd
099bffa8 77e76c97 rpcrt4!BaseCachedThreadRoutine+0×79
099bffb4 7c80b729 rpcrt4!ThreadStartRoutine+0×1a
099bffec 00000000 kernel32!BaseThreadStart+0×37

0:001> dd 099bfe14
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006
099bfe74  00000006 e1f79698 e39b8b60 00000000
099bfe84  fbe33c40 00000001 e5ce12f8 b279db9c

0:001> dd 099bfe14-20
099bfdf4  ???????? ???????? ???????? ????????
099bfe04  ???????? ???????? ???????? ????????
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006

As a warning here it is possible to conclude that minidumps can also reveal private information especially when ASCII or Unicode buffers are seen on raw stack data.

I was thinking how to name this pattern and Oxford Thesaurus of English suggested the following name: Abridged Dump by analogy with an abridged book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to strong and weak process coupling patterns we also have another variant that I call semantic coupling. Some processes (not necessarily from the same vendor) cooperate to provide certain functionality. The cooperation might not involve trackable and visible inter-process communication such as (A)LPC/RPC or pipes but involve events, shared memory and other possible mechanisms not explicitly visible when we look at memory dumps. In many cases, after finding problems in one or several processes from a semantic group we also look at the remaining processes from that group to see if there are some anomalies there as well. The one example I encounter often can be generalized as follows: we have an ALPC wait chain ProcessA -> ProcessB <-> ProcessC (not necessarily a deadlock) but the crucial piece of functionality is also implemented in ProcessD. Sometimes ProcessD is healthy and the problem resides in ProcessC or ProcessB, and sometimes, when we look at ProcessD we find evidence of an earlier problem pattern there so the focus of recommendations shifts to one of ProcessD modules. The case study is coming soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Started testing marketing materials to increase Dublin people awareness of memory dump analysis. Here are the first exemplars of what I’m going to wear by the end of this week while commuting and in local pubs

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we show the possible signs of the classical Thread Starvation.
Suppose we have two running threads with a priority 8:

0: kd> !running

System Processors 3 (affinity mask)
  Idle Processors 0

Prcbs  Current   Next   
  0    ffdff120  89a92020            O...............
  1    f7737120  89275020            W...............

0: kd> !thread 89a92020
THREAD 89a92020  Cid 11d8.27d8  Teb: 7ffd9000 Win32Thread: bc1e6860 RUNNING on processor 0
[...]
Priority 8 BasePriority 8 PriorityDecrement 0

0: kd> !thread 89275020
THREAD 89275020  Cid 1cd0.2510  Teb: 7ffa9000 Win32Thread: bc343180 RUNNING on processor 1
[...]
Priority 8 BasePriority 8 PriorityDecrement 0

If we have other threads ready with the same priority contending for the same processors other threads with less priority might starve (shown in red):

0: kd> !ready
Processor 0: Ready Threads at priority 8
    THREAD 894a1db0  Cid 1a98.25c0  Teb: 7ffde000 Win32Thread: bc19cea8 READY
    THREAD 897c4818  Cid 11d8.1c5c  Teb: 7ffa2000 Win32Thread: bc2c5ba8 READY
    THREAD 8911fd18  Cid 2730.03f4  Teb: 7ffd9000 Win32Thread: bc305830 READY
Processor 0: Ready Threads at priority 7
    THREAD 8a9e5ab0  Cid 0250.0470  Teb: 7ff9f000 Win32Thread: 00000000 READY
    THREAD 8a086838  Cid 0250.0654  Teb: 7ff93000 Win32Thread: 00000000 READY
    THREAD 8984b8b8  Cid 0250.1dc4  Teb: 7ff99000 Win32Thread: 00000000 READY
    THREAD 8912a4c0  Cid 0f4c.2410  Teb: 7ff81000 Win32Thread: 00000000 READY
    THREAD 89e5c570  Cid 0f4c.01c8  Teb: 00000000 Win32Thread: 00000000 READY
Processor 0: Ready Threads at priority 6
    THREAD 8a9353b0  Cid 1584.1598  Teb: 7ff8b000 Win32Thread: bc057698 READY
    THREAD 8aba2020  Cid 1584.15f0  Teb: 7ff9f000 Win32Thread: bc2a0ea8 READY
    THREAD 8aab17a0  Cid 1584.01a8  Teb: 7ff92000 Win32Thread: bc316ea8 READY
    THREAD 8a457020  Cid 1584.0634  Teb: 7ff8d000 Win32Thread: bc30fea8 READY
    THREAD 8a3d4020  Cid 1584.1510  Teb: 7ff8f000 Win32Thread: bc15b8a0 READY
    THREAD 8a5f5db0  Cid 1584.165c  Teb: 7ff9d000 Win32Thread: bc171be8 READY
    THREAD 8a297020  Cid 0f4c.0f54  Teb: 7ffde000 Win32Thread: bc20fda0 READY
    THREAD 8a126020  Cid 1584.175c  Teb: 7ffa9000 Win32Thread: 00000000 READY
    THREAD 8a548478  Cid 0250.07b0  Teb: 7ff9a000 Win32Thread: 00000000 READY
    THREAD 8a478020  Cid 0944.0988  Teb: 7ffd9000 Win32Thread: 00000000 READY
    THREAD 8986ad08  Cid 1d2c.1cf0  Teb: 7ffa8000 Win32Thread: bc285800 READY
    THREAD 897f4db0  Cid 1d2c.2554  Teb: 7ffdb000 Win32Thread: bc238e80 READY
    THREAD 89a2e618  Cid 1d2c.1de4  Teb: 7ffdd000 Win32Thread: bc203908 READY
Processor 0: Ready Threads at priority 0
    THREAD 8b184db0  Cid 0004.0008  Teb: 00000000 Win32Thread: 00000000 READY
Processor 1: Ready Threads at priority 8
    THREAD 89d89db0  Cid 1b10.20ac  Teb: 7ffd7000 Win32Thread: bc16e680 READY
    THREAD 891f24a8  Cid 1e2c.20d0  Teb: 7ffda000 Win32Thread: bc1b9ea8 READY
    THREAD 89214db0  Cid 1e2c.24d4  Teb: 7ffd7000 Win32Thread: bc24ed48 READY
    THREAD 89a28020  Cid 1b10.21b4  Teb: 7ffa7000 Win32Thread: bc25b3b8 READY
    THREAD 891e03b0  Cid 1a98.05c4  Teb: 7ffdb000 Win32Thread: bc228bb0 READY
    THREAD 891b0020  Cid 1cd0.0144  Teb: 7ffde000 Win32Thread: bc205ea8 READY
Processor 1: Ready Threads at priority 7
    THREAD 898367a0  Cid 0f4c.1cd4  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 8a1ac020  Cid 0f4c.1450  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 8aa1ab90  Cid 0f4c.11b0  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 89cc92e0  Cid 0f4c.1b34  Teb: 00000000 Win32Thread: 00000000 READY
    THREAD 89579020  Cid 0f4c.2220  Teb: 00000000 Win32Thread: 00000000 READY
Processor 1: Ready Threads at priority 6
    THREAD 8a487db0  Cid 1584.14bc  Teb: 7ffa2000 Win32Thread: bc304ea8 READY
    THREAD 8a3ce020  Cid 1584.0630  Teb: 7ff8e000 Win32Thread: bc293c20 READY
    THREAD 8a1b6db0  Cid 1584.1590  Teb: 7ff8c000 Win32Thread: bc310ea8 READY
    THREAD 8a1fe6e0  Cid 1584.15ec  Teb: 7ffa1000 Win32Thread: bc15bea8 READY
    THREAD 8ac0adb0  Cid 1584.156c  Teb: 7ff8a000 Win32Thread: bc153be8 READY
    THREAD 8b1e35a0  Cid 1584.15f4  Teb: 7ff9e000 Win32Thread: bc0567e8 READY
    THREAD 8a3288e8  Cid 1584.14b8  Teb: 7ff9a000 Win32Thread: bc2fbea8 READY
    THREAD 8a5056a0  Cid 1584.1518  Teb: 7ff91000 Win32Thread: bc337ea8 READY
    THREAD 891afdb0  Cid 1d2c.27e8  Teb: 7ffaf000 Win32Thread: bc217c18 READY
    THREAD 8a07d308  Cid 1d2c.2548  Teb: 7ffae000 Win32Thread: bc235750 READY
    THREAD 8a055d18  Cid 1584.17d0  Teb: 7ffd5000 Win32Thread: 00000000 READY
    THREAD 8ac0b770  Cid 0250.0268  Teb: 7ffde000 Win32Thread: bc2349d8 READY
    THREAD 8a0eeb40  Cid 1584.1560  Teb: 7ffdc000 Win32Thread: 00000000 READY

Here we should also analyze stack traces for running and ready threads with priority 8 and check kernel and user times. If we find anything common between them we should also check ready threads with lower priority to see if that commonality is unique to threads with priority 8. See also the similar pattern: Busy System and the similar starvation pattern resulted from realtime priority threads.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I accidentally use du WinDbg command (to interpret memory as Unicode) instead of da (to interpret memory as ASCII):

0:000> db 07329f28 l20
07329f28  68 69 64 70 6e 73 74 3d-26 74 78 74 74 6f 3d 26  hidpnst=&txtto=&
07329f38  74 78 74 63 63 3d 26 74-78 74 62 63 63 3d 26 74  txtcc=&txtbcc=&t

0:000> da 07329f28 l20
07329f28  "hidpnst=&txtto=&txtcc=&txtbcc=&t"

0:000> du 07329f28 l10
07329f28  "楨灤獮㵴琦瑸潴.硴捴㵣琦瑸换㵣琦"

This time I tried to get extra hidden meaning from a process dump taken after the process suffered a CPU spike by using Google translator and got this text (I put more lengthy Unicode sequence and removed some offensive words):

&quot;Luan Xian Zhen Qi-bin 㵴 cisternae. Huasong 㵣 Qi, Qi-bin-bin for 㵣 pull 㵪 䕒 .. 䉉 Ya Hui material. Hong SHIKA King. Huajiayuyan nuts .. 䐰 〥 䅁 evil force. Rafter Hui Qi 䤫 Mi cat deterrent Junying hydrogen walk. cisternae Huzhao Man cat Wuzhou Wen Zhen Zhao Zhen Pan scene file Shan. prison Shang Tang. Jue Shi Pan. sewage knock Xi. generous Zhen. 䤫. ice. conflict. cisternae Zhao askance nuts. rafter .. On unfeigned domain knock. Kagesue Mankuo. 㜲 Ruo Yi enemy luster of gems. cisternae Yu Wei Shan scene. Tan knock Shan. tally Xia Pan Ying. rafter. Xia. luster of gems tumultuous. Jing Feng-Tou Airuo enemy luster of gems Yixian … additionally . Tu. civet eliminating the lot Shan Ying RB Thieme, Jr.-Voltage trapping Feng-潷 Man. Tan knock Ruo Yi Xian cat enemy luster of gems. rafter Shi Feng-Tou. Mu. Minli Bang domain sewage Huitangyuzhao Su-hai.-Voltage Jiumi. rafter. Qing Wei Jun. 歳 Mi hai 䤫 Panyu. Zhucuoqufang .. 䐰 〥. 䐰 〥 䥁 hydrogen walk. rafter. Mount Zao Man. .. Run-Voltage Rendering. Tang Ying Yi. Shisuqingshi Fangmaosheji Yu Zhao 䤫 Su-. tide. tatami knock Feng-generous. rafter. Min luster of gems. Que Tu Mei Shi Tang Pan Ying. Jijue-Voltage. rafter. Wei Hui Mongoose Feng-. hunting. rafter. revolves Recent-Voltage sewage 䤫. stay Jiao RB Thieme, Jr soup.潷 Han.’m setback Xun. Han Tun petty. Liaohe. 䥔 end of Tu Feng-generous. rafter Xiang Shan Li Tu. trapping the end of sleep ZHEJIANG NORMAL Feng-Tou Yu Xun Jing Wen Fang 䤫 .. 䠫 pine and methods of disease. tatami knock Feng-generous. apply Feng-evil force fell Junying Su-Ao Po .. knock .. Tan Li Shan Jie look askance alone. ㅆ Guang Tang rafter. pool just cultural and&quot;

From the translation I see previously hidden notions of gems, disease and evil forces :-)

ASCII->Unicode->translation->ASCII

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

8

A separating digit of user space and kernel space in equi-bipartitional 32-bit virtual memory, for example, on x86 Windows systems:

RetAddr
80833491 nt!KiSwapContext+0×26
80828f2b nt!KiSwapThread+0×2e5
8099490a nt!KeDelayExecutionThread+0×2ab
808897cc nt!NtDelayExecution+0×84
7c82860c nt!KiFastCallEntry+0xfc
7c826f69 ntdll!KiFastSystemCallRet
7c83d454 ntdll!NtDelayExecution+0xc
77e6482f ntdll!RtlpTimerThread+0×47
00000000 kernel32!BaseThreadStart+0×34

lkd> lm
start end module name
[…]
77bd0000 77c19000 GDI32
77c20000 77cbf000 RPCRT4
77cd0000 77d5b000 OLEAUT32
77e20000 77eb2000 USER32
77ec0000 77f12000 SHLWAPI
77f30000 77fdc000 ADVAPI32
7c800000 7c915000 kernel32
7c920000 7c9e8000 ntdll
7c9f0000 7d1f9000 SHELL32
7e020000 7e02f000 SAMLIB
80800000 80a5d000 nt
80a5d000 80a89000 hal
bf800000 bf9d0000 win32k
[…]

This number doesn’t continue to be used in the same way on x64 Windows systems, for example:

RetAddr Call Site
fffff800`0167e082 nt!KiSwapContext+0×7a
fffff800`0167a34b nt!KiCommitThreadWait+0×1d2
fffff800`01973def nt!KeWaitForMultipleObjects+0×271
fffff800`019746f6 nt!ObpWaitForMultipleObjects+0×294
fffff800`01675ed3 nt!NtWaitForMultipleObjects+0xe5
00000000`77ca030a nt!KiSystemServiceCopyEnd+0×13

lkd> lm
start end module name
00000000`774a0000 00000000`7756d000 USER32
00000000`77570000 00000000`7769d000 kernel32
00000000`776a0000 00000000`77826000 ntdll
00000000`ffe60000 00000000`ffec0000 services
000007fe`fca90000 000007fe`fca97000 wshtcpip
000007fe`fcae0000 000007fe`fcb0c000 NTMARTA
000007fe`fcbe0000 000007fe`fcc34000 schannel
000007fe`fce10000 000007fe`fce5f000 mswsock
000007fe`fcf60000 000007fe`fcf69000 credssp
000007fe`fcf90000 000007fe`fcf97000 wship6
000007fe`fd0d0000 000007fe`fd207000 CRYPT32
000007fe`fd260000 000007fe`fd279000 MSASN1
000007fe`fd5a0000 000007fe`fd5b6000 NCObjAPI
000007fe`fd5c0000 000007fe`fd625000 SCESRV
000007fe`fd630000 000007fe`fd665000 apphelp
000007fe`fd680000 000007fe`fd6a6000 AUTHZ
000007fe`fd950000 000007fe`fd96c000 Secur32
000007fe`fd970000 000007fe`fd996000 USERENV
000007fe`fd9a0000 000007fe`fd9bc000 SAMLIB
000007fe`fdc80000 000007fe`fdce4000 GDI32
000007fe`fdd10000 000007fe`fdd1d000 LPK
000007fe`fdd20000 000007fe`fde28000 ADVAPI32
000007fe`fde30000 000007fe`fdecc000 msvcrt
000007fe`fded0000 000007fe`fdefd000 IMM32
000007fe`fdf00000 000007fe`fe0d8000 ole32
000007fe`fe0e0000 000007fe`fe0e7000 NSI
000007fe`fe190000 000007fe`fe22a000 USP10
000007fe`fe230000 000007fe`fe373000 RPCRT4
000007fe`ff340000 000007fe`ff395000 WLDAP32
000007fe`ff540000 000007fe`ff584000 WS2_32
000007fe`ff7d0000 000007fe`ff8d2000 MSCTF
fffff800`01802000 fffff800`01848000 hal
fffff800`01848000 fffff800`01d60000 nt
fffff960`000f0000 fffff960`003a1000 win32k
[…]

Synonyms:

Antonyms:

Also: virtual memory, user space, kernel space, x86, x64, 7

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -