This pattern description is short. Pleiades is a cluster of modules in lm WinDbg command output that serve similar function, like print drivers in print spooler or Citrix printing services. Usually we know that anyone of them could be at fault. Another example is a group of process modules in a complete memory dump serving the same function in separate terminal services sessions.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
For analysis of memory dumps from coupled processes or, in general, memory fibers from fiber bundle memory spaces we need to know their creation times (called debug session time). In some cases we need to know their time sequence: which process memory dump was saved first and how much time had passed before the second process memory dump was saved. Beside an initial output when we open a dump .time and version WinDbg commands can be used to check this information at any time during analysis.
In one example involving printing we see a blocking thread trying to contact a print spooler service using LPC. Its thread age is no more than 3 seconds. We also have the print spooler service process memory dump supposedly taken at the same time. However, when we check we see it was saved 2 minutes before. Moreover, PrintIsolationHost.exe process memory dump was saved even earlier. So the whole sequence was reversed because the printing application calls the spooler and it calls the appropriate driver, not the way around. We call this pattern Unsynchronized Dumps.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Having Fun - Having too many functions to analyze, reverse engineer, or simply having too many of them on a call stack.
Examples: We were having fun all night. A thread had fun culminating in a double fault. Why do I love debugging?
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Adult Debugging - The act of debugging when you are dealing with Functions you don’t know much about using public sources. Comes from famous xxx calls, for example:
win32k!xxxHkCallHook
win32k!xxxCallHook2
win32k!xxxCallHook
win32k!xxxReceiveMessage
win32k!xxxRealSleepThread
win32k!xxxSleepThread
win32k!xxxInterSendMsgEx
win32k!xxxSendMessageTimeout
win32k!xxxWrapSendMessage
Examples: We are doing adult debugging from time to time.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes, it is useful to know how much time ago a thread was created in order to understand when other behavioral patterns possibly started to appear. For example, in user process memory dumps with saved thread time information we can see using !runaway WinDbg extension command or using .ttime command. Looking at a stack trace collection we notice a thread blocked in LPC call:
0:000> ~40 kc
Call Site
ntdll!NtAlpcSendWaitReceivePort
rpcrt4!LRPC_CCALL::SendReceive
rpcrt4!NdrpClientCall3
rpcrt4!NdrClientCall3
[...]
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
We are interested when all this started because we want to compare with other coupled process memory dumps saved at different times:
0:000> !runaway f
User Mode Time
[...]
Kernel Mode Time
Thread Time
[...]
Elapsed Time
Thread Time
0:8ac 4 days 11:42:14.484
1:8b4 4 days 11:42:14.296
[...]
35:868 4 days 10:18:48.255
36:73ec 0 days 15:55:31.938
37:c0bc 0 days 10:36:53.447
38:782c 0 days 0:02:01.683
39:5864 0 days 0:00:52.236
40:5ffc 0 days 0:00:02.565
0:000> ~40s
ntdll!NtAlpcSendWaitReceivePort+0xa:
00000000`76d3ff0a c3 ret
0:040> .ttime
Created: Tue Jun 14 15:15:28.444 2011
Kernel: 0 days 0:00:00.000
User: 0 days 0:00:00.000
0:040> .time
Debug session time: Tue Jun 14 15:15:31.000 2011
System Uptime: 4 days 11:43:21.389
Process Uptime: 4 days 11:42:15.000
Kernel time: 0 days 0:00:10.000
User time: 0 days 0:04:22.000
We call this pattern Thread Age. We see that our blocked thread had only recently started compared to other threads and actually started after other dumps were saved when we look at their debug session time.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
As the number of crash dump analysis pattern icons approaches 100 with software trace analysis patterns (and other troubleshooting and debugging pattern categories) on the horizon with every icon as a symbol signifying certain computer memory signs (computer memory semiotics is called memiotics, as a part of memoretics discipline) I found it necessary to name the new form of art. I decided upon the name Memioart. If you have a better name for it please let me know. I also plan to write commentaries on memioicons (semioicons) to explain elements on pattern icons and reasons why I chose them.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Today we introduce an icon for Handle Limit (GDI) pattern:
B/W
Color
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
In this part we introduce Coupled Modules pattern. Often we identify a pattern that points to a particular module such as a driver or DLL other modules could use functional services from and, therefore, the latter modules can be implicated in abnormal software behavior. For example, detected insufficient kernel paged pool memory pointed to a driver that owns a pool tag DRV:
1: kd> !poolused 4
Sorting by Paged Pool Consumed
Tag Allocs Frees Diff Used
DRV 1466496 1422361 44135 188917256 UNKNOWN pooltag ‘DRV ‘, please update pooltag.txt
File 6334830 6284036 50794 6735720 File objects
Thre 53721 45152 8569 4346432 Thread objects , Binary: nt!ps
[…]
This module is known to be a directing module to other drivers (from stack trace perspective) but we also know that other (directed) modules use its services that require memory allocation.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
A page to reference all different kinds of stack traces is necessary, so I created this post:
I’ll update it as soon as I add more similar patterns.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
A service written in one of .NET languages was described as crashing and a process dump was collected for analysis. The default analysis command couldn’t an exception so we had to look at stack trace collection to find any anomalies. Indeed, there was a thread with message box code:
0:010> kL
Child-SP RetAddr Call Site
00000000`1f69e808 00000000`774b4bc4 user32!ZwUserWaitMessage+0xa
00000000`1f69e810 00000000`774b4edd user32!DialogBox2+0x274
00000000`1f69e8a0 00000000`77502920 user32!InternalDialogBox+0x135
00000000`1f69e900 00000000`77501c15 user32!SoftModalMessageBox+0x9b4
00000000`1f69ea30 00000000`7750146b user32!MessageBoxWorker+0x31d
00000000`1f69ebf0 00000000`77501362 user32!MessageBoxTimeoutW+0xb3
00000000`1f69ecc0 000007fe`f1590ce7 user32!MessageBoxW+0×4e
00000000`1f69ed00 000007fe`eb0f5c59 mscorwks!DoNDirectCall__PatchGetThreadCall+0×7b
[…]
MessageBoxW parameters were showing self-diagnosis pattern with a stack trace:
0:010> du 00000000`085f90c8
00000000`085f90c8 "...... at ClassA.foo()”
[…]
0:010> du 00000000`085f9c40
00000000`085f9c40 "Assertion Failed: Abort=Quit, Re"
00000000`085f9c80 "try=Debug, Ignore=Continue"
We tried unsuccessfully to load CLR extensions and requested the copy of .NET Framework from the affected computer. After that we were able to load version-specific extension, see managed stack trace and managed code exception:
0:010> !DumpStack
OS Thread Id: 0x8dc (15)
Child-SP RetAddr Call Site
000000001f69e808 00000000774b4bc4 user32!ZwUserWaitMessage+0xa
000000001f69e810 00000000774b4edd user32!DialogBox2+0x274
000000001f69e8a0 0000000077502920 user32!InternalDialogBox+0x135
000000001f69e900 0000000077501c15 user32!SoftModalMessageBox+0x9b4
000000001f69ea30 000000007750146b user32!MessageBoxWorker+0x31d
000000001f69ebf0 0000000077501362 user32!MessageBoxTimeoutW+0xb3
000000001f69ecc0 000007fef1590ce7 user32!MessageBoxW+0x4e
000000001f69ed00 000007feeb0f5c59 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
[...]
000000001f69e030 000007ff00a9ba1c ModuleA!ClassA.foo()+0×47
[…]
000000001f69fe30 000000007781c521 kernel32!BaseThreadInitThunk+0xd
000000001f69fe60 0000000000000000 ntdll!RtlUserThreadStart+0×1d
0:010> ~0s
0:000> !pe
Exception object: 0000000005a976b8
Exception type: System.FormatException
Message: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
InnerException: <none>
StackTrace (generated):
SP IP Function
0000000000D0BE40 000007FEEC2153B0 mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0x999280
0000000000D0BEE0 000007FEEB87C0FA mscorlib_ni!System.String.Format(System.IFormatProvider, System.String, System.Object[])+0x5a
0000000000D0BF30 000007FF00AB336B ModuleA!ClassB.get()+0xeb
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
We have CLR Thread and Managed Code Exception patterns. But something is missing in this picture especially when we try to communicate an analysis. While writing today a case study I found the need to name another pattern as Managed Stack Trace. Typical examples are stack traces from !CLRStack and !pe extension commands or subtraces from !DumpStack and !EEStack extension commands:
0:000> !pe
Exception object: 0000000005a976b8
Exception type: System.FormatException
Message: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
InnerException: <none>
StackTrace (generated):
SP IP Function
0000000000D0BE40 000007FEEC2153B0 mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider, System.String, System.Object[])+0×999280
0000000000D0BEE0 000007FEEB87C0FA mscorlib_ni!System.String.Format(System.IFormatProvider, System.String, System.Object[])+0×5a
0000000000D0BF30 000007FF00AB336B ModuleA!ClassB.get()+0xeb
0:010> !DumpStack
OS Thread Id: 0x8dc (15)
Child-SP RetAddr Call Site
000000001f69e808 00000000774b4bc4 user32!ZwUserWaitMessage+0xa
000000001f69e810 00000000774b4edd user32!DialogBox2+0x274
000000001f69e8a0 0000000077502920 user32!InternalDialogBox+0x135
000000001f69e900 0000000077501c15 user32!SoftModalMessageBox+0x9b4
000000001f69ea30 000000007750146b user32!MessageBoxWorker+0x31d
000000001f69ebf0 0000000077501362 user32!MessageBoxTimeoutW+0xb3
000000001f69ecc0 000007fef1590ce7 user32!MessageBoxW+0x4e
000000001f69ed00 000007feeb0f5c59 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
[...]
000000001f69e030 000007ff00a9ba1c ModuleA!ClassA.foo()+0×47
[…]
000000001f69fe30 000000007781c521 kernel32!BaseThreadInitThunk+0xd
000000001f69fe60 0000000000000000 ntdll!RtlUserThreadStart+0×1d
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
It was my dream since the school days to learn physics in its entirety. Whereas The Feynman Lectures on Physics: Commemorative Issue, Three Volume Set
- Dmitry Vostokov @ LiterateScientist.com -
“… further” applications “suffered from over-expansion owing to the exhaustion of their” computational “opportunities…”
Graeme Snooks, The Dynamic Society: The Sources of Global Change
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
To generate crash dumps for my recent presentation The Old New Crash: Cloud Memory Dump Analysis I created the tool similar to TestWER. It is a .NET worker service role for Windows Azure that does a NULL pointer assignment. Here is a source code from WorkerRole.cs (I left EnableCollection twice just in case :-)):
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.Diagnostics.Management;
namespace TestWAERWorker
{
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
Trace.WriteLine("TestWAERWorker entry point called", "Information");
Thread.Sleep(60000);
unsafe
{
int* p = null;
*p = 1;
};
while (true)
{
Thread.Sleep(10000);
Trace.WriteLine("Working", "Information");
}
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
CrashDumps.EnableCollection(true);
DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();
string conn_str = RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins. Diagnostics.ConnectionString");
CloudStorageAccount account = CloudStorageAccount.Parse(conn_str);
config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(account, config);
CrashDumps.EnableCollection(true);
return base.OnStart();
}
}
}
The following forum discussion really helped me to get things right: http://stackoverflow.com/questions/5727692/azure-crash-dumps-not-appearing. So I’d like to thank Oliver Bock here.
If you want to deploy it to Windows Azure then don’t forget to add your storage account name and its key in the service configuration file. For your development environment you will need to replace the whole value with “UseDevelopmentStorage=true”. The full case study with detailed steps is forthcoming.
You can download the package from here: TestWAER.zip
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Sometimes we don’t have symbols (No Component Symbols pattern) or have only a restricted set that we call No Data Types pattern. For example, in a base OS we have data types:
0:016> dt ntdll!*
ntdll!LIST_ENTRY64
ntdll!LIST_ENTRY32
ntdll!_KUSER_SHARED_DATA
ntdll!_KSYSTEM_TIME
ntdll!_KSYSTEM_TIME
ntdll!_NT_PRODUCT_TYPE
[...]
In the “private” version we don’t have them although the symbol file exists:
0:015> dt ntdll!*
0:015> !lmi ntdll
Loaded Module Info: [ntdll]
Module: ntdll
Base Address: 0000000076de0000
Image Name: ntdll.dll
Machine Type: 34404 (X64)
Time Stamp: 4dcd9861 Fri May 13 21:45:21 2011
Size: 17f000
CheckSum: 188814
Characteristics: 2022 perf
Debug Data Dirs: Type Size VA Pointer
CODEVIEW 22, f72a8, f66a8 RSDS - GUID: {05A648A7-625D-42E7-B736-7816F0CA1E0C}
Age: 2, Pdb: ntdll.pdb
CLSID 8, f72a0, f66a0 [Data not mapped]
Image Type: MEMORY - Image read successfully from loaded memory.
Symbol Type: PDB - Symbols loaded successfully from symbol server.
c:\mss\ntdll.pdb\05A648A7625D42E7B7367816F0CA1E0C2\ntdll.pdb
Load Report: public symbols , not source indexed
c:\mss\ntdll.pdb\05A648A7625D42E7B7367816F0CA1E0C2\ntdll.pdb
In such cases manually loading a proximate module might help: Coping with missing symbolic information (although I haven’t yet tested it on x64 systems). I also thought of naming the pattern as Private Modification but that would not cover many other cases where types were missing from the very beginning.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
After getting my hands on crash dumps from Windows Azure cloud platform I discerned a few patterns and the first one is just called Cloud Environment. The pattern covers both development (emulator, if it exists) and real (staging and deployment) environments. This pattern is best diagnosed by looking at specific infrastructure modules:
0:016> lm m Wa*
start end module name
00000000`00b00000 00000000`00b0c000 WaWorkerHost
00000000`74fb0000 00000000`74fbd000 WaRuntimeProxy
0:016> lm m *Azure*
start end module name
00000000`57cd0000 00000000`57d26000 Microsoft_WindowsAzure_StorageClient
00000000`58820000 00000000`5886c000 Microsoft_WindowsAzure_Diagnostics
00000000`5c750000 00000000`5c764000 Microsoft_WindowsAzure_ServiceRuntime
Development platform can be distinguished for now by looking at ntdll version:
0:016> lmv m ntdll
start end module name
00000000`76de0000 00000000`76f5f000 ntdll
Loaded symbol image file: ntdll.dll
Image path: D:\Windows\System32\ntdll.dll
Image name: ntdll.dll
Timestamp: Fri May 13 21:45:21 2011 (4DCD9861)
CheckSum: 00188814
ImageSize: 0017F000
File version: 6.0.6002.18446
Product version: 6.0.6002.18446
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: ntdll.dll
OriginalFilename: ntdll.dll
ProductVersion: 6.0.6002.18446
FileVersion: 6.0.6002.18446 (rd_os_v1.110513-1321)
FileDescription: NT Layer DLL
LegalCopyright: © Microsoft Corporation. All rights reserved.
0:016> lmv m ntdll
start end module name
00000000`775a0000 00000000`7774b000 ntdll
Loaded symbol image file: ntdll.dll
Image path: C:\Windows\System32\ntdll.dll
Image name: ntdll.dll
Timestamp: Tue Jul 14 02:32:27 2009 (4A5BE02B)
CheckSum: 001B1CB5
ImageSize: 001AB000
File version: 6.1.7600.16385
Product version: 6.1.7600.16385
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: ntdll.dll
OriginalFilename: ntdll.dll
ProductVersion: 6.1.7600.16385
FileVersion: 6.1.7600.16385 (win7_rtm.090713-1255)
FileDescription: NT Layer DLL
LegalCopyright: © Microsoft Corporation. All rights reserved.
We see that real Windows Azure is currently RD OS V1 (at least in a datacenter chosen for West European affinity during deployment).
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Today we introduce an icon for Custom Exception Handler (kernel space) pattern:
B/W
Color
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Committed to Cloud
I’ve been thinking for some time about a service that allows to ”Memory Dump It” easily. Finally my thoughts overflowed me and I memory dumped a solution (name) 
Jokes apart, I’m deadly serious and the forthcoming service will allow everyone to memory dump their devices from any location and store memory dumps securely in a cloud.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Motivated by reading from The Black Swan: The Impact of the Highly Improbable
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
Found it funny that President’s Daily Brief is abbreviated as PDB. For intelligence analysts who might be reading this post there are a few links explaining PDB files:
I also suggest to deabbreviate PDB files as Programmer’s Daily Briefs in the context of nightly builds on Windows platforms.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -
