Securitism (Part 1)

February 19th, 2008

“Do Not Tolerate Easter Eggs” - hidden gem on page 293 in the book Secure Programming with Static Analysis prompted me to define this phenomenon as Software Securitism. I was looking for its definition but couldn’t find anything except the fact that the word was sometimes used in political discourse. Hence a definition is required and it is based on an analogy with Scientism with at least two meanings:

  1. The view that Security has authority over other software engineering and support best practices usually at the expense of customers.

  2. Improper usage of security terminology regardless of the project scope and common sense as a counter-argument.

If you heard about other abuses of security perhaps due to its current fashionable usage you can contact me anonymously using this contact form:

http://www.dumpanalysis.org/contact

- Dmitry Vostokov @ DumpAnalysis.org -

Debugger Log Analyzer: Inception

February 18th, 2008

Comparing reference stack traces with the output of !process 0 ff command or just visually inspecting the long log and trying to spot anomalies is very difficult and largely based on personal experience with prior problem cases. A tool is needed and I’m currently writing the one. It will compare logs from problem memory dumps with reference stack traces and other information and automatically pinpoint any anomalies and highlight areas for more detailed manual inspection. This is similar to Kernel Memory Space Analyzer original intent but far more useful. Originally I thought about calling it WinDbg Log Analyzer but later decided to make it more general and extendable to other types of logs from different debuggers like GDB. Some people asked me the question: won’t a WinDbg debugger extension suffice? My answer was no - some companies cannot send complete, kernel and process memory dumps due to security considerations but they can send logs free from sensitive data as explained in my previous article:

Resolving security issues with crash dumps

Additionally I want it to be debugger independent at least in the second version and I want it to be web-based too and free from the choice of the hosting platform.  

Stay tuned because the working prototype will be soon as a command line tool first. I personally need it for my day-to-day job. The latter always was my primary motivation to create various tools to automate or semi-automate data gathering and improve customer problem analysis.

The next version will have front-end GUI and I still haven’t decided yet whether to employ embedded HTML control like IE, RichEdit or revive my old text processor project. I’m inclined to choose the former due to endless possibilities with HTML and its platform independence. The choice of command line tool written in C++/STL will help to port it to FreeBSD/Linux/Solaris and adapt to other debuggers like GDB/ADB. The latter is my “wild fantasy” at the moment but its good to think towards other platforms that slowly increase their presence in my professional life :-)

Any suggestions are very welcome especially if you have dealt with large debugger logs including not only backtraces but also various synchronization objects, module information, timing and I/O packet distribution.

- Dmitry Vostokov @ DumpAnalysis.org -

Yet another great WinDbg resource

February 16th, 2008

Just found this wonderful WinDbg command reference thematically grouped with examples written by Robert Kuster:

Common WinDbg Commands (Thematically Grouped)

and booklet

WinDbg. From A to Z!

I’m adding this to windbg.org

- Dmitry Vostokov @ DumpAnalysis.org -

Windows Internals University Course

February 15th, 2008

Tons of PPTs based on Windows Internals book and David Solomon Expert Seminars. All can be found here:

The Windows Operating Systems Internals Curriculum Resource Kit

Lots of other materials including Windows Core Architecture by Dave Probert can be found here too:

Academic Resource Center - Operating Systems

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 50)

February 15th, 2008

Beginner users of WinDbg sometimes confuse the first 3 parameters (or 4 for x64) displayed by kb or kv commands with real function parameters: 

0:000> kbnL
 # ChildEBP RetAddr  Args to Child
00 002df5f4 0041167b 002df97c 00000000 7efdf000 ntdll!DbgBreakPoint
01 002df6d4 004115c9 00000000 40000000 00000001 CallingConventions!A::thiscallFunction+0×2b
02 002df97c 004114f9 00000001 40001000 00000002 CallingConventions!fastcallFunction+0×69
03 002dfbf8 0041142b 00000000 40000000 00000001 CallingConventions!cdeclFunction+0×59
04 002dfe7c 004116e8 00000000 40000000 00000001 CallingConventions!stdcallFunction+0×5b
05 002dff68 00411c76 00000001 005a2820 005a28c8 CallingConventions!wmain+0×38
06 002dffb8 00411abd 002dfff0 7d4e7d2a 00000000 CallingConventions!__tmainCRTStartup+0×1a6
07 002dffc0 7d4e7d2a 00000000 00000000 7efdf000 CallingConventions!wmainCRTStartup+0xd
08 002dfff0 00000000 00411082 00000000 000000c8 kernel32!BaseProcessStart+0×28

The calling sequence for it is:

stdcallFunction(0, 0x40000000, 1, 0x40001000, 2, 0x40002000) ->
cdeclFunction(0, 0x40000000, 1, 0x40001000, 2, 0x40002000) ->
fastcallFunction(0, 0x40000000, 1, 0x40001000, 2, 0x40002000) ->
A::thiscallFunction(0, 0x40000000, 1, 0x40001000, 2, 0x40002000)

and we see that only in the case of fastcall calling convention we have discrepancy due to the fact that the first 2 parameters are passed not via stack but through ECX and EDX:

0:000> ub 004114f9
CallingConventions!cdeclFunction+0x45
004114e5 push    ecx
004114e6 mov     edx,dword ptr [ebp+14h]
004114e9 push    edx
004114ea mov     eax,dword ptr [ebp+10h]
004114ed push    eax
004114ee mov     edx,dword ptr [ebp+0Ch]
004114f1 mov     ecx,dword ptr [ebp+8]
004114f4 call    CallingConventions!ILT+475(?fastcallFunctionYIXHHHHHHZ) (004111e0)

However if we have full symbols we can see all parameters:

0:000> .frame 2
02 002df97c 004114f9 CallingConventions!fastcallFunction+0x69

0:000> dv /i /V
prv param  002df974 @ebp-0x08               a = 0
prv param  002df968 @ebp-0x14               b = 1073741824
prv param  002df984 @ebp+0x08               c = 1
prv param  002df988 @ebp+0x0c               d = 1073745920
prv param  002df98c @ebp+0x10               e = 2
prv param  002df990 @ebp+0x14               f = 1073750016
prv local  002df7c7 @ebp-0x1b5             obj = class A
prv local  002df7d0 @ebp-0x1ac           dummy = int [100]

How does dv command know about values in ECX and EDX which were definitely overwritten by later code? This is because the called function prolog saved them as local variables which you can notice as negative offsets for EBP register in dv output above:

0:000> uf CallingConventions!fastcallFunction
CallingConventions!fastcallFunction
   32 00411560 push    ebp
   32 00411561 mov     ebp,esp
   32 00411563 sub     esp,27Ch
   32 00411569 push    ebx
   32 0041156a push    esi
   32 0041156b push    edi
   32 0041156c push    ecx
   32 0041156d lea     edi,[ebp-27Ch]
   32 00411573 mov     ecx,9Fh
   32 00411578 mov     eax,0CCCCCCCCh
   32 0041157d rep stos dword ptr es:[edi]
   32 0041157f pop     ecx
   32 00411580 mov     dword ptr [ebp-14h],edx
   32 00411583 mov     dword ptr [ebp-8],ecx


I call this pattern as False Function Parameters where double checks and knowledge of calling conventions are required. Sometimes this pattern is a consequence of another pattern that I previously called Optimized Code.

x64 stack traces don’t show any discrepancies except the fact that thiscall function parameters are shifted to the right:

0:000> kbL
RetAddr           : Args to Child                                                           : Call Site
00000001`40001397 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : ntdll!DbgBreakPoint
00000001`40001233 : 00000000`0012fa94 cccccccc`00000000 cccccccc`40000000 cccccccc`00000001 : CallingConventions!A::thiscallFunction+0×37
00000001`40001177 : cccccccc`00000000 cccccccc`40000000 cccccccc`00000001 cccccccc`40001000 : CallingConventions!fastcallFunction+0×93
00000001`400010c7 : cccccccc`00000000 cccccccc`40000000 cccccccc`00000001 cccccccc`40001000 : CallingConventions!cdeclFunction+0×87
00000001`400012ae : cccccccc`00000000 cccccccc`40000000 cccccccc`00000001 cccccccc`40001000 : CallingConventions!stdcallFunction+0×87
00000001`400018ec : 00000001`00000001 00000000`00481a80 00000000`00000000 00000001`400026ee : CallingConventions!wmain+0×4e
00000001`4000173e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : CallingConventions!__tmainCRTStartup+0×19c
00000000`77d5964c : 00000000`77d59620 00000000`00000000 00000000`00000000 00000000`0012ffa8 : CallingConventions!wmainCRTStartup+0xe
00000000`00000000 : 00000001`40001730 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseProcessStart+0×29

How this can happen if the standard x64 calling convention passes the first 4 parameters via ECX, EDX, R8 and R9? This is because the called function prolog saved them on stack (this might not be true in the case of optimized code):

0:000> uf CallingConventions!fastcallFunction
CallingConventions!fastcallFunction
   32 00000001`400011a0 44894c2420      mov     dword ptr [rsp+20h],r9d
   32 00000001`400011a5 4489442418      mov     dword ptr [rsp+18h],r8d
   32 00000001`400011aa 89542410        mov     dword ptr [rsp+10h],edx
   32 00000001`400011ae 894c2408        mov     dword ptr [rsp+8],ecx
...
...
...

A::thiscallFunction function passes this pointer via ECX too and this explains the right shift of parameters.

Here is the C++ code I used for experimentation:

#include "stdafx.h"
#include <windows.h>

void __stdcall stdcallFunction (int, int, int, int, int, int);
void __cdecl cdeclFunction (int, int, int, int, int, int);
void __fastcall fastcallFunction (int, int, int, int, int, int);

class A
{
public:
 void thiscallFunction (int, int, int, int, int, int) { DebugBreak(); };
};

void __stdcall stdcallFunction (int a, int b, int c, int d, int e, int f)
{
 int dummy[100] = {0};

 cdeclFunction (a, b, c, d, e, f);
}

void __cdecl cdeclFunction (int a, int b, int c, int d, int e, int f)
{
 int dummy[100] = {0};

 fastcallFunction (a, b, c, d, e, f);
}

void __fastcall fastcallFunction (int a, int b, int c, int d, int e, int f)
{
 int dummy[100] = {0};

 A obj;

 obj.thiscallFunction (a, b, c, d, e, f);
}

int _tmain(int argc, _TCHAR* argv[])
{
 stdcallFunction (0, 0x40000000, 1, 0x40001000, 2, 0x40002000);

 return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org -

Pocket Reference Stack Traces for Vista x86

February 14th, 2008

Previously announced volume is available in trade paperback and hardcover versions at the nominal price to cover manufacturing costs:

Small print (paperback)

Buy

Small print (hardcover)

Buy

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 49)

February 13th, 2008

Frame Pointer Omission pattern is the most visible compiler optimization technique and you can notice it in verbose stack traces:

0:000> kv
ChildEBP RetAddr
0012ee10 004737a7 application!MemCopy+0x17 (FPO: [3,0,2])
0012ef0c 35878c5b application!ProcessData+0×97 (FPO: [Uses EBP] [3,59,4])
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012ef1c 72a0015b 0×35878c5b
0012ef20 a625e1b0 0×72a0015b
0012ef24 d938bcfe 0xa625e1b0
0012ef28 d4f91bb4 0xd938bcfe
0012ef2c c1c035ce 0xd4f91bb4


To recall FPO is a compiler optimization where ESP register is used to address local variables and parameters instead of EBP. EBP may or may not be used for other purposes. When it is used you will notice

FPO: [Uses EBP]

as in the trace above. For description of other FPO number triplets please see Debugging Tools for Windows help section “k, kb, kd, kp, kP, kv (Display Stack Backtrace)”

Running analysis command points to possible stack corruption:

PRIMARY_PROBLEM_CLASS:  STACK_CORRUPTION

BUGCHECK_STR:  APPLICATION_FAULT_STACK_CORRUPTION

FAULTING_IP:
application!MemCopy+17
00438637 f3a5 rep movs dword ptr es:[edi],dword ptr [esi]

Looking at EBP and ESP shows that they are mismatched:

0:000> r
eax=00000100 ebx=00a027f3 ecx=00000040 edx=0012ee58 esi=d938bcfe edi=0012ee58
eip=00438637 esp=0012ee0c ebp=00a02910 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
application!MemCopy+0×17:
00438637 f3a5 rep movs dword ptr es:[edi],dword ptr [esi] es:0023:0012ee58=00000010 ds:0023:d938bcfe=????????

We might think about Local Buffer Overflow pattern here but two top stack trace lines are in accordance with each other:

0:000> ub 004737a7
application!ProcessData+0x80:
00473790 cmp     eax,edi
00473792 jb      application!ProcessData+0x72 (00473782)
00473794 mov     ecx,dword ptr [esp+104h]
0047379b push    esi
0047379c lea     edx,[esp+38h]
004737a0 push    ecx
004737a1 push    edx
004737a2 call    application!MemCopy (00438620)

So perhaps EBP value differs greatly from ESP due to its usage as general purpose register and in fact there was no any stack corruption. Despite using public symbols we have the instance of Incorrect Stack Trace pattern and we might want to reconstruct it manually. Let’s search for EBP value on raw stack below the crash point:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0012ffb0
    StackBase:            00130000
    StackLimit:           00126000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             0000660c . 00005890
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffd9000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds  00126000 00130000
00126000  00000000
00126004  00000000
00126008  00000000
...
...
...
0012eb0c  00a02910
0012eb10  7c90e25e ntdll!NtRaiseException+0xc
0012eb14  7c90eb15 ntdll!KiUserExceptionDispatcher+0x29
0012eb18  0012eb24
0012eb1c  0012eb40
0012eb20  00000000
0012eb24  c0000005
0012eb28  00000000
0012eb2c  00000000
0012eb30  00438637 application!MemCopy+0x17
0012eb34  00000002
0012eb38  00000000
0012eb3c  d938bcfe
...
...
...
0012ebf4  00a02910
0012ebf8  00438637 application!MemCopy+0×17
0012ebfc  0000001b



0012f134  00436323 application!ConstructInfo+0×113
0012f138  00a02910
0012f13c  0000011c


Let’s see what functions ConstructInfo calls:

0:000> ub 00436323
application!ConstructInfo+0x103
00436313 lea     edx,[esp+10h]
00436317 push    edx
00436318 push    eax
00436319 push    30h
0043631b push    ecx
0043631c push    ebx
0043631d push    ebp
0043631e call    application!EnvelopeData (00438bf0)

We notice EBP was pushed prior to calling EnvelopeData. If we disassemble this function we would see that it calls ProcessData from our partial stack trace: 

0:000> uf 00438bf0
application!EnvelopeData:
00438bf0 sub     esp,1F4h
00438bf6 push    ebx
00438bf7 mov     ebx,dword ptr [esp+20Ch]
00438bfe test    ebx,ebx
00438c00 push    ebp
...
...
...
00438c76 rep stos byte ptr es:[edi]
00438c78 lea     eax,[esp+14h]
00438c7c push    eax
00438c7d push    ebp
00438c7e call    application!ProcessData (00473710)
00438c83 pop     edi
00438c84 pop     esi

Let’s try elaborate form of k command and supply it with custom ESP and EBP values pointing to

0012f134  00436323 application!ConstructInfo+0×113

and also EIP of the fault:

0:000> k L=0012f134 0012f134 00438637
ChildEBP RetAddr 
0012f1cc 00435a65 application!MemCopy+0×17
0012f28c 0043532e application!ClientHandleServerRequest+0×395
0012f344 00434fcd application!Accept+0×23
0012f374 0042e4f3 application!DataArrival+0×17d
0012f43c 0041aea9 application!ProcessInput+0×98
0012ff0c 0045b278 application!AppMain+0xda
0012ff24 0041900e application!WinMain+0×78
0012ffc0 7c816fd7 application!WinMainCRTStartup+0×134
0012fff0 00000000 kernel32!BaseProcessStart+0×23

We see that although it misses some initial frames after MemCopy we aided WinDbg to walk to the bottom of the stack and reconstruct the plausible stack trace for us.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 48)

February 12th, 2008

Special Stack Trace pattern is about stack traces not present in normal crash dumps. The similar pattern is called Special Process which is about processes not running during normal operation or highly domain specific processes that are the sign of certain software environment, for example, OS running inside VMWare or VirtualPC. Here I’ll show one example when identifying specific process led to successful problem identification inside a complete memory dump.

Inspection of running processes shows the presence of Dr. Watson:

5: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 89d9b648  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 54d5d020  ObjectTable: e1000e20  HandleCount: 1711.
    Image: System

PROCESS 8979b758  SessionId: none  Cid: 01b0    Peb: 7ffdd000  ParentCid: 0004
    DirBase: 54d5d040  ObjectTable: e181d8b0  HandleCount:  29.
    Image: smss.exe

PROCESS 89793cf0  SessionId: 0  Cid: 01e0    Peb: 7ffde000  ParentCid: 01b0
    DirBase: 54d5d060  ObjectTable: e13eea10  HandleCount: 1090.
    Image: csrss.exe

...
...
...

PROCESS 8797a600  SessionId: 1  Cid: 17d0    Peb: 7ffdc000  ParentCid: 1720
    DirBase: 54d5d8c0  ObjectTable: e2870af8  HandleCount: 243.
    Image: explorer.exe

PROCESS 87966d88  SessionId: 2  Cid: 0df0    Peb: 7ffd4000  ParentCid: 01b0
    DirBase: 54d5d860  ObjectTable: e284cd48  HandleCount:  53.
    Image: csrss.exe

PROCESS 879767c8  SessionId: 0  Cid: 0578    Peb: 7ffde000  ParentCid: 0ca8
    DirBase: 54d5d8a0  ObjectTable: e2c05268  HandleCount: 180.
    Image: drwtsn32.exe

Inspecting stack traces shows that drwtsn32.exe is waiting for a debugger event so there must be some debugging target (debuggee):

5: kd> .process /r /p 879767c8
Implicit process is now 879767c8
Loading User Symbols

5: kd> !process 879767c8
PROCESS 879767c8  SessionId: 0  Cid: 0578    Peb: 7ffde000  ParentCid: 0ca8
    DirBase: 54d5d8a0  ObjectTable: e2c05268  HandleCount: 180.
    Image: drwtsn32.exe
    VadRoot 88a33cd0 Vads 59 Clone 0 Private 1737. Modified 10792. Locked 0.
    DeviceMap e10028e8
    Token                             e2ee2330
    ElapsedTime                       00:01:12.703
    UserTime                          00:00:00.203
    KernelTime                        00:00:00.031
    QuotaPoolUsage[PagedPool]         52092
    QuotaPoolUsage[NonPagedPool]      2360
    Working Set Sizes (now,min,max)  (2488, 50, 345) (9952KB, 200KB, 1380KB)
    PeakWorkingSetSize                2534
    VirtualSize                       34 Mb
    PeakVirtualSize                   38 Mb
    PageFaultCount                    13685
    MemoryPriority                    BACKGROUND
    BasePriority                      6
    CommitCharge                      1927

THREAD 87976250  Cid 0578.04bc  Teb: 7ffdd000 Win32Thread: bc14a008 WAIT: (Unknown) UserMode Non-Alertable
    87976558  Thread
Not impersonating
DeviceMap                 e10028e8
Owning Process            879767c8       Image:         drwtsn32.exe
Wait Start TickCount      13460          Ticks: 4651 (0:00:01:12.671)
Context Switch Count      15                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address drwtsn32!mainCRTStartup (0x01007c1d)
Start Address kernel32!BaseProcessStartThunk (0x7c8217f8)
Stack Init f433b000 Current f433ac60 Base f433b000 Limit f4337000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr 
f433ac78 80833465 nt!KiSwapContext+0x26
f433aca4 80829a62 nt!KiSwapThread+0x2e5
f433acec 80938d0c nt!KeWaitForSingleObject+0x346
f433ad50 8088978c nt!NtWaitForSingleObject+0x9a
f433ad50 7c9485ec nt!KiFastCallEntry+0xfc
0007fe98 7c821c8d ntdll!KiFastSystemCallRet
0007feac 01005557 kernel32!WaitForSingleObject+0x12
0007ff0c 01003ff8 drwtsn32!NotifyWinMain+0x1ba
0007ff44 01007d4c drwtsn32!main+0x31
0007ffc0 7c82f23b drwtsn32!mainCRTStartup+0x12f
0007fff0 00000000 kernel32!BaseProcessStart+0x23

THREAD 87976558  Cid 0578.0454  Teb: 7ffdc000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    89de2e50  NotificationEvent
    879765d0  NotificationTimer
Not impersonating
DeviceMap                 e10028e8
Owning Process            879767c8       Image:         drwtsn32.exe
Wait Start TickCount      18102          Ticks: 9 (0:00:00:00.140)
Context Switch Count      1163            
UserTime                  00:00:00.203
KernelTime                00:00:00.031
Win32 Start Address drwtsn32!DispatchDebugEventThread (0x01003d6d)
Start Address kernel32!BaseThreadStartThunk (0x7c8217ec)
Stack Init f4e26000 Current f4e25be8 Base f4e26000 Limit f4e23000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr 
f4e25c00 80833465 nt!KiSwapContext+0x26
f4e25c2c 80829a62 nt!KiSwapThread+0x2e5
f4e25c74 809a06ab nt!KeWaitForSingleObject+0x346
f4e25d4c 8088978c nt!NtWaitForDebugEvent+0xd5
f4e25d4c 7c9485ec nt!KiFastCallEntry+0xfc
0095ed20 60846f8f ntdll!KiFastSystemCallRet
0095ee6c 60816ecf dbgeng!LiveUserTargetInfo::WaitForEvent+0x1fa
0095ee88 608170d3 dbgeng!WaitForAnyTarget+0x45
0095eecc 60817270 dbgeng!RawWaitForEvent+0x15f
0095eee4 01003f8d dbgeng!DebugClient::WaitForEvent+0×80
0095ffb8 7c824829 drwtsn32!DispatchDebugEventThread+0×220
0095ffec 00000000 kernel32!BaseThreadStart+0×34

Knowing that a debugger suspends threads in a debuggee (Suspended Thread pattern) we see the problem process indeed:

5: kd> !process 0 2
**** NT ACTIVE PROCESS DUMP ****

...
...
...

PROCESS 898285b0  SessionId: 0  Cid: 0ca8    Peb: 7ffda000  ParentCid: 022c
    DirBase: 54d5d500  ObjectTable: e2776880  HandleCount:   2.
    Image: svchost.exe

THREAD 888b8668  Cid 0ca8.1448  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
SuspendCount 2
  888b87f8  Semaphore Limit 0×2

Dumping its thread stacks shows only one system thread where we normally expect plenty of them originated from user space. There is also the presence of a debug port:

5: kd> .process /r /p 898285b0
Implicit process is now 898285b0
Loading User Symbols

5: kd> !process 898285b0
PROCESS 898285b0  SessionId: 0  Cid: 0ca8    Peb: 7ffda000  ParentCid: 022c
    DirBase: 54d5d500  ObjectTable: e2776880  HandleCount:   2.
    Image: svchost.exe
    VadRoot 88953220 Vads 209 Clone 0 Private 901. Modified 3. Locked 0.
    DeviceMap e10028e8
    Token                             e27395b8
    ElapsedTime                       00:03:25.640
    UserTime                          00:00:00.156
    KernelTime                        00:00:00.234
    QuotaPoolUsage[PagedPool]         82988
    QuotaPoolUsage[NonPagedPool]      8824
    Working Set Sizes (now,min,max)  (2745, 50, 345) (10980KB, 200KB, 1380KB)
    PeakWorkingSetSize                2819
    VirtualSize                       82 Mb
    PeakVirtualSize                   83 Mb
    PageFaultCount                    4519
    MemoryPriority                    BACKGROUND
    BasePriority                      6
    CommitCharge                      1380
    DebugPort                         89de2e50

THREAD 888b8668  Cid 0ca8.1448  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
SuspendCount 2
    888b87f8  Semaphore Limit 0x2
Not impersonating
DeviceMap                 e10028e8
Owning Process            898285b0       Image:         svchost.exe
Wait Start TickCount      13456          Ticks: 4655 (0:00:01:12.734)
Context Switch Count      408            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address driverA!DriverThread (0xf6fb8218)
Stack Init f455b000 Current f455a3ac Base f455b000 Limit f4558000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr 
f455a3c4 80833465 nt!KiSwapContext+0x26
f455a3f0 80829a62 nt!KiSwapThread+0x2e5
f455a438 80833178 nt!KeWaitForSingleObject+0x346
f455a450 8082e01f nt!KiSuspendThread+0x18
f455a498 80833480 nt!KiDeliverApc+0x117
f455a4d0 80829a62 nt!KiSwapThread+0x300
f455a518 f6fb7f13 nt!KeWaitForSingleObject+0x346
f455a548 f4edd457 driverA!WaitForSingleObject+0x75
f455a55c f4edcdd8 driverB!DeviceWaitForRead+0x19
f455ad90 f6fb8265 driverB!InputThread+0x17e
f455adac 80949b7c driverA!DriverThread+0x4d
f455addc 8088e062 nt!PspSystemThreadStartup+0x2e
00000000 00000000 nt!KiThreadStartup+0x16

The most likely scenario was that svchost.exe experienced an unhandled exception that triggered the launch of a postmortem debugger such as Dr. Watson.

Other similar examples this pattern might include the presence of WerFault.exe on Vista, NTSD and other JIT debuggers running.

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg.Org: WinDbg Quick Links

February 11th, 2008

Sometimes I need a quick link to install Debugging Tools for Windows and for other related information such as standard symbol server path, common commands, etc. For this purpose I’ve setup windbg.org domain and hope it will be handy. Currently its main page has the following links:

  • Download links to 32-bit and 64-bit versions
  • My favourite standard symbol path
  • Link to HTML version of Crash Dump Analysis Poster
  • Link to Crash Dump Analysis Checklist

Help menu on dumpanalysis.org used to point to CDA Poster now points to this page too.

I’ll add more useful information there soon. Any suggestions are welcome!

- Dmitry Vostokov @ DumpAnalysis.org -

Myths and Facts about Software Support (Part 1)

February 11th, 2008

I started these new series to debunk widespread myths about software technical support. The first one is: 

Technical support engineers can’t and don’t write code (myth). Technical support engineers do write code and sometimes fairly advanced one (fact).

There is a prevalent view of a technical support engineer spending all the time on the phone as a shield from introvert software engineers who hate customers. This is not true. There are usually several layers of support from very basic ones requiring only customer communication and foreign language skills to very advanced problem identification and troubleshooting skills that requires a thousand page knowledge from Windows Internals book. My point here is that advanced troubleshooting requires tools that sometimes don’t exist and this prompts support engineers to develop their own. Sometimes it is easy to query information from the customer environment and/or fix the problem on the spot by writing a tool or a script. And this is pure unconstrained development limited only by individual imagination, skills and complexity of the task.

The weak form of this myth is the view of a support engineer using only Visual Basic or its scripting variant.

What do you think about this? The idea of these series came from the following book that I’m reading at the moment:

Facts and Fallacies of Software Engineering (Agile Software Development)

Buy from Amazon

- Dmitry Vostokov @ DumpAnalysis.org -

Dump2Wave v1.2.1 source code

February 8th, 2008

Since the first release of Dump2Wave I was under pressure to publish its source code and today I released it under GPL. I have to apologize that it doesn’t always use secure string manipulation functions, error handling is copy/pasted several times and there are no comments. I promise better code in the next version. :-)

If you plan to make changes and improvements please let me know so I could enjoy your versions of memory sounds too. I used ancient Visual C++ 6.0 to compile and build the project.

// Dump2Wave version 1.2.1 (c) Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt

#include <iostream>
#include <process.h>
#include <windows.h>

#pragma pack(1)

typedef struct {

 unsigned int   riff;   // 'RIFF'
 unsigned int   length; // dump size + sizeof(WAVEFILEHDR) - 8
 unsigned int   wave;   // 'WAVE' 
 unsigned int   fmt;    // 'fmt '
 unsigned int   fmtlen; // 16  
 unsigned short code;   // 1
 unsigned short channels;  // 2
 unsigned int   sps;    // 44100
 unsigned int   avgbps; // sps*channels*(sbits/8)
 unsigned short alignment; // 4
 unsigned short sbits;  // 16
 unsigned int   data;   // 'data'
 unsigned int   datalen;  

} WAVEFILEHDR;

#pragma pack()

WAVEFILEHDR hdr = {'FFIR', sizeof(WAVEFILEHDR) - 8, 'EVAW',
  ' tmf', 16, 1, 2, 44100, 44100*4, 4, 16, 'atad', 0};

void DisplayError (LPCSTR szPrefix)
{
 LPSTR errMsg;
 CHAR  szMsg[256];
 strncpy(szMsg, szPrefix, 128);
 DWORD gle = GetLastError();
 if (gle && FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER|
  FORMAT_MESSAGE_FROM_SYSTEM, NULL, gle, 0,
  (LPSTR)&errMsg, 0, NULL))
 {
  strcat(szMsg, ": ");
  strncat(szMsg, errMsg, 120);
 }  
 std::cout << szMsg << std::endl; 
 LocalFree(errMsg);
}

int main(int argc, char* argv[])
{
 std::cout << std::endl << "Dump2Wave version 1.2.1" <<
    std::endl << "Written by Dmitry Vostokov, 2006" <<
    std::endl << std::endl;
 if (argc < 3)
 {
  std::cout << "Usage: Dump2Wave dumpfile wavefile [44100|22050|11025|8000 16|8 2|1]" << std::endl;
  return -1;
 }

 HANDLE hFile = CreateFile(argv[1],
  GENERIC_READ, FILE_SHARE_READ, NULL,
  OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
 if (hFile == INVALID_HANDLE_VALUE)
 {
  DisplayError("Cannot read dump file"); 
  return -1;
 }

 DWORD dwDumpSizeHigh = 0;
 DWORD dwDumpSizeLow = GetFileSize(hFile, &dwDumpSizeHigh);
 CloseHandle(hFile);

 if (dwDumpSizeHigh)
 {
  std::cout << "The dump file must be less than 4Gb" <<
  std::endl;
  return -1;
 }

 if (argc == 6)
 {
  hdr.channels = atoi(argv[5]);
  hdr.sps = atoi(argv[3]);
  hdr.sbits = atoi(argv[4]);
  hdr.avgbps = hdr.sps*hdr.channels*(hdr.sbits/8);
  hdr.alignment = hdr.channels*(hdr.sbits/8);
 }

 dwDumpSizeLow = (dwDumpSizeLow/hdr.alignment)*(hdr.alignment);

 hdr.length += dwDumpSizeLow;
 hdr.datalen = dwDumpSizeLow;

 hFile = CreateFile(argv[2], GENERIC_WRITE, 0, 
  NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
 if (hFile == INVALID_HANDLE_VALUE)
 {
  DisplayError("Cannot create wave header file"); 
  return -1;
 }

 DWORD dwWritten;
 if (!WriteFile(hFile, &hdr, sizeof(hdr), &dwWritten, NULL))
 {
  DisplayError("Cannot write wave header file"); 
  CloseHandle(hFile);
  return -1;
 }

 
 CloseHandle(hFile);

 std::string str = "copy \"";
 str += argv[2];
 str += "\" /B + \"";
 str += argv[1];
 str += "\" /B \"";
 str += argv[2];
 str += "\" /B";

 system(str.c_str());
 
 return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg

February 8th, 2008

Google shows different cheat sheets for WinDbg and I want to remind about my own version that is geared towards postmortem dump analysis of native code:

Memory Dump Analysis Checklist

This post was motivated by WinDbg blog post written by Volker von Einem :-)

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump Analysis Anthology, Volume 1

February 7th, 2008

It is very easy to become a publisher nowadays. Much easier than I thought. I registered myself as a publisher under the name of OpenTask which is my registered business name in Ireland. I also got the list of ISBN numbers and therefore can announce product details for the first volume of Memory Dump Analysis Anthology series:

Memory Dump Analysis Anthology, Volume 1

  • Paperback: 720 pages (*)
  • ISBN-13: 978-0-9558328-0-2
  • Hardcover: 720 pages (*)
  • ISBN-13: 978-0-9558328-1-9
  • Author: Dmitry Vostokov
  • Publisher: Opentask (15 Apr 2008)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

(*) subject to change 

PDF file will be available for download too.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 47)

February 6th, 2008

Most of the time threads are not suspended explicitly. If you look at active and waiting threads in kernel and complete memory dumps their SuspendCount member is 0:

THREAD 88951bc8  Cid 03a4.0d24  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
    889d6a78  Semaphore Limit 0x7fffffff
    88951c40  NotificationTimer
Not impersonating
DeviceMap                 e1b80b98
Owning Process            888a9d88       Image:         svchost.exe
Wait Start TickCount      12669          Ticks: 5442 (0:00:01:25.031)
Context Switch Count      3            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c4b0f5)
Start Address kernel32!BaseThreadStartThunk (0x7c8217ec)
Stack Init f482f000 Current f482ec0c Base f482f000 Limit f482c000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr 
f482ec24 80833465 nt!KiSwapContext+0x26
f482ec50 80829a62 nt!KiSwapThread+0x2e5
f482ec98 809226bd nt!KeWaitForSingleObject+0x346
f482ed48 8088978c nt!NtReplyWaitReceivePortEx+0x521
f482ed48 7c9485ec nt!KiFastCallEntry+0xfc (TrapFrame @ f482ed64)
00efff84 77c58792 ntdll!KiFastSystemCallRet
00efff8c 77c5872d RPCRT4!RecvLotsaCallsWrapper+0xd
00efffac 77c4b110 RPCRT4!BaseCachedThreadRoutine+0x9d
00efffb8 7c824829 RPCRT4!ThreadStartRoutine+0x1b
00efffec 00000000 kernel32!BaseThreadStart+0x34

5: kd> dt _KTHREAD 88951bc8
ntdll!_KTHREAD
   +0x000 Header           : _DISPATCHER_HEADER
   +0x010 MutantListHead   : _LIST_ENTRY [ 0x88951bd8 - 0x88951bd8 ]
   +0x018 InitialStack     : 0xf482f000
   +0x01c StackLimit       : 0xf482c000
   +0x020 KernelStack      : 0xf482ec0c
   +0x024 ThreadLock       : 0
   +0x028 ApcState         : _KAPC_STATE
...
...
...
   +0x14f FreezeCount      : 0 ''
   +0×150 SuspendCount     : 0 ”


You won’t find SuspendCount in reference stack traces. Only when some other thread explicitly suspends another thread the latter has non-zero suspend count. Suspended threads are excluded from thread scheduling and therefore can be considered as blocked. This might be the sign of a debugger present, for example, all threads in a process are suspended when a user debugger is processing a debugger event like a breakpoint or access violation exception. In this case !process 0 ff command output shows SuspendCount value:

THREAD 888b8668  Cid 0ca8.1448  Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
SuspendCount 2
    888b87f8  Semaphore Limit 0×2
Not impersonating
DeviceMap                 e10028e8
Owning Process            898285b0       Image:         processA.exe
Wait Start TickCount      13456          Ticks: 4655 (0:00:01:12.734)
Context Switch Count      408            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address driver!DriverThread (0xf6fb8218)
Stack Init f455b000 Current f455a3ac Base f455b000 Limit f4558000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr 
f455a3c4 80833465 nt!KiSwapContext+0×26
f455a3f0 80829a62 nt!KiSwapThread+0×2e5
f455a438 80833178 nt!KeWaitForSingleObject+0×346
f455a450 8082e01f nt!KiSuspendThread+0×18
f455a498 80833480 nt!KiDeliverApc+0×117
f455a4d0 80829a62 nt!KiSwapThread+0×300
f455a518 f6fb7f13 nt!KeWaitForSingleObject+0×346
f455a548 f4edd457 driver!WaitForSingleObject+0×75
f455a55c f4edcdd8 driver!DeviceWaitForRead+0×19
f455ad90 f6fb8265 driver!InputThread+0×17e
f455adac 80949b7c driver!DriverThread+0×4d
f455addc 8088e062 nt!PspSystemThreadStartup+0×2e
00000000 00000000 nt!KiThreadStartup+0×16

5: kd> dt _KTHREAD 888b8668
ntdll!_KTHREAD
   +0x000 Header           : _DISPATCHER_HEADER
   +0x010 MutantListHead   : _LIST_ENTRY [ 0x888b8678 - 0x888b8678 ]
   +0x018 InitialStack     : 0xf455b000
   +0x01c StackLimit       : 0xf4558000
   +0x020 KernelStack      : 0xf455a3ac
   +0x024 ThreadLock       : 0
...
...
...
   +0x14f FreezeCount      : 0 ''
   +0×150 SuspendCount     : 2 ”


I call this pattern Suspended Thread. It should rise suspicion bar and in some cases coupled with Special Process pattern can lead to immediate problem identification.

- Dmitry Vostokov @ DumpAnalysis.org -

Dump2Picture v1.1 source code

February 5th, 2008

Since the first release of Dump2Picture I was under pressure to publish its source code and today I released it under GPL. I have to apologize that it doesn’t always use secure string manipulation functions, error handling is copy/pasted several times and there are no comments. I promise better code in the next version. :-)

If you plan to make changes and improvements please let me know so I could enjoy your versions of memory visuals too. I used ancient Visual C++ 6.0 to compile and build the project.

// Dump2Picture version 1.1 (c) Dmitry Vostokov
// GNU GENERAL PUBLIC LICENSE
// http://www.gnu.org/licenses/gpl-3.0.txt

#include <math.h>
#include <iostream>
#include <process.h>
#include <windows.h>

BITMAPFILEHEADER bmfh = { 'MB', 0, 0, 0,
   sizeof(BITMAPFILEHEADER) + sizeof(BITMAPINFOHEADER) };
BITMAPINFOHEADER bmih = { sizeof(BITMAPINFOHEADER), 0, 0, 1, 32,
   0, 0, 0, 0, 0, 0 };
RGBQUAD rgb[256];

void DisplayError (LPCSTR szPrefix)
{
 LPSTR errMsg;
 CHAR  szMsg[256];
 strncpy(szMsg, szPrefix, 128);
 DWORD gle = GetLastError();
 if (gle && FormatMessage(
    FORMAT_MESSAGE_ALLOCATE_BUFFER|FORMAT_MESSAGE_FROM_SYSTEM,
    NULL, gle, 0, (LPSTR)&errMsg, 0, NULL))
 {
  strcat(szMsg, ": ");
  strncat(szMsg, errMsg, 120);
 }  
 std::cout << szMsg << std::endl;
 LocalFree(errMsg);
}

int main(int argc, char* argv[])
{
 std::cout << std::endl << "Dump2Picture version 1.1"
    << std::endl << "Written by Dmitry Vostokov, 2007"
    << std::endl << std::endl;
 if (argc < 3)
 {
  std::cout << "Usage: Dump2Picture dumpfile bmpfile [8|16|24|32]" << std::endl;
  return -1;
 }

 HANDLE hFile = CreateFile(argv[1], GENERIC_READ,
    FILE_SHARE_READ, NULL, OPEN_EXISTING,
    FILE_ATTRIBUTE_NORMAL, NULL);
 if (hFile == INVALID_HANDLE_VALUE)
 {
  DisplayError("Cannot read dump file"); 
  return -1;
 }

 DWORD dwDumpSizeHigh = 0;
 DWORD dwDumpSizeLow = GetFileSize(hFile, &dwDumpSizeHigh);
 CloseHandle(hFile);

 if (dwDumpSizeHigh)
 {
  std::cout << "The dump file must be less than 4Gb"
     << std::endl;
  return -1;
 }

 if (argc == 4)
 {
  if (!strcmp(argv[argc-1],"8"))
  {
   bmih.biBitCount = 8;
   for (int i = 0; i < 256; ++i)
   {
    rgb[i].rgbBlue = rgb[i].rgbGreen = rgb[i].rgbRed = i;
    rgb[i].rgbReserved = 0;
   }
  }
  else if (!strcmp(argv[argc-1],"16"))
  {
   bmih.biBitCount = 16;
  }
  else if (!strcmp(argv[argc-1],"24"))
  {
   bmih.biBitCount = 24;
  }
  else
  {
   bmih.biBitCount = 32;
  }
 }

 bmih.biWidth = bmih.biHeight = sqrt((double)(dwDumpSizeLow/
    (bmih.biBitCount/8)));
 bmih.biWidth -= bmih.biWidth%2;
 if (bmih.biBitCount == 8 )
 {
  bmih.biWidth -= bmih.biWidth%8;
 }
 bmih.biHeight -= bmih.biHeight%2;
 bmih.biSizeImage = bmih.biWidth*bmih.biHeight*
    (bmih.biBitCount/8);
 if (bmih.biBitCount == 8 )
 {
  bmfh.bfOffBits += sizeof(rgb);
 }
 bmfh.bfSize = bmfh.bfOffBits + bmih.biSizeImage;

 hFile = CreateFile(argv[2], GENERIC_WRITE, 0, NULL,
    CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
 if (hFile == INVALID_HANDLE_VALUE)
 {
  DisplayError("Cannot create bitmap header file"); 
  return -1;
 }

 DWORD dwWritten;
 if (!WriteFile(hFile, &bmfh, sizeof(bmfh), &dwWritten, NULL))
 {
  DisplayError("Cannot write bitmap header file"); 
  CloseHandle(hFile);
  return -1;
 }

 if (!WriteFile(hFile, &bmih, sizeof(bmih), &dwWritten, NULL))
 {
  DisplayError("Cannot write bitmap header file"); 
  CloseHandle(hFile);
  return -1;
 }

 if (bmih.biBitCount == 8 )
 {
  if (!WriteFile(hFile, &rgb, sizeof(rgb), &dwWritten, NULL))
  {
   DisplayError("Cannot write bitmap header file"); 
   CloseHandle(hFile);
   return -1;
  }
 }
 
 CloseHandle(hFile);

 std::string str = "copy \"";
 str += argv[2];
 str += "\" /B + \"";
 str += argv[1];
 str += "\" /B \"";
 str += argv[2];
 str += "\" /B";

 system(str.c_str());
 
 return 0;
}

- Dmitry Vostokov @ DumpAnalysis.org -

Memoretics

February 4th, 2008

I’ve been trying to put memory dump analysis on relevant scientific grounds for some time and now this branch of science needs its own name. After considering different alternative names I finally chose the word Memoretics. Here is the brief definition:

Computer Memoretics studies computer memory snapshots and their evolution in time.

Obviously this domain of research has many links with application and system debugging. However its scope is wider than debugging because it doesn’t necessarily study memory snapshots from systems and applications experiencing faulty behaviour.

Initially I was thinking about Memogenics word but its suffix is heavily associated with genes metaphor which I’m currently trying to avoid although I personally re-discovered software genes approach to software disorders when thinking about Memoretics vs. Memogenics. Later I found some research efforts going on but seems they are based on constructing software genes artificially. On the contrary I would try to discover genes in computer memories first.

genic

Also Memoretics has longer prefix almost resembling Memory word. This had the final influence on my decision.

PS. I was also thinking about Memorology word but it has negative connotations with Astrology or Numerology and was coined already by someone like Memology and Memorics

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 13d)

February 4th, 2008

In order to maintain virtual to physical address translation OS needs page tables. These tables occupy memory too. If there is not enough memory for new tables the system will fail to create processes, allocate I/O buffers and memory from pools. You might see the following diagnostic message from WinDbg:

4: kd> !vm

*** Virtual Memory Usage ***
 Physical Memory:      851422 (   3405688 Kb)
 Page File: \??\C:\pagefile.sys
   Current:   2095104 Kb  Free Space:   2081452 Kb
   Minimum:   2095104 Kb  Maximum:      4190208 Kb
 Available Pages:      683464 (   2733856 Kb)
 ResAvail Pages:       800927 (   3203708 Kb)
 Locked IO Pages:         145 (       580 Kb)
 Free System PTEs:      23980 (     95920 Kb)

 ******* 356363 system PTE allocations have failed ******

 Free NP PTEs:           6238 (     24952 Kb)
 Free Special NP:           0 (         0 Kb)
 Modified Pages:          482 (      1928 Kb)
 Modified PF Pages:       482 (      1928 Kb)
 NonPagedPool Usage:    18509 (     74036 Kb)
 NonPagedPool Max:      31970 (    127880 Kb)
 PagedPool 0 Usage:      8091 (     32364 Kb)
 PagedPool 1 Usage:      2495 (      9980 Kb)
 PagedPool 2 Usage:      2580 (     10320 Kb)
 PagedPool 3 Usage:      2552 (     10208 Kb)
 PagedPool 4 Usage:      2584 (     10336 Kb)
 PagedPool Usage:       18302 (     73208 Kb)
 PagedPool Maximum:     39936 (    159744 Kb)

 ********** 48530 pool allocations have failed **********

 Shared Commit:          5422 (     21688 Kb)
 Special Pool:              0 (         0 Kb)
 Shared Process:         5762 (     23048 Kb)
 PagedPool Commit:      18365 (     73460 Kb)
 Driver Commit:          2347 (      9388 Kb)
 Committed pages:      129014 (    516056 Kb)
 Commit limit:        1342979 (   5371916 Kb)

We also see another diagnostic message about pool allocation failures which could be the consequence of PTE allocation failures.

The cause of system PTE allocation failures might be incorrect value of SystemPages registry key that needs to be adjusted as explained in the following TechNet article:

The number of free page table entries is low, which can cause system instability

Another cause would be /3GB boot option on x86 systems especially used for hosting terminal sessions. This case is explained in Brad Rutkowski’s blog post which also shows how to detect /3GB kernel and complete memory dumps:

Consequences of running 3GB and PAE together  

In our case the system was booted with /3GB:

4: kd> vertarget
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer
Built by: 3790.srv03_sp2_gdr.070304-2240
Kernel base = 0xe0800000 PsLoadedModuleList = 0xe08af9c8
Debug session time: Fri Feb  1 09:10:17.703 2008 (GMT+0)
System Uptime: 6 days 17:14:45.528

Normal Windows 2003 systems have different kernel base address which can be checked from Reference Stack Traces for Windows Server 2003 (Virtual Memory section): 

kd> vertarget
Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 3790.srv03_sp2_rtm.070216-1710
Kernel base = 0×80800000 PsLoadedModuleList = 0×8089ffa8
Debug session time: Wed Jan 30 17:54:13.390 2008 (GMT+0)
System Uptime: 0 days 0:30:12.000

- Dmitry Vostokov @ DumpAnalysis.org -

Optometrics and Crashes

February 4th, 2008

What’s the relation? During my eye test today an optometrist complained that he has to re-enter data because his program crashed. Later on I looked at the screen and saw familiar Borland style GUI. I resisted temptation to offer on the spot crash dump analysis assistance. Now I regret that - perhaps he might have offered better discount for me :-)

- Dmitry Vostokov @ DumpAnalysis.org -

2007 in Retrospection (Part 3)

February 1st, 2008

Out of more than 13,000 organizations including more than 450 universities and colleges I selected top 10 visited my blog. Here is the graph showing the number of visits vs. company name:

- Dmitry Vostokov @ DumpAnalysis.org -

LiterateScientist update (January, 2008)

February 1st, 2008

As promised here is the first monthly summary of my Literate Scientist blog:

- Dmitry Vostokov @ DumpAnalysis.org -