Archive for June, 2008

Music for Debugging: Biography of a Bug

Monday, June 16th, 2008

Klaus Schulze was one of my favorite electronic composers in 90s. Now I’m revisiting my past collection and recommend ”X” album featuring several musical biographies including Nietzsche. “X” here stands for 10th album not for “x” WinDbg command. Good for meditation during slow debugging sessions. All music is very sad and mystical. I even think about this title: “Musical Biography of a Bug“:

X

Buy from Amazon

- Dmitry Vostokov @ DumpAnalysis.org -

AERA (Automated Error Reporting and Analysis)

Monday, June 16th, 2008

Due to a request I opened AERA subforum:

http://www.dumpanalysis.org/forum/viewforum.php?f=29

Everyone is welcome to comment and add thoughts.

IBCDAS code name has been changed to AERA :-)

- Dmitry Vostokov @ DumpAnalysis.org

Patterns for Concurrent Programming on Vista

Monday, June 16th, 2008

This new book will be published later this year:

Concurrent Programming on Windows Vista: Architecture, Principles, and Patterns (Microsoft .NET Development Series)

Buy from Amazon

Although it is featured in Microsoft .NET Development Series I believe that it might be very interesting read for both unmanaged and managed code developers and escalation engineers. Understanding concurrency and having the ability to see its patterns is the crucial skill for crash dump analysis and debugging of complex software issues. I pre-ordered it myself and write a review once it is in print.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump File Examples

Thursday, June 12th, 2008

I decided to post selected crash dumps for some case studies and patterns after getting feedback from readers of my MDAA book. They are custom made from my toy programs and preprocessed to clear sensitive information that might have leaked from my home computers otherwise. They can be found at the following public FTP address:

ftp://dumpanalysis.org/pub/

The first example was published today for Missing Component (static linking) pattern. I also created the following page to list them all including the ones that I add for some previously published patterns:

Memory Dump File Samples

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 59b)

Thursday, June 12th, 2008

Previously I introduced Missing Component pattern where the example and emphasis was on dynamically loaded modules. In this part I cover statically linked modules. Failure for a loader to find one of them results in a software exception. The most frequent of them are (numbers were taken from Google search):

C0000142 918
C0000143 919
C0000145 1,530
C0000135 24,900

0:001> !error c0000142
Error code: (NTSTATUS) 0xc0000142 (3221225794) - {DLL Initialization Failed}  Initialization of the dynamic link library %hs failed. The process is terminating abnormally.

0:001> !error c0000143
Error code: (NTSTATUS) 0xc0000143 (3221225795) - {Missing System File}  The required system file %hs is bad or missing.

0:001> !error c0000145
Error code: (NTSTATUS) 0xc0000145 (3221225797) - {Application Error}  The application failed to initialize properly (0x%lx). Click on OK to terminate the application.

0:000> !error c0000135
Error code: (NTSTATUS) 0xc0000135 (3221225781) - {Unable To Locate Component}  This application has failed to start because %hs was not found. Re-installing the application may fix this problem.

In this part I only consider user mode exceptions. If we have a default debugger configured it will usually save a crash dump. To model this problem I modified one of my applications by changing all occurrences of KERNEL32.DLL to  KERNEL32.DL using Visual Studio Binary Editor. CDB was configured as a default postmortem debugger (see Custom postmortem debuggers on Vista). When the application was launched CDB attached to it and saved a crash dump. If we open it in WinDbg we get characteristic Special Stack Trace involving loader functions:

Loading Dump File [C:\UserDumps\CDAPatternMissingComponent.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Vista Version 6000 MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Debug session time: Thu Jun 12 12:03:28.000 2008 (GMT+1)
System Uptime: 1 days 8:46:23.167
Process Uptime: 0 days 0:00:48.000

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(da4.f60): Wake debugger - code 80000007 (first/second chance not available)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!_LdrpInitialize+0x6d:
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

0:000> kL
ChildEBP RetAddr 
0017fd00 77b937ea ntdll!_LdrpInitialize+0×6d
0017fd10 00000000 ntdll!LdrInitializeThunk+0×10

Verbose analysis command doesn’t give us an indication of what had happened so we need to dig further:

0:000> !analyze -v
[...]

FAULTING_IP:
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000007 (Wake debugger)
  ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR:  80000007

PROCESS_NAME:  StackOverflow.exe

ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened}  the system debugger was awakened by an interrupt.

NTGLOBALFLAG:  400

APPLICATION_VERIFIER_FLAGS:  0

DERIVED_WAIT_CHAIN: 

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   0   da4.f60 Unknown               

WAIT_CHAIN_COMMAND:  ~0s;k;;

BLOCKING_THREAD:  00000f60

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BusyHang

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BusyHang

LAST_CONTROL_TRANSFER:  from 77b937ea to 77bcf1d1

FAULTING_THREAD:  00000000

STACK_TEXT: 
0017fd00 77b937ea 0017fd24 77b60000 00000000 ntdll!_LdrpInitialize+0x6d
0017fd10 00000000 0017fd24 77b60000 00000000 ntdll!LdrInitializeThunk+0x10

FOLLOWUP_IP:
ntdll!_LdrpInitialize+6d
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!_LdrpInitialize+6d

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4549bdf8

STACK_COMMAND:  ~0s ; kb

BUCKET_ID:  80000007_ntdll!_LdrpInitialize+6d

FAILURE_BUCKET_ID:  ntdll.dll!_LdrpInitialize_80000007_APPLICATION_HANG_BusyHang

Followup: MachineOwner

Last event and error code are not helpful too: 

0:000> .lastevent
Last event: da4.f60: Wake debugger - code 80000007 (first/second chance not available)
  debugger time: Thu Jun 12 15:04:38.917 2008 (GMT+1)

0:000> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0

Let’s search thread raw stack data for any signs of exceptions:

0:000> !teb
TEB at 7efdd000
    ExceptionList:        0017fcf0
    StackBase:            00180000
    StackLimit:           0017e000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efdd000
    EnvironmentPointer:   00000000
    ClientId:             00000da4 . 00000f60
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 0017e000 00180000
[...]
0017f8d8  7efdd000
0017f8dc  0017f964
0017f8e0  77c11c78 ntdll!_except_handler4
0017f8e4  00000000
0017f8e8  0017f988
0017f8ec  0017f900
0017f8f0  77ba1ddd ntdll!RtlCallVectoredContinueHandlers+0x15
0017f8f4  0017f988
0017f8f8  0017f9d8
0017f8fc  77c40370 ntdll!RtlpCallbackEntryList
0017f900  0017f970
0017f904  77ba1db5 ntdll!RtlDispatchException+0×11f
0017f908  0017f988
0017f90c  0017f9d8
0017f910  7efde028
0017f914  00000001
0017f918  77630000 kernel32!_imp___aullrem <PERF> (kernel32+0×0)
0017f91c  00000001
0017f920  776ced81 kernel32!_DllMainCRTStartupForGS2+0×10
0017f924  0017f938
0017f928  7765d4d9 kernel32!BaseDllInitialize+0×18
0017f92c  76042340 user32!$$VProc_ImageExportDirectory
0017f930  00000001
0017f934  00000000
0017f938  0017f9e0
0017f93c  77b8f890 ntdll!LdrpSnapThunk+0xc9
0017f940  0040977a StackOverflow+0×977a
0017f944  0000030b
0017f948  76030000 user32!_imp__RegSetValueExW <PERF> (user32+0×0)
0017f94c  76042f94 user32!$$VProc_ImageExportDirectory+0xc54
0017f950  77bb8881 ntdll!LdrpSnapThunk+0×40d
0017f954  0017bb30
0017f958  00409770 StackOverflow+0×9770
0017f95c  00881a50
0017f960  004098b2 StackOverflow+0×98b2
0017f964  77bac282 ntdll!ZwRaiseException+0×12
0017f968  00180000
0017f96c  0017fc48
0017f970  0017fd00
0017f974  77bac282 ntdll!ZwRaiseException+0×12
0017f978  77b7ee72 ntdll!KiUserExceptionDispatcher+0×2a

0017f97c  0017f988 ; exception record
0017f980  0017f9d8 ; exception context
0017f984  00000000
0017f988  c0000135
0017f98c  00000001
0017f990  00000000
0017f994  77bcf1d1 ntdll!_LdrpInitialize+0×6d
0017f998  00000000
0017f99c  77c11c78 ntdll!_except_handler4
0017f9a0  77b8dab8 ntdll!RtlpRunTable+0×218
0017f9a4  fffffffe
0017f9a8  77ba2515 ntdll!vDbgPrintExWithPrefixInternal+0×214
0017f9ac  77ba253b ntdll!DbgPrintEx+0×1e
0017f9b0  77b7f356 ntdll! ?? ::FNODOBFM::`string’
0017f9b4  00000055
0017f9b8  00000003
0017f9bc  77b809c2 ntdll! ?? ::FNODOBFM::`string’
0017f9c0  0017fc9c
0017f9c4  00000001
0017f9c8  0017fd00
0017f9cc  77bcf28e ntdll!_LdrpInitialize+0×12a
0017f9d0  00000055
0017f9d4  75ce3cf9
0017f9d8  0001003f
0017f9dc  00000000
0017f9e0  00000000
0017f9e4  00000000
0017f9e8  00000000
0017f9ec  00000000
0017f9f0  00000000
0017f9f4  0000027f
[…]

We see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get one:

0:000> .exr 0017f988
ExceptionAddress: 77bcf1d1 (ntdll!_LdrpInitialize+0x0000006d)
   ExceptionCode: c0000135
  ExceptionFlags: 00000001
NumberParameters: 0

Error c0000135 means that the loader was unable to locate a component. Now we try to examine the same raw stack data for any string patterns. For example, the following UNICODE pattern is clearly visible:

0017f2fc  00000000
0017f300  00880ec4
0017f304  77b910d7 ntdll!RtlpDosPathNameToRelativeNtPathName_Ustr+0x344
0017f308  00000000
0017f30c  43000043
0017f310  0042002a
0017f314  0017f33c
0017f318  00000000
0017f31c  00000002
0017f320  00000008
0017f324  00000000
0017f328  0000008c
0017f32c  000a0008
0017f330  77b91670 ntdll!`string'
0017f334  00b92bd6
0017f338  0017f5d4
0017f33c  003a0043
0017f340  0050005c
0017f344  006f0072
0017f348  00720067
0017f34c  006d0061
0017f350  00460020
0017f354  006c0069
0017f358  00730065
0017f35c  00280020
0017f360  00380078
0017f364  00290036
0017f368  0043005c
0017f36c  006d006f
0017f370  006f006d
0017f374  0020006e
0017f378  00690046
0017f37c  0065006c
0017f380  005c0073
0017f384  006f0052
0017f388  00690078
0017f38c  0020006f
0017f390  00680053
0017f394  00720061
0017f398  00640065
0017f39c  0044005c
0017f3a0  004c004c
0017f3a4  00680053
0017f3a8  00720061
0017f3ac  00640065
0017f3b0  004b005c
0017f3b4  00520045
0017f3b8  0045004e
0017f3bc  0033004c
0017f3c0  002e0032
0017f3c4  006c0064

0017f3c8  00000000
0017f3cc  00000000

It is a path to DLL that was probably missing:

0:000> du 0017f33c
0017f33c  "C:\Program Files (x86)\Common Fi"
0017f37c  "les\Roxio Shared\DLLShared\KERNE"
0017f3bc  "L32.dl"

I think the loader was trying to find KERNEL32.dl following the DLL search order and this was the last path element:

0:000> !peb
PEB at 7efde000
    InheritedAddressSpace:    No
    ReadImageFileExecOptions: No
    BeingDebugged:            Yes
    ImageBaseAddress:         00400000
    Ldr                       77c40080
    Ldr.Initialized:          Yes
    Ldr.InInitializationOrderModuleList: 00881ad0 . 008831b8
    Ldr.InLoadOrderModuleList:           00881a50 . 00883dc8
    Ldr.InMemoryOrderModuleList:         00881a58 . 00883dd0
            Base TimeStamp                     Module
[...]
    Environment:  00881de8
[...] 
Path=C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
[…]

In  similar situations !dlls command might help that shows the load order (-l switch) and points to the last processed DLL:

0:001> !dlls -l

0x004740e8: C:\Program Files\Application\Application.exe
      Base   0x012a0000  EntryPoint  0x012b0903  Size        0x00057000
      Flags  0x00004010  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_ENTRY_PROCESSED

0x00474158: C:\Windows\SysWOW64\ntdll.dll
      Base   0x77d00000  EntryPoint  0x00000000  Size        0x00160000
      Flags  0x00004014  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED

0x00474440: C:\Windows\syswow64\kernel32.dll
      Base   0x77590000  EntryPoint  0x775a1f3e  Size        0x00110000
      Flags  0x00084014  LoadCount   0x0000ffff  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_PROCESS_ATTACH_CALLED

[…] 

0x00498ff8: C:\Windows\WinSxS\x86_microsoft.windows.common-controls_...\comctl32.dll
      Base   0x74d90000  EntryPoint  0x74dc43e5  Size        0x0019e000
      Flags  0x100c4014  LoadCount   0x00000003  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED
             LDRP_DONT_CALL_FOR_THREADS
             LDRP_PROCESS_ATTACH_CALLED

0x004991b8: C:\Windows\WinSxS\x86_microsoft.vc80.mfcloc_...\MFC80ENU.DLL
      Base   0x71b10000  EntryPoint  0x00000000  Size        0x0000e000
      Flags  0x10004014  LoadCount   0x00000001  TlsIndex    0x00000000
             LDRP_IMAGE_DLL
             LDRP_ENTRY_PROCESSED

If it is difficult to identify what had really happened in crash dumps we can enable loader snaps using gflags and run the application under a debugger. For example, for notepad.exe we have:

Microsoft (R) Windows Debugger Version 6.8.0004.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Windows\notepad.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00000000`ffac0000 00000000`ffaef000   notepad.exe
ModLoad: 00000000`779b0000 00000000`77b2a000   ntdll.dll
LDR: NEW PROCESS
     Image Path: C:\Windows\notepad.exe (notepad.exe)
     Current Directory: C:\Program Files\Debugging Tools for Windows 64-bit\
     Search Path: C:\Windows; C:\Windows\system32; C:\Windows\system; C:\Windows;.; C:\Program Files\Debugging Tools for Windows 64-bit\winext\arcade; C:\Windows\system32;C:\Windows; C:\Windows\System32\Wbem; C:\Program Files\ATI Technologies\ATI.ACE; c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\; C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\
LDR: LdrLoadDll, loading kernel32.dll from
ModLoad: 00000000`777a0000 00000000`778d1000   C:\Windows\system32\kernel32.dll
LDR: kernel32.dll bound to ntdll.dll
LDR: kernel32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from kernel32.dll
LDR: LdrGetProcedureAddress by NAME - BaseThreadInitThunk
[3d8,1278] LDR: Real INIT LIST for process C:\Windows\notepad.exe pid 984 0x3d8
[3d8,1278]    C:\Windows\system32\kernel32.dll init routine 00000000777DC960
[3d8,1278] LDR: kernel32.dll loaded - Calling init routine at 00000000777DC960
LDR: notepad.exe bound to ADVAPI32.dll
ModLoad: 000007fe`fe520000 000007fe`fe61f000   C:\Windows\system32\ADVAPI32.dll
LDR: ADVAPI32.dll bound to ntdll.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from ADVAPI32.dll
LDR: ADVAPI32.dll bound to KERNEL32.dll
LDR: ADVAPI32.dll has stale binding to KERNEL32.dll
LDR: ADVAPI32.dll bound to ntdll.dll via forwarder(s) from kernel32.dll
LDR: ADVAPI32.dll has stale binding to ntdll.dll
LDR: Stale Bind KERNEL32.dll from ADVAPI32.dll
LDR: LdrGetProcedureAddress by NAME - RtlAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlReAllocateHeap
LDR: LdrGetProcedureAddress by NAME - RtlEncodePointer
LDR: LdrGetProcedureAddress by NAME - RtlDecodePointer
LDR: LdrGetProcedureAddress by NAME - RtlSizeHeap
LDR: LdrGetProcedureAddress by NAME - RtlDeleteCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlEnterCriticalSection
LDR: LdrGetProcedureAddress by NAME - RtlLeaveCriticalSection
LDR: ADVAPI32.dll bound to RPCRT4.dll
[...]

This technique only works for native platform loader snaps. For example, it doesn’t show loader snaps for 32-bit modules loaded under WOW64 (to the best of my knowledge):

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Apps\StackOverflow.exe
Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00418000   StackOverflow.exe
ModLoad: 77b60000 77cb0000   ntdll.dll
LDR: NEW PROCESS
     Image Path: C:\Apps\StackOverflow.exe (StackOverflow.exe)
[...]
LDR: Loading (STATIC, NON_REDIRECTED) C:\Windows\system32\wow64cpu.dll
LDR: wow64cpu.dll bound to ntdll.dll
LDR: wow64cpu.dll has stale binding to ntdll.dll
LDR: Stale Bind ntdll.dll from wow64cpu.dll
LDR: wow64cpu.dll bound to wow64.dll
LDR: wow64cpu.dll has stale binding to wow64.dll
LDR: Stale Bind wow64.dll from wow64cpu.dll
LDR: wow64.dll has stale binding to wow64cpu.dll
LDR: Stale Bind wow64cpu.dll from wow64.dll
LDR: Refcount wow64cpu.dll (1)
LDR: Refcount wow64.dll (2)
LDR: Refcount wow64win.dll (1)
LDR: Refcount wow64.dll (3)
LDR: LdrGetProcedureAddress by NAME - Wow64LdrpInitialize
[...]
ModLoad: 77630000 77740000   C:\Windows\syswow64\kernel32.dll
ModLoad: 76030000 76100000   C:\Windows\syswow64\USER32.dll
ModLoad: 775a0000 77630000   C:\Windows\syswow64\GDI32.dll
ModLoad: 76d00000 76dbf000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 76df0000 76ee0000   C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75d60000 75dc0000   C:\Windows\syswow64\Secur32.dll

(1ec.1290): Unknown exception - code c0000135 (first chance)
(1ec.1290): Unknown exception - code c0000135 (!!! second chance !!!)
eax=00000000 ebx=77c4a174 ecx=75ce3cf9 edx=00000000 esi=7efde028 edi=7efdd000
eip=77bcf1d1 esp=0017fca4 ebp=0017fd00 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!_LdrpInitialize+0×6d:
77bcf1d1 8b45b8          mov     eax,dword ptr [ebp-48h] ss:002b:0017fcb8=7efde000

The dump file that I used was modified to remove sensitive information (see Data Hiding in Crash Dumps for this technique). It can be downloaded from FTP to play with:

ftp://dumpanalysis.org/pub/CDAPatternMissingComponent.zip

- Dmitry Vostokov @ DumpAnalysis.org -

Data Hiding in Crash Dumps

Tuesday, June 10th, 2008

Suppose we want to send a complete memory dump to a vendor but want to remove certain sensitive details or perhaps the whole process or image from it. In this case we can use f WinDbg command (virtual addresses) or fp (physical addresses) to fill pages with zeroes. Let’s open a complete memory dump and erase environment variables for a process:

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS fffffadfe7afd8e0
    SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 0014a000  ObjectTable: fffffa8000000c10  HandleCount: 730.
    Image: System

PROCESS fffffadfe6edc040
    SessionId: none  Cid: 0130    Peb: 7fffffdf000  ParentCid: 0004
    DirBase: 34142000  ObjectTable: fffffa80009056d0  HandleCount:  19.
    Image: smss.exe

[...]

PROCESS fffffadfe67905a0
    SessionId: 0  Cid: 085c    Peb: 7fffffd4000  ParentCid: 0acc
    DirBase: 232e2000  ObjectTable: fffffa8000917e10  HandleCount:  55.
    Image: SystemDump.exe

kd> .process /r /p fffffadfe7287610
Implicit process is now fffffadf`e7287610
Loading User Symbols

kd> !peb
PEB at 000007fffffd4000
[...]
    Environment:  0000000000010000

kd> dd 10000
00000000`00010000  004c0041 0055004c 00450053 00530052
00000000`00010010  00520050 0046004f 004c0049 003d0045
00000000`00010020  003a0043 0044005c 0063006f 006d0075
00000000`00010030  006e0065 00730074 00610020 0064006e
00000000`00010040  00530020 00740065 00690074 0067006e
00000000`00010050  005c0073 006c0041 0020006c 00730055
00000000`00010060  00720065 002e0073 00320057 0033004b
00000000`00010070  00410000 00500050 00410044 00410054

kd> f 10000 10000+1000 0
Filled 0x1000 bytes

kd> dd 10000
00000000`00010000  00000000 00000000 00000000 00000000
00000000`00010010  00000000 00000000 00000000 00000000
00000000`00010020  00000000 00000000 00000000 00000000
00000000`00010030  00000000 00000000 00000000 00000000
00000000`00010040  00000000 00000000 00000000 00000000
00000000`00010050  00000000 00000000 00000000 00000000
00000000`00010060  00000000 00000000 00000000 00000000
00000000`00010070  00000000 00000000 00000000 00000000

Now we can save the modified complete dump file:

kd> .dump /f c:\Dumps\SecuredDump.dmp

If we want to find and erase read-write pages, for example, we can use !vad WinDbg command to get the description of virtual address ranges:

kd> !process
PROCESS fffffadfe67905a0
    SessionId: 0  Cid: 085c    Peb: 7fffffd4000  ParentCid: 0acc
    DirBase: 232e2000  ObjectTable: fffffa8000917e10  HandleCount:  55.
    Image: SystemDump.exe
    VadRoot fffffadfe6f293e0 Vads 65 Clone 0 Private 388. Modified 84. Locked 0.
    DeviceMap fffffa80020777c0
    Token                             fffffa80008e5b50
    ElapsedTime                       00:00:06.265
    UserTime                          00:00:00.031
    KernelTime                        00:00:00.062
    QuotaPoolUsage[PagedPool]         113464
    QuotaPoolUsage[NonPagedPool]      5152
    Working Set Sizes (now,min,max)  (1429, 50, 345) (5716KB, 200KB, 1380KB)
    PeakWorkingSetSize                1429
    VirtualSize                       61 Mb
    PeakVirtualSize                   63 Mb
    PageFaultCount                    1555
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      471

kd> !vad fffffadfe6f293e0
VAD             level      start      end    commit
fffffadfe682bdf0 ( 6)         10       10         1 Private      READWRITE
fffffadfe73a0e10 ( 5)         20       20         1 Private      READWRITE
fffffadfe73a0dd0 ( 4)         30      12f         8 Private      READWRITE
fffffadfe71a4770 ( 5)        130      134         0 Mapped       READONLY
fffffadfe781bbe0 ( 3)        140      141         0 Mapped       READONLY
[…]
fffffadfe772d630 (-2)   7fffffdc 7fffffdd         2 Private      READWRITE
fffffadfe788e180 (-1)   7fffffde 7fffffdf         2 Private      READWRITE

Total VADs:    65  average level: 66076419  maximum depth: -1

In the output start and end columns refer to virtual page numbers (VPN). To get an address we need to multiply by 0×1000, for example,  7fffffde000

Filling memory with zeroes to hide data with subsequent saving of a modified crash dump is applicable to user dumps too. Please also check for additional security-related flags in .dump command:

WinDbg is privacy-aware

Another application for data hiding and modification could be the creation of the customized crash dumps for digital forensics exercises and contests.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 16b)

Tuesday, June 10th, 2008

I’ve just found that although I covered Stack Overflow in kernel mode I didn’t do this for user mode. In fact this is one of the simplest patterns to see in crash dumps. It has its own characteristic exception code and stack trace:

FAULTING_IP:
StackOverflow!SoFunction+27
00401317 6a00            push    0

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00401300 (StackOverflow!SoFunction+0x00000010)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00082ffc

0:000> kL
ChildEBP RetAddr 
00083000 00401317 StackOverflow!SoFunction+0x10
00083010 00401317 StackOverflow!SoFunction+0×27
00083020 00401317 StackOverflow!SoFunction+0×27
00083030 00401317 StackOverflow!SoFunction+0×27
00083040 00401317 StackOverflow!SoFunction+0×27
00083050 00401317 StackOverflow!SoFunction+0×27
00083060 00401317 StackOverflow!SoFunction+0×27
00083070 00401317 StackOverflow!SoFunction+0×27
00083080 00401317 StackOverflow!SoFunction+0×27
00083090 00401317 StackOverflow!SoFunction+0×27
000830a0 00401317 StackOverflow!SoFunction+0×27
000830b0 00401317 StackOverflow!SoFunction+0×27
000830c0 00401317 StackOverflow!SoFunction+0×27
000830d0 00401317 StackOverflow!SoFunction+0×27
000830e0 00401317 StackOverflow!SoFunction+0×27
000830f0 00401317 StackOverflow!SoFunction+0×27
00083100 00401317 StackOverflow!SoFunction+0×27
00083110 00401317 StackOverflow!SoFunction+0×27
00083120 00401317 StackOverflow!SoFunction+0×27
00083130 00401317 StackOverflow!SoFunction+0×27

There could be thousands of stack frames:

0:000> kL 2000
[...]
000a2fa0 00401317 StackOverflow!SoFunction+0x27
000a2fb0 00401317 StackOverflow!SoFunction+0x27
000a2fc0 00401317 StackOverflow!SoFunction+0x27
000a2fd0 00401317 StackOverflow!SoFunction+0x27
000a2fe0 00401317 StackOverflow!SoFunction+0x27
000a2ff0 00401317 StackOverflow!SoFunction+0x27

To reach the bottom and avoid over scrolling we can dump the raw stack data, search for the end of the repeating pattern of StackOverflow!SoFunction+0×27 and try to manually reconstruct the bottom of the stack trace:

0:000> !teb
TEB at 7efdd000
    ExceptionList:        0017fdf0
    StackBase:            00180000
    StackLimit:           00081000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efdd000
    EnvironmentPointer:   00000000
    ClientId:             00001dc4 . 00001b74
    RpcHandle:            00000000
    Tls Storage:          7efdd02c
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      c0000034
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00081000 00180000
[...]
0017fc74  00401317 StackOverflow!SoFunction+0×27
0017fc78  00000000
0017fc7c  a3a8ea65
0017fc80  0017fc90
0017fc84  00401317 StackOverflow!SoFunction+0×27
0017fc88  10001843
0017fc8c  a3a8ea95
0017fc90  0017fca0
0017fc94  00401317 StackOverflow!SoFunction+0×27
0017fc98  0017fcb8
0017fc9c  a3a8ea85
0017fca0  0017fcb0
0017fca4  00401317 StackOverflow!SoFunction+0×27
0017fca8  00000003
0017fcac  a3a8eab5
0017fcb0  0017fcc0
0017fcb4  00401317 StackOverflow!SoFunction+0×27
0017fcb8  76c68738 user32!_EndUserApiHook+0×11
0017fcbc  a3a8eaa5
0017fcc0  0017fcd0
0017fcc4  00401317 StackOverflow!SoFunction+0×27
0017fcc8  76c6a6cc user32!DefWindowProcW+0×94
0017fccc  a3a8ead5
0017fcd0  0017fce0
0017fcd4  00401317 StackOverflow!SoFunction+0×27
0017fcd8  0037311e
0017fcdc  a3a8eac5
0017fce0  0017fcf0
0017fce4  00401317 StackOverflow!SoFunction+0×27
0017fce8  0017fcd0
0017fcec  a3a8eaf5
0017fcf0  0017fd00
0017fcf4  00401317 StackOverflow!SoFunction+0×27
0017fcf8  76c6ad0f user32!NtUserBeginPaint+0×15
0017fcfc  a3a8eae5
0017fd00  0017fd5c
0017fd04  00401272 StackOverflow!WndProc+0xe2
0017fd08  00401190 StackOverflow!WndProc
0017fd0c  00000003
0017fd10  cf017ada
[…]

We use the extended version of k WinDbg command and supply EBP, ESP and EIP to see in what function it started:

0:000> r
eax=a3b739e5 ebx=00000000 ecx=ac430000 edx=ffefd944 esi=0037311e edi=00000000
eip=00401300 esp=00082ff8 ebp=00083000 iopl=0         nv up ei ng nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
StackOverflow!SoFunction+0×10:
00401300 89442404        mov     dword ptr [esp+4],eax ss:002b:00082ffc=00000000

0:000> k L=0017fcf0 00082ff8 00401300
ChildEBP RetAddr 
0017fcb0 00401317 StackOverflow!SoFunction+0×10
0017fd00 00401272 StackOverflow!SoFunction+0×27

0017fd5c 76c687af StackOverflow!WndProc+0xe2
0017fd88 76c68936 user32!InternalCallWinProc+0×23
0017fe00 76c6a571 user32!UserCallWinProcCheckWow+0×109
0017fe5c 76c6a5dd user32!DispatchClientMessage+0xe0
0017fe98 77ccee2e user32!__fnDWORD+0×2b
0017fedc 0040107d ntdll!KiUserCallbackDispatcher+0×2e
0017ff08 0040151e StackOverflow!wWinMain+0×7d
00402ba0 20245c8b StackOverflow!__tmainCRTStartup+0×176

- Dmitry Vostokov @ DumpAnalysis.org -

Data Recovery with Memory Dump Analysis

Monday, June 9th, 2008

My friend was typing a long message in IE to one of his old schoolmates that he had just found on Internet. He spent about an hour writing and rewriting and when finally hit the Send button he got a page saying that connection was probably lost. Going back in URL history brought the empty edit box and all data was lost. Or was it? He called me and I immediately advised him to save a crash dump of iexplore.exe using Task Manager (Vista). I also asked him for a word he used to start his message. It was “Hello” in Russian. I got his dump file and opened it in WinDbg. Because the language of his message was Russian I assumed that it was still there in local buffers or heap entries in UNICODE format so I typed “ello” in Notepad and saved this in a Unicode text file. Loading it in a binary editor (I used Visual C++) showed the following sequence of bytes:

40 04 38 04 32 04 35 04 42 04

Then I did a search in WinDbg for this sequence from the first loaded module address till the end of user space:

0:000> lm
start    end        module name
003c0000 0045b000   iexplore
[...]

0:000> s 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04
[...]
048971e4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04  @.8.2.5.B.,. ...
[...]
08530fe4 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...]
201ea65c 40 04 38 04 32 04 35 04-42 04 2c 00 20 00 1c 04 @.8.2.5.B.,. ...
[...]
 

The number of found entries was big and I decided to output every entry into a file using the following script: 

.foreach ( address { s-[1]b 003c0000 L?7FFFFFFF 40 04 38 04 32 04 35 04 42 04 }) {.writemem c:\dmitry\ieout${address}.txt ${address}-10 ${address}+1000}

I got numerous files:

C:\dmitry>dir ieout*.txt
[...]
09/06/2008  08:53               4112 ieout0x048971e4.txt
09/06/2008  08:53               4112 ieout0x0489784c.txt
09/06/2008  08:53               4112 ieout0x0489b854.txt
09/06/2008  08:53               4112 ieout0x0489bc5c.txt
[...]

I combined all of them into one big file and sent it to my friend:

C:\dmitry>type ieout0x*.txt >ieoutall.txt

The file contained not only the final message but all intermediate typing histories too. He was very happy.

- Dmitry Vostokov @ DumpAnalysis.org -

Memoidealism

Monday, June 9th, 2008

Looking at memory dumps every day and writing about them has an unfortunate implication: every state of the world looks like a gigantic memory dump to me. Everything is memory and every state is memory dump. The current state of the world is an infinite (or an immense) number of memuons*. Infinite can be any cardinal number greater or equal to that of natural numbers. In any case we can say it is N bits where this number is either finite or ∞. Therefore we have 2N possible memory states (S). The set of possible transitions between them (S -> S) has the number of 2N ^ 2N elements. Which is the memory itself and we have transitions between its states too. Ad infinitum we have a limiting process from which arises the perceived flow of events. Anyway there is much to elaborate here. I’ll come back to this later.

(*) Memuon is an indivisible entity similar to a bit of information.

This is my current philosophy :-) I’m dead serious.

- Dmitry Vostokov @ DumpAnalysis.org -

Review of Memory Analysis album

Friday, June 6th, 2008

If you remember I promised to review this CD:

 The first Memory Analysis CD album

Finally it arrived by post and here it is:

I’ve just finished listening to it. It is wonderful! Clearly belongs to my Music for Debugging collection. Here is my commentary (italics) on track titles:

1. Chameleon 
   It crashes then hangs then spikes then leaks. You never know what happens next…
2. Silence Before The Storm 
   Waiting for the problem to start dumping memory
3. Appearances 
   Multiple occurrences of the issue ease crash dump collection
4. Memory Analysis 
   Memory dump analysis activity
5. Voices From Heaven 
   It’s my blog full of crash dump analysis patterns
6. Figure In The Fog 
   It’s that DLL! I see it clearly!!!
7. Diamond Tear 
   Customer is happy
8. The Toy Soldier 
   WinDbg
9. Tears In Dragon’s Eyes 
   Operating system vendor is happy too
10. Forgotten Song 
   No pasaran (bugs)
11. Skies
   We are debugging gods!

Enjoy :-) 

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 63)

Friday, June 6th, 2008

Another pattern that we need to pay attention is called Thread Starvation. This happens when some threads or processes have higher priority and favored by OS thread dispatcher effectively starving other threads. If prioritized threads are CPU-bound we also see Spiking Thread pattern. However, if their thread priorities were normal they would have been preempted by other threads and latter threads would not be starved. Here is one example where 2 threads from 2 different applications but from one user session are spiking on 2 processors (threads from other processors have above normal and normal priority):

System Uptime: 0 days 6:40:41.143

0: kd> !running

System Processors f (affinity mask)
  Idle Processors None

     Prcb      Current   Next    
  0  f773a120  89864c86 ................
  1  f773d120  89f243d2 ................
  2  f7737120  89f61398 ................
  3  f773f120  897228a0 ................

0: kd> .thread /r /p 89f61398
Implicit thread is now 89f61398
Implicit process is now 88bcc2e0
Loading User Symbols

0: kd> !thread 89f61398 1f
THREAD 89f61398  Cid 16f8.2058  Teb: 7ffdf000 Win32Thread: bc41aea8 RUNNING on processor 2
Not impersonating
DeviceMap                 e48a6508
Owning Process            88bcc2e0       Image:         application.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      7201654                 LargeStack
UserTime                  01:24:06.687
KernelTime                00:14:53.828

Win32 Start Address application (0×0040a52c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba336000 Current ba335d00 Base ba336000 Limit ba330000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012e09c 762c3b7d USER32!IsWindowVisible
0012e0c4 762d61bb MSVBVM50!RbyCountVisibleDesks+0×3c
0012e0d0 004831f6 MSVBVM50!rtcDoEvents+0×7
0012e348 0046d1ae application+0×831f6
0012e3a0 762ce5a9 application+0×6d1ae
0012e3dc 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012e3f8 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012e434 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012e514 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012e55c 762b1aa3 MSVBVM50!EvtErrFire+0×18
0012e5ac 7739bffa MSVBVM50!CThreadPool::GetThreadData+0xf
0012e58c 762cd13b USER32!CallHookWithSEH+0×21
0012e5ac 7739bffa MSVBVM50!VBDefControlProc_2787+0xad
0012e618 762d3348 USER32!CallHookWithSEH+0×21
0012e640 762cda44 MSVBVM50!PushCtlProc+0×2e
0012e668 762cd564 MSVBVM50!CommonGizWndProc+0×4e
0012e6b8 7739b6e3 MSVBVM50!StdCtlWndProc+0×171
0012e6e4 7739b874 USER32!InternalCallWinProc+0×28
0012e75c 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012e7c4 773a16e5 USER32!DispatchMessageWorker+0×327
0012e7d4 762d616e USER32!DispatchMessageA+0xf
0012e828 762d6054 MSVBVM50!ThunderMsgLoop+0×97
0012e874 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012e8b4 004831f6 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012e8c0 00d3b3c8 application+0×831f6
00184110 00000000 0xd3b3c8

0: kd> .thread /r /p 897228a0
Implicit thread is now 897228a0
Implicit process is now 897348a8
Loading User Symbols

0: kd> !thread 897228a0 1f 100
THREAD 897228a0  Cid 2984.2988  Teb: 7ffdf000 Win32Thread: bc381488 RUNNING on processor 3
IRP List:
    89794bb8: (0006,0220) Flags: 00000000  Mdl: 8a145878
Not impersonating
DeviceMap                 e3ec0360
Owning Process            897348a8       Image:         application2.exe
Wait Start TickCount      1569737        Ticks: 0
Context Switch Count      10239625                 LargeStack
UserTime                  02:38:18.890
KernelTime                00:29:36.187

Win32 Start Address application2 (0×00442e4c)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init f1d90000 Current f1d8fd00 Base f1d90000 Limit f1d88000 Call 0
Priority 24 BasePriority 24 PriorityDecrement 0
ChildEBP RetAddr 
0012f66c 762d61bb USER32!_SEH_prolog+0×5
0012f678 00fdb0b9 MSVBVM50!rtcDoEvents+0×7
0012f92c 00fca760 application2+0xbdb0b9
0012fa20 762ce5a9 application2+0xbca760
0012fa40 762ce583 MSVBVM50!CallProcWithArgs+0×20
0012fa5c 762db781 MSVBVM50!InvokeVtblEvent+0×33
0012fa98 762cfbc2 MSVBVM50!InvokeEvent+0×32
0012fb78 762cfa4a MSVBVM50!EvtErrFireWorker+0×175
0012fb90 76330b2b MSVBVM50!EvtErrFire+0×18
0012fbf0 762cd13b MSVBVM50!ErrDefMouse_100+0×16d
0012fca4 762cda44 MSVBVM50!VBDefControlProc_2787+0xad
0012fccc 7631c826 MSVBVM50!CommonGizWndProc+0×4e
0012fd08 762cd523 MSVBVM50!StdCtlPreFilter_50+0×9e
0012fd5c 7739b6e3 MSVBVM50!StdCtlWndProc+0×130
0012fd88 7739b874 USER32!InternalCallWinProc+0×28
0012fe00 7739ba92 USER32!UserCallWinProcCheckWow+0×151
0012fe68 773a16e5 USER32!DispatchMessageWorker+0×327
0012fe78 762d616e USER32!DispatchMessageA+0xf
0012fea8 762bb78f MSVBVM50!ThunderMsgLoop+0×97
0012feb8 762d60cb MSVBVM50!MsoFInitPx+0×39
0012fecc 762d6054 MSVBVM50!CMsoCMHandler::FPushMessageLoop+0×1a
0012ff18 762d5f55 MSVBVM50!SCM::FPushMessageLoop+0xaf
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012fef8 762d5f8e nt!KiDeliverApc+0×11f
0012ff18 762d5f55 MSVBVM50!SCM_MsoCompMgr::FPushMessageLoop+0×2f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ff18 762d5f55 nt!KiDeliverApc+0×11f
0012ffa0 8082ea41 MSVBVM50!CMsoComponent::PushMsgLoop+0×24
0012ffd4 0012ffc8 nt!KiDeliverApc+0×11f
00000000 00000000 0×12ffc8

What we see here is unusually high Priority and BasePriority values (24 and 24). This means that the base priority for these processes was most likely artificially increased to Realtime. Most processes have base priority 8 (Normal):

0: kd> !thread 88780db0 1f
THREAD 88780db0  Cid 44a8.1b8c  Teb: 7ffaf000 Win32Thread: bc315d20 WAIT: (Unknown) UserMode Non-Alertable
    887b8650  Semaphore Limit 0x7fffffff
    88780e28  NotificationTimer
Not impersonating
DeviceMap                 e1085298
Owning Process            889263a0       Image:         explorer.exe
Wait Start TickCount      1565543        Ticks: 4194 (0:00:01:05.531)
Context Switch Count      7                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b6754000 Current b6753c0c Base b6754000 Limit b6750000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
b6753c24 80833e8d nt!KiSwapContext+0×26
b6753c50 80829b74 nt!KiSwapThread+0×2e5
b6753c98 809249cd nt!KeWaitForSingleObject+0×346
b6753d48 8088ac4c nt!NtReplyWaitReceivePortEx+0×521
b6753d48 7c8285ec nt!KiFastCallEntry+0xfc
01a2fe18 7c82783b ntdll!KiFastSystemCallRet
01a2fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
01a2ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0×198
01a2ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
01a2ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0×9d
01a2ffb8 77e64829 RPCRT4!ThreadStartRoutine+0×1b
01a2ffec 00000000 kernel32!BaseThreadStart+0×34

Some important system processes like csrss.exe have base priority 13 (High) but their threads wait most of the time and this doesn’t create any problems:

0: kd> !thread 887eb3d0 1f
THREAD 887eb3d0  Cid 4cf4.2bd4  Teb: 7ffaf000 Win32Thread: bc141cc0 WAIT: (Unknown) UserMode Non-Alertable
    888769b0  SynchronizationEvent
Not impersonating
DeviceMap                 e1000930
Owning Process            8883f7c0       Image:         csrss.exe
Wait Start TickCount      1540456        Ticks: 29281 (0:00:07:37.515)
Context Switch Count      40                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Start Address winsrv!ConsoleInputThread (0×75a81b18)
Stack Init b5c5a000 Current b5c59bac Base b5c5a000 Limit b5c55000 Call 0
Priority 15 BasePriority 13 PriorityDecrement 0
Kernel stack not resident.
ChildEBP RetAddr 
b5c59bc4 80833e8d nt!KiSwapContext+0×26
b5c59bf0 80829b74 nt!KiSwapThread+0×2e5
b5c59c38 bf89b1c3 nt!KeWaitForSingleObject+0×346
b5c59c94 bf89b986 win32k!xxxSleepThread+0×1be
b5c59cec bf89da22 win32k!xxxRealInternalGetMessage+0×46a
b5c59d4c 8088ac4c win32k!NtUserGetMessage+0×3f
b5c59d4c 7c8285ec nt!KiFastCallEntry+0xfc
00ffff64 7739c811 ntdll!KiFastSystemCallRet
00ffff84 75a81c47 USER32!NtUserGetMessage+0xc
00fffff4 00000000 winsrv!ConsoleInputThread+0×16c

However it is very unusual for a process to have Realtime base priority. I can speculate what had really happened before the system crash was forced. The system administrator noticed two applications consuming CPU over the long period of time and decided to intervene. Unfortunately his hand slipped when he was browsing Task Manager Set Priority menu and Realtime was selected instead of Low. Human error…

See also the similar starvation pattern resulted from threads with normal priority.

- Dmitry Vostokov @ DumpAnalysis.org -

PFD lectures are available in PDF format

Thursday, June 5th, 2008

Due to the request from blog readers I made my old Practical Foundations of Debugging lectures available in PDF format:

  1. Memory, registers and simple arithmetic
  2. Number representations and pointers
  3. Bytes, words, double words and pointers to memory
  4. Instruction pointer and disassembling a program with pointers
  5. Memory and stacks
  6. Frame pointer and local variables (Part 1)
  7. Frame pointer and local variables (Part 2)
  8. Function parameters
  9. Function pointer parameters (Part 1)
  10. Function pointer parameters (Part 2)
  11. Virtual Memory, Processes and Threads (Part 1)
  12. Virtual Memory, Processes and Threads (Part 2)
  13. Arrays and structures in memory (Part 1)
  14. Arrays and structures in memory (Part 2)

x64 version:

  1. Memory, registers and simple arithmetic
  2. Number representations and pointers

I keep both versions (HTMP and PDF) on the following pages where updates or corrections will be posted in the future:

Practical Foundations of Debugging (x86)
Practical Foundations of Debugging (x64)

- Dmitry Vostokov @ DumpAnalysis.org -

Software Debugging book from China

Thursday, June 5th, 2008

Just discovered that China has its own 1000 page Windows debugging book:

ISBN: 9787121064074 (June, 2008)

Author is Zhang Yinkui (Raymond Zhang).

Google translation is quite impressive:

Automatically translated

Automatically translated Table of Contents

Original

- Dmitry Vostokov @ DumpAnalysis.org -

What is the difference between AMD and Intel?

Thursday, June 5th, 2008

A. Both have online versions of processor manuals. But Intel also ships them in paper format for free (the paper is worse in the latest revision than 2 years ago but books are a bit lighter now):

Information on how to order them can be found here:

http://www.intel.com/products/processor/manuals/index.htm

AMD docs can be downloaded from here:

http://developer.amd.com/documentation/guides/Pages/default.aspx

Comparing both online manuals I see sometimes that certain concepts are explained better in AMD docs and vice versa, so it is recommended to check both. It is also evident that AMD and Intel had to rephrase instruction descriptions differently when they talk about the same things, for example, BOUND instruction:

AMD:  Check Array Bound
Intel:  Check Array Index Against Bounds

- Dmitry Vostokov @ DumpAnalysis.org -

The ASS Book

Wednesday, June 4th, 2008

Crash dump analysis is a support activity. Therefore understanding software support business is important. I recently started abbreviating book titles for my personal book reading accounting purposes and here accidentally emerges the ASS abbreviation that reminds me every time that support guys cover “asses” of software engineers designing and developing software and expose their own “asses” when talking to customers :-)

I’m sorry if I offended someone here…

The Art of Software Support

Buy from Amazon

The book was on my shelf for 4 years and only today I got the message :-) 

- Dmitry Vostokov @ DumpAnalysis.org -

The Science of Dr. Watson

Tuesday, June 3rd, 2008

Motivated by The Science of Sherlock Holmes I plan to write a book about the history of debugging with the following preliminary product details and tentative release date (which may come earlier if I have enough time):

  • Title: The Science of Dr. Watson: An Illustrated History of Debugging
  • Author: Dmitry Vostokov
  • Publisher: Opentask (01 September 2010)
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • ISBN-13: 978-1-906717-07-0
  • Paperback: 256 pages

- Dmitry Vostokov @ DumpAnalysis.org -

Computer Memory Visualization

Tuesday, June 3rd, 2008

More books to come in 2009. One of them is full-color book illustrated with beautiful visual images emerging from inherent modularized structure of modern operating systems and applications. Preliminary product details:

  • Title: Computer Memory Visualization
  • Authors: Jamie Fenton, Dmitry Vostokov
  • Publisher: Opentask (01 February 2009)
  • Language: English
  • Product Dimensions: 28 x 21.6
  • ISBN-13: 978-1-906717-06-3
  • Paperback: 64 pages

- Dmitry Vostokov @ DumpAnalysis.org -

Microsoft Campus in Redmond

Tuesday, June 3rd, 2008

I was visiting Microsoft in March and here are some pictures I took outside the building where I worked (click to enlarge):

 

I also visited the company store where people were buying “Microsoft Brain” toy and saw one of the thickest software engineering books (1488 pages) there. Initially I hesitated to buy it because of its price but subsequently bought it nevertheless upon my return to Ireland:

Software Engineering Foundations: A Software Science Perspective

- Dmitry Vostokov @ DumpAnalysis.org -

MDAA Volume 1 Full-Color Collector’s Edition

Tuesday, June 3rd, 2008

Full-color special edition is available now. PART 6: Fun with Crash Dumps that features memory dump visualization pictures is the most impressive there. All screenshots and diagrams are color too. The book is thicker, heavier and much more expensive. Print on demand color books are very pricey. It is only available on Lulu because my Ingram distributor, Lightning Source, doesn’t print color books with more than 480 pages:

Memory Dump Analysis Anthology Collector’s Edition, Volume 1 

- Dmitry Vostokov @ DumpAnalysis.org -

The first Memory Analysis CD album

Sunday, June 1st, 2008

Just discovered this album from Polish band Gutter Sirens on Amazon:

Memory Analysis

Buy from Amazon

I haven’t listened to it yet. Will post a review as soon as it arrives :-)

- Dmitry Vostokov @ DumpAnalysis.org -