Bugchecks: SYSTEM_SERVICE_EXCEPTION

April 22nd, 2008

Bugcheck 0×3B is forced on x64 Windows platforms when an exception happens during a system service and unwind leads to a transition from a kernel to a user mode. Let’s see this in a complete memory dump:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff80001048a1d, Address of the exception record for the exception that caused the bugcheck
Arg3: fffffade643f6870, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

CONTEXT: fffffade643f6870 -- (.cxr 0xfffffade643f6870)
rax=005300450053005c rbx=0000000000000048 rcx=0000000000000020
rdx=fffffa8007c9da20 rsi=0000000000000048 rdi=fffffade643f71d0
rip=fffff80001048a1d rsp=fffffade643f7088 rbp=0000000000000000
 r8=0000000000000048 r9=0000000000000002 r10=00490046002d0054
r11=0000000000000000 r12=fffffadf19744010 r13=fffffade643f7a78
r14=0000000000000800 r15=fffffadf1da71ee8
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202
nt!memmove+0xbd:
fffff800`01048a1d 488941e0 mov qword ptr [rcx-20h],rax ds:002b:00000000`00000000=????????????????

0: kd> kL
Child-SP RetAddr Call Site
fffffade`643f5eb8 fffff800`0104e834 nt!KeBugCheckEx
fffffade`643f5ec0 fffff800`0104e2fb nt!KiBugCheckDispatch+0x74
fffffade`643f6040 fffff800`0105c09d nt!KiSystemServiceHandler+0x7b
fffffade`643f6080 fffff800`01031561 nt!RtlpExecuteHandlerForException+0xd
fffffade`643f60b0 fffff800`010174fa nt!RtlDispatchException+0x2c0
fffffade`643f6770 fffff800`0104e92f nt!KiDispatchException+0xd9
fffffade`643f6d70 fffff800`0104d7e1 nt!KiExceptionExit
fffffade`643f6ef0 fffff800`01048a1d nt!KiPageFault+0x1e1
fffffade`643f7088 fffff800`01025977 nt!memmove+0xbd
fffffade`643f7090 fffffadf`101f858d nt!RtlAppendUnicodeStringToString+0x67
fffffade`643f70c0 fffffadf`101f8a1d driver+0x558d
fffffade`643f7a20 fffff800`012c3b21 driver+0x5a1d
fffffade`643f7a70 fffff800`012c3bd6 nt!IopXxxControlFile+0xa6b
fffffade`643f7b90 fffff800`0104e5fd nt!NtDeviceIoControlFile+0x56
fffffade`643f7c00 00000000`77ef12ca nt!KiSystemServiceCopyEnd+0×3
00000000`00e6ba08 00000000`77d67963 ntdll!ZwDeviceIoControlFile+0xa
00000000`00e6ba10 00000000`6340239f kernel32!DeviceIoControl+0×237
00000000`00e6bbf0 00000000`0000000e application!DllUnregisterServer+0×40f
[…]

On x64 Windows platforms KiSystemServiceCopyEnd has the similar purpose as KiFastSystemCallRet on x86 platforms.

We see that the chain of exception handlers spans protection boundary where KiSystemServiceCopyEnd has KiSystemServiceHandler as its exception handler:

0: kd> !exchain
100 stack frames, scanning for handlers...
Frame 0x03: nt!RtlpExecuteHandlerForException+0xd (fffff800`0105c09d)
  ehandler nt!RtlpExceptionHandler (fffff800`0105c060)
Frame 0x05: nt!KiDispatchException+0xd9 (fffff800`010174fa)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0x0a: driver+0x558d (fffffadf`101f858d)
  ehandler driver+0x1242 (fffffadf`101f4242)
Frame 0x0c: nt!IopXxxControlFile+0xa6b (fffff800`012c3b21)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0×0e: nt!KiSystemServiceCopyEnd+0×3 (fffff800`0104e5fd)
  ehandler nt!KiSystemServiceHandler (fffff800`0104e280)

Frame 0×10: kernel32!DeviceIoControl+0×237 (00000000`77d67963)
  ehandler kernel32!_C_specific_handler (00000000`77d92200)

If we disassemble KiSystemServiceHandler we get this code with bugcheck 3B branch:

kd> uf nt!KiSystemServiceHandler
nt!KiSystemServiceHandler:
[...]
fffff800`01040ddc cmp     byte ptr [rax+153h],0
fffff800`01040de3 je      nt!KiSystemServiceHandler+0×7b (fffff800`01040dfb)

nt!KiSystemServiceHandler+0x65:
fffff800`01040de5 xor     r10,r10
fffff800`01040de8 mov     r9,r8
fffff800`01040deb mov     r8,qword ptr [rcx+10h]
fffff800`01040def mov     edx,dword ptr [rcx]
fffff800`01040df1 mov     ecx,3Bh
fffff800`01040df6 call    nt!KiBugCheckDispatch (fffff800`01041300)

nt!KiSystemServiceHandler+0x7b:
fffff800`01040dfb mov     eax,1
fffff800`01040e00 add     rsp,38h
fffff800`01040e04 ret
[...]

Here we see that the code checks if the previous mode for a thread was UserMode and if this is the case it bugchecks the system because transitioning back to  user space in exception unwind would have had disastrous consequences. The system wants to save a controlled crash dump for later problem analysis:

kd> dt _KTHREAD
ntdll!_KTHREAD
   +0x000 Header           : _DISPATCHER_HEADER
   +0x018 MutantListHead   : _LIST_ENTRY
   +0x028 InitialStack     : Ptr64 Void
   +0x030 StackLimit       : Ptr64 Void
   +0x038 KernelStack      : Ptr64 Void
[...]
   +0×153 PreviousMode     : Char
[…]

Note that _KTHREAD.PreviousMode should not be confused with _KTRAP_FRAME.PreviousMode. The latter has KernelMode value if an exception happened while CPU was in kernel mode but the former structure field shows the previous CPU mode of a thread, for example, it has UserMode value if a user space thread called a system service.

kd> dt _KTRAP_FRAME
ntdll!_KTRAP_FRAME
  +0x000 P1Home : Uint8B
  +0x008 P2Home : Uint8B
  +0x010 P3Home : Uint8B
  +0x018 P4Home : Uint8B
  +0x020 P5 : Uint8B
  +0×028 PreviousMode : Char
  +0×029 PreviousIrql : UChar
  +0×02a FaultIndicator : UChar
[…]

I put all of this on a colored sequence UML diagram:

 

- Dmitry Vostokov @ DumpAnalysis.org -

What was this process doing?

April 21st, 2008

This is a common question we have when faced with stack traces for which we don’t have symbols. Consider the following stack trace from a complete memory dump where a bugcheck thread belongs to one graphical application:

2: kd> kL 100
ChildEBP RetAddr 
aa1999b4 8082d800 nt!KeBugCheckEx+0x1b
aa199d78 8088a262 nt!KiDispatchException+0x3a2
aa199de0 8088a216 nt!CommonDispatchException+0x4a
aa199e5c bfe7e5b7 nt!KiExceptionExit+0x186
[...]
aa19a110 bf8b2fe6 win32k!GrePolyPatBlt+0x45
aa19a148 bf89422b win32k!FillRect+0x58
aa19a16c bf8942f7 win32k!xxxPaintRect+0x70
aa19a19c bf8942ac win32k!xxxFillWindow+0x3e
aa19a1b4 bf8adf6e win32k!xxxDWP_EraseBkgnd+0x51
aa19a214 bf884771 win32k!xxxRealDefWindowProc+0x318
aa19a22c bf8847a1 win32k!xxxWrapRealDefWindowProc+0x16
aa19a248 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
aa19a280 8088978c win32k!NtUserMessageCall+0xc0
aa19a280 7c8285ec nt!KiFastCallEntry+0xfc
0013f68c 7739d1ec ntdll!KiFastSystemCallRet
0013f6e0 7739c6ae USER32!NtUserMessageCall+0xc
0013f6fc 7739c718 USER32!RealDefWindowProcW+0x47
0013f744 3003a5b3 USER32!DefWindowProcW+0x72
0013f75c 300a0d72 Application+0x3a5b3
0013f7bc 300a0cb2 Application+0xa0d72
0013f7f4 7739b6e3 Application+0xa0cb2
0013f820 7739b874 USER32!InternalCallWinProc+0x28
0013f898 7739c8b8 USER32!UserCallWinProcCheckWow+0x151
0013f8f4 7739c9c6 USER32!DispatchClientMessage+0xd9
0013f91c 7c828536 USER32!__fnDWORD+0x24
0013f91c 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
aa19a564 8091d6d1 nt!KiCallUserMode+0x4
aa19a5bc bf8a2622 nt!KeUserModeCallback+0x8f
aa19a640 bf8a242d win32k!SfnDWORD+0xb4
aa19a688 bf8a13d9 win32k!xxxSendMessageToClient+0x176
aa19a6d4 bf8a12ee win32k!xxxSendMessageTimeout+0x1a6
aa19a6f8 bf8c1342 win32k!xxxSendMessage+0x1b
aa19a71c bf85e0a1 win32k!xxxSendEraseBkgnd+0x5c
aa19a73c bf85dee1 win32k!xxxSimpleDoSyncPaint+0xc6
aa19a758 bf8ae16d win32k!xxxInternalDoSyncPaint+0x12
aa19a7b4 bf884771 win32k!xxxRealDefWindowProc+0x753
aa19a7cc bf8847a1 win32k!xxxWrapRealDefWindowProc+0x16
aa19a7e8 bf8c1459 win32k!NtUserfnNCDESTROY+0x27
aa19a820 8088978c win32k!NtUserMessageCall+0xc0
aa19a820 7c8285ec nt!KiFastCallEntry+0xfc
0013f91c 7c828536 ntdll!KiFastSystemCallRet
0013f91c 808308f4 ntdll!KiUserCallbackDispatcher+0x2e
aa19ab00 8091d6d1 nt!KiCallUserMode+0x4
aa19ab58 bf8a2622 nt!KeUserModeCallback+0x8f
aa19abdc bf8a242d win32k!SfnDWORD+0xb4
aa19ac24 bf8c4177 win32k!xxxSendMessageToClient+0x176
aa19ac94 bf89b829 win32k!xxxReceiveMessage+0x2b5
aa19ace4 bf89c4d9 win32k!xxxRealInternalGetMessage+0x1da
aa19ad48 8088978c win32k!NtUserPeekMessage+0x42
aa19ad48 7c8285ec nt!KiFastCallEntry+0xfc
0013fbd8 7c828536 ntdll!KiFastSystemCallRet
0013fc04 7739bde5 ntdll!KiUserCallbackDispatcher+0x2e
0013fc30 7739be5e USER32!NtUserPeekMessage+0xc
0013fc5c 3002baa0 USER32!PeekMessageW+0xab
0013fc84 3002b556 Application+0x2baa0
0013fca8 3000abf5 Application+0x2b556
0013fcf4 30005dfd Application+0xabf5
0013ff34 3000248c Application+0x5dfd
0013ffc0 77e6f23b Application+0x248c
0013fff0 00000000 kernel32!BaseProcessStart+0x23

The thread seems to be doing some drawing in response to WM_ERASEBKGND message generated from the code processing WM_TIMER:

2: kd> kv 100
[...]
aa19a6f8 bf8c1342 be63f8b8 00000014 91010979 win32k!xxxSendMessage+0×1b
aa19a71c bf85e0a1 be63f8b8 00000000 00000001 win32k!xxxSendEraseBkgnd+0×5c
[…]
0013fc5c 3002baa0 0013fcc0 00000000 00000000 USER32!PeekMessageW+0xab
[…]

2: kd> dd 0013fcc0 l4
0013fcc0  00000000 00000113 000066c2 00000000

The first parameter to PeekMessage function is a pointer to MSG structure whose second member is a message code (from MSDN): 

BOOL PeekMessage(
    LPMSG lpMsg,
    HWND hWnd,
    UINT wMsgFilterMin,
    UINT wMsgFilterMax,
    UINT wRemoveMsg
);

typedef struct {
    HWND hwnd;
    UINT message;
    WPARAM wParam;
    LPARAM lParam;
    DWORD time;
    POINT pt;
} MSG, *PMSG;

In WinUser.h we can find message codes:

#define WM_ERASEBKGND  0x0014
#define WM_TIMER       0x0113

Now we can ask the next troubleshooting question: what was the application file loaded before the system crash? We know that the application uses EXT file extension for its data. If we look at the handle table we find the only one such instance of File object:

2: kd> !handle
processor number 2, process a31a4a08
PROCESS a31a4a08  SessionId: 1  Cid: 2440    Peb: 7ffd7000  ParentCid: 1180
    DirBase: bffca720  ObjectTable: ddc38eb8  HandleCount: 291.
    Image: Application.EXE

Handle table at dcb65000 with 291 Entries in use

[...]

03f4: Object: a2ee85b0  GrantedAccess: 00120089 Entry: dcb657e8
Object: a2ee85b0  Type: (a55c8ca0) File
    ObjectHeader: a2ee8598 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \Profiles\MYNAME\LOCALS~1\Temp\APPDATA\MyFile.ext {HarddiskVolume3}

[...]

Now we can check other crash dumps to see whether there is any consistency in file names.

- Dmitry Vostokov @ DumpAnalysis.org -

What does this function do?

April 19th, 2008

Often I’m asked about what a particular function that we see on a stack trace does. Over the time I found the following function name and purpose mining techniques and resources useful:

  • - We might need to strip or replace prefixes and suffixes like

NtUserGetMessage

GetMessageW

ZwReadFile <-> NtReadFile

  • - Search in MSDN, Platform SDK and WDK (formerly DDK) help
  • - Various blogs like this excellent summary:

A catalog of NTDLL kernel mode to user mode callbacks

  • - Reverse engineering and logical deduction:

 What is KiFastSystemCallRet?

  • - Various books like this:

Windows NT/2000 Native API Reference

Buy from Amazon

  • - Win32 API emulators like WINE
  • - and finally Windows source code if you are a Microsoft source code licensee or a participant in Windows Academic Program.
  • - Sometimes Internet search finds the description of the whole stack trace collection from the class of common processes like this one:

Production Debugging for .NET Framework Applications 

- Dmitry Vostokov @ DumpAnalysis.org -

Microsoft DLL Help Database

April 17th, 2008

Just to remind about this sometimes useful resource where we can check the product and indirectly any updates for the particular module if we have its file name and version from a crash dump, for example:

http://support.microsoft.com/dllhelp/

We can event see exports and component dependencies if we need to quickly check them without running depends.exe:

- Dmitry Vostokov @ DumpAnalysis.org -

Colorimetric Computer Memory Dating (Part 1)

April 16th, 2008

Similar to radiometric dating using isotopes we can use memory visualization techniques to see distribution of allocated buffers and their retention over time. The key is to allocate colored memory. For example, to append a red buffer that contains RGBA values 0xFF000000 to specific allocations. I call these colored memory marks isomemotopes

We can either inject a different isomemotope for a different data or change the isomemotope over time to mark specific allocation times. I created a test program that allocates buffers marked by a different amount of different isomemotopes every time: 

#include "stdafx.h"
#include <stdlib.h>
#include <memory.h>
#include <windows.h>

typedef unsigned int ISOMEMOTOPE;

void *alloc_and_mark_with_isomemotope(size_t size,
                                     ISOMEMOTOPE color,
                                     size_t amount)
{
  char *p = (char *)malloc(size+amount);

  for (char *isop = p+size;
       p && isop  < p+size+amount;
       isop+=sizeof(ISOMEMOTOPE))
  {
    *(ISOMEMOTOPE *)isop=color;
  }

  return p;
}

int _tmain(int argc, _TCHAR* argv[])
{
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFF000000, // red 
                                 0x10000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x00FF0000, // green
                                 0x20000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0x0000FF00, // blue
                                 0x30000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFFFF00, // white
                                 0x40000);
  alloc_and_mark_with_isomemotope(0x1000,
                                 0xFFFF0000, // yellow
                                 0x50000);

  DebugBreak();

  return 0;
}

Corresponding Dump2Picture image is this (0×00000000 address is at the bottom):

 

:-)

- Dmitry Vostokov @ DumpAnalysis.org -

The First Windows® Memory Dump Analysis Book!

April 15th, 2008

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg as a Binary Editor

April 15th, 2008

Sometimes we have a binary file or even a text file where we want to alter some bytes but we don’t have any binary editor at hand. We can use WinDbg for this purpose. To illustrate this, I created hello.bin file with “Hello World!” in its contents. Suppose we want to change it to “Hello WinDbg!”. First, we need to open any available full process user dump file and then get the list of valid address ranges by either using !address or lm command:

0:000> lm
start             end                 module name
00000000`00400000 00000000`0044d000   TestDefaultDebugger64
00000000`77850000 00000000`77981000   kernel32
00000000`77990000 00000000`77a5a000   user32
00000000`77a60000 00000000`77bda000   ntdll
000007fe`f8940000 000007fe`f8997000   winspool
000007fe`fcb00000 000007fe`fccf0000   comctl32
000007fe`fcfc0000 000007fe`fd012000   uxtheme
000007fe`fe1d0000 000007fe`fe2d4000   msctf
000007fe`fe380000 000007fe`fe3f1000   shlwapi
000007fe`fe660000 000007fe`fe799000   rpcrt4
000007fe`fe9f0000 000007fe`feac8000   oleaut32
000007fe`fead0000 000007fe`ff704000   shell32
000007fe`ff880000 000007fe`ff91a000   usp10
000007fe`ff920000 000007fe`ff9c1000   msvcrt
000007fe`ff9d0000 000007fe`ff9f8000   imm32
000007fe`ffa00000 000007fe`ffbe0000   ole32
000007fe`ffbe0000 000007fe`ffcdf000   advapi32
000007fe`ffce0000 000007fe`ffcec000   lpk
000007fe`ffcf0000 000007fe`ffd51000   gdi32

Let’s choose 00000000`00400000 address. It points to the following memory data:

0:000> dc 00000000`00400000
00000000`00400000  00905a4d 00000003 00000004 0000ffff  MZ..............
00000000`00400010  000000b8 00000000 00000040 00000000  ........@.......
00000000`00400020  00000000 00000000 00000000 00000000  ................
00000000`00400030  00000000 00000000 00000000 000000e8  ................
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ........!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode....$.......

Now we load our hello.bin by specifying the this address and the number of bytes to load:

0:000> .readmem c:\dmitri\hello.bin 00000000`00400000 L0n12
Reading c bytes.

We see the new memory data immediately:

0:000> dc 00000000`00400000
00000000`00400000  6c6c6548 6f57206f 21646c72 0000ffff  Hello World!….
00000000`00400010  000000b8 00000000 00000040 00000000  ……..@…….
00000000`00400020  00000000 00000000 00000000 00000000  …………….
00000000`00400030  00000000 00000000 00000000 000000e8  …………….
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ……..!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode….$…….

Then we can change it immediately using any of e* commands: 

0:000> ea 00000000`00400000+6 "WinDbg!"

0:000> dc 00000000`00400000
00000000`00400000  6c6c6548 6957206f 6762446e 2100ffff  Hello WinDbg!
00000000`00400010  000000b8 00000000 00000040 00000000  ……..@…….
00000000`00400020  00000000 00000000 00000000 00000000  …………….
00000000`00400030  00000000 00000000 00000000 000000e8  …………….
00000000`00400040  0eba1f0e cd09b400 4c01b821 685421cd  ……..!..L.!Th
00000000`00400050  70207369 72676f72 63206d61 6f6e6e61  is program canno
00000000`00400060  65622074 6e757220 206e6920 20534f44  t be run in DOS
00000000`00400070  65646f6d 0a0d0d2e 00000024 00000000  mode….$…….

Alternatively we can use GUI memory editor:

Now we can write memory contents back to our file:

0:000> .writemem c:\dmitri\hello.bin 00000000`00400000 L0n13
Writing d bytes.

- Dmitry Vostokov @ DumpAnalysis.org -

Final Back Cover for MDAA V1

April 14th, 2008

To avoid controversial pictures I decided to put an image of TestDefaultDebugger crash dump generated by Dump2Picture:

Final Back Cover for Memory Dump Analysis Anthology, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -

Controversial Book Cover?

April 13th, 2008

Some people commented that by placing an image of a complete memory dump on the back cover of a book both violates copyright and intellectual property rights, as the picture is generated from copyrighted material. Instead they suggested to put a picture of a freeware program. Here is my response:

I disagree to the best of my understanding. This picture is just the visualized physical memory for illustration purposes only. What about disassembling a function to illustrate a bug? Or dumping memory, for example, a thread structure? Or printing a screenshot from Performance Monitor or Task Manager to illustrate CPU spike? Or a stack trace from a complete memory dump? Does it violate copyright and intellectual property rights because it is generated from copyrighted material? What about the front cover then, showing book spines of hundreds of copyrighted books? If Microsoft asks me to remove the picture, certainly, I’ll do it and reprint the book. And, surely, a memory dump of a freeware program will definitely contain portions of copyrighted material, like ntdll.dll, kernel32.dll or accidental 3rd-party hooks. Regarding a complete memory dump copyrighted material might have been paged out from physical memory and not included in file contents. Do you admit that printing a CRC number violates property rights because it was generated from copyrighted material? Due to the mathematical nature of involved algorithms it is not possible to reconstruct binary code from the printed cover picture which could have been created artificially as well.

What do you think?

- Dmitry Vostokov @ DumpAnalysis.org -

Final TOC for MDAA Volume 1

April 12th, 2008

I’ve posted the final Table of Contents and additional information for the soon-to-be-published book:

Memory Dump Analysis Anthology, Volume 1

Note that the proposed back cover image is the picture of a 1Gb complete physical memory dump generated by Dump2Picture:

Back Cover for Memory Dump Analysis Anthology, Volume 1

- Dmitry Vostokov @ DumpAnalysis.org -

CDF Traces: Using Performance Analyzer

April 12th, 2008

Because Citrix CDF traces are ETW based it should be possible to process and analyze them using XPrerf and XPerfView from the latest Microsoft Windows Performance Tools:

Windows Performance Tools Kit, v.4.1.1

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 58a)

April 9th, 2008

Among various memory leaks leading to Insufficient Memory pattern there is so called session pool leak briefly touched in the previous kernel pool leaks post. I decided to factor it out separately because it also involves GDI handles and structures allocated per user session that has the limit on how many of them can be created and this case should rather be called Handle Limit. Such leaks can result in poor visual application behavior after some time when drawing requests are not satisfied anymore. In severe cases, when the same bugs are present in a display driver, it can result in bugchecks like

BugCheck AB: SESSION_HAS_VALID_POOL_ON_EXIT

or, if a handle allocation request was not satisfied, it may result in a NULL pointer stored somewhere with the subsequent Invalid Pointer access:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)

CONTEXT:  b791e010 -- (.cxr 0xffffffffb791e010)
eax=00000000 ebx=bc43d004 ecx=a233add8 edx=00000000 esi=bc430fff edi=00000000
eip=bfe7d380 esp=b791e3dc ebp=b791e480 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
DisplayDriver+0×3e380:
bfe7d380 8a4702          mov     al,byte ptr [edi+2]        ds:0023:00000002=??

Rather than providing examples from specific kernel dump files I created 3 Win32 applications in Visual C++ that simulate GDI leaks. All of them create GDI objects in a loop and select them into their current graphics device context (DC) on Windows Server 2003 x64 SP2. Before running them I got the following session paged pool statistics:

lkd> !poolused c

   Sorting by Session Paged Pool Consumed

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       257  3629056 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 NV          0        0       203  1347648 nVidia video driver
 Ttfd        0        0       233  1053152 TrueType Font driver
 Gh05        0        0       391  1050400 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla1        0        0       348   785088 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gcac        0        0        25   640880 Gdi glyph cache
 Gla5        0        0       631   323072 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gdrs        0        0        33   172288 Gdi GDITAG_DRVSUP
 Gla:        0        0       212   139072 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla4        0        0       487   116880 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Usti        0        0       148    97088 THREADINFO , Binary: win32k!AllocateW32Thread
 Gla8        0        0       383    91920 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla@        0        0       339    70512 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gbaf        0        0        48    67584 UNKNOWN pooltag ‘Gbaf’, please update pooltag.txt
 knlf        0        0        20    66496 UNKNOWN pooltag ‘knlf’, please update pooltag.txt
 GDev        0        0         7    57344 Gdi pdev
 Usqu        0        0       152    53504 Q , Binary: win32k!InitQEntryLookaside
 Uscu        0        0       334    53440 CURSOR , Binary: win32k!_CreateEmptyCursorObject
 Bmfd        0        0        21    50224 Font related stuff
 Uspi        0        0       153    40000 PROCESSINFO , Binary: win32k!MapDesktop
 Gfnt        0        0        47    39856 UNKNOWN pooltag ‘Gfnt’, please update pooltag.txt
 Ggb         0        0        34    39088 Gdi glyph bits
 Gh08        0        0        33    38656 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ghab        0        0       228    32832 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ovfl        0        0         1    32768 The internal pool tag table has overflowed - usually this is a result of nontagged allocations being made
 Gpff        0        0        88    27712 Gdi physical font file
 Gpfe        0        0        88    27600 UNKNOWN pooltag ‘Gpfe’, please update pooltag.txt
 thdd        0        0         1    20480 DirectDraw/3D handle manager table
 Gebr        0        0        17    19776 Gdi ENGBRUSH
 Gh0@        0        0        86    19264 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gsp         0        0        79    18960 Gdi sprite
 HT40        0        0         2    16384 UNKNOWN pooltag ‘HT40′, please update pooltag.txt
 Gpat        0        0         4    16192 UNKNOWN pooltag ‘Gpat’, please update pooltag.txt
 Ggls        0        0       169    12944 Gdi glyphset
 Glnk        0        0       371    11872 Gdi PFELINK
 Gldv        0        0         9    11248 Gdi Ldev
 Gffv        0        0        84     9408 Gdi FONTFILEVIEW
 Gfsb        0        0         1     8192 Gdi font sustitution list
 Uskt        0        0         2     7824 KBDTABLE , Binary: win32k!ReadLayoutFile
 Gh04        0        0         7     5856 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gdcf        0        0        51     5712 UNKNOWN pooltag ‘Gdcf’, please update pooltag.txt
 Gh0<        0        0        88     5632 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gglb        0        0         1     4096 Gdi temp buffer
 Ustm        0        0        30     3360 TIMER , Binary: win32k!InternalSetTimer
 Gspm        0        0        39     3120 UNKNOWN pooltag ‘Gspm’, please update pooltag.txt
 Usac        0        0        16     3056 ACCEL , Binary: win32k!_CreateAcceleratorTable
 Usqm        0        0        25     2800 QMSG , Binary: win32k!InitQEntryLookaside
 Ghas        0        0         3     2592 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Uscl        0        0        20     2128 CLASS , Binary: win32k!ClassAlloc
 Uswl        0        0         1     2032 WINDOWLIST , Binary: win32k!BuildHwndList
 Gmul        0        0        19     1520 UNKNOWN pooltag ‘Gmul’, please update pooltag.txt
 Dddp        0        0         8     1472 UNKNOWN pooltag ‘Dddp’, please update pooltag.txt
 Ggdv        0        0         8     1472 Gdi GDITAG_GDEVICE
 UsDI        0        0         4     1408 DEVICEINFO , Binary: win32k!CreateDeviceInfo
 Vtfd        0        0         4     1312 Font file/context
 Ushk        0        0        20     1280 HOTKEY , Binary: win32k!_RegisterHotKey
 Gspr        0        0         3     1264 Gdi sprite grow range
 Gtmw        0        0        13     1248 Gdi TMW_INTERNAL
 Gxlt        0        0         8     1152 Gdi Xlate
 Gpft        0        0         2      944 Gdi font table
 Uspp        0        0         5      944 PNP , Binary: win32k!AllocateAndLinkHidTLCInf
 Ussm        0        0         7      896 SMS , Binary: win32k!InitSMSLookaside
 Gdbr        0        0        10      800 Gdi driver brush realization
 Usdc        0        0         8      768 DCE , Binary: win32k!CreateCacheDC
 Usct        0        0        12      768 CHECKPT , Binary: win32k!CkptRestore
 Usim        0        0         2      736 IME , Binary: win32k!CreateInputContext
 Usci        0        0         3      720 CLIENTTHREADINFO , Binary: win32k!InitSystemThread
 Gh09        0        0         1      640 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ussy        1       80         4      608 SYSTEM , Binary: win32k!xxxDesktopThread
 Urdr        0        0         9      576 REDIRECT , Binary: win32k!SetRedirectionBitmap
 Uswd        0        0         2      576 WINDOW , Binary: win32k!xxxCreateWindowEx
 Uscb        0        0         3      544 CLIPBOARD , Binary: win32k!_ConvertMemHandle
 Gcsl        0        0         1      496 Gdi string resource script names
 Ustx        0        0        10      496 TEXT , Binary: win32k!NtUserDrawCaptionTemp
 Ussw        0        0         1      496 SWP , Binary: win32k!_BeginDeferWindowPos
 Gdev        0        0         2      480 Gdi GDITAG_DEVMODE
 Usih        0        0        10      480 IMEHOTKEY , Binary: win32k!SetImeHotKey
 Gdrv        0        0         1      368 UNKNOWN pooltag ‘Gdrv’, please update pooltag.txt
 GVdv        0        0         1      320 UNKNOWN pooltag ‘GVdv’, please update pooltag.txt
 Gmap        0        0         1      320 Gdi font map signature table
 Uskb        0        0         2      288 KBDLAYOUT , Binary: win32k!xxxLoadKeyboardLayoutEx
 Uskf        0        0         2      288 KBDFILE , Binary: win32k!LoadKeyboardLayoutFile
 Uswe        0        0         2      224 WINEVENT , Binary: win32k!_SetWinEventHook
 Gddf        0        0         2      224 Gdi ddraw driver heaps
 Gddv        0        0         2      192 Gdi ddraw driver video memory list
 GFil        0        0         2      192 Gdi engine descriptor list
 Gdwd        0        0         2       96 Gdi watchdog support objects , Binary: win32k.sys
 Usd9        0        0         1       80 DDE9 , Binary: win32k!xxxCsDdeInitialize
 Gvds        0        0         1       64 UNKNOWN pooltag ‘Gvds’, please update pooltag.txt
 GreA        0        0         1       64 UNKNOWN pooltag ‘GreA’, please update pooltag.txt
 Usse        0        0         1       48 SECURITY , Binary: win32k!SetDisconnectDesktopSecu
 Usvl        0        0         1       48 VWPL , Binary: win32k!VWPLAdd
 Mdxg        1      112         0        0 UNKNOWN pooltag ‘Mdxg’, please update pooltag.txt
 Gini        3      128         0        0 Gdi fast mutex
 Usev        1       64         0        0 EVENT , Binary: win32k!xxxPollAndWaitForSingleO
 Gdde        3      240         0        0 Gdi ddraw event
 TOTAL           9      624      6256 24408704

The first application leaks fonts:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
     hdc = BeginPaint(hWnd, &ps);
     while (true)
     {
        HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
        SelectObject(ps.hdc, hf);
     }
     EndPaint(hWnd, &ps);
     break;

We clearly see the leak as the greatly increased number of allocations for ”Gla:” pool tag:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 Gla:        0        0     10194  6687264 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 BIG         0        0       248  3690496 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 NV          0        0       203  1347648 nVidia video driver
 Gh05        0        0       396  1057888 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ttfd        0        0       226  1043264 TrueType Font driver

The second application leaks fonts and pens:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
     hdc = BeginPaint(hWnd, &ps);
     while (true)
     {
       HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
       HPEN hp = CreatePen(0, 10, RGB(10, 20, 30));
       SelectObject(ps.hdc, hf);
       SelectObject(ps.hdc, hp);
     }
     EndPaint(hWnd, &ps);
     break;

We see that the roughly the same number of allocations is split between ”Gla:” and “Gh0@” pool tags:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       262  3874816 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 Gla:        0        0      5203  3413168 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 NV          0        0       203  1347648 nVidia video driver
 Gh0@        0        0      5077  1137248 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Ttfd        0        0       233  1053152 TrueType Font driver

The third program leaks fonts, pens and brushes:

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
 int wmId, wmEvent;
 PAINTSTRUCT ps;
 HDC hdc;

 switch (message)
 {
   case WM_PAINT:
   hdc = BeginPaint(hWnd, &ps);
   while (true)
   {
     HFONT hf = CreateFont(10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, L"Arial");
     HPEN hp = CreatePen(0, 10, RGB(10, 20, 30));
     HBRUSH hb = CreateSolidBrush(RGB(10, 20, 30));
     SelectObject(ps.hdc, hf);
     SelectObject(ps.hdc, hp);
     SelectObject(ps.hdc, hb);
   }
   EndPaint(hWnd, &ps);
   break;

Now we see that the same number of allocations is almost equally split between ”Gla:”, “Gh0@” and “Gla@” pool tags:

  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 NV_x        0        0         5 14024704 UNKNOWN pooltag 'NV_x', please update pooltag.txt
 BIG         0        0       262  3874816 Large session pool allocations (ntos\ex\pool.c) , Binary: nt!mm
 Gla:        0        0      3539  2321584 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 NV          0        0       203  1347648 nVidia video driver
 Ttfd        0        0       233  1053152 TrueType Font driver
 Gh05        0        0       392  1052768 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla1        0        0       353   796368 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gh0@        0        0      3414   764736 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gla@        0        0      3665   762320 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys
 Gcac        0        0        25   640880 Gdi glyph cache

When the certain amount of handles is reached all subsequent GDI Create calls fail and other applications start showing various visual defects. Print screen operation also fails with insufficient memory message.
                                                                                                                                                                                                        
- Dmitry Vostokov @ DumpAnalysis.org -

Jobs: Citrix Global Escalation Engineer

April 5th, 2008

I posted a job ad on the portal:

Citrix Escalation Engineer (Dublin, Ireland)

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 57)

April 3rd, 2008

Another pattern that occurs frequently is Hardware Error. This can be internal CPU malfunction due to overheating, RAM or hard disk I/O problem. It usually results in the appropriate bugcheck and the most frequent one is the 6th from the top of Bug Check Frequency Table:

  • BugCheck 9C: MACHINE_CHECK_EXCEPTION

Other relevant bugchecks include:

  • BugCheck 7B: INACCESSIBLE_BOOT_DEVICE

  • BugCheck 77: KERNEL_STACK_INPAGE_ERROR

  • BugCheck 7A: KERNEL_DATA_INPAGE_ERROR

Another bugcheck from this category can also be triggered on purpose to get a crash dump of a hanging or slow system:

Please also note that other popular bugchecks like  

  • BugCheck 7F: UNEXPECTED_KERNEL_MODE_TRAP

  • BugCheck 50: PAGE_FAULT_IN_NONPAGED_AREA

can result from RAM problems but we should try to find a software cause first.

Sometimes the following bugchecks like

  • BugCheck 7E: SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

report EXCEPTION_DOESNOT_MATCH_CODE where read or write address doesn’t correspond to faulted instruction at EIP:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: bf802671, The address that the exception occurred at
Arg3: f10b8c74, Exception Record Address
Arg4: f10b88c4, Context Record Address

FAULTING_IP:
driver!AcquireSemaphoreShared+4
bf802671 90 nop

EXCEPTION_RECORD: f10b8c74 -- (.exr fffffffff10b8c74)
ExceptionAddress: bf802671 (driver!AcquireSemaphoreShared+0x00000004)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 0000000c
Attempt to write to address 0000000c

CONTEXT: f10b88c4 -- (.cxr fffffffff10b88c4)
eax=884d2d01 ebx=0000000c ecx=00000000 edx=80010031 esi=8851ef60 edi=bc3846d4
eip=bf802671 esp=f10b8d3c ebp=f10b8d70 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206
driver!AcquireSemaphoreShared+0x4:
bf802671 90 nop
Resetting default scope

WRITE_ADDRESS: 0000000c

EXCEPTION_DOESNOT_MATCH_CODE: This indicates a hardware error.
Instruction at bf802671 does not read/write to 0000000c

Code mismatch can also happen in user mode but from my experience it usually results from improper Hooked Function or similar corruption: 

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c848768 (ntdll!_LdrpInitialize+0x00000184)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 0

DEFAULT_BUCKET_ID: CODE_ADDRESS_MISMATCH

WRITE_ADDRESS: f774f120

FAULTING_IP:
ntdll!_LdrpInitialize+184
7c848768 cc int 3

EXCEPTION_DOESNOT_MATCH_CODE: This indicates a hardware error.
Instruction at 7c848768 does not read/write to f774f120

STACK_TEXT:
0012fd14 7c8284c5 0012fd28 7c800000 00000000 ntdll!_LdrpInitialize+0x184
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x25

In such cases EIP might point to the middle of the expected instruction (Wild Code):

FAULTING_IP:
+59c3659
059c3659 86990508f09b xchg bl,byte ptr [ecx-640FF7FBh]

Here is an example of the real hardware error (note the concatenated error code for bugcheck 0×9C):

MACHINE_CHECK_EXCEPTION (9c)
A fatal Machine Check Exception has occurred.
KeBugCheckEx parameters;
    x86 Processors
        If the processor has ONLY MCE feature available (For example Intel
        Pentium), the parameters are:
        1 - Low  32 bits of P5_MC_TYPE MSR
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of P5_MC_ADDR MSR
        4 - Low  32 bits of P5_MC_ADDR MSR
        If the processor also has MCA feature available (For example Intel
        Pentium Pro), the parameters are:
        1 - Bank number
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
        4 - Low  32 bits of MCi_STATUS MSR for the MCA bank that had the error
    IA64 Processors
        1 - Bugcheck Type
            1 - MCA_ASSERT
            2 - MCA_GET_STATEINFO
                SAL returned an error for SAL_GET_STATEINFO while processing MCA.
            3 - MCA_CLEAR_STATEINFO
                SAL returned an error for SAL_CLEAR_STATEINFO while processing MCA.
            4 - MCA_FATAL
                FW reported a fatal MCA.
            5 - MCA_NONFATAL
                SAL reported a recoverable MCA and we don't support currently
                support recovery or SAL generated an MCA and then couldn't
                produce an error record.
            0xB - INIT_ASSERT
            0xC - INIT_GET_STATEINFO
                  SAL returned an error for SAL_GET_STATEINFO while processing INIT event.
            0xD - INIT_CLEAR_STATEINFO
                  SAL returned an error for SAL_CLEAR_STATEINFO while processing INIT event.
            0xE - INIT_FATAL
                  Not used.
        2 - Address of log
        3 - Size of log
        4 - Error code in the case of x_GET_STATEINFO or x_CLEAR_STATEINFO
    AMD64 Processors
        1 - Bank number
        2 - Address of MCA_EXCEPTION structure
        3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
        4 - Low  32 bits of MCi_STATUS MSR for the MCA bank that had the error
Arguments:
Arg1: 00000000
Arg2: 808a07a0
Arg3: be000300
Arg4: 1008081f

Debugging Details:
------------------

   NOTE:  This is a hardware error.  This error was reported by the CPU
   via Interrupt 18.  This analysis will provide more information about
   the specific error.  Please contact the manufacturer for additional
   information about this error and troubleshooting assistance.

   This error is documented in the following publication:

      - IA-32 Intel(r) Architecture Software Developer's Manual
        Volume 3: System Programming Guide

   Bit Mask:

    MA                           Model Specific       MCA
 O  ID      Other Information      Error Code     Error Code
VV  SDP ___________|____________ _______|_______ _______|______
AEUECRC|                        |               |             
LRCNVVC|                        |               |             
^^^^^^^|                        |               |              
   6         5         4         3         2         1
3210987654321098765432109876543210987654321098765432109876543210
----------------------------------------------------------------
1011111000000000000000110000000000010000000010000000100000011111 

VAL   - MCi_STATUS register is valid
        Indicates that the information contained within the IA32_MCi_STATUS
        register is valid.  When this flag is set, the processor follows the
        rules given for the OVER flag in the IA32_MCi_STATUS register when
        overwriting previously valid entries.  The processor sets the VAL
        flag and software is responsible for clearing it.

UC    - Error Uncorrected
        Indicates that the processor did not or was not able to correct the
        error condition.  When clear, this flag indicates that the processor
        was able to correct the error condition.

EN    - Error Enabled
        Indicates that the error was enabled by the associated EEj bit of the
        IA32_MCi_CTL register.

MISCV - IA32_MCi_MISC Register Valid
        Indicates that the IA32_MCi_MISC register contains additional
        information regarding the error.  When clear, this flag indicates
        that the IA32_MCi_MISC register is either not implemented or does
        not contain additional information regarding the error.

ADDRV - IA32_MCi_ADDR register valid
        Indicates that the IA32_MCi_ADDR register contains the address where
        the error occurred.

PCC   - Processor Context Corrupt
        Indicates that the state of the processor might have been corrupted
        by the error condition detected and that reliable restarting of the
        processor may not be possible.

BUSCONNERR - Bus and Interconnect Error   BUS{LL}_{PP}_{RRRR}_{II}_{T}_err
        These errors match the format 0000 1PPT RRRR IILL

   Concatenated Error Code:
   --------------------------
   _VAL_UC_EN_MISCV_ADDRV_PCC_BUSCONNERR_1F

   This error code can be reported back to the manufacturer.
   They may be able to provide additional information based upon
   this error.  All questions regarding STOP 0x9C should be
   directed to the hardware manufacturer.

BUGCHECK_STR:  0x9C_IA32_GenuineIntel

DEFAULT_BUCKET_ID:  DRIVER_FAULT

PROCESS_NAME:  Idle

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 80a7fbd8 to 8087b6be

STACK_TEXT: 
f773d280 80a7fbd8 0000009c 00000000 f773d2b0 nt!KeBugCheckEx+0x1b
f773d3b4 80a7786f f7737fe0 00000000 00000000 hal!HalpMcaExceptionHandler+0x11e
f773d3b4 f75a9ca2 f7737fe0 00000000 00000000 hal!HalpMcaExceptionHandlerWrapper+0x77
f78c6d50 8083abf2 00000000 0000000e 00000000 intelppm!AcpiC1Idle+0x12
f78c6d54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0xa

- Dmitry Vostokov @ DumpAnalysis.org -

Updated: Selected Citrix Tools

April 2nd, 2008

The last year presentation has been updated and now available in both HTML and PDF formats:

Selected Citrix Troubleshooting and Debugging Tools (HTML)

Selected Citrix Troubleshooting and Debugging Tools (PDF)

- Dmitry Vostokov @ DumpAnalysis.org -

The Inception of Debugging Studio

April 2nd, 2008

Born out of 2008 April Fool’s joke (although I was thinking about it long before) the idea is to have a general framework and associated IDE for analysis activities for various tools that produce logs. Similar to planned GUI Monitor integrating WindowHistory, MesageHistory and ScreenHistory tools but also including Debugger Log Analyzer, Debugging Notebooks, various memory and trace log visualization subsystems. It also has the working title UTF (Unified Tools Framework).

Seems I’m very easy on inceptions :-) All this will be covered in great detail in the forthcoming book which I tirelessly advertise:

DebugWare: The Art and Craft of Writing Troubleshooting Tools

- Dmitry Vostokov @ DumpAnalysis.org -

Microsoft gives sneak preview of Debugging Studio

April 1st, 2008

This looks really cool:

http://www.microsoft.com/whdc/devstudio/default.mspx 

- Dmitry Vostokov @ DumpAnalysis.org -

CDF Traces: Analyzing Process Launch Sequence

March 31st, 2008

Citrix CDF traces are based on ETW (Event Tracing for Windows) and therefore Citrix customers, their support personnel and developers can use MS TraceView tool for troubleshooting Citrix terminal service environments:

Viewing Common Diagnostics Facility (CDF) Traces Using TraceView

In cases with slow logon or slow process startup we can analyze process launch sequence to determine delays. In the output trace we can filter tzhook module messages which also contain session id (this is quite handy to differentiate between different sessions), for example: 

PID      TID      TIME          MESSAGE     
21864    21912    06:34:53.598  tzhook: Attach on process - cmd.exe session=51         
21620    20372    06:34:59.754  tzhook: Attach on process - acregl.exe session=51         
18668    21240    06:35:02.704  tzhook: Attach on process - cmstart.exe session=51         
18560    18832    06:35:02.735  tzhook: Attach on process - wfshell.exe session=51         
18204    20060    06:35:06.575  tzhook: Attach on process - icast.exe session=51         
20640    21104    06:35:07.717  tzhook: Attach on process - LOGON.EXE session=51         
21188    21032    06:35:07.938  tzhook: Attach on process - cscript.exe session=51         
21888    19592    06:35:11.157  tzhook: Attach on process - WScript.exe session=51         
20600    20732    06:35:11.780  tzhook: Attach on process - admin.exe session=51         
17976    20456    06:35:18.752  tzhook: Attach on process - winlogon.exe session=53         
21332    13156    06:35:51.625  tzhook: Attach on process - mpnotify.exe session=53         
10988    10732    06:35:57.043  tzhook: Attach on process - rundll32.exe session=53

Here is another process launch sequence for published Notepad application:

PID     TID     TIME            MESSAGE
15828   18720   15:34:02.637    tzhook: Attach on process - winlogon.exe session=2 
5300    18508   15:34:03.043    tzhook: Attach on process - mpnotify.exe session=2 
17948   19300   15:34:03.356    tzhook: Attach on process - userinit.exe session=2 
17956   19316   15:34:03.415    tzhook: Attach on process - cmd.exe session=2 
5384    5324    15:34:03.443    tzhook: Attach on process - cmd.exe session=2 
19432   19264   15:34:03.461    tzhook: Attach on process - SSONSVR.EXE session=2 
12480   7472    15:34:03.466    tzhook: Attach on process - cmd.exe session=2 
19448   19364   15:34:03.474    tzhook: Attach on process - net.exe session=2 
19416   19656   15:34:03.489    tzhook: Attach on process - acregl.exe session=2 
19480   19596   15:34:03.544    tzhook: Attach on process - cmstart.exe session=2 
664     19512   15:34:03.559    tzhook: Attach on process - wfshell.exe session=2 
19904   13140   15:34:03.610    tzhook: Attach on process - net.exe session=2 
6864    20036   15:34:03.746    tzhook: Attach on process - icast.exe session=2 
19540   20016   15:34:03.749    tzhook: Attach on process - ctfmon.exe session=2 
19944   19032   15:34:03.757    tzhook: Attach on process - net.exe session=2 
10232   18356   15:34:03.787    tzhook: Attach on process - notepad.exe session=2

Such sequences are also useful to determine a process upon which the session initialization or startup sequence hangs. In this case a user dump of that process might be useful.

Of course we can do all this with Process Monitor and other similar tools but here we get other Citrix related trace messages as well. All in one.

- Dmitry Vostokov @ DumpAnalysis.org -

LiterateScientist and ManagementBits update (March, 2008)

March 31st, 2008

I was very busy this month with the forthcoming Memory Dump Analysis Anthology Volume 1 plus I had a business trip to Redmond and therefore I really didn’t have enough time to contribute well to other my blogs. Nevertheless I finished reading Incompleteness book during my transatlantic flights, started reading two others and here is the small update:

LiterateScientist Blog:

Incompleteness: The Proof and Paradox of Kurt Godel

The Philosophers Toolkit

ManagementBits Blog:

The Science of Career Promotions

Management Bit and Tip 0×200

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Forum has been reopened!

March 30th, 2008

This is an attempt to revive the forum closed a year ago due to the lack of interest during that time. Now the number of visitors has increased by an order of magnitude and I get more and more requests to help with crash dump analysis but I cannot guarantee immediate response. Therefore I upgraded and reopened Crash Dump Analysis and Debugging Forum and encourage using it. In order to prevent spammers gaining access any new registration has to be approved by me (usually in less than 24 hours).

- Dmitry Vostokov @ DumpAnalysis.org -