Software Diagnostics Library

ManagementBits Blog:

Management Bit and Tip 0×400

Managing Reading via Cooperative Multireading

LiterateScientist Blog:

The Science of Sherlock Holmes

Is it Safe to Eat?

- Dmitry Vostokov @ DumpAnalysis.org -

Just discovered that priority goes to Mikel Lasa

Memory Dump: 1960-1990 (Poesía vasca, hoy)

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes certain files are opened but not closed when not needed and this prevents other applications and users from using, deleting or replacing them. Sometimes on behalf of certain API calls another process opens them. One of the questions I was asked recently is how to find that process. The answer that I found is very simple: just list all handles from all processes and search for that file name there. This works in kernel and complete memory dumps and also in live kernel debugging session including its local kernel debugging variant. The following WinDbg command lists all objects (we need to open a log because the output is usually of several megabytes):

3: kd> !for_each_process "!handle 0 3 @#Process"

!handle 0 3 @#Process
processor number 3, process 8a392818
PROCESS 8a392818  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: bfc51000  ObjectTable: e1001e00  HandleCount: 2650.
    Image: System

Handle table at e16a3000 with 2650 Entries in use
0004: Object: 8a392818  GrantedAccess: 001f0fff Entry: e1004008
Object: 8a392818  Type: (8a392e70) Process
    ObjectHeader: 8a392800 (old version)
        HandleCount: 3  PointerCount: 235

0008: Object: 8a391db0  GrantedAccess: 00000000 Entry: e1004010
Object: 8a391db0  Type: (8a392ca0) Thread
    ObjectHeader: 8a391d98 (old version)
        HandleCount: 1  PointerCount: 1

000c: Object: e101bca0  GrantedAccess: 00000000 Entry: e1004018
Object: e101bca0  Type: (8a37db00) Key
    ObjectHeader: e101bc88 (old version)
        HandleCount: 1  PointerCount: 3
        Directory Object: 00000000  Name: \REGISTRY

[...]

1fac: Object: 88ec72b0  GrantedAccess: 00000003 (Protected) Entry: e1ed7f58
Object: 88ec72b0  Type: (8a36f900) File
    ObjectHeader: 88ec7298 (old version)
        HandleCount: 1  PointerCount: 2
        Directory Object: 00000000  Name: \Documents and Settings\MyUserName\NTUSER.DAT {HarddiskVolume1}

[...]

07fc: Object: e1000768  GrantedAccess: 00000003 Entry: e2fefff8
Object: e1000768  Type: (8a387e70) KeyedEvent
    ObjectHeader: e1000750 (old version)
        HandleCount: 273  PointerCount: 274
        Directory Object: e1001a48  Name: CritSecOutOfMemoryEvent

processor number 3, process 8873f3b8
PROCESS 8873f3b8  SessionId: 6  Cid: 4c1c    Peb: 7ffdf000  ParentCid: 42bc
    DirBase: 49dbb000  ObjectTable: e325f788  HandleCount:  90.
    Image: PROFLWIZ.EXE

Handle table at e36c3000 with 90 Entries in use
0004: Object: e1000768  GrantedAccess: 00000003 Entry: e36c3008
Object: e1000768  Type: (8a387e70) KeyedEvent
    ObjectHeader: e1000750 (old version)
        HandleCount: 273  PointerCount: 274
        Directory Object: e1001a48  Name: CritSecOutOfMemoryEvent

[...]

- Dmitry Vostokov @ DumpAnalysis.org -

Following the introductory division of memory analysis into two broad categories I decided to plan yet another book with the following title and preliminary product details:

• Title: Memory Analysis Forensics and Intelligence: An Integral Approach
• Author: Dmitry Vostokov
• Publisher: Opentask (01 September 2009)
• Language: English
• Product Dimensions: 22.86 x 15.24
• ISBN-13: 978-1-906717-05-6
• Paperback: 256 pages

As you might have noticed, I prefer to put 128, 256 or 512 pages in my book announcements. What that would say about the author’s background? I would like to set 1024 pages for my memory dump analysis anthology series but 740 or 800 pages is the limit for POD technology I use.

- Dmitry Vostokov @ DumpAnalysis.org -

Software ageing can be the cause of the problem. Sometimes a look at the following WinDbg output can give irresistible temptation to suggest periodic reboots:

Debug session time: Wed April 28 15:36:52.330 2008 (GMT+0)
System Uptime: 124 days 6:27:16.658

The suggested pattern name is Overaged System.

- Dmitry Vostokov @ DumpAnalysis.org -

While working on WDN book I noticed that it is possible to analyze managed code in complete memory dumps. We just need to switch to the process in question and load SOS DLL (if memory dumps are from 64-bit Windows we need to run 64-bit WinDbg because it needs to load 64-bit SOS from Microsoft.NET \ Framework64 \ vX.X.XXXXX folder).

Here is some command output from a complete memory dump generated from SystemDump when running TestDefaultDebugger.NET application where we try to find and disassemble IL code for this function:

namespace WindowsApplication1
{
  public partial class Form1 : Form
  {
[...]
  private void button1_Click(object sender, EventArgs e)
  {
  System.Collections.Stack stack = new System.Collections.Stack();stack.Pop();
  }
  }
}

Loading Dump File [C:\W2K3\MEMORY_NET.DMP]
Kernel Complete Dump File: Full address space is available

Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 (Service Pack 2) UP Free x64
Product: Server, suite: Enterprise TerminalServer
Built by: 3790.srv03_sp2_gdr.070321-2337
Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`01198b00

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
[...]
PROCESS fffffadfe7287610
    SessionId: 0  Cid: 0ad8    Peb: 7fffffdf000  ParentCid: 0acc
    DirBase: 12cd9000  ObjectTable: fffffa800163c260  HandleCount: 114.
    Image: TestDefaultDebugger.NET.exe

PROCESS fffffadfe67905a0
    SessionId: 0  Cid: 085c    Peb: 7fffffd4000  ParentCid: 0acc
    DirBase: 232e2000  ObjectTable: fffffa8000917e10  HandleCount:  55.
    Image: SystemDump.exe

kd> .process /r /p fffffadfe7287610
Implicit process is now fffffadf`e7287610
Loading User Symbols

kd> .loadby sos mscorwks

kd> !threads

ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                              PreEmptive                                                Lock
       ID OSID        ThreadOBJ     State   GC     GC Alloc Context                  Domain           Count APT Exception
       1  a94 0000000000161150      6020 Enabled  0000000000000000:0000000000000000 000000000014ccb0     0 STA
       2  604 00000000001688b0      b220 Enabled  0000000000000000:0000000000000000 000000000014ccb0     0 MTA (Finalizer)

kd> !process fffffadfe7287610 4
PROCESS fffffadfe7287610
    SessionId: 0  Cid: 0ad8    Peb: 7fffffdf000  ParentCid: 0acc
    DirBase: 12cd9000  ObjectTable: fffffa800163c260  HandleCount: 114.
    Image: TestDefaultDebugger.NET.exe

        THREAD fffffadfe668cbf0  Cid 0ad8.0a94  Teb: 000007fffffdd000 Win32Thread: fffff97ff4df2830 WAIT
        THREAD fffffadfe727e6d0  Cid 0ad8.0f54  Teb: 000007fffffdb000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe72d5bf0  Cid 0ad8.0604  Teb: 000007fffffd9000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe679cbf0  Cid 0ad8.06b0  Teb: 000007fffffd7000 Win32Thread: 0000000000000000 WAIT
        THREAD fffffadfe67d23d0  Cid 0ad8.0b74  Teb: 000007fffffd5000 Win32Thread: fffff97ff4b99010 WAIT

kd> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000000002a41030
generation 1 starts at 0x0000000002a41018
generation 2 starts at 0x0000000002a41000
ephemeral segment allocation context: (0x0000000002a8d528, 0x0000000002a8dfe8)
         segment            begin         allocated             size
00000000001a1260 0000064274e28f60  0000064274e5f610 0x00000000000366b0(222896)
00000000001a1070 000006427692ffe8  000006427695af20 0x000000000002af38(175928)
0000000000164f60 00000642787c7380  0000064278809150 0x0000000000041dd0(269776)
0000000002a40000 0000000002a41000  0000000002a8dfe8 0x000000000004cfe8(315368)
Large object heap starts at 0x0000000012a41000
         segment            begin         allocated             size
0000000012a40000 0000000012a41000  0000000012a4e738 0x000000000000d738(55096)
Total Size           0xfdad8(1039064)
------------------------------
GC Heap Size           0xfdad8(1039064)

kd> !gchandles
Bad MethodTable for Obj at 0000000002a7a7b8
Bad MethodTable for Obj at 0000000002a7a750
Bad MethodTable for Obj at 0000000002a445b0
GC Handle Statistics:
Strong Handles: 25
Pinned Handles: 7
Async Pinned Handles: 0
Ref Count Handles: 1
Weak Long Handles: 30
Weak Short Handles: 63
Other Handles: 0
Statistics:
              MT    Count    TotalSize Class Name
[...]
0000064280016580        1          464 WindowsApplication1.Form1
[…]

kd> !dumpmt -md 0000064280016580
EEClass: 0000064280143578
Module: 0000064280012e00
Name: WindowsApplication1.Form1
mdToken: 02000002  (C:\TestDefaultDebugger.NET.exe)
BaseSize: 0×1d0
ComponentSize: 0×0
Number of IFaces in IFaceMap: 15
Slots in VTable: 375
————————————–
MethodDesc Table
           Entry       MethodDesc      JIT Name
[…]
0000064280150208 00000642800164d0      JIT WindowsApplication1.Form1.InitializeComponent()
0000064280150210 00000642800164e0      JIT WindowsApplication1.Form1..ctor()
0000064280150218 00000642800164f0     NONE WindowsApplication1.Form1.button1_Click(System.Object, System.EventArgs)

kd> !dumpil 00000642800164f0
ilAddr = 00000000004021bc
IL_0000: newobj System.Collections.Stack::.ctor
IL_0005: stloc.0
IL_0006: ldloc.0
IL_0007: callvirt System.Collections.Stack::Pop
IL_000c: pop
IL_000d: ret

- Dmitry Vostokov @ DumpAnalysis.org -

Motivated by the following book title that I bought a month ago:

The Oxford Book of Modern Science Writing

I decided to convert my library into a beautifully illustrated color book called

The Opentask Guide to Modern Software Engineering Writing

I plan to update it yearly with new titles that come to my attention. The first edition is planned for this summer and has the following draft product details:

• Title: The Opentask Guide to Modern Software Engineering Writing, 2008 Edition
• Author: Dmitry Vostokov
• Publisher: Opentask (25 August 2008)
• Language: English
• Product Dimensions: 21.6 x 14.0
• ISBN-13: 978-1-906717-04-9
• Paperback: 128 pages

Draft table of contents and sample chapter will be posted soon. 

- Dmitry Vostokov @ DumpAnalysis.org -

Mistake #1 - Not looking at full stack traces

By default WinDbg cuts off stack traces after 20th line and an analyst misses essential information when looking at Stack Trace or Stack Trace Collection. Consider this thread stack trace taken from a user process dump where runaway information was not saved but customers reported CPU spikes:

0:000> ~3kvL
ChildEBP RetAddr
0290f864 773976f2 user32!_SEH_prolog+0xb
0290f86c 0047f9ec user32!EnableMenuItem+0xf
0290f884 00488f6d Application!Close+0x142c
0290f8a4 0047a9c6 Application!EnableMenu+0x5d
0290f8b8 0048890d Application!EnableWindows+0x106
0290f8d0 0048cc2b Application!SetHourGlass+0xbd
0290f8fc 0046327a Application!WriteDataStream+0x24b
0290f924 0048d8f9 Application!WriteDataStream+0x21a
0290fa68 00479811 Application!WriteDataStream+0xcb9
0290fadc 5b5e976c Application!OnWrite+0x3c1
0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2
0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22
0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91
0290fc10 00516454 mfc71!AfxWndProc+0x46
0290fc3c 7739c3b7 Application!ExitCheck+0x28f34
0290fc68 7739c484 user32!InternalCallWinProc+0x28
0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151
0290fd10 773ad03f user32!CallWindowProcAorW+0x98
0290fd30 0047a59a user32!CallWindowProcA+0x1b

We can see that it uses MFC libraries and window messaging API but was it caught accidentally? Is it a typical message loop like idle message loops in Passive Thread pattern using GetMessage or it is an active GUI message pump using PeekMessage? If we expand stack trace we would see that the thread is actually MFC GUI thread that spins according to MFC source code:

int CWinThread::Run()
{
  for (;;)
  {
    while (bIdle &&
      !::PeekMessage(&(pState->m_msgCur), 
         NULL, NULL, NULL, PM_NOREMOVE))
     {

0:000> ~3kvL 100
ChildEBP RetAddr
0290f864 773976f2 user32!_SEH_prolog+0xb
0290f86c 0047f9ec user32!EnableMenuItem+0xf
0290f884 00488f6d Application!Close+0x142c
0290f8a4 0047a9c6 Application!EnableMenu+0x5d
0290f8b8 0048890d Application!EnableWindows+0x106
0290f8d0 0048cc2b Application!SetHourGlass+0xbd
0290f8fc 0046327a Application!WriteDataStream+0x24b
0290f924 0048d8f9 Application!WriteDataStream+0x21a
0290fa68 00479811 Application!WriteDataStream+0xcb9
0290fadc 5b5e976c Application!OnWrite+0x3c1
0290fb70 5b60e0b0 mfc71!CWnd::OnWndMsg+0x4f2
0290fb90 5b60e14f mfc71!CWnd::WindowProc+0x22
0290fbf0 5b60e1b8 mfc71!AfxCallWndProc+0x91
0290fc10 00516454 mfc71!AfxWndProc+0x46
0290fc3c 7739c3b7 Application!ExitCheck+0x28f34
0290fc68 7739c484 user32!InternalCallWinProc+0x28
0290fce0 77395563 user32!UserCallWinProcCheckWow+0x151
0290fd10 773ad03f user32!CallWindowProcAorW+0x98
0290fd30 0047a59a user32!CallWindowProcA+0x1b
0290fdb0 7739c3b7 Application!OnOK+0x77a
0290fddc 7739c484 user32!InternalCallWinProc+0x28
0290fe54 7739c73c user32!UserCallWinProcCheckWow+0x151
0290febc 7738e406 user32!DispatchMessageWorker+0x327
0290fecc 5b609076 user32!DispatchMessageA+0xf
0290fedc 5b60913e mfc71!AfxInternalPumpMessage+0x3e
0290fef8 004ba7cf mfc71!CWinThread::Run+0×54
0290ff04 5b61b30c Application!CMyThread::Run+0xf
0290ff84 5b869565 mfc71!_AfxThreadEntry+0×100
0290ffb8 77e66063 msvcr71!_endthreadex+0xa0
0290ffec 00000000 kernel32!BaseThreadStart+0×34

There is also WinDbg .kframes meta-command that can change default stack trace depth:

2: kd> .kframes 0n100
Default stack trace depth is 0n100 frames

- Dmitry Vostokov @ DumpAnalysis.org -

It now becomes a tradition to publish a sample chapter about heap. As promised here are two draft pages from the forthcoming Windows® Debugging Notebook:

05 (0×05): Heap / Pool / Paged / NonPaged

- Dmitry Vostokov @ DumpAnalysis.org -

01:00 am 24.05 I noticed it reached #1 bestseller status in Assembly Language Programming category:

Because the status is updated hourly you might not see the same status when you read this post

- Dmitry Vostokov @ DumpAnalysis.org -

Recently being interested in forensic science and intelligence I decided to elaborate on memory analysis classification and came up with the following division:

- Memory Analysis Forensics

Answering questions related to a committed computer crime. The suspect may be a human or a software / hardware component. Incident response, troubleshooting and debugging belong to this category. Postmortem memory analysis is usually analysis of dump files saved and detached from the original system or operating conditions.

- Memory Analysis Intelligence

Monitoring memory state for behavioural and structural patterns to prevent certain events from occurring. Usually done in situ. However digital dumpster divers and spies may also collect and analyze memory data that was detached from the original computer system.

Each category can be further subdivided into:

– Functional Memory Analysis

Tracing of events.

– Memoretics 

Analysis of memory states and their evolution.

The latter can be subdivided into: 

— Static Memory Analysis

Traditional memory dump analysis.

— Dynamic Memory Analysis

Live debugging.

- Dmitry Vostokov @ DumpAnalysis.org -

There is a bit of confusion around the division of exceptions between the first- and second-chance. For example, I am often asked whether saving crash dumps on first-chance exceptions should be disabled or enabled. So I decided to clarify this issue.

First, let me say that the concept of first-chance exceptions is purely a debugger-related. There is only one exception that happens when we access an invalid address, for example. However, that exception may be handled or may not be handled by exception handlers. Or it might be handled in peculiar way and terminate the thread that caused the exception, for example. And if it was not handled then an unhandled exception filter might be called. The default one might launch a postmortem debugger (or any process that can read process memory) to save a postmortem memory dump. Any thread can replace the default filter with a custom exception filter that might also do peculiar things and quietly terminate or exit. Even the properly configured postmortem debugger can fail to save a dump file. Therefore we have this question: how can we catch the exception and examine the process state as earlier as possible, before the execution flow goes through the exception handling mechanism?

Here we have the concept of the first chance exception dispatched to the attached user-mode debugger. if it wasn’t handled we have the same exception but called the second chance that is dispatched to the same debugger again. We see that it has nothing to do with the postmortem debugger although the attached live debugger can save crash dump files too, which what ADPlus does, for example.

- Dmitry Vostokov @ DumpAnalysis.org -

In the article about memory dump analysis as forensic science CSI was proposed to mean “Crashed Server Investigation”. I’m interested in general forensic science as well and I’ve almost finished reading the book about the emergence of forensic science in 19th and 20th centuries:

The Science of Sherlock Holmes: From Baskerville Hall to the Valley of Fear, the Real Forensics Behind the Great Detective’s Greatest Cases

As a result, yesterday I was rethinking CSI again and found these similar meanings:

- Crashed Software Investigation
- Crashed System Investigation

Any more suggestions? :-)

- Dmitry Vostokov @ DumpAnalysis.org -

The book has finally made it through Google Search program and is available for search here:

http://www.google.com/books?id=RR5whfK1BYsC

Also, yesterday I approved the hardcover version for distribution on Amazon and it should be available there in a week or so.

- Dmitry Vostokov @ DumpAnalysis.org -

Invalid Handle exception (0xC0000008) can frequently be seen in crash dumps. It results from an invalid handle value passed to CloseHandle and other Win32 API or when a handle or return status is checked manually for validity and the same exception is raised via RaiseException or internally via RtlRaiseStatus. Therefore it merits its own separate crash dump analysis pattern with the same name.

For example, critical sections are implemented using events and invalid event handle can result in this exception:

STACK_TEXT:
025bff00 7c94243c c0000008 7c9010ed 00231af0 ntdll!RtlRaiseStatus+0×26
025bff80 7c90104b 0015b4ac 77e76a6f 0015b4ac ntdll!RtlpWaitForCriticalSection+0×204
025bff88 77e76a6f 0015b4ac 010d2040 00000000 ntdll!RtlEnterCriticalSection+0×46
025bffa8 77e76c0a 0015b420 025bffec 7c80b683 rpcrt4!BaseCachedThreadRoutine+0xad
025bffb4 7c80b683 001feae8 010d2040 00000000 rpcrt4!ThreadStartRoutine+0×1a
025bffec 00000000 77e76bf0 001feae8 00000000 kernel32!BaseThreadStart+0×37

By default, unless raised manually, this exception doesn’t result in a default postmortem debugger called to save a crash dump. In order to do this we need to run the application under a debugger and save a crash dump upon this exception or use exception monitoring tools that save first-chance exceptions like Debug Diagnostics, ADPlus or Exception Monitor (see Early Crash Dump pattern):

0:002> g
(7b0.d1c): Invalid handle - code c0000008 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400          ret     4

0:000> g
(7b0.d1c): Invalid handle - code c0000008 (!!! second chance !!!)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=7d999906 edi=00403378
eip=7d61c92d esp=0012ff68 ebp=0012ff70 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
ntdll!NtClose+0×12:
7d61c92d c20400          ret     4

In order to catch it using postmortem debuggers we can use Application Verifier and configure its basic checks to include invalid handles. Then we will have crash dumps if a postmortem debugger or WER is properly configured. The typical stack might look like this and pointing straight to the problem component:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 6b006369
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 00000000

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT

0:000> kL
ChildEBP RetAddr 
0301ff44 0489a480 ntdll!NtClose+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0301ff54 7d4d8e4f vfbasics+0xa480
0301ff60 04894df9 kernel32!CloseHandle+0×59
0301ff70 00401022 vfbasics+0×4df9
0301ffc0 7d4e7d2a BadHandle+0×1022
0301fff0 00000000 kernel32!BaseProcessStart+0×28

or like this:

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0012ed58 00000000`01f9395a ntdll!DbgBreakPoint
00000000`0012ed60 00000000`023e29a7 vrfcore!VerifierStopMessageEx+0×846
00000000`0012f090 00000000`023d9384 vfbasics+0×129a7
00000000`0012f0f0 00000000`77f251ec vfbasics+0×9384
00000000`0012f180 00000000`77ee5f36 ntdll!RtlpCallVectoredHandlers+0×26f
00000000`0012f210 00000000`77ee6812 ntdll!RtlDispatchException+0×46
00000000`0012f8c0 00000000`77ef325a ntdll!RtlRaiseException+0xae
00000000`0012fe00 00000000`77d6e314 ntdll!KiRaiseUserExceptionDispatcher+0×3a
00000000`0012fed0 00000001`40001028 kernel32!CloseHandle+0×5f
00000000`0012ff00 00000001`40001294 BadHandle+0×1028
00000000`0012ff30 00000000`77d5964c BadHandle+0×1294
00000000`0012ff80 00000000`00000000 kernel32!BaseProcessStart+0×29

vfbasics and vrfcore are Application Verifier DLLs that might translate an invalid handle exception to a breakpoint exception and therefore trigger the launch of a postmortem debugger from an unhandled exception filter. Application Verifier version (x64 or x86) must match the application platform (64-bit or 32-bit).

If invalid handle exception is raised manually we get the status code and possibly problem component immediately from !analyze command:

FAULTING_IP:
kernel32!RaiseException+53
7d4e2366 5e              pop     esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7d4e2366 (kernel32!RaiseException+0x00000053)
   ExceptionCode: c0000008 (Invalid handle)
  ExceptionFlags: 00000000
NumberParameters: 0
Thread tried to close a handle that was invalid or illegal to close

DEFAULT_BUCKET_ID:  STATUS_INVALID_HANDLE

PROCESS_NAME:  BadHandle.exe

ERROR_CODE: (NTSTATUS) 0xc0000008 - An invalid HANDLE was specified.

STACK_TEXT: 
0012ff64 00401043 c0000008 00000000 00000000 kernel32!RaiseException+0×53
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 BadHandle+0×1043
0012fff0 00000000 004012f9 00000000 00000000 kernel32!BaseProcessStart+0×28

FAULTING_THREAD:  00000b64

PRIMARY_PROBLEM_CLASS:  STATUS_INVALID_HANDLE

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_INVALID_HANDLE

Because we have WinDbg warning about stack unwind we can double check the disassembly of RaiseException return address:

0:000> ub 00401043
BadHandle+0×1029:
00401029 push    offset BadHandle+0×212c (0040212c)
0040102e push    0
00401030 call    esi
00401032 push    0
00401034 push    0
00401036 push    0
00401038 push    0C0000008h
0040103d call    dword ptr [BadHandle+0×2004 (00402004)]

0:000> dps 00402004 l1
00402004  7d4e2318 kernel32!RaiseException

Beware that in such cases the real problem might have been memory corruption overwriting stored valid handle values.

- Dmitry Vostokov @ DumpAnalysis.org -

After Bug Check Frequencies post I was curious enough to do the same Google counting procedure for exceptions. Here are my results for exceptions listed in Visual C++ Debug \ Exceptions dialog:

The corresponding graph:

- Dmitry Vostokov @ DumpAnalysis.org -

Continuing Music for Debugging post I would like to recommend the following album that I used to listen to long time ago and have just found buried in the old corner. In it various movements correspond to different debugging sessions, some glorious and some filled with tension and worries, gradually building up the ultimate problem resolution. I think almost all Vangelis music is very suitable to accompany debugging.

El Greco

- Dmitry Vostokov @ DumpAnalysis.org -

A small note from the field. Sometimes on x64 Windows platform we set a default postmortem debugger or configure WER and then install Microsoft Application Verifier to do some checks. However no crash dump files are saved. The reason for this might be that we installed and configured amd64 bit version of Application Verifier but the problem application was 32-bit. For this application we need to install and configure x86 version of Application Verifier.

 - Dmitry Vostokov @ DumpAnalysis.org -

This is a new Xen book that I got in the post a few days ago and it seems to be so good that I took it home from work to start reading it during this weekend:

Running Xen: A Hands-On Guide to the Art of Virtualization

I’ll post a review once I read past the first chapter which is the excellent overview of different virtualization technologies.

See also another good Xen book that I discovered previously:

XEN from a system programmer’s perspective

- Dmitry Vostokov @ DumpAnalysis.org -

Seems dt command wildcards doesn’t work anymore for ntdll and nt modules (I haven’t tested other modules):

1: kd> dt nt!*
1: kd>

Previous version of WinDbg on the same crash dump works fine. I’m listing its sorted output to preserve types for future reference in case Microsoft decided to remove this feature (NTDLL types can be found here):

  nt!LIST_ENTRY64
  nt!_ACCESS_STATE
  nt!_ACL
  nt!_ACTIVATION_CONTEXT
  nt!_ACTIVATION_CONTEXT_DATA
  nt!_ACTIVATION_CONTEXT_STACK
  nt!_ADAPTER_OBJECT
  nt!_ALPC_COMMUNICATION_INFO
  nt!_ALPC_COMPLETION_LIST
  nt!_ALPC_COMPLETION_LIST_HEADER
  nt!_ALPC_COMPLETION_LIST_STATE
  nt!_ALPC_COMPLETION_PACKET_LOOKASIDE
  nt!_ALPC_COMPLETION_PACKET_LOOKASIDE_ENTRY
  nt!_ALPC_DISPATCH_CONTEXT
  nt!_ALPC_HANDLE_ENTRY
  nt!_ALPC_HANDLE_TABLE
  nt!_ALPC_MESSAGE_ATTRIBUTES
  nt!_ALPC_MESSAGE_ZONE
  nt!_ALPC_PORT
  nt!_ALPC_PORT_ATTRIBUTES
  nt!_ALPC_PROCESS_CONTEXT
  nt!_ALPHA_LOADER_BLOCK
  nt!_ALTERNATIVE_ARCHITECTURE_TYPE
  nt!_AMD64_DBGKD_CONTROL_SET
  nt!_ARBITER_ACTION
  nt!_ARBITER_ADD_RESERVED_PARAMETERS
  nt!_ARBITER_ALLOCATION_STATE
  nt!_ARBITER_ALTERNATIVE
  nt!_ARBITER_BOOT_ALLOCATION_PARAMETERS
  nt!_ARBITER_CONFLICT_INFO
  nt!_ARBITER_INSTANCE
  nt!_ARBITER_INTERFACE
  nt!_ARBITER_LIST_ENTRY
  nt!_ARBITER_ORDERING
  nt!_ARBITER_ORDERING_LIST
  nt!_ARBITER_PARAMETERS
  nt!_ARBITER_QUERY_ALLOCATED_RESOURCES_PARAMETERS
  nt!_ARBITER_QUERY_ARBITRATE_PARAMETERS
  nt!_ARBITER_QUERY_CONFLICT_PARAMETERS
  nt!_ARBITER_REQUEST_SOURCE
  nt!_ARBITER_RESULT
  nt!_ARBITER_RETEST_ALLOCATION_PARAMETERS
  nt!_ARBITER_TEST_ALLOCATION_PARAMETERS
  nt!_ARC_DISK_INFORMATION
  nt!_ARM_DBGKD_CONTROL_SET
  nt!_ASSEMBLY_STORAGE_MAP
  nt!_BITMAP_RANGE
  nt!_BLOB
  nt!_BLOB_ID
  nt!_BLOB_TYPE
  nt!_BUS_EXTENSION_LIST
  nt!_CACHE_DESCRIPTOR
  nt!_CACHE_MANAGER_CALLBACKS
  nt!_CACHE_UNINITIALIZE_EVENT
  nt!_CACHED_CHILD_LIST
  nt!_CACHED_KSTACK_LIST
  nt!_CALL_HASH_ENTRY
  nt!_CALL_PERFORMANCE_DATA
  nt!_CELL_DATA
  nt!_CHILD_LIST
  nt!_CLIENT_ID
  nt!_CLS_LSN
  nt!_CM_BIG_DATA
  nt!_CM_CACHED_VALUE_INDEX
  nt!_CM_CELL_REMAP_BLOCK
  nt!_CM_FULL_RESOURCE_DESCRIPTOR
  nt!_CM_INDEX_HINT_BLOCK
  nt!_CM_INTENT_LOCK
  nt!_CM_KCB_UOW
  nt!_CM_KEY_BODY
  nt!_CM_KEY_CONTROL_BLOCK
  nt!_CM_KEY_HASH
  nt!_CM_KEY_INDEX
  nt!_CM_KEY_NODE
  nt!_CM_KEY_REFERENCE
  nt!_CM_KEY_SECURITY
  nt!_CM_KEY_SECURITY_CACHE
  nt!_CM_KEY_SECURITY_CACHE_ENTRY
  nt!_CM_KEY_VALUE
  nt!_CM_NAME_CONTROL_BLOCK
  nt!_CM_NAME_HASH
  nt!_CM_NOTIFY_BLOCK
  nt!_CM_PARTIAL_RESOURCE_DESCRIPTOR
  nt!_CM_PARTIAL_RESOURCE_LIST
  nt!_CM_RESOURCE_LIST
  nt!_CM_RM
  nt!_CM_TRANS
  nt!_CM_VIEW_OF_FILE
  nt!_CM_WORKITEM
  nt!_CMHIVE
  nt!_COMPRESSED_DATA_INFO
  nt!_CONFIGURATION_CLASS
  nt!_CONFIGURATION_COMPONENT
  nt!_CONFIGURATION_COMPONENT_DATA
  nt!_CONFIGURATION_TYPE
  nt!_CONTEXT
  nt!_CONTROL_AREA
  nt!_CURDIR
  nt!_DBGKD_ANY_CONTROL_SET
  nt!_DBGKD_BREAKPOINTEX
  nt!_DBGKD_CONTINUE
  nt!_DBGKD_CONTINUE2
  nt!_DBGKD_FILL_MEMORY
  nt!_DBGKD_GET_CONTEXT
  nt!_DBGKD_GET_INTERNAL_BREAKPOINT32
  nt!_DBGKD_GET_INTERNAL_BREAKPOINT64
  nt!_DBGKD_GET_SET_BUS_DATA
  nt!_DBGKD_GET_VERSION32
  nt!_DBGKD_GET_VERSION64
  nt!_DBGKD_LOAD_SYMBOLS32
  nt!_DBGKD_LOAD_SYMBOLS64
  nt!_DBGKD_MANIPULATE_STATE32
  nt!_DBGKD_MANIPULATE_STATE64
  nt!_DBGKD_QUERY_MEMORY
  nt!_DBGKD_QUERY_SPECIAL_CALLS
  nt!_DBGKD_READ_MEMORY32
  nt!_DBGKD_READ_MEMORY64
  nt!_DBGKD_READ_WRITE_IO_EXTENDED32
  nt!_DBGKD_READ_WRITE_IO_EXTENDED64
  nt!_DBGKD_READ_WRITE_IO32
  nt!_DBGKD_READ_WRITE_IO64
  nt!_DBGKD_READ_WRITE_MSR
  nt!_DBGKD_RESTORE_BREAKPOINT
  nt!_DBGKD_SEARCH_MEMORY
  nt!_DBGKD_SET_CONTEXT
  nt!_DBGKD_SET_INTERNAL_BREAKPOINT32
  nt!_DBGKD_SET_INTERNAL_BREAKPOINT64
  nt!_DBGKD_SET_SPECIAL_CALL32
  nt!_DBGKD_SET_SPECIAL_CALL64
  nt!_DBGKD_SWITCH_PARTITION
  nt!_DBGKD_WRITE_BREAKPOINT32
  nt!_DBGKD_WRITE_BREAKPOINT64
  nt!_DBGKD_WRITE_MEMORY32
  nt!_DBGKD_WRITE_MEMORY64
  nt!_DBGKM_EXCEPTION32
  nt!_DBGKM_EXCEPTION64
  nt!_DEFERRED_WRITE
  nt!_DESCRIPTOR
  nt!_DEVICE_CAPABILITIES
  nt!_DEVICE_FLAGS
  nt!_DEVICE_MAP
  nt!_DEVICE_NODE
  nt!_DEVICE_OBJECT
  nt!_DEVICE_OBJECT_POWER_EXTENSION
  nt!_DEVICE_POWER_STATE
  nt!_DEVICE_RELATION_TYPE
  nt!_DEVICE_RELATIONS
  nt!_DEVICE_USAGE_NOTIFICATION_TYPE
  nt!_DEVOBJ_EXTENSION
  nt!_DEVPROPKEY
  nt!_DISPATCHER_HEADER
  nt!_DPFLTR_TYPE
  nt!_DRIVER_EXTENSION
  nt!_DRIVER_OBJECT
  nt!_DUAL
  nt!_DUMP_INITIALIZATION_CONTEXT
  nt!_DUMP_STACK_CONTEXT
  nt!_ECP_LIST
  nt!_EFI_FIRMWARE_INFORMATION
  nt!_EJOB
  nt!_EPROCESS
  nt!_EPROCESS_QUOTA_BLOCK
  nt!_ERESOURCE
  nt!_ETHREAD
  nt!_ETIMER
  nt!_ETW_BUFFER_CONTEXT
  nt!_ETW_BUFFER_STATE
  nt!_ETW_GUID_ENTRY
  nt!_ETW_GUID_TYPE
  nt!_ETW_KERNEL_TRACE_TIMESTAMP
  nt!_ETW_LAST_ENABLE_INFO
  nt!_ETW_PROVIDER_STATE
  nt!_ETW_PROVIDER_TABLE_ENTRY
  nt!_ETW_REF_CLOCK
  nt!_ETW_REG_ENTRY
  nt!_ETW_REPLY_QUEUE
  nt!_ETW_RT_EVENT_LOSS
  nt!_ETW_SYSTEMTIME
  nt!_EVENT_DATA_DESCRIPTOR
  nt!_EVENT_DESCRIPTOR
  nt!_EVENT_TYPE
  nt!_EX_FAST_REF
  nt!_EX_PUSH_LOCK
  nt!_EX_PUSH_LOCK_CACHE_AWARE
  nt!_EX_PUSH_LOCK_WAIT_BLOCK
  nt!_EX_RUNDOWN_REF
  nt!_EX_WORK_QUEUE
  nt!_EXCEPTION_DISPOSITION
  nt!_EXCEPTION_POINTERS
  nt!_EXCEPTION_RECORD
  nt!_EXCEPTION_RECORD32
  nt!_EXCEPTION_RECORD64
  nt!_EXCEPTION_REGISTRATION_RECORD
  nt!_FAST_IO_DISPATCH
  nt!_FAST_MUTEX
  nt!_FILE_BASIC_INFORMATION
  nt!_FILE_GET_QUOTA_INFORMATION
  nt!_FILE_INFORMATION_CLASS
  nt!_FILE_NETWORK_OPEN_INFORMATION
  nt!_FILE_OBJECT
  nt!_FILE_STANDARD_INFORMATION
  nt!_FIRMWARE_INFORMATION_LOADER_BLOCK
  nt!_flags
  nt!_FLOATING_SAVE_AREA
  nt!_FLS_CALLBACK_INFO
  nt!_FNSAVE_FORMAT
  nt!_FREE_DISPLAY
  nt!_FS_FILTER_CALLBACK_DATA
  nt!_FS_FILTER_CALLBACKS
  nt!_FS_FILTER_PARAMETERS
  nt!_FS_FILTER_SECTION_SYNC_TYPE
  nt!_FS_FILTER_STREAM_FO_NOTIFICATION_TYPE
  nt!_FSINFOCLASS
  nt!_FX_SAVE_AREA
  nt!_FXSAVE_FORMAT
  nt!_GDI_TEB_BATCH
  nt!_GENERAL_LOOKASIDE
  nt!_GENERAL_LOOKASIDE_POOL
  nt!_GENERIC_MAPPING
  nt!_GUID
  nt!_HANDLE_TABLE
  nt!_HANDLE_TABLE_ENTRY
  nt!_HANDLE_TABLE_ENTRY_INFO
  nt!_HANDLE_TRACE_DB_ENTRY
  nt!_HANDLE_TRACE_DEBUG_INFO
  nt!_HARDWARE_PTE
  nt!_HBASE_BLOCK
  nt!_HEADLESS_LOADER_BLOCK
  nt!_HEAP
  nt!_HEAP_COUNTERS
  nt!_HEAP_DEBUGGING_INFORMATION
  nt!_HEAP_ENTRY
  nt!_HEAP_ENTRY_EXTRA
  nt!_HEAP_FAILURE_TYPE
  nt!_HEAP_FREE_ENTRY
  nt!_HEAP_FREE_ENTRY_EXTRA
  nt!_HEAP_LIST_LOOKUP
  nt!_HEAP_LOCAL_SEGMENT_INFO
  nt!_HEAP_LOCK
  nt!_HEAP_LOOKASIDE
  nt!_HEAP_PSEUDO_TAG_ENTRY
  nt!_HEAP_SEGMENT
  nt!_HEAP_STOP_ON_TAG
  nt!_HEAP_STOP_ON_VALUES
  nt!_HEAP_SUBSEGMENT
  nt!_HEAP_TAG_ENTRY
  nt!_HEAP_TUNING_PARAMETERS
  nt!_HEAP_UCR_DESCRIPTOR
  nt!_HEAP_USERDATA_HEADER
  nt!_HEAP_VIRTUAL_ALLOC_ENTRY
  nt!_HHIVE
  nt!_HIVE_LIST_ENTRY
  nt!_HMAP_DIRECTORY
  nt!_HMAP_ENTRY
  nt!_HMAP_TABLE
  nt!_I386_LOADER_BLOCK
  nt!_IA64_DBGKD_CONTROL_SET
  nt!_IA64_LOADER_BLOCK
  nt!_IMAGE_DATA_DIRECTORY
  nt!_IMAGE_DEBUG_DIRECTORY
  nt!_IMAGE_DOS_HEADER
  nt!_IMAGE_FILE_HEADER
  nt!_IMAGE_NT_HEADERS
  nt!_IMAGE_OPTIONAL_HEADER
  nt!_IMAGE_ROM_OPTIONAL_HEADER
  nt!_IMAGE_SECTION_HEADER
  nt!_IMAGE_SECURITY_CONTEXT
  nt!_INITIAL_PRIVILEGE_SET
  nt!_INTERFACE
  nt!_INTERFACE_TYPE
  nt!_INTERLOCK_SEQ
  nt!_IO_ALLOCATION_ACTION
  nt!_IO_CLIENT_EXTENSION
  nt!_IO_COMPLETION_CONTEXT
  nt!_IO_DRIVER_CREATE_CONTEXT
  nt!_IO_PAGING_PRIORITY
  nt!_IO_PRIORITY_HINT
  nt!_IO_PRIORITY_INFO
  nt!_IO_RESOURCE_DESCRIPTOR
  nt!_IO_RESOURCE_LIST
  nt!_IO_RESOURCE_REQUIREMENTS_LIST
  nt!_IO_SECURITY_CONTEXT
  nt!_IO_STACK_LOCATION
  nt!_IO_STATUS_BLOCK
  nt!_IO_TIMER
  nt!_iobuf
  nt!_IOV_FORCED_PENDING_TRACE
  nt!_IRP
  nt!_IRQ_DEVICE_POLICY
  nt!_IRQ_PRIORITY
  nt!_JOB_ACCESS_STATE
  nt!_KALPC_HANDLE_DATA
  nt!_KALPC_MESSAGE
  nt!_KALPC_MESSAGE_ATTRIBUTES
  nt!_KALPC_REGION
  nt!_KALPC_RESERVE
  nt!_KALPC_SECTION
  nt!_KALPC_SECURITY_DATA
  nt!_KALPC_VIEW
  nt!_KAPC
  nt!_KAPC_STATE
  nt!_KDEVICE_QUEUE
  nt!_KDEVICE_QUEUE_ENTRY
  nt!_KDPC
  nt!_KDPC_DATA
  nt!_KENLISTMENT
  nt!_KENLISTMENT_HISTORY
  nt!_KENLISTMENT_STATE
  nt!_KERNEL_STACK_CONTROL
  nt!_KERNEL_STACK_SEGMENT
  nt!_KEVENT
  nt!_KEXECUTE_OPTIONS
  nt!_KGATE
  nt!_KGDTENTRY
  nt!_KGUARDED_MUTEX
  nt!_KIDTENTRY
  nt!_KiIoAccessMap
  nt!_KINTERRUPT
  nt!_KINTERRUPT_MODE
  nt!_KINTERRUPT_POLARITY
  nt!_KLOCK_QUEUE_HANDLE
  nt!_KMUTANT
  nt!_KNODE
  nt!_KOBJECTS
  nt!_KPCR
  nt!_KPRCB
  nt!_KPROCESS
  nt!_KPROCESSOR_STATE
  nt!_KQUEUE
  nt!_KRESOURCEMANAGER
  nt!_KRESOURCEMANAGER_COMPLETION_BINDING
  nt!_KRESOURCEMANAGER_STATE
  nt!_KSEMAPHORE
  nt!_KSPECIAL_REGISTERS
  nt!_KSPIN_LOCK_QUEUE
  nt!_KSPIN_LOCK_QUEUE_NUMBER
  nt!_KSYSTEM_TIME
  nt!_KTHREAD
  nt!_KTHREAD_STATE
  nt!_KTIMER
  nt!_KTIMER_TABLE_ENTRY
  nt!_KTM
  nt!_KTMNOTIFICATION_PACKET
  nt!_KTMOBJECT_NAMESPACE
  nt!_KTMOBJECT_NAMESPACE_LINK
  nt!_KTRANSACTION
  nt!_KTRANSACTION_HISTORY
  nt!_KTRANSACTION_OUTCOME
  nt!_KTRANSACTION_STATE
  nt!_KTRAP_FRAME
  nt!_KTSS
  nt!_KUSER_SHARED_DATA
  nt!_KWAIT_BLOCK
  nt!_KWAIT_REASON
  nt!_LARGE_INTEGER
  nt!_LAZY_WRITER
  nt!_LDR_DATA_TABLE_ENTRY
  nt!_LIST_ENTRY
  nt!_LOADER_PARAMETER_BLOCK
  nt!_LOADER_PARAMETER_EXTENSION
  nt!_LOADER_PERFORMANCE_DATA
  nt!_LOOKASIDE_LIST_EX
  nt!_LPCP_MESSAGE
  nt!_LPCP_NONPAGED_PORT_QUEUE
  nt!_LPCP_PORT_OBJECT
  nt!_LPCP_PORT_QUEUE
  nt!_LUID
  nt!_LUID_AND_ATTRIBUTES
  nt!_MAILSLOT_CREATE_PARAMETERS
  nt!_MAPPED_FILE_SEGMENT
  nt!_MBCB
  nt!_MDL
  nt!_MEMORY_ALLOCATION_DESCRIPTOR
  nt!_MEMORY_CACHING_TYPE
  nt!_MEMORY_CACHING_TYPE_ORIG
  nt!_MI_COLOR_BASE
  nt!_MI_DYNAMIC_MEMORY_LOCKTYPE
  nt!_MI_EXTRA_IMAGE_INFORMATION
  nt!_MI_IMAGE_SECURITY_REFERENCE
  nt!_MI_PAGEFILE_TRACES
  nt!_MI_PER_SESSION_PROTOS
  nt!_MI_PFN_CACHE_ATTRIBUTE
  nt!_MI_SECTION_CREATION_GATE
  nt!_MI_SECTION_IMAGE_INFORMATION
  nt!_MI_SPECIAL_POOL
  nt!_MI_SYSTEM_PTE_TYPE
  nt!_MI_SYSTEM_VA_TYPE
  nt!_MI_VAD_TYPE
  nt!_MI_VERIFIER_DRIVER_ENTRY
  nt!_MI_VERIFIER_POOL_HEADER
  nt!_MM_AVL_TABLE
  nt!_MM_DRIVER_VERIFIER_DATA
  nt!_MM_PAGE_ACCESS_INFO
  nt!_MM_PAGE_ACCESS_INFO_FLAGS
  nt!_MM_PAGE_ACCESS_INFO_HEADER
  nt!_MM_PAGE_ACCESS_TYPE
  nt!_MM_PAGED_POOL_INFO
  nt!_MM_POOL_FAILURE_REASONS
  nt!_MM_POOL_PRIORITIES
  nt!_MM_POOL_TYPES
  nt!_MM_PREEMPTIVE_TRIMS
  nt!_MM_SESSION_SPACE
  nt!_MM_SESSION_SPACE_FLAGS
  nt!_MM_SUBSECTION_AVL_TABLE
  nt!_MMADDRESS_LIST
  nt!_MMADDRESS_NODE
  nt!_MMBANKED_SECTION
  nt!_MMEXTEND_INFO
  nt!_MMLISTS
  nt!_MMMOD_WRITER_MDL_ENTRY
  nt!_MMPAGING_FILE
  nt!_MMPAGING_FILE_FREE_ENTRY
  nt!_MMPFN
  nt!_MMPFNENTRY
  nt!_MMPFNLIST
  nt!_MMPTE
  nt!_MMPTE_FLUSH_LIST
  nt!_MMPTE_HARDWARE
  nt!_MMPTE_HIGHLOW
  nt!_MMPTE_LIST
  nt!_MMPTE_PROTOTYPE
  nt!_MMPTE_SOFTWARE
  nt!_MMPTE_SUBSECTION
  nt!_MMPTE_TIMESTAMP
  nt!_MMPTE_TRANSITION
  nt!_MMSECTION_FLAGS
  nt!_MMSECURE_FLAGS
  nt!_MMSESSION
  nt!_MMSUBSECTION_FLAGS
  nt!_MMSUBSECTION_NODE
  nt!_MMSUPPORT
  nt!_MMSUPPORT_FLAGS
  nt!_MMVAD
  nt!_MMVAD_FLAGS
  nt!_MMVAD_FLAGS2
  nt!_MMVAD_FLAGS3
  nt!_MMVAD_LONG
  nt!_MMVAD_SHORT
  nt!_MMVIEW
  nt!_MMWSL
  nt!_MMWSLE
  nt!_MMWSLE_FREE_ENTRY
  nt!_MMWSLE_HASH
  nt!_MMWSLE_NONDIRECT_HASH
  nt!_MMWSLENTRY
  nt!_MODE
  nt!_MSUBSECTION
  nt!_NAMED_PIPE_CREATE_PARAMETERS
  nt!_NETWORK_LOADER_BLOCK
  nt!_NLS_DATA_BLOCK
  nt!_NPAGED_LOOKASIDE_LIST
  nt!_NT_PRODUCT_TYPE
  nt!_NT_TIB
  nt!_OB_DUPLICATE_OBJECT_STATE
  nt!_OB_OPEN_REASON
  nt!_OBJECT_ATTRIBUTES
  nt!_OBJECT_CREATE_INFORMATION
  nt!_OBJECT_DIRECTORY
  nt!_OBJECT_DIRECTORY_ENTRY
  nt!_OBJECT_DUMP_CONTROL
  nt!_OBJECT_HANDLE_COUNT_DATABASE
  nt!_OBJECT_HANDLE_COUNT_ENTRY
  nt!_OBJECT_HANDLE_INFORMATION
  nt!_OBJECT_HEADER
  nt!_OBJECT_HEADER_CREATOR_INFO
  nt!_OBJECT_HEADER_HANDLE_INFO
  nt!_OBJECT_HEADER_NAME_INFO
  nt!_OBJECT_HEADER_QUOTA_INFO
  nt!_OBJECT_INFORMATION_CLASS
  nt!_OBJECT_NAME_INFORMATION
  nt!_OBJECT_REF_INFO
  nt!_OBJECT_REF_STACK_INFO
  nt!_OBJECT_REF_TRACE
  nt!_OBJECT_SYMBOLIC_LINK
  nt!_OBJECT_TYPE
  nt!_OBJECT_TYPE_INITIALIZER
  nt!_OBP_LOOKUP_CONTEXT
  nt!_OWNER_ENTRY
  nt!_PAGED_LOOKASIDE_LIST
  nt!_PAGEFAULT_HISTORY
  nt!_PCAT_FIRMWARE_INFORMATION
  nt!_PCI_HOTPLUG_SLOT_INTERRUPT
  nt!_PEB
  nt!_PEB_LDR_DATA
  nt!_PERFINFO_GROUPMASK
  nt!_PERFINFO_HARDPAGEFAULT_INFORMATION
  nt!_PERFINFO_TRACE_HEADER
  nt!_PF_FILE_ACCESS_TYPE
  nt!_PF_HARD_FAULT_INFO
  nt!_PF_KERNEL_GLOBALS
  nt!_PHYSICAL_MEMORY_DESCRIPTOR
  nt!_PHYSICAL_MEMORY_RUN
  nt!_PI_BUS_EXTENSION
  nt!_PI_RESOURCE_ARBITER_ENTRY
  nt!_PLUGPLAY_EVENT_BLOCK
  nt!_PLUGPLAY_EVENT_CATEGORY
  nt!_PNP_ASSIGN_RESOURCES_CONTEXT
  nt!_PNP_DEVICE_COMPLETION_QUEUE
  nt!_PNP_DEVICE_EVENT_ENTRY
  nt!_PNP_DEVICE_EVENT_LIST
  nt!_PNP_DEVNODE_STATE
  nt!_PNP_RESOURCE_REQUEST
  nt!_PNP_VETO_TYPE
  nt!_PO_DEVICE_NOTIFY
  nt!_PO_DEVICE_NOTIFY_ORDER
  nt!_PO_HIBER_PERF
  nt!_PO_IRP_MANAGER
  nt!_PO_IRP_QUEUE
  nt!_PO_MEMORY_RANGE_ARRAY
  nt!_PO_MEMORY_RANGE_ARRAY_LINK
  nt!_PO_MEMORY_RANGE_ARRAY_RANGE
  nt!_PO_NOTIFY_ORDER_LEVEL
  nt!_POLICY_AUDIT_EVENT_TYPE
  nt!_POOL_BLOCK_HEAD
  nt!_POOL_DESCRIPTOR
  nt!_POOL_HACKER
  nt!_POOL_HEADER
  nt!_POOL_TRACKER_BIG_PAGES
  nt!_POOL_TRACKER_TABLE
  nt!_POOL_TYPE
  nt!_POP_ACTION_TRIGGER
  nt!_POP_CPU_INFO
  nt!_POP_DEVICE_SYS_STATE
  nt!_POP_DISPLAY_RESUME_CONTEXT
  nt!_POP_HIBER_CONTEXT
  nt!_POP_POWER_ACTION
  nt!_POP_SHUTDOWN_BUG_CHECK
  nt!_POP_THERMAL_ZONE
  nt!_POP_TRIGGER_WAIT
  nt!_PORT_MESSAGE
  nt!_POWER_CHANNEL_SUMMARY
  nt!_POWER_SEQUENCE
  nt!_POWER_STATE
  nt!_POWER_STATE_TYPE
  nt!_PP_LOOKASIDE_LIST
  nt!_PP_NPAGED_LOOKASIDE_NUMBER
  nt!_PPM_DIA_STATS
  nt!_PPM_IDLE_STATE
  nt!_PPM_IDLE_STATES
  nt!_PPM_PERF_STATE
  nt!_PPM_PERF_STATES
  nt!_PRIVATE_CACHE_MAP
  nt!_PRIVATE_CACHE_MAP_FLAGS
  nt!_PRIVILEGE_SET
  nt!_PROCESSOR_CACHE_TYPE
  nt!_PROCESSOR_POWER_STATE
  nt!_PROFILE_PARAMETER_BLOCK
  nt!_PROXY_CLASS
  nt!_PS_CLIENT_SECURITY_CONTEXT
  nt!_PS_RESOURCE_TYPE
  nt!_PSP_RATE_APC
  nt!_PTE_TRACKER
  nt!_QUAD
  nt!_REG_NOTIFY_CLASS
  nt!_REMOTE_PORT_VIEW
  nt!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
  nt!_RTL_ATOM_TABLE
  nt!_RTL_ATOM_TABLE_ENTRY
  nt!_RTL_AVL_TABLE
  nt!_RTL_BALANCED_LINKS
  nt!_RTL_BITMAP
  nt!_RTL_CRITICAL_SECTION
  nt!_RTL_CRITICAL_SECTION_DEBUG
  nt!_RTL_DRIVE_LETTER_CURDIR
  nt!_RTL_GENERIC_COMPARE_RESULTS
  nt!_RTL_HANDLE_TABLE
  nt!_RTL_HANDLE_TABLE_ENTRY
  nt!_RTL_RANGE
  nt!_RTL_RANGE_LIST
  nt!_RTL_SRWLOCK
  nt!_RTL_USER_PROCESS_PARAMETERS
  nt!_RTLP_RANGE_LIST_ENTRY
  nt!_SCSI_REQUEST_BLOCK
  nt!_SE_AUDIT_PROCESS_CREATION_INFO
  nt!_SECTION_IMAGE_INFORMATION
  nt!_SECTION_OBJECT
  nt!_SECTION_OBJECT_POINTERS
  nt!_SECURITY_CLIENT_CONTEXT
  nt!_SECURITY_DESCRIPTOR
  nt!_SECURITY_DESCRIPTOR_RELATIVE
  nt!_SECURITY_IMPERSONATION_LEVEL
  nt!_SECURITY_OPERATION_CODE
  nt!_SECURITY_QUALITY_OF_SERVICE
  nt!_SECURITY_SUBJECT_CONTEXT
  nt!_SECURITY_TOKEN_AUDIT_DATA
  nt!_SECURITY_TOKEN_PROXY_DATA
  nt!_SEGMENT
  nt!_SEGMENT_FLAGS
  nt!_SEGMENT_OBJECT
  nt!_SEP_AUDIT_POLICY
  nt!_SEP_LOGON_SESSION_REFERENCES
  nt!_SEP_TOKEN_PRIVILEGES
  nt!_SETUP_LOADER_BLOCK
  nt!_SHARED_CACHE_MAP
  nt!_SHARED_CACHE_MAP_LIST_CURSOR
  nt!_SID
  nt!_SID_AND_ATTRIBUTES
  nt!_SID_AND_ATTRIBUTES_HASH
  nt!_SID_IDENTIFIER_AUTHORITY
  nt!_SINGLE_LIST_ENTRY
  nt!_SLIST_HEADER
  nt!_SMBIOS_TABLE_HEADER
  nt!_STACK_TABLE
  nt!_STRING
  nt!_SUBSECTION
  nt!_SYSPTES_HEADER
  nt!_SYSTEM_POWER_POLICY
  nt!_SYSTEM_POWER_STATE
  nt!_SYSTEM_POWER_STATE_CONTEXT
  nt!_SYSTEM_TRACE_HEADER
  nt!_TEB
  nt!_TEB_ACTIVE_FRAME
  nt!_TEB_ACTIVE_FRAME_CONTEXT
  nt!_TERMINATION_PORT
  nt!_THERMAL_INFORMATION
  nt!_THERMAL_INFORMATION_EX
  nt!_TIME_FIELDS
  nt!_TOKEN
  nt!_TOKEN_AUDIT_POLICY
  nt!_TOKEN_CONTROL
  nt!_TOKEN_SOURCE
  nt!_TOKEN_TYPE
  nt!_TP_CALLBACK_ENVIRON
  nt!_TP_CALLBACK_INSTANCE
  nt!_TP_CLEANUP_GROUP
  nt!_TP_DIRECT
  nt!_TP_POOL
  nt!_TP_TASK
  nt!_TP_TASK_CALLBACKS
  nt!_TRACE_ENABLE_CONTEXT
  nt!_TRACE_ENABLE_CONTEXT_EX
  nt!_TRACE_ENABLE_INFO
  nt!_TXN_PARAMETER_BLOCK
  nt!_TYPE_OF_MEMORY
  nt!_u
  nt!_ULARGE_INTEGER
  nt!_UNICODE_STRING
  nt!_VACB
  nt!_VACB_ARRAY_HEADER
  nt!_VACB_LEVEL_REFERENCE
  nt!_VF_BTS_DATA_MANAGEMENT_AREA
  nt!_VF_BTS_RECORD
  nt!_VF_POOL_TRACE
  nt!_VF_TRACKER
  nt!_VF_TRACKER_STAMP
  nt!_VI_CANCEL_GLOBALS
  nt!_VI_CNT_INDEX_TYPE
  nt!_VI_DEADLOCK_ADDRESS_RANGE
  nt!_VI_DEADLOCK_GLOBALS
  nt!_VI_DEADLOCK_NODE
  nt!_VI_DEADLOCK_RESOURCE
  nt!_VI_DEADLOCK_RESOURCE_TYPE
  nt!_VI_DEADLOCK_THREAD
  nt!_VI_FAULT_TRACE
  nt!_VI_POOL_ENTRY
  nt!_VI_POOL_ENTRY_INUSE
  nt!_VI_POOL_PAGE_HEADER
  nt!_VI_TRACK_IRQL
  nt!_VI_VERIFIER_ISSUE
  nt!_VIRTUAL_EFI_RUNTIME_SERVICES
  nt!_VOLUME_CACHE_MAP
  nt!_VPB
  nt!_WAIT_CONTEXT_BLOCK
  nt!_WAIT_TYPE
  nt!_WHEA_ERROR_PACKET
  nt!_WHEA_ERROR_PACKET_FLAGS
  nt!_WHEA_ERROR_RECORD
  nt!_WHEA_ERROR_RECORD_HEADER
  nt!_WHEA_ERROR_RECORD_HEADER_FLAGS
  nt!_WHEA_ERROR_RECORD_HEADER_VALIDBITS
  nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR
  nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR_FLAGS
  nt!_WHEA_ERROR_RECORD_SECTION_DESCRIPTOR_VALIDBITS
  nt!_WHEA_ERROR_SEVERITY
  nt!_WHEA_ERROR_SOURCE_TYPE
  nt!_WHEA_ERROR_STATUS
  nt!_WHEA_ERROR_TYPE
  nt!_WHEA_GENERIC_PROCESSOR_ERROR
  nt!_WHEA_GENERIC_PROCESSOR_ERROR_VALIDBITS
  nt!_WHEA_MEMORY_ERROR
  nt!_WHEA_MEMORY_ERROR_VALIDBITS
  nt!_WHEA_NMI_ERROR
  nt!_WHEA_NMI_ERROR_FLAGS
  nt!_WHEA_PCIEXPRESS_BRIDGE_CONTROL_STATUS
  nt!_WHEA_PCIEXPRESS_COMMAND_STATUS
  nt!_WHEA_PCIEXPRESS_DEVICE_ID
  nt!_WHEA_PCIEXPRESS_ERROR
  nt!_WHEA_PCIEXPRESS_ERROR_VALIDBITS
  nt!_WHEA_PCIEXPRESS_VERSION
  nt!_WHEA_PCIXBUS_COMMAND
  nt!_WHEA_PCIXBUS_ERROR
  nt!_WHEA_PCIXBUS_ERROR_VALIDBITS
  nt!_WHEA_PCIXBUS_ID
  nt!_WHEA_PCIXDEVICE_ERROR
  nt!_WHEA_PCIXDEVICE_ERROR_VALIDBITS
  nt!_WHEA_PCIXDEVICE_ID
  nt!_WHEA_PERSISTENCE_INFO
  nt!_WHEA_RAW_DATA_FORMAT
  nt!_WHEA_REVISION
  nt!_WHEA_TIMESTAMP
  nt!_WMI_BUFFER_HEADER
  nt!_WMI_LOGGER_CONTEXT
  nt!_WMI_TRACE_PACKET
  nt!_WNODE_HEADER
  nt!_WORK_QUEUE_ENTRY
  nt!_WORK_QUEUE_ITEM
  nt!_WOW64_SHARED_INFORMATION
  nt!_X86_DBGKD_CONTROL_SET
  nt!BATTERY_REPORTING_SCALE
  nt!BUS_QUERY_ID_TYPE
  nt!CMP_OFFSET_ARRAY
  nt!DEVICE_TEXT_TYPE
  nt!EX_QUEUE_WORKER_INFO
  nt!HSTORAGE_TYPE
  nt!KTM_STATE
  nt!LIST_ENTRY32
  nt!LIST_ENTRY64
  nt!LSA_FOREST_TRUST_RECORD_TYPE
  nt!PO_MEMORY_IMAGE
  nt!POP_POLICY_DEVICE_TYPE
  nt!POWER_ACTION
  nt!POWER_ACTION_POLICY
  nt!PPM_IDLE_ACCOUNTING
  nt!PPM_IDLE_STATE_ACCOUNTING
  nt!PROCESSOR_IDLESTATE_INFO
  nt!PROCESSOR_IDLESTATE_POLICY
  nt!PROCESSOR_PERFSTATE_POLICY
  nt!PROFILE_STATUS
  nt!ReplacesCorHdrNumericDefines
  nt!SYSTEM_POWER_CAPABILITIES
  nt!SYSTEM_POWER_CONDITION
  nt!SYSTEM_POWER_LEVEL
  nt!UoWActionType
  nt!WHEA_PCIEXPRESS_DEVICE_TYPE
  nt!WHEA_PCIXDEVICE_REGISTER_PAIR

- Dmitry Vostokov @ DumpAnalysis.org -