Software Diagnostics Library

Recently, a reader of this blog sent me a minidump file and a debugger log of an application that had about 300 modules loaded in a process address space. What was interesting is the huge amount of ModLoad / Unload module debugger events in the log prior to an access violation exception. Some modules were loaded / unloaded many times, for example (only included lines for just one module but there were many other):

[...]
ModLoad: 16640000 16649000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 16640000
[...]
ModLoad: 192b0000 192b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 192b0000
[...]
ModLoad: 192b0000 192b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 192b0000
[...]
ModLoad: 161b0000 161b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161b0000
[...]
ModLoad: 161e0000 161e9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161e0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 161f0000 161f9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 161f0000
[...]
ModLoad: 171b0000 171b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 171b0000
[...]
ModLoad: 25180000 25189000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 25180000
[...]
ModLoad: 171b0000 171b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 171b0000
[...]
ModLoad: 171b0000 171b9000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 171b0000
[...]
[...]
[...]
ModLoad: 0df60000 0df69000   X:\Client\Bin\ModuleA.dll
[...]
Unload module X:\Client\Bin\ModuleA.dll at 0df60000
[...]
(f38.560): Access violation - code c0000005 (first chance)
---
--- 1st chance AccessViolation exception ----
[...]

We see the component ModuleA was loaded at different addresses and this looks similar to a singleton object factory with Create / Destroy operations that resembles heap operations Alloc and Free where every allocation can place the same object at a different address. This is why I call all this a component or module heap. The application was COM-based and every domain-specific object was implemented in a separate in-proc COM DLL. There were thousands of such objects.

PS. This story reminds me that when I learnt about COM in 90s I wanted to redesign my word processor written in C to have every paragraph, line and even word to be implemented as a COM object.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Thinking again about prescriptive value-added debugging and the sin of requesting yet another memory dump from a customer for the sake of curiosity (the so called “further analysis”):

Dump analysis matters, but business results matter more.

Aaron Erickson, The Nomadic Developer

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Optimized Code pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I recently got a crying request from a reader of my blog to analyze the source of frequent bugchecks on a newly bought computer running Windows 7. I got 8 kernel minidumps with 5 different bugchecks. However, inspection of the default analysis revealed common Fault Context pattern of high resource consumption flight simulator processes in 6 minidumps. Most fault IPs were showing signs of Wild Code pattern and that most probably implicated Hardware Error (Looks like WinDbg suggests that MISALIGNED_IP implicates hardware). Here is the listing of relevant output fragments with attempts to disassemble code around IP (Instruction Pointer) to see if code make any sense (magenta color means the valid that should have been instead of misaligned code highlighted in red):

Windows 7 Kernel Version 7600 MP (4 procs) Free x86 compatible

Debug session time: Fri Jan  8 20:31:15.121 2010 (GMT+0)
System Uptime: 0 days 2:54:44.916

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  2

TRAP_FRAME:  807e6ea4 -- (.trap 0xffffffff807e6ea4)
ErrCode = 00000002
eax=872082a7 ebx=80028d5f ecx=b3348635 edx=87208638 esi=80280001 edi=000082a7
eip=8d613485 esp=807e6f18 ebp=6f248635 iopl=0  nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
USBPORT!USBPORT_Xdpc_End+0xa6:
8d613485 897904          mov     dword ptr [ecx+4],edi ds:0023:b3348639=????????
Resetting default scope

STACK_TEXT: 
807e6ea4 8d613485 badb0d00 87208638 82a7b334 nt!KiTrap0E+0x2cf
807e6f24 8d613d18 00000000 86358720 86358002 USBPORT!USBPORT_Xdpc_End+0xa6
807e6f48 82aa33b5 8635872c 86358002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
807e6fa4 82aa3218 807c6120 87e7e950 00000000 nt!KiExecuteAllDpcs+0xf9
807e6ff4 82aa29dc 9f7e1ce4 00000000 00000000 nt!KiRetireDpcList+0xd5
807e6ff8 9f7e1ce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa29dc 00000000 0000001a 00d6850f bb830000 0x9f7e1ce4

Debug session time: Fri Jan  8 20:42:16.395 2010 (GMT+0)
System Uptime: 0 days 0:10:22.815

2: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

TRAP_FRAME:  8d91cbc4 -- (.trap 0xffffffff8d91cbc4)
ErrCode = 00000002
eax=00000000 ebx=8d901a00 ecx=86570108 edx=86570108 esi=8d905884 edi=86573920
eip=911e5f5d esp=8d91cc38 ebp=8d91cc78 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???
Resetting default scope

IMAGE_NAME:  hardware

2: kd> u HDAudBus!HdaController::NotificationDpc+14d
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???
911e5f5e ff              ???
911e5f5f ff6a00          jmp     fword ptr [edx]
911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4

2: kd> uf HDAudBus!HdaController::NotificationDpc
[...]
HDAudBus!HdaController::NotificationDpc+0x135:
911e5f45 8b45d8          mov     eax,dword ptr [ebp-28h]
911e5f48 c6405400        mov     byte ptr [eax+54h],0
911e5f4c 8b4dd8          mov     ecx,dword ptr [ebp-28h]
911e5f4f 83c148          add     ecx,48h
911e5f52 8a55e7          mov     dl,byte ptr [ebp-19h]
911e5f55 ff1510a01e91    call    dword ptr [HDAudBus!_imp_KfReleaseSpinLock (911ea010)]

HDAudBus!HdaController::NotificationDpc+0x14b:
911e5f5b e909ffffff      jmp     HDAudBus!HdaController::NotificationDpc+0x59 (911e5e69)

HDAudBus!HdaController::NotificationDpc+0x150:
911e5f60 6a00            push    0
911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4
911e5f6f 6a08            push    8
911e5f71 6a02            push    2
911e5f73 e818180000      call    HDAudBus!HDABusWmiLogETW (911e7790)
911e5f78 8b4df0          mov     ecx,dword ptr [ebp-10h]
911e5f7b 64890d00000000  mov     dword ptr fs:[0],ecx
911e5f82 59              pop     ecx
911e5f83 5f              pop     edi
911e5f84 5e              pop     esi
911e5f85 5b              pop     ebx
911e5f86 8be5            mov     esp,ebp
911e5f88 5d              pop     ebp
911e5f89 c21000          ret     10h

Debug session time: Fri Jan  8 21:32:04.096 2010 (GMT+0)
System Uptime: 0 days 0:49:10.517

1: kd> !analyze -v

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)

Arg1: c000001d, The exception code that was not handled

EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

TRAP_FRAME:  a99e3644 -- (.trap 0xffffffffa99e3644)
ErrCode = 00000000
eax=000000fe ebx=8556a2b0 ecx=754764cd edx=00000001 esi=858ad008 edi=858ad048
eip=82ada4c2 esp=a99e36b8 ebp=a99e3704 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
nt!IopCompleteRequest+0×3ac:
82ada4c2 02cd            add     cl,ch

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  1

MISALIGNED_IP:
nt!IopCompleteRequest+3ac
82ada4c2 02cd            add     cl,ch

IMAGE_NAME:  hardware

1: kd> uf nt!IopCompleteRequest+3ac
nt!IopCompleteRequest+0×3a9:
82ada4bf 82680002        sub     byte ptr [eax],2
82ada4c3 cd82            int     82h
82ada4c5 50              push    eax
82ada4c6 ff75e0          push    dword ptr [ebp-20h]
82ada4c9 57              push    edi
82ada4ca e881830100      call    nt!KeInitializeApc (82af2850)
82ada4cf 6a02            push    2
82ada4d1 6a00            push    0
82ada4d3 ff7628          push    dword ptr [esi+28h]
82ada4d6 57              push    edi
82ada4d7 e8d2830100      call    nt!KeInsertQueueApc (82af28ae)
82ada4dc 33ff            xor     edi,edi
82ada4de eb5f            jmp     nt!IopCompleteRequest+0×429 (82ada53f)

1: kd> ub nt!IopCompleteRequest+3ac
                                  ^ Unable to find valid previous instruction for 'ub nt!IopCompleteRequest+3ac'

Debug session time: Sat Jan  9 07:45:24.155 2010 (GMT+0)
System Uptime: 0 days 2:09:39.576

0: kd> !analyze -v

UNEXPECTED_KERNEL_MODE_TRAP (7f)

Arg1: 0000000d, EXCEPTION_GP_FAULT

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  6

STACK_TEXT: 
a24b3bd8 90f9e956 badb0d00 00000000 ddf1ba50 nt!KiSystemFatalException+0xf
a24b3cc4 90f93f2b 00000001 00000004 00000004 HDAudBus!HDABusWmiLogETW+0x1c6
a24b3d08 82a817ad 864a6280 86541000 a24b3d34 HDAudBus!HdaController::Isr+0x2b
a24b3d08 20c40d61 864a6280 86541000 a24b3d34 nt!KiInterruptDispatch+0x6d
WARNING: Frame IP not in any known module. Following frames may be wrong.
1343f8ea 00000000 00000000 00000000 00000000 0x20c40d61

Debug session time: Sat Jan  9 08:52:03.454 2010 (GMT+0)
System Uptime: 0 days 1:05:54.249

0: kd> !analyze -v

IRQL_NOT_LESS_OR_EQUAL (a)

CURRENT_IRQL:  2

PROCESS_NAME:  FlightSimulatorA.exe

TRAP_FRAME:  8078adf0 -- (.trap 0xffffffff8078adf0)
ErrCode = 00000002
eax=8632e2a6 ebx=00000000 ecx=880fb200 edx=00000118 esi=00000007 edi=8632e27c
eip=82a0c967 esp=8078ae64 ebp=c1e2baa0 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
hal!HalBuildScatterGatherList+0xf3:
82a0c967 8901            mov     dword ptr [ecx],eax  ds:0023:880fb200=????????
Resetting default scope

STACK_TEXT: 
8078adf0 82a0c967 badb0d00 00000118 82b5f466 nt!KiTrap0E+0x2cf
8078ae78 82a0cc16 880fb218 86379028 8632e260 hal!HalBuildScatterGatherList+0xf3
8078aea8 909b3e70 8651c6b0 86379028 8632e260 hal!HalGetScatterGatherList+0x26
8078aef4 909b3807 86379028 86379970 00000007 USBPORT!USBPORT_Core_iMapTransfer+0x21e
8078af24 909add18 86379028 86379970 86379002 USBPORT!USBPORT_Core_UsbMapDpc_Worker+0x1e3
8078af48 82aa73b5 8637997c 86379002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8078afa4 82aa7218 82b68d20 88139a98 00000000 nt!KiExecuteAllDpcs+0xf9
8078aff4 82aa69dc 9fd8cce4 00000000 00000000 nt!KiRetireDpcList+0xd5
8078aff8 9fd8cce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa69dc 00000000 0000001a 00d6850f bb830000 0x9fd8cce4

Debug session time: Sat Jan  9 16:34:48.134 2010 (GMT+0)
System Uptime: 0 days 1:53:05.929

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

PROCESS_NAME:  firefox.exe

TRAP_FRAME:  bb92449c -- (.trap 0xffffffffbb92449c)
ErrCode = 00000000
eax=000005b4 ebx=0db19ba0 ecx=80000000 edx=00000001 esi=85fdff29 edi=bb924530
eip=8bc7e2c7 esp=bb924510 ebp=bb924638 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
tcpip!TcpBeginTcbSend+0xa83:
8bc7e2c7 eb06            jmp     tcpip!TcpBeginTcbSend+0xa8b (8bc7e2cf)
Resetting default scope

STACK_TEXT: 
bb92449c 8bc7e2c7 badb0d00 00000001 00000000 nt!KiTrap0E+0x2cf
bb924638 8bc7d2bf 87b39c78 00000000 00000001 tcpip!TcpBeginTcbSend+0xa83
bb92479c 8bc814b5 87b39c78 00000000 00000001 tcpip!TcpTcbSend+0x426
bb9247bc 8bc7f349 87b39c78 87fa6c38 00000000 tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0x157
bb92481c 8bc81846 87b39c78 bb92491c 00000000 tcpip!TcpEnqueueTcbSend+0x3ca
bb924838 82a95f8a bb9248c8 96d9c9d2 00000000 tcpip!TcpTlConnectionSendCalloutRoutine+0x17
bb9248a0 8bc80a0b 8bc8182f bb9248c8 00000000 nt!KeExpandKernelStackAndCalloutEx+0x132
bb9248d8 908b5d27 87b39c01 bb924900 85572e18 tcpip!TcpTlConnectionSend+0x73
bb92493c 908bb2e3 00d4f1e0 85572e18 85572eac tdx!TdxSendConnection+0x1d7
bb924958 82a424bc 86236b80 85572e18 862389c0 tdx!TdxTdiDispatchInternalDeviceControl+0x115
bb924970 908d65ca 86d0e0c8 00000000 86238990 nt!IofCallDriver+0x63
WARNING: Stack unwind information not available. Following frames may be wrong.
bb9249c8 908d17f8 86238990 85572e18 85572ed0 aswTdi+0x55ca
bb924a28 82a424bc 862388d8 85572e18 8623f0e8 aswTdi+0x7f8
bb924a40 90935310 8623f030 82a424bc 8623f030 nt!IofCallDriver+0x63
bb924a60 90900a0e 2b1c89ba bb924b20 00000001 aswRdr+0x310
bb924ab0 908ed542 00000000 908ed542 87a5c530 afd!AfdFastConnectionSend+0x2a6
bb924c28 82c608f7 87ec6701 00000001 02b5f8cc afd!AfdFastIoDeviceControl+0x53d
bb924cd0 82c634ac 85a89c10 0000024c 00000000 nt!IopXxxControlFile+0x2d0
bb924d04 82a4942a 00000240 0000024c 00000000 nt!NtDeviceIoControlFile+0x2a
bb924d04 774464f4 00000240 0000024c 00000000 nt!KiFastCallEntry+0x12a
02b5f920 00000000 00000000 00000000 00000000 0x774464f4

1: kd> u 8bc7e2cf
tcpip!TcpBeginTcbSend+0xa8b:
8bc7e2cf 83bd18ffffff00  cmp     dword ptr [ebp-0E8h],0
8bc7e2d6 0f84d1000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2dc 8d85f8feffff    lea     eax,[ebp-108h]
8bc7e2e2 3bf8            cmp     edi,eax
8bc7e2e4 0f85c3000000    jne     tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2ea 83bd54ffffff00  cmp     dword ptr [ebp-0ACh],0
8bc7e2f1 0f84b6000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2f7 f7433c00002000  test    dword ptr [ebx+3Ch],200000h

Debug session time: Sat Jan  9 19:42:50.817 2010 (GMT+0)
System Uptime: 0 days 3:07:23.612

3: kd> !analyze -v

BUGCODE_USB_DRIVER (fe)
USB Driver bugcheck, first parameter is USB bugcheck code.
Arguments:
Arg1: 00000006, USBBUGCODE_BAD_SIGNATURE An Internal data structure (object)
 has been corrupted.
Arg2: 864b20e0, Object address
Arg3: 4f444648, Signature that was expected
Arg4: 00000000

PROCESS_NAME:  System

CURRENT_IRQL:  2

STACK_TEXT: 
8d952b8c 90fa1025 000000fe 00000006 864b20e0 nt!KeBugCheckEx+0x1e
8d952ba8 90fa6672 864b20e0 4f444668 4f444648 USBPORT!USBPORT_AssertSig+0x20
8d952bc8 90fa4553 864b2028 85c57d10 82a8b334 USBPORT!USBPORT_FlushAdapterDBs+0x1b
8d952c00 90fa5178 00000001 856e3ab8 87fb98c0 USBPORT!USBPORT_Core_iCompleteDoneTransfer+0x3cb
8d952c2c 90fa89af 864b2028 864b20f0 864b2a98 USBPORT!USBPORT_Core_iIrpCsqCompleteDoneTransfer+0x33b
8d952c54 90fa2d18 864b2028 864b2a98 864b2002 USBPORT!USBPORT_Core_UsbIocDpc_Worker+0xbc
8d952c78 82ab33b5 864b2aa4 864b2002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8d952cd4 82ab3218 8d936120 8d93b800 00000000 nt!KiExecuteAllDpcs+0xf9
8d952d20 82ab3038 00000000 0000000e 00000000 nt!KiRetireDpcList+0xd5
8d952d24 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x38

Debug session time: Sun Jan 10 04:06:19.856 2010 (GMT+0)
System Uptime: 0 days 0:23:05.651

1: kd> !analyze -v

PAGE_FAULT_IN_NONPAGED_AREA (50)

PROCESS_NAME:  FlightSimulatorB.exe

CURRENT_IRQL:  0

TRAP_FRAME:  a127fa30 -- (.trap 0xffffffffa127fa30)
ErrCode = 00000000
eax=a127fec8 ebx=00000000 ecx=00000011 edx=86488ba0 esi=86488b78 edi=00000000
eip=8b83b87d esp=a127faa4 ebp=a127fab8 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
fltmgr!TreeFindNodeOrParent+0×9:
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0×764c9) (8b4974c9)[ebp],al ss:0010:2c716f81=??
Resetting default scope

MISALIGNED_IP:
fltmgr!TreeFindNodeOrParent+9
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0x764c9) (8b4974c9)[ebp],al

STACK_TEXT: 
a127fa18 82a8d5f8 00000000 8b497414 00000000 nt!MmAccessFault+0x106
a127fa18 8b83b87d 00000000 8b497414 00000000 nt!KiTrap0E+0xdc
a127fab8 8b834340 86488ba4 86e5e458 00000000 fltmgr!TreeFindNodeOrParent+0x9
a127faf8 8b83440a 86488b78 86e5e458 00000000 fltmgr!GetContextFromStreamList+0x50
a127fb14 8b86c6da 86e5e458 86488b78 a127fb40 fltmgr!FltGetStreamContext+0x34
a127fb44 8b866b35 87f30718 a127fb98 a127fba8 fileinfo!FIStreamGet+0x36
a127fbac 8b833aeb 87f30718 a127fbcc a127fbf8 fileinfo!FIPreReadWriteCallback+0xf1
a127fc18 8b83617b a127fc54 85cfd738 a127fcac fltmgr!FltpPerformPreCallbacks+0x34d
a127fc30 8b848c37 0027fc54 8b848ad4 00000000 fltmgr!FltpPassThroughFastIo+0x3d
a127fc74 82c96b32 85cfd738 a127fcb4 00001000 fltmgr!FltpFastIoRead+0x163
a127fd08 82a8a42a 86e484c0 00000000 00000000 nt!NtReadFile+0x2d5
a127fd08 775864f4 86e484c0 00000000 00000000 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0202fc8c 00000000 00000000 00000000 00000000 0x775864f4

IMAGE_NAME:  hardware

1: kd> u fltmgr!TreeFindNodeOrParent
fltmgr!TreeFindNodeOrParent:
8b83b874 8bff            mov     edi,edi
8b83b876 55              push    ebp
8b83b877 8bec            mov     ebp,esp
8b83b879 8b4508          mov     eax,dword ptr [ebp+8]
8b83b87c 8b08            mov     ecx,dword ptr [eax]
8b83b87e 85c9            test    ecx,ecx
8b83b880 7449            je      fltmgr!TreeFindNodeOrParent+0×57 (8b83b8cb)
8b83b882 8b5510          mov     edx,dword ptr [ebp+10h]

1: kd> ub 8b834340
fltmgr!GetContextFromStreamList+0x37:
8b834327 8bcb            mov     ecx,ebx
8b834329 ff15a4d0838b    call    dword ptr [fltmgr!_imp_ExfAcquirePushLockShared (8b83d0a4)]
8b83432f 33db            xor     ebx,ebx
8b834331 895dfc          mov     dword ptr [ebp-4],ebx
8b834334 ff7510          push    dword ptr [ebp+10h]
8b834337 ff750c          push    dword ptr [ebp+0Ch]
8b83433a 57              push    edi
8b83433b e896750000      call    fltmgr!TreeLookup (8b83b8d6)

1: kd> uf 8b83b8d6
fltmgr!TreeLookup:
8b83b8d6 8bff            mov     edi,edi
8b83b8d8 55              push    ebp
8b83b8d9 8bec            mov     ebp,esp
8b83b8db 8d4510          lea     eax,[ebp+10h]
8b83b8de 50              push    eax
8b83b8df ff7510          push    dword ptr [ebp+10h]
8b83b8e2 ff750c          push    dword ptr [ebp+0Ch]
8b83b8e5 ff7508          push    dword ptr [ebp+8]
8b83b8e8 e887ffffff      call    fltmgr!TreeFindNodeOrParent (8b83b874)
8b83b8ed 48              dec     eax
8b83b8ee f7d8            neg     eax
8b83b8f0 1bc0            sbb     eax,eax
8b83b8f2 f7d0            not     eax
8b83b8f4 234510          and     eax,dword ptr [ebp+10h]
8b83b8f7 5d              pop     ebp
8b83b8f8 c20c00          ret     0Ch

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In the case of multiple different faults like bugchecks and/or different crash points, stack traces and modules we can look at what is common among them. It could be their process context, which can easily be seen from the default analysis:

1: kd> !analyze -v

[...]

PROCESS_NAME:  Application.exe

We give this pattern a name Fault Context. Then we can look whether an application is resource consumption intensive (could implicate hardware faults) like games and simulators or uses its own drivers (implicates latent corruption). In a production environment it can also be removed if it is functionally non-critical and can be avoided or replaced. See also a forthcoming case study.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Limiting high-priority ready threads by a processor affinity (p. 391)

Thread dispatch reasons: ready, leaves running state, priority change, processor affinity change (p. 392)

Thread vs. process scheduling granularity (pp. 392 - 393)

Thread priority level 0 is reserved for zero page thread (p. 393)

2 pespectives on thread priority levels (pp. 393 - 394)

Changing CPU-intensive process base priority instead of priority of individual threads (p. 395)

Increased based priority for special processes (p. 395) - here is a comparison of base priorities between lsm.exe and smss.exe from x64 W2K8:

0: kd> !process fffffa80047ffc10
PROCESS fffffa80047ffc10
SessionId: 0  Cid: 0294    Peb: 7fffffd6000  ParentCid: 0238
DirBase: b1c4e000  ObjectTable: fffff88007f05cd0  HandleCount: 173.
Image: lsm.exe
VadRoot fffffa80046dd720 Vads 68 Clone 0 Private 462. Modified 0. Locked 0.
DeviceMap fffff88000007310
Token                             fffff88007f376f0
ElapsedTime                       00:04:17.552
UserTime                          00:00:00.015
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         69000
QuotaPoolUsage[NonPagedPool]      7072
Working Set Sizes (now,min,max)  (1314, 50, 345) (5256KB, 200KB, 1380KB)
PeakWorkingSetSize                1318
VirtualSize                       36 Mb
PeakVirtualSize                   38 Mb
PageFaultCount                    1375
MemoryPriority                    BACKGROUND
    BasePriority                      8
CommitCharge                      756

0: kd> !process fffffa80046d9040
PROCESS fffffa80046d9040
SessionId: none  Cid: 019c    Peb: 7fffffdf000  ParentCid: 0004
DirBase: bccd5000  ObjectTable: fffff880005f45b0  HandleCount:  33.
Image: smss.exe
VadRoot fffffa80046d97e0 Vads 19 Clone 0 Private 96. Modified 24. Locked 0.
DeviceMap fffff88000007310
Token                             fffff88000964af0
ElapsedTime                       00:04:40.343
UserTime                          00:00:00.000
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         10392
QuotaPoolUsage[NonPagedPool]      1728
Working Set Sizes (now,min,max)  (254, 50, 345) (1016KB, 200KB, 1380KB)
PeakWorkingSetSize                254
VirtualSize                       6 Mb
PeakVirtualSize                   16 Mb
PageFaultCount                    458
MemoryPriority                    BACKGROUND
    BasePriority                      11
CommitCharge                      127

Sleep(0) to relinquish the rest of quantum (p. 396) 

Realtime Notepad (pp. 397 - 398) - I’m often asked why it doesn’t affect performance? This is because most threads in a system are waiting and notepad is waiting for window messages to process like keyboard and mouse. It is more noticeable when a realtime thread starts looping - it becomes scheduled every time

WSRM (Windows System Resource Manager) (pp. 398 - 399) - Looks good to prevent CPU spikes and memory leaks to come out of control

Thread priorities and IRQL (pp. 399 - 400) - in another words these concepts are orthogonal (independent from each other)

On The Same Page - coming to the same conclusion as another engineer when looking at a memory dump or a software trace. Literally means the same page of memory where an exception occurred or a stack trace is reconstructed or the same “page” when browsing a software trace output using a viewer.

Examples: Aha, we are on the same page!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Lateral Damage pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a new methodology I’m working upon. The idea came from reading “About the Author” page in a book I got yesterday in my post:

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting

I post a review here and on Amazon when finished reading. Just a few words now. This is the first career book I’m reading where I find pages in roman numerals useful. The page xiii itself looks like a good template (or an example) for a business-oriented CV summary. Thinking now about updating my CV book (2nd edition?):

Resume and CV: As a Book

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for False Positive Dump pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

While reading two balanced books about Trotsky I started to admire the Russian signature “С коммунистическим приветом, <имя>” (”S kommunisticheskim privetom, <name>”) that can be translated as “With communist greetings, <name>”. Did they laugh in their red sleeves? When I was at a primary school I loved History (that was long before I saw a computer at Moscow University and I loved Chemistry in secondary and high schools). In fact, to realize my childhood dream, OpenTask, an iterative and incremental publisher, plans to publish a centennial balanced 2 volume bilingual history of Russian revolutions (the work has began already): http://www.opentask.com/history-titles

While commuting today I devised a similar but rectangular 2×2 greeting:

With fix-privet,
Dr. Debuglove

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When the performance is a performative and the performative is a performance we think about performability:

A virtualization is virtualization.

Peter Brook, The Empty Space

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Clock cycle counter for measuring CPU activity  (p. 382)

Process Explorer usage to inspect hung threads (p. 383) - useful for coupled processes (http://www.dumpanalysis.org/blog/index.php/2007/09/26/crash-dump-analysis-patterns-part-28/) and could be great with simultaneous WinDbg session to inspect wait chains (http://www.dumpanalysis.org/blog/index.php/2009/02/17/wait-chain-patterns/)

Process Explorer shows both thread and WOW64 thread stacks on x64 (p. 384)

Thread stack and context query limitations for protected processes (pp. 384 - 386)

Thread pool mechanism was moved into kernel space in Vista (p. 387)

TpWorkerFactory and I/O completion ports and KQUEUE (pp. 387 - 388) - see also a “brief guide” to I/O completion ports: http://www.dumpanalysis.org/blog/index.php/2007/11/27/understanding-io-completion-ports/ 

The mystery of ntdll!TppWorkerThread in stack traces (pp. 389 - 390)

Today we introduce an icon for Dynamic Memory Corruption (kernel pool) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

W32THREAD (p. 371) - One candidate in _ETHREAD that points to it is Tcb.Win32Thread. One interesting code I found on how to extract window message queues from it: http://www.cc.gatech.edu/~brendan/volatility/dl/threadqueues.py. _W32THREAD structure on x64 W2K8 (we also see that is points to _ETHREAD):

0: kd> dt _W32THREAD
win32k!_W32THREAD
+0x000 pEThread         : Ptr64 _ETHREAD
+0x008 RefCount         : Uint4B
+0x010 ptlW32           : Ptr64 _TL
+0x018 pgdiDcattr       : Ptr64 Void
+0x020 pgdiBrushAttr    : Ptr64 Void
+0x028 pUMPDObjs        : Ptr64 Void
+0x030 pUMPDHeap        : Ptr64 Void
+0x038 pUMPDObj         : Ptr64 Void
+0x040 pProxyPort       : Ptr64 Void
+0x048 pClientID        : Ptr64 Void
+0x050 GdiTmpTgoList    : _LIST_ENTRY

!thread output fields (p. 376) - Stack Base and Limit fields can be useful to dump raw stack data via dps command to see execution residue or when reconstructing stack trace, see, for example, this pattern: http://www.dumpanalysis.org/blog/index.php/2009/10/23/crash-dump-analysis-patterns-part-88/

tlist utility (p. 377)

Thread creation calls (pp. 380 - 381) - a condensed view of top level function calls on x64 W2K8:

0: kd> uf /c CreateThread
kernel32!CreateThread (00000000`7731c1c0)
kernel32!CreateThread+0x28 (00000000`7731c1e8):
call to kernel32!CreateRemoteThread (00000000`7731c200)

0: kd> uf /c CreateRemoteThread
Flow analysis was incomplete, some code may be missing
kernel32!CreateRemoteThread (00000000`7731c200)
kernel32!CreateRemoteThread+0x134 (00000000`7731c334):
    call to ntdll!NtCreateThreadEx (00000000`77477790)
  kernel32!CreateRemoteThread+0×166 (00000000`7731c366):
call to ntdll!RtlAllocateActivationContextStack (00000000`77456900)
kernel32!CreateRemoteThread+0×1b4 (00000000`7731c3b4):
call to ntdll!RtlQueryInformationActivationContext (00000000`77456b20)
kernel32!CreateRemoteThread+0×241 (00000000`7731c441):
    call to ntdll!CsrClientCallServer (00000000`7747a460)
  kernel32!CreateRemoteThread+0×281 (00000000`7731c47d):
    call to ntdll!ZwResumeThread (00000000`77477230)
  kernel32!CreateRemoteThread+0×38b (00000000`7731c4ae):
call to kernel32!_security_check_cookie (00000000`7732c200)

0: kd> uf /c NtCreateThreadEx
ntdll!NtCreateThreadEx (00000000`77477790)
no calls found

0: kd> uf NtCreateThreadEx
ntdll!NtCreateThreadEx:
00000000`77477790 4c8bd1          mov     r10,rcx
00000000`77477793 b8a5000000      mov     eax,0A5h
00000000`77477798 0f05            syscall
00000000`7747779a c3              ret

0: kd> uf /c nt!NtCreateThreadEx
nt!NtCreateThreadEx (fffff800`01af60fc)
nt!NtCreateThreadEx+0x3d (fffff800`01af6139):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0x5b (fffff800`01af6157):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0x99 (fffff800`01af6195):
call to nt!memset (fffff800`0187a4d0)
nt!NtCreateThreadEx+0xc8 (fffff800`01af61c4):
call to nt!PspBuildCreateProcessContext (fffff800`01af5204)
nt!NtCreateThreadEx+0x1e1 (fffff800`01af62dd):
    call to nt!PspCreateThread (fffff800`01af5d40)
  nt!NtCreateThreadEx+0×1f0 (fffff800`01af62ec):
call to nt!PspDeleteCreateProcessContext (fffff800`01af68f0)

0: kd> uf /c nt!PspCreateThread
nt!PspCreateThread (fffff800`01af5d40)
nt!PspCreateThread+0x102 (fffff800`01af5e42):
call to nt!ObReferenceObjectByHandle (fffff800`01ad8110)
nt!PspCreateThread+0x15b (fffff800`01af5e9b):
call to nt!ObfReferenceObject (fffff800`01883250)
nt!PspCreateThread+0x22f (fffff800`01af5f6f):
call to nt!PspAllocateThread (fffff800`01af6338)
nt!PspCreateThread+0x243 (fffff800`01af5f83):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateThread+0x2a6 (fffff800`01af5fe6):
call to nt!PspInsertThread (fffff800`01af4c10)
nt!PspCreateThread+0x318 (fffff800`01af6058):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt!PspCreateThread+0x32a (fffff800`01af606a):
call to nt!_security_check_cookie (fffff800`01895e50)
nt!PspCreateThread+0x36a (fffff800`01af60aa):
call to nt!ObfReferenceObject (fffff800`01883250)
nt!PspCreateThread+0x3a2 (fffff800`01af60e2):
call to nt!ExfAcquireRundownProtection (fffff800`0184f66c)
nt! ?? ::NNGAKEGL::`string'+0x2816e (fffff800`01b3628e):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x281ad (fffff800`01b362ca):
call to nt!ExfReleaseRundownProtection (fffff800`0184f690)
nt! ?? ::NNGAKEGL::`string'+0x281ce (fffff800`01b362eb):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x281d8 (fffff800`01b362f5):
call to nt!ObfDereferenceObject (fffff800`0187cde0)
nt! ?? ::NNGAKEGL::`string'+0x281e7 (fffff800`01b36304):
call to nt!ExfReleaseRundownProtection (fffff800`0184f690)
nt! ?? ::NNGAKEGL::`string'+0x281ff (fffff800`01b3631c):
call to nt!KiCheckForKernelApcDelivery (fffff800`0183c754)
nt! ?? ::NNGAKEGL::`string'+0x2821a (fffff800`01b36337):
call to nt!PspTerminateThreadByPointer (fffff800`01ad30dc)

Today we introduce an icon for Dynamic Memory Corruption (process heap) pattern:

B/W

Color

Another alternative I considered to use is a chain metaphor but decided that it is more appropriate for linked lists.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just launched my game Blackjack, Jackpot and Slot applets written in pure Java 1.0 in 1997 and they still work in IE (I do launch them occasionally every few years):

http://vostokov.opentask.com/Applets.htm

Can’t believe it!

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Today we introduce an icon for Multiple Exceptions (kernel mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just noticed that Memory Dump Analysis Anthology, Volume 3 is on a metaphysics bestseller list on Amazon DE today (the volume indeed has a few articles related to Memoidealism and memory dumps + memory traces worldview):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I borrowed a pattern icon idea from the book I’m reading now: Algorithms in a Nutshell

From now on, every memory dump analysis pattern (and later trace analysis patterns) will have platform-independent pictorial representation. Today we introduce an icon for Multiple Exceptions (user mode) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -