Fault context, wild code and hardware error: pattern cooperation

I recently got a crying request from a reader of my blog to analyze the source of frequent bugchecks on a newly bought computer running Windows 7. I got 8 kernel minidumps with 5 different bugchecks. However, inspection of the default analysis revealed common Fault Context pattern of high resource consumption flight simulator processes in 6 minidumps. Most fault IPs were showing signs of Wild Code pattern and that most probably implicated Hardware Error (Looks like WinDbg suggests that MISALIGNED_IP implicates hardware). Here is the listing of relevant output fragments with attempts to disassemble code around IP (Instruction Pointer) to see if code make any sense (magenta color means the valid that should have been instead of misaligned code highlighted in red):

Windows 7 Kernel Version 7600 MP (4 procs) Free x86 compatible

Debug session time: Fri Jan  8 20:31:15.121 2010 (GMT+0)
System Uptime: 0 days 2:54:44.916

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  2

TRAP_FRAME:  807e6ea4 -- (.trap 0xffffffff807e6ea4)
ErrCode = 00000002
eax=872082a7 ebx=80028d5f ecx=b3348635 edx=87208638 esi=80280001 edi=000082a7
eip=8d613485 esp=807e6f18 ebp=6f248635 iopl=0  nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
USBPORT!USBPORT_Xdpc_End+0xa6:
8d613485 897904          mov     dword ptr [ecx+4],edi ds:0023:b3348639=????????
Resetting default scope

STACK_TEXT: 
807e6ea4 8d613485 badb0d00 87208638 82a7b334 nt!KiTrap0E+0x2cf
807e6f24 8d613d18 00000000 86358720 86358002 USBPORT!USBPORT_Xdpc_End+0xa6
807e6f48 82aa33b5 8635872c 86358002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
807e6fa4 82aa3218 807c6120 87e7e950 00000000 nt!KiExecuteAllDpcs+0xf9
807e6ff4 82aa29dc 9f7e1ce4 00000000 00000000 nt!KiRetireDpcList+0xd5
807e6ff8 9f7e1ce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa29dc 00000000 0000001a 00d6850f bb830000 0x9f7e1ce4

Debug session time: Fri Jan  8 20:42:16.395 2010 (GMT+0)
System Uptime: 0 days 0:10:22.815

2: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

TRAP_FRAME:  8d91cbc4 -- (.trap 0xffffffff8d91cbc4)
ErrCode = 00000002
eax=00000000 ebx=8d901a00 ecx=86570108 edx=86570108 esi=8d905884 edi=86573920
eip=911e5f5d esp=8d91cc38 ebp=8d91cc78 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???

Resetting default scope

IMAGE_NAME:  hardware

2: kd> u HDAudBus!HdaController::NotificationDpc+14d
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???
911e5f5e ff              ???
911e5f5f ff6a00          jmp     fword ptr [edx]

911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4

2: kd> uf HDAudBus!HdaController::NotificationDpc
[...]
HDAudBus!HdaController::NotificationDpc+0x135:
911e5f45 8b45d8          mov     eax,dword ptr [ebp-28h]
911e5f48 c6405400        mov     byte ptr [eax+54h],0
911e5f4c 8b4dd8          mov     ecx,dword ptr [ebp-28h]
911e5f4f 83c148          add     ecx,48h
911e5f52 8a55e7          mov     dl,byte ptr [ebp-19h]
911e5f55 ff1510a01e91    call    dword ptr [HDAudBus!_imp_KfReleaseSpinLock (911ea010)]

HDAudBus!HdaController::NotificationDpc+0x14b:
911e5f5b e909ffffff      jmp     HDAudBus!HdaController::NotificationDpc+0x59 (911e5e69)

HDAudBus!HdaController::NotificationDpc+0x150:
911e5f60 6a00            push    0
911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4
911e5f6f 6a08            push    8
911e5f71 6a02            push    2
911e5f73 e818180000      call    HDAudBus!HDABusWmiLogETW (911e7790)
911e5f78 8b4df0          mov     ecx,dword ptr [ebp-10h]
911e5f7b 64890d00000000  mov     dword ptr fs:[0],ecx
911e5f82 59              pop     ecx
911e5f83 5f              pop     edi
911e5f84 5e              pop     esi
911e5f85 5b              pop     ebx
911e5f86 8be5            mov     esp,ebp
911e5f88 5d              pop     ebp
911e5f89 c21000          ret     10h

Debug session time: Fri Jan  8 21:32:04.096 2010 (GMT+0)
System Uptime: 0 days 0:49:10.517

1: kd> !analyze -v

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)

Arg1: c000001d, The exception code that was not handled

EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

TRAP_FRAME:  a99e3644 -- (.trap 0xffffffffa99e3644)
ErrCode = 00000000
eax=000000fe ebx=8556a2b0 ecx=754764cd edx=00000001 esi=858ad008 edi=858ad048
eip=82ada4c2 esp=a99e36b8 ebp=a99e3704 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
nt!IopCompleteRequest+0×3ac:
82ada4c2 02cd            add     cl,ch

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  1

MISALIGNED_IP:
nt!IopCompleteRequest+3ac
82ada4c2 02cd            add     cl,ch

IMAGE_NAME:  hardware

1: kd> uf nt!IopCompleteRequest+3ac
nt!IopCompleteRequest+0×3a9:
82ada4bf 82680002        sub     byte ptr [eax],2
82ada4c3 cd82            int     82h

82ada4c5 50              push    eax
82ada4c6 ff75e0          push    dword ptr [ebp-20h]
82ada4c9 57              push    edi
82ada4ca e881830100      call    nt!KeInitializeApc (82af2850)
82ada4cf 6a02            push    2
82ada4d1 6a00            push    0
82ada4d3 ff7628          push    dword ptr [esi+28h]
82ada4d6 57              push    edi
82ada4d7 e8d2830100      call    nt!KeInsertQueueApc (82af28ae)
82ada4dc 33ff            xor     edi,edi
82ada4de eb5f            jmp     nt!IopCompleteRequest+0×429 (82ada53f)

1: kd> ub nt!IopCompleteRequest+3ac
                                  ^ Unable to find valid previous instruction for 'ub nt!IopCompleteRequest+3ac'

Debug session time: Sat Jan  9 07:45:24.155 2010 (GMT+0)
System Uptime: 0 days 2:09:39.576

0: kd> !analyze -v

UNEXPECTED_KERNEL_MODE_TRAP (7f)

Arg1: 0000000d, EXCEPTION_GP_FAULT

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  6

STACK_TEXT: 
a24b3bd8 90f9e956 badb0d00 00000000 ddf1ba50 nt!KiSystemFatalException+0xf
a24b3cc4 90f93f2b 00000001 00000004 00000004 HDAudBus!HDABusWmiLogETW+0x1c6
a24b3d08 82a817ad 864a6280 86541000 a24b3d34 HDAudBus!HdaController::Isr+0x2b
a24b3d08 20c40d61 864a6280 86541000 a24b3d34 nt!KiInterruptDispatch+0x6d
WARNING: Frame IP not in any known module. Following frames may be wrong.
1343f8ea 00000000 00000000 00000000 00000000 0x20c40d61

Debug session time: Sat Jan  9 08:52:03.454 2010 (GMT+0)
System Uptime: 0 days 1:05:54.249

0: kd> !analyze -v

IRQL_NOT_LESS_OR_EQUAL (a)

CURRENT_IRQL:  2

PROCESS_NAME:  FlightSimulatorA.exe

TRAP_FRAME:  8078adf0 -- (.trap 0xffffffff8078adf0)
ErrCode = 00000002
eax=8632e2a6 ebx=00000000 ecx=880fb200 edx=00000118 esi=00000007 edi=8632e27c
eip=82a0c967 esp=8078ae64 ebp=c1e2baa0 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
hal!HalBuildScatterGatherList+0xf3:
82a0c967 8901            mov     dword ptr [ecx],eax  ds:0023:880fb200=????????
Resetting default scope

STACK_TEXT: 
8078adf0 82a0c967 badb0d00 00000118 82b5f466 nt!KiTrap0E+0x2cf
8078ae78 82a0cc16 880fb218 86379028 8632e260 hal!HalBuildScatterGatherList+0xf3
8078aea8 909b3e70 8651c6b0 86379028 8632e260 hal!HalGetScatterGatherList+0x26
8078aef4 909b3807 86379028 86379970 00000007 USBPORT!USBPORT_Core_iMapTransfer+0x21e
8078af24 909add18 86379028 86379970 86379002 USBPORT!USBPORT_Core_UsbMapDpc_Worker+0x1e3
8078af48 82aa73b5 8637997c 86379002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8078afa4 82aa7218 82b68d20 88139a98 00000000 nt!KiExecuteAllDpcs+0xf9
8078aff4 82aa69dc 9fd8cce4 00000000 00000000 nt!KiRetireDpcList+0xd5
8078aff8 9fd8cce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa69dc 00000000 0000001a 00d6850f bb830000 0x9fd8cce4

Debug session time: Sat Jan  9 16:34:48.134 2010 (GMT+0)
System Uptime: 0 days 1:53:05.929

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

PROCESS_NAME:  firefox.exe

TRAP_FRAME:  bb92449c -- (.trap 0xffffffffbb92449c)
ErrCode = 00000000
eax=000005b4 ebx=0db19ba0 ecx=80000000 edx=00000001 esi=85fdff29 edi=bb924530
eip=8bc7e2c7 esp=bb924510 ebp=bb924638 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
tcpip!TcpBeginTcbSend+0xa83:
8bc7e2c7 eb06            jmp     tcpip!TcpBeginTcbSend+0xa8b (8bc7e2cf)

Resetting default scope

STACK_TEXT: 
bb92449c 8bc7e2c7 badb0d00 00000001 00000000 nt!KiTrap0E+0x2cf
bb924638 8bc7d2bf 87b39c78 00000000 00000001 tcpip!TcpBeginTcbSend+0xa83
bb92479c 8bc814b5 87b39c78 00000000 00000001 tcpip!TcpTcbSend+0x426
bb9247bc 8bc7f349 87b39c78 87fa6c38 00000000 tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0x157
bb92481c 8bc81846 87b39c78 bb92491c 00000000 tcpip!TcpEnqueueTcbSend+0x3ca
bb924838 82a95f8a bb9248c8 96d9c9d2 00000000 tcpip!TcpTlConnectionSendCalloutRoutine+0x17
bb9248a0 8bc80a0b 8bc8182f bb9248c8 00000000 nt!KeExpandKernelStackAndCalloutEx+0x132
bb9248d8 908b5d27 87b39c01 bb924900 85572e18 tcpip!TcpTlConnectionSend+0x73
bb92493c 908bb2e3 00d4f1e0 85572e18 85572eac tdx!TdxSendConnection+0x1d7
bb924958 82a424bc 86236b80 85572e18 862389c0 tdx!TdxTdiDispatchInternalDeviceControl+0x115
bb924970 908d65ca 86d0e0c8 00000000 86238990 nt!IofCallDriver+0x63
WARNING: Stack unwind information not available. Following frames may be wrong.
bb9249c8 908d17f8 86238990 85572e18 85572ed0 aswTdi+0x55ca
bb924a28 82a424bc 862388d8 85572e18 8623f0e8 aswTdi+0x7f8
bb924a40 90935310 8623f030 82a424bc 8623f030 nt!IofCallDriver+0x63
bb924a60 90900a0e 2b1c89ba bb924b20 00000001 aswRdr+0x310
bb924ab0 908ed542 00000000 908ed542 87a5c530 afd!AfdFastConnectionSend+0x2a6
bb924c28 82c608f7 87ec6701 00000001 02b5f8cc afd!AfdFastIoDeviceControl+0x53d
bb924cd0 82c634ac 85a89c10 0000024c 00000000 nt!IopXxxControlFile+0x2d0
bb924d04 82a4942a 00000240 0000024c 00000000 nt!NtDeviceIoControlFile+0x2a
bb924d04 774464f4 00000240 0000024c 00000000 nt!KiFastCallEntry+0x12a
02b5f920 00000000 00000000 00000000 00000000 0x774464f4

1: kd> u 8bc7e2cf
tcpip!TcpBeginTcbSend+0xa8b:
8bc7e2cf 83bd18ffffff00  cmp     dword ptr [ebp-0E8h],0

8bc7e2d6 0f84d1000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2dc 8d85f8feffff    lea     eax,[ebp-108h]
8bc7e2e2 3bf8            cmp     edi,eax
8bc7e2e4 0f85c3000000    jne     tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2ea 83bd54ffffff00  cmp     dword ptr [ebp-0ACh],0
8bc7e2f1 0f84b6000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2f7 f7433c00002000  test    dword ptr [ebx+3Ch],200000h

Debug session time: Sat Jan  9 19:42:50.817 2010 (GMT+0)
System Uptime: 0 days 3:07:23.612

3: kd> !analyze -v

BUGCODE_USB_DRIVER (fe)
USB Driver bugcheck, first parameter is USB bugcheck code.
Arguments:
Arg1: 00000006, USBBUGCODE_BAD_SIGNATURE An Internal data structure (object)
 has been corrupted.
Arg2: 864b20e0, Object address
Arg3: 4f444648, Signature that was expected
Arg4: 00000000

PROCESS_NAME:  System

CURRENT_IRQL:  2

STACK_TEXT: 
8d952b8c 90fa1025 000000fe 00000006 864b20e0 nt!KeBugCheckEx+0x1e
8d952ba8 90fa6672 864b20e0 4f444668 4f444648 USBPORT!USBPORT_AssertSig+0x20
8d952bc8 90fa4553 864b2028 85c57d10 82a8b334 USBPORT!USBPORT_FlushAdapterDBs+0x1b
8d952c00 90fa5178 00000001 856e3ab8 87fb98c0 USBPORT!USBPORT_Core_iCompleteDoneTransfer+0x3cb
8d952c2c 90fa89af 864b2028 864b20f0 864b2a98 USBPORT!USBPORT_Core_iIrpCsqCompleteDoneTransfer+0x33b
8d952c54 90fa2d18 864b2028 864b2a98 864b2002 USBPORT!USBPORT_Core_UsbIocDpc_Worker+0xbc
8d952c78 82ab33b5 864b2aa4 864b2002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8d952cd4 82ab3218 8d936120 8d93b800 00000000 nt!KiExecuteAllDpcs+0xf9
8d952d20 82ab3038 00000000 0000000e 00000000 nt!KiRetireDpcList+0xd5
8d952d24 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x38

Debug session time: Sun Jan 10 04:06:19.856 2010 (GMT+0)
System Uptime: 0 days 0:23:05.651

1: kd> !analyze -v

PAGE_FAULT_IN_NONPAGED_AREA (50)

PROCESS_NAME:  FlightSimulatorB.exe

CURRENT_IRQL:  0

TRAP_FRAME:  a127fa30 -- (.trap 0xffffffffa127fa30)
ErrCode = 00000000
eax=a127fec8 ebx=00000000 ecx=00000011 edx=86488ba0 esi=86488b78 edi=00000000
eip=8b83b87d esp=a127faa4 ebp=a127fab8 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
fltmgr!TreeFindNodeOrParent+0×9:
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0×764c9) (8b4974c9)[ebp],al ss:0010:2c716f81=??

Resetting default scope

MISALIGNED_IP:
fltmgr!TreeFindNodeOrParent+9
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0x764c9) (8b4974c9)[ebp],al

STACK_TEXT: 
a127fa18 82a8d5f8 00000000 8b497414 00000000 nt!MmAccessFault+0x106
a127fa18 8b83b87d 00000000 8b497414 00000000 nt!KiTrap0E+0xdc
a127fab8 8b834340 86488ba4 86e5e458 00000000 fltmgr!TreeFindNodeOrParent+0x9
a127faf8 8b83440a 86488b78 86e5e458 00000000 fltmgr!GetContextFromStreamList+0x50
a127fb14 8b86c6da 86e5e458 86488b78 a127fb40 fltmgr!FltGetStreamContext+0x34
a127fb44 8b866b35 87f30718 a127fb98 a127fba8 fileinfo!FIStreamGet+0x36
a127fbac 8b833aeb 87f30718 a127fbcc a127fbf8 fileinfo!FIPreReadWriteCallback+0xf1
a127fc18 8b83617b a127fc54 85cfd738 a127fcac fltmgr!FltpPerformPreCallbacks+0x34d
a127fc30 8b848c37 0027fc54 8b848ad4 00000000 fltmgr!FltpPassThroughFastIo+0x3d
a127fc74 82c96b32 85cfd738 a127fcb4 00001000 fltmgr!FltpFastIoRead+0x163
a127fd08 82a8a42a 86e484c0 00000000 00000000 nt!NtReadFile+0x2d5
a127fd08 775864f4 86e484c0 00000000 00000000 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0202fc8c 00000000 00000000 00000000 00000000 0x775864f4

IMAGE_NAME:  hardware

1: kd> u fltmgr!TreeFindNodeOrParent
fltmgr!TreeFindNodeOrParent:
8b83b874 8bff            mov     edi,edi
8b83b876 55              push    ebp
8b83b877 8bec            mov     ebp,esp
8b83b879 8b4508          mov     eax,dword ptr [ebp+8]
8b83b87c 8b08            mov     ecx,dword ptr [eax]

8b83b87e 85c9            test    ecx,ecx
8b83b880 7449            je      fltmgr!TreeFindNodeOrParent+0×57 (8b83b8cb)
8b83b882 8b5510          mov     edx,dword ptr [ebp+10h]

1: kd> ub 8b834340
fltmgr!GetContextFromStreamList+0x37:
8b834327 8bcb            mov     ecx,ebx
8b834329 ff15a4d0838b    call    dword ptr [fltmgr!_imp_ExfAcquirePushLockShared (8b83d0a4)]
8b83432f 33db            xor     ebx,ebx
8b834331 895dfc          mov     dword ptr [ebp-4],ebx
8b834334 ff7510          push    dword ptr [ebp+10h]
8b834337 ff750c          push    dword ptr [ebp+0Ch]
8b83433a 57              push    edi
8b83433b e896750000      call    fltmgr!TreeLookup (8b83b8d6)

1: kd> uf 8b83b8d6
fltmgr!TreeLookup:
8b83b8d6 8bff            mov     edi,edi
8b83b8d8 55              push    ebp
8b83b8d9 8bec            mov     ebp,esp
8b83b8db 8d4510          lea     eax,[ebp+10h]
8b83b8de 50              push    eax
8b83b8df ff7510          push    dword ptr [ebp+10h]
8b83b8e2 ff750c          push    dword ptr [ebp+0Ch]
8b83b8e5 ff7508          push    dword ptr [ebp+8]
8b83b8e8 e887ffffff      call    fltmgr!TreeFindNodeOrParent (8b83b874)
8b83b8ed 48              dec     eax
8b83b8ee f7d8            neg     eax
8b83b8f0 1bc0            sbb     eax,eax
8b83b8f2 f7d0            not     eax
8b83b8f4 234510          and     eax,dword ptr [ebp+10h]
8b83b8f7 5d              pop     ebp
8b83b8f8 c20c00          ret     0Ch

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply

You must be logged in to post a comment.