Archive for the ‘Minidump Analysis’ Category

Software Diagnostics Services

Friday, July 13th, 2012

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming 2nd edition of Memory Dump Analysis Anthology, Volume 1

Sunday, April 15th, 2012

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

  • Title: Memory Dump Analysis Anthology, Volume 1
  • Author: Dmitry Vostokov
  • Publisher: OpenTask (Summer 2012)
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 800 pages
  • ISBN-13: 978-1-908043-35-1
  • Hardcover: 800 pages
  • ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 125)

Sunday, January 2nd, 2011

Similar to Blocking Module pattern we also have Top Module pattern where the difference is in stack trace syntax only. A top module is any module we choose that is simply on top of a stack trace. Most of the time it is likely to be a non-OS vendor module. Whether the stack trace is well-formed and semantically sound or incorrect is irrelevant:

0:005> kL
ChildEBP RetAddr 
01abc4d8 6efba23d ntdll!KiFastSystemCallRet
WARNING: Stack unwind information not available. Following frames may be wrong.
01abc988 7c820833 ModuleB+0×2a23d
01abcbe4 7c8207f6 kernel32!GetVolumeNameForRoot+0×26
01abcc0c 7c82e6de kernel32!BasepGetVolumeNameForVolumeMountPoint+0×75
01abcc54 6efaf70b kernel32!GetVolumePathNameW+0×18a
01abccdc 6efbd1a6 ModuleB+0×1f70b
01abcce0 00000000 ModuleB+0×2d1a6

Here we can also check the validity of ModuleB code by backwards disassembly of 6efba23d return address (ub command) unless we have an abridged dump file (minidump) and we need to specify the image file path in WinDbg,

Why a top module is important? In various troubleshooting scenarious we can check the module timestamp (Not My Version pattern) and other useful information (lmv and !lmi WinDbg commands). If we suspect the module belonging to hooksware we can also recommend removing it or its software vendor package for testing purposes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Analysis as a Service

Tuesday, November 30th, 2010

MAaaS includes 2 complementary DA+TA services:

1. Dump Analysis as a Service (DAaaS)
2. Trace Analysis as a Service (TAaaS)

Memory Dump Analysis Services is the first organization to provide such a service at an audit and certification levels.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Audit Service

Thursday, November 25th, 2010

Memory Dump Analysis Services announces the launch of the first Memory Dump Analysis Audit Service:

http://www.dumpanalysis.com/memory-dump-analysis-audit-service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Memory Dump Analysis Anthology, Volume 5

Friday, November 12th, 2010

Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca

The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 5
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 400 pages
  • Publisher: Opentask (10 December 2010)
  • ISBN-13: 978-1-906717-96-4
  • Hardcover: 400 pages
  • Publisher: Opentask (10 December 2010)
  • ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Webinars in Q4, 2010

Tuesday, October 5th, 2010

The choice of webinars below mostly reflects my personal preferences and long time desire to speak on topics like systems thinking, troubleshooting tool design and development. The other topic about BSOD minidump analysis was requested by participants in an ongoing survey. There will be more topics in 2011. All forthcoming webinars will be hosted by Memory Dump Analysis Services. The planning list includes:

Systems Thinking in Memory Dump and Software Trace Analysis

Software Troubleshooting and Debugging Tools: Objects, Components, Patterns and Frameworks with UML

  • UML basics
  • DebugWare patterns
  • Unified Troubleshooting Framework
  • RADII software development process
  • Hands-on exercise: designing and building a tool

Blue Screen Of Death Analysis Done Right: Minidump Investigation for System Administrators

  • Making sense of !analyze –v output
  • Get extra troubleshooting information with additional WinDbg commands
  • Guessing culprits with raw stack analysis
  • Who’s responsible: hardware or software?
  • Checklist and patterns
  • Including hands-on exercises: send your own minidumps

More detailed information will be available soon. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 104)

Wednesday, August 4th, 2010

Sometimes we get memory dumps that are difficult to analyze in full because some if not most of information was omitted during saving. These are usually small memory dumps (contrasted with kernel and complete) and user process minidumps. We can easily recognize that when we open a dump:

User Mini Dump File: Only registers, stack and portions of memory are available

or

Mini Kernel Dump File: Only registers and stack trace are available

The same also applies to user dumps where thread times information is omitted so it is not possible to use !runaway WinDbg command or to a dump saved with various options of .dump command (including privacy-aware) instead of /ma or deprecated /f option. On the contrary, manually erased data in crash dumps looks more like an example of another pattern called Lateral Damage.

The similar cases of abridged dumps are discussed in Wrong Dump and Missing Space antipatterns.

Anyway, we shouldn’t dismiss such dumps and should try to analyze them. For example, some approaches (including using image binaries) are listed in kernel minidump analysis series. We can even see portions of raw stack data in search of execution residue:

0: kd> !thread
GetPointerFromAddress: unable to read from 81d315b0
THREAD 82f49020  Cid 0004.0034  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
    Unable to read nt!_IRP @ 8391e008
Not impersonating
GetUlongFromAddress: unable to read from 81d0ad90
Owning Process            82f00ab0       Image:         System
Attached Process          N/A            Image:         N/A
ffdf0000: Unable to get shared data
Wait Start TickCount      4000214     
Context Switch Count      21886            
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0x81c78ea3)
Stack Init 85be0000 Current 85bdf7c0 Base 85be0000 Limit 85bdd000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
[…]

0: kd> dps 85bdd000 85be0000
85bdd000  ????????
85bdd004  ????????
85bdd008  ????????
85bdd00c  ????????
85bdd010  ????????
85bdd014  ????????
85bdd018  ????????
85bdd01c  ????????
85bdd020  ????????
85bdd024  ????????
85bdd028  ????????
[...]
85bdf8c4  ????????
85bdf8c8  ????????
85bdf8cc  ????????
85bdf8d0  0000000a
85bdf8d4  a112883e
85bdf8d8  0000001b
85bdf8dc  00000000
85bdf8e0  81c28750 nt!KeSetEvent+0x4d
85bdf8e4  85bdf8e8
85bdf8e8  85bdf970
85bdf8ec  81c28750 nt!KeSetEvent+0x4d
85bdf8f0  badb0d00
85bdf8f4  00000000
85bdf8f8  00000000
85bdf8fc  81cf4820 nt!KiInitialPCR+0x120
85bdf900  00000000
85bdf904  85bdf938
85bdf908  81cf4820 nt!KiInitialPCR+0x120
85bdf90c  00000000
85bdf910  81d32300 nt!IopTimerLock
85bdf914  00000000
85bdf918  81fa0000 nt!_NULL_IMPORT_DESCRIPTOR <PERF> (nt+0x3a0000)
85bdf91c  85bd0023
85bdf920  00000023
85bdf924  00000000
85bdf928  81d323c0 nt!KiDispatcherLock
85bdf92c  a1128828
85bdf930  85bdf9b4
85bdf934  85bdfdb0
85bdf938  00000030
85bdf93c  84ca6f40
85bdf940  84ca6f38
85bdf944  00000001
85bdf948  85bdf970
85bdf94c  00000000
85bdf950  81c28750 nt!KeSetEvent+0x4d
85bdf954  00000008
85bdf958  00010246
85bdf95c  00000000
85bdf960  84ca68a0
[...]
85bdfd2c  82f49020
85bdfd30  835ca4d0
85bdfd34  a6684538
85bdfd38  81cfde7c nt!ExWorkerQueue+0x3c
85bdfd3c  00000001
85bdfd40  00000000
85bdfd44  85bdfd7c
85bdfd48  81c78fa0 nt!ExpWorkerThread+0xfd
85bdfd4c  835ca4d0
85bdfd50  00000000
85bdfd54  82f49020
85bdfd58  00000000
85bdfd5c  00000000
85bdfd60  0069000b
85bdfd64  00000000
85bdfd68  00000001
85bdfd6c  00000000
85bdfd70  835ca4d0
85bdfd74  81da9542 nt!PnpDeviceEventWorker
85bdfd78  00000000
85bdfd7c  85bdfdc0
85bdfd80  81e254e0 nt!PspSystemThreadStartup+0x9d
85bdfd84  835ca4d0
85bdfd88  85bd4680
85bdfd8c  00000000
85bdfd90  00000000
85bdfd94  00000000
85bdfd98  00000002
85bdfd9c  00000000
85bdfda0  00000000
85bdfda4  00000001
85bdfda8  85bdfd88
85bdfdac  85bdfdbc
85bdfdb0  ffffffff
85bdfdb4  81c8aad5 nt!_except_handler4
85bdfdb8  81c9ddb8 nt!`string'+0x4
85bdfdbc  00000000
85bdfdc0  00000000
85bdfdc4  81c9159e nt!KiThreadStartup+0x16
85bdfdc8  81c78ea3 nt!ExpWorkerThread
85bdfdcc  00000001
85bdfdd0  00000000
85bdfdd4  00000000
85bdfdd8  002e0069
85bdfddc  006c0064
85bdfde0  004c006c
85bdfde4  00000000
85bdfde8  000007f0
85bdfdec  00010000
85bdfdf0  0000027f
85bdfdf4  00000000
85bdfdf8  00000000
85bdfdfc  00000000
85bdfe00  00000000
85bdfe04  00000000
85bdfe08  00001f80
85bdfe0c  0000ffff
85bdfe10  00000000
85bdfe14  00000000
85bdfe18  00000000
[...]
85bdffe4  00000000
85bdffe8  00000000
85bdffec  00000000
85bdfff0  00000000
85bdfff4  00000000
85bdfff8  00000000
85bdfffc  00000000
85be0000  ????????

User minidumps are similar here:

0:001> k
ChildEBP RetAddr 
099bfe147c90daaa ntdll!KiFastSystemCallRet
099bfe18 77e765e3 ntdll!NtReplyWaitReceivePortEx+0xc
099bff80 77e76caf rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×12a
099bff88 77e76ad1 rpcrt4!RecvLotsaCallsWrapper+0xd
099bffa8 77e76c97 rpcrt4!BaseCachedThreadRoutine+0×79
099bffb4 7c80b729 rpcrt4!ThreadStartRoutine+0×1a
099bffec 00000000 kernel32!BaseThreadStart+0×37

0:001> dd 099bfe14
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006
099bfe74  00000006 e1f79698 e39b8b60 00000000
099bfe84  fbe33c40 00000001 e5ce12f8 b279db9c

0:001> dd 099bfe14-20
099bfdf4  ???????? ???????? ???????? ????????
099bfe04  ???????? ???????? ???????? ????????
099bfe14  099bfe24 7c90daaa 77e765e3 00000224
099bfe24  099bff74 00000000 2db87ae8 099bff48
099bfe34  fbf58e18 00000040 fd629338 b279dbbc
099bfe44  fd5928b8 fbf58ebc b279dbbc e0c1e002
099bfe54  00000000 00000006 00000001 00000000
099bfe64  e637d218 00000000 00000006 00000006

As a warning here it is possible to conclude that minidumps can also reveal private information especially when ASCII or Unicode buffers are seen on raw stack data.

I was thinking how to name this pattern and Oxford Thesaurus of English suggested the following name: Abridged Dump by analogy with an abridged book.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Welcome to Memory Dump Analysis Services!

Sunday, July 11th, 2010

Our future sponsor has been registered in Ireland and has its own independent website and logo: DumpAnalysis.com

More information will be available later this month.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Introducing Crash and Hang Analysis Audit Service

Wednesday, June 16th, 2010

Adding AI. Analysis Improvement.

After reading earlier today Windows Internals pages about system audit an idea came to my mind in the evening to provide audit services for memory dump and software trace analysis. One mind is good but two are better, especially if the second is a pattern-driven AI. Here are possible problem scenarios:

Problem: You are not satisfied with a crash report.

Problem: Your critical issue is escalated to the VP level. Engineers analyze memory dumps and software traces. No definite conclusion so far. You want to be sure that nothing has been omitted from the analysis.

Problem: You analyze a system dump or a software trace. You need a second pair of eyes but don’t want to send your memory dump due to your company security policies.

Other scenarios (use cases) will be added as soon as I see the service fit to the realities of software technical support.

I plan to make this service operational in July - August, 2010. Prices to be announced soon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Modern Memory Dump and Software Trace Analysis: Volumes 1-3

Sunday, April 18th, 2010

OpenTask to offer first 3 volumes of Memory Dump Analysis Anthology in one set:

The set is available exclusively from OpenTask e-Commerce web site starting from June. Individual volumes are also available from Amazon, Barnes & Noble and other bookstores worldwide.

Product information:

  • Title: Modern Memory Dump and Software Trace Analysis: Volumes 1-3
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 1600 pages
  • Publisher: Opentask (31 May 2010)
  • ISBN-13: 978-1-906717-99-5

Information about individual volumes:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump and Software Trace Analysis Training and Seminars

Friday, April 9th, 2010

Plan to start providing training and seminars in my free time. If you are interested please answer these questions (you can either respond here in comments or use this form for private communication http://www.dumpanalysis.org/contact):

  • Are you interested in on-site training, prefer traveling or attending webinars?
  • Are you interested in software trace analysis as well?
  • What specific topics are you interested in?
  • What training level (beginner, intermediate, advanced) are you interested in? (please provide an example, if possible)

Additional topics of expertise that can be integrated into training include Source Code Reading and Analysis, Debugging, Windows Architecture, Device Drivers, Troubleshooting Tools Design and Implementation, Multithreading, Deep Down C and C++, x86 and x64 Assembly Language Reading.

Looking forward to your responses. Any suggestions are welcome.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Korean Edition of Memory Dump Analysis Anthology, Volume 1

Monday, April 5th, 2010

I’m very pleased to announce that the Korean edition is available:

The book can be found on: 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Fault context, wild code and hardware error: pattern cooperation

Tuesday, March 16th, 2010

I recently got a crying request from a reader of my blog to analyze the source of frequent bugchecks on a newly bought computer running Windows 7. I got 8 kernel minidumps with 5 different bugchecks. However, inspection of the default analysis revealed common Fault Context pattern of high resource consumption flight simulator processes in 6 minidumps. Most fault IPs were showing signs of Wild Code pattern and that most probably implicated Hardware Error (Looks like WinDbg suggests that MISALIGNED_IP implicates hardware). Here is the listing of relevant output fragments with attempts to disassemble code around IP (Instruction Pointer) to see if code make any sense (magenta color means the valid that should have been instead of misaligned code highlighted in red):

Windows 7 Kernel Version 7600 MP (4 procs) Free x86 compatible

Debug session time: Fri Jan  8 20:31:15.121 2010 (GMT+0)
System Uptime: 0 days 2:54:44.916

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  2

TRAP_FRAME:  807e6ea4 -- (.trap 0xffffffff807e6ea4)
ErrCode = 00000002
eax=872082a7 ebx=80028d5f ecx=b3348635 edx=87208638 esi=80280001 edi=000082a7
eip=8d613485 esp=807e6f18 ebp=6f248635 iopl=0  nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
USBPORT!USBPORT_Xdpc_End+0xa6:
8d613485 897904          mov     dword ptr [ecx+4],edi ds:0023:b3348639=????????
Resetting default scope

STACK_TEXT: 
807e6ea4 8d613485 badb0d00 87208638 82a7b334 nt!KiTrap0E+0x2cf
807e6f24 8d613d18 00000000 86358720 86358002 USBPORT!USBPORT_Xdpc_End+0xa6
807e6f48 82aa33b5 8635872c 86358002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
807e6fa4 82aa3218 807c6120 87e7e950 00000000 nt!KiExecuteAllDpcs+0xf9
807e6ff4 82aa29dc 9f7e1ce4 00000000 00000000 nt!KiRetireDpcList+0xd5
807e6ff8 9f7e1ce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa29dc 00000000 0000001a 00d6850f bb830000 0x9f7e1ce4

Debug session time: Fri Jan  8 20:42:16.395 2010 (GMT+0)
System Uptime: 0 days 0:10:22.815

2: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

TRAP_FRAME:  8d91cbc4 -- (.trap 0xffffffff8d91cbc4)
ErrCode = 00000002
eax=00000000 ebx=8d901a00 ecx=86570108 edx=86570108 esi=8d905884 edi=86573920
eip=911e5f5d esp=8d91cc38 ebp=8d91cc78 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???

Resetting default scope

IMAGE_NAME:  hardware

2: kd> u HDAudBus!HdaController::NotificationDpc+14d
HDAudBus!HdaController::NotificationDpc+0×14d:
911e5f5d ff              ???
911e5f5e ff              ???
911e5f5f ff6a00          jmp     fword ptr [edx]

911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4

2: kd> uf HDAudBus!HdaController::NotificationDpc
[...]
HDAudBus!HdaController::NotificationDpc+0x135:
911e5f45 8b45d8          mov     eax,dword ptr [ebp-28h]
911e5f48 c6405400        mov     byte ptr [eax+54h],0
911e5f4c 8b4dd8          mov     ecx,dword ptr [ebp-28h]
911e5f4f 83c148          add     ecx,48h
911e5f52 8a55e7          mov     dl,byte ptr [ebp-19h]
911e5f55 ff1510a01e91    call    dword ptr [HDAudBus!_imp_KfReleaseSpinLock (911ea010)]

HDAudBus!HdaController::NotificationDpc+0x14b:
911e5f5b e909ffffff      jmp     HDAudBus!HdaController::NotificationDpc+0x59 (911e5e69)

HDAudBus!HdaController::NotificationDpc+0x150:
911e5f60 6a00            push    0
911e5f62 6a00            push    0
911e5f64 6a00            push    0
911e5f66 68ff000000      push    0FFh
911e5f6b 6a03            push    3
911e5f6d 6a04            push    4
911e5f6f 6a08            push    8
911e5f71 6a02            push    2
911e5f73 e818180000      call    HDAudBus!HDABusWmiLogETW (911e7790)
911e5f78 8b4df0          mov     ecx,dword ptr [ebp-10h]
911e5f7b 64890d00000000  mov     dword ptr fs:[0],ecx
911e5f82 59              pop     ecx
911e5f83 5f              pop     edi
911e5f84 5e              pop     esi
911e5f85 5b              pop     ebx
911e5f86 8be5            mov     esp,ebp
911e5f88 5d              pop     ebp
911e5f89 c21000          ret     10h

Debug session time: Fri Jan  8 21:32:04.096 2010 (GMT+0)
System Uptime: 0 days 0:49:10.517

1: kd> !analyze -v

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)

Arg1: c000001d, The exception code that was not handled

EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

TRAP_FRAME:  a99e3644 -- (.trap 0xffffffffa99e3644)
ErrCode = 00000000
eax=000000fe ebx=8556a2b0 ecx=754764cd edx=00000001 esi=858ad008 edi=858ad048
eip=82ada4c2 esp=a99e36b8 ebp=a99e3704 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
nt!IopCompleteRequest+0×3ac:
82ada4c2 02cd            add     cl,ch

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  1

MISALIGNED_IP:
nt!IopCompleteRequest+3ac
82ada4c2 02cd            add     cl,ch

IMAGE_NAME:  hardware

1: kd> uf nt!IopCompleteRequest+3ac
nt!IopCompleteRequest+0×3a9:
82ada4bf 82680002        sub     byte ptr [eax],2
82ada4c3 cd82            int     82h

82ada4c5 50              push    eax
82ada4c6 ff75e0          push    dword ptr [ebp-20h]
82ada4c9 57              push    edi
82ada4ca e881830100      call    nt!KeInitializeApc (82af2850)
82ada4cf 6a02            push    2
82ada4d1 6a00            push    0
82ada4d3 ff7628          push    dword ptr [esi+28h]
82ada4d6 57              push    edi
82ada4d7 e8d2830100      call    nt!KeInsertQueueApc (82af28ae)
82ada4dc 33ff            xor     edi,edi
82ada4de eb5f            jmp     nt!IopCompleteRequest+0×429 (82ada53f)

1: kd> ub nt!IopCompleteRequest+3ac
                                  ^ Unable to find valid previous instruction for 'ub nt!IopCompleteRequest+3ac'

Debug session time: Sat Jan  9 07:45:24.155 2010 (GMT+0)
System Uptime: 0 days 2:09:39.576

0: kd> !analyze -v

UNEXPECTED_KERNEL_MODE_TRAP (7f)

Arg1: 0000000d, EXCEPTION_GP_FAULT

PROCESS_NAME:  FlightSimulatorA.exe

CURRENT_IRQL:  6

STACK_TEXT: 
a24b3bd8 90f9e956 badb0d00 00000000 ddf1ba50 nt!KiSystemFatalException+0xf
a24b3cc4 90f93f2b 00000001 00000004 00000004 HDAudBus!HDABusWmiLogETW+0x1c6
a24b3d08 82a817ad 864a6280 86541000 a24b3d34 HDAudBus!HdaController::Isr+0x2b
a24b3d08 20c40d61 864a6280 86541000 a24b3d34 nt!KiInterruptDispatch+0x6d
WARNING: Frame IP not in any known module. Following frames may be wrong.
1343f8ea 00000000 00000000 00000000 00000000 0x20c40d61

Debug session time: Sat Jan  9 08:52:03.454 2010 (GMT+0)
System Uptime: 0 days 1:05:54.249

0: kd> !analyze -v

IRQL_NOT_LESS_OR_EQUAL (a)

CURRENT_IRQL:  2

PROCESS_NAME:  FlightSimulatorA.exe

TRAP_FRAME:  8078adf0 -- (.trap 0xffffffff8078adf0)
ErrCode = 00000002
eax=8632e2a6 ebx=00000000 ecx=880fb200 edx=00000118 esi=00000007 edi=8632e27c
eip=82a0c967 esp=8078ae64 ebp=c1e2baa0 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
hal!HalBuildScatterGatherList+0xf3:
82a0c967 8901            mov     dword ptr [ecx],eax  ds:0023:880fb200=????????
Resetting default scope

STACK_TEXT: 
8078adf0 82a0c967 badb0d00 00000118 82b5f466 nt!KiTrap0E+0x2cf
8078ae78 82a0cc16 880fb218 86379028 8632e260 hal!HalBuildScatterGatherList+0xf3
8078aea8 909b3e70 8651c6b0 86379028 8632e260 hal!HalGetScatterGatherList+0x26
8078aef4 909b3807 86379028 86379970 00000007 USBPORT!USBPORT_Core_iMapTransfer+0x21e
8078af24 909add18 86379028 86379970 86379002 USBPORT!USBPORT_Core_UsbMapDpc_Worker+0x1e3
8078af48 82aa73b5 8637997c 86379002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8078afa4 82aa7218 82b68d20 88139a98 00000000 nt!KiExecuteAllDpcs+0xf9
8078aff4 82aa69dc 9fd8cce4 00000000 00000000 nt!KiRetireDpcList+0xd5
8078aff8 9fd8cce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
82aa69dc 00000000 0000001a 00d6850f bb830000 0x9fd8cce4

Debug session time: Sat Jan  9 16:34:48.134 2010 (GMT+0)
System Uptime: 0 days 1:53:05.929

1: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)

CURRENT_IRQL:  2

PROCESS_NAME:  firefox.exe

TRAP_FRAME:  bb92449c -- (.trap 0xffffffffbb92449c)
ErrCode = 00000000
eax=000005b4 ebx=0db19ba0 ecx=80000000 edx=00000001 esi=85fdff29 edi=bb924530
eip=8bc7e2c7 esp=bb924510 ebp=bb924638 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
tcpip!TcpBeginTcbSend+0xa83:
8bc7e2c7 eb06            jmp     tcpip!TcpBeginTcbSend+0xa8b (8bc7e2cf)

Resetting default scope

STACK_TEXT: 
bb92449c 8bc7e2c7 badb0d00 00000001 00000000 nt!KiTrap0E+0x2cf
bb924638 8bc7d2bf 87b39c78 00000000 00000001 tcpip!TcpBeginTcbSend+0xa83
bb92479c 8bc814b5 87b39c78 00000000 00000001 tcpip!TcpTcbSend+0x426
bb9247bc 8bc7f349 87b39c78 87fa6c38 00000000 tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0x157
bb92481c 8bc81846 87b39c78 bb92491c 00000000 tcpip!TcpEnqueueTcbSend+0x3ca
bb924838 82a95f8a bb9248c8 96d9c9d2 00000000 tcpip!TcpTlConnectionSendCalloutRoutine+0x17
bb9248a0 8bc80a0b 8bc8182f bb9248c8 00000000 nt!KeExpandKernelStackAndCalloutEx+0x132
bb9248d8 908b5d27 87b39c01 bb924900 85572e18 tcpip!TcpTlConnectionSend+0x73
bb92493c 908bb2e3 00d4f1e0 85572e18 85572eac tdx!TdxSendConnection+0x1d7
bb924958 82a424bc 86236b80 85572e18 862389c0 tdx!TdxTdiDispatchInternalDeviceControl+0x115
bb924970 908d65ca 86d0e0c8 00000000 86238990 nt!IofCallDriver+0x63
WARNING: Stack unwind information not available. Following frames may be wrong.
bb9249c8 908d17f8 86238990 85572e18 85572ed0 aswTdi+0x55ca
bb924a28 82a424bc 862388d8 85572e18 8623f0e8 aswTdi+0x7f8
bb924a40 90935310 8623f030 82a424bc 8623f030 nt!IofCallDriver+0x63
bb924a60 90900a0e 2b1c89ba bb924b20 00000001 aswRdr+0x310
bb924ab0 908ed542 00000000 908ed542 87a5c530 afd!AfdFastConnectionSend+0x2a6
bb924c28 82c608f7 87ec6701 00000001 02b5f8cc afd!AfdFastIoDeviceControl+0x53d
bb924cd0 82c634ac 85a89c10 0000024c 00000000 nt!IopXxxControlFile+0x2d0
bb924d04 82a4942a 00000240 0000024c 00000000 nt!NtDeviceIoControlFile+0x2a
bb924d04 774464f4 00000240 0000024c 00000000 nt!KiFastCallEntry+0x12a
02b5f920 00000000 00000000 00000000 00000000 0x774464f4

1: kd> u 8bc7e2cf
tcpip!TcpBeginTcbSend+0xa8b:
8bc7e2cf 83bd18ffffff00  cmp     dword ptr [ebp-0E8h],0

8bc7e2d6 0f84d1000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2dc 8d85f8feffff    lea     eax,[ebp-108h]
8bc7e2e2 3bf8            cmp     edi,eax
8bc7e2e4 0f85c3000000    jne     tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2ea 83bd54ffffff00  cmp     dword ptr [ebp-0ACh],0
8bc7e2f1 0f84b6000000    je      tcpip!TcpBeginTcbSend+0xb68 (8bc7e3ad)
8bc7e2f7 f7433c00002000  test    dword ptr [ebx+3Ch],200000h

Debug session time: Sat Jan  9 19:42:50.817 2010 (GMT+0)
System Uptime: 0 days 3:07:23.612

3: kd> !analyze -v

BUGCODE_USB_DRIVER (fe)
USB Driver bugcheck, first parameter is USB bugcheck code.
Arguments:
Arg1: 00000006, USBBUGCODE_BAD_SIGNATURE An Internal data structure (object)
 has been corrupted.
Arg2: 864b20e0, Object address
Arg3: 4f444648, Signature that was expected
Arg4: 00000000

PROCESS_NAME:  System

CURRENT_IRQL:  2

STACK_TEXT: 
8d952b8c 90fa1025 000000fe 00000006 864b20e0 nt!KeBugCheckEx+0x1e
8d952ba8 90fa6672 864b20e0 4f444668 4f444648 USBPORT!USBPORT_AssertSig+0x20
8d952bc8 90fa4553 864b2028 85c57d10 82a8b334 USBPORT!USBPORT_FlushAdapterDBs+0x1b
8d952c00 90fa5178 00000001 856e3ab8 87fb98c0 USBPORT!USBPORT_Core_iCompleteDoneTransfer+0x3cb
8d952c2c 90fa89af 864b2028 864b20f0 864b2a98 USBPORT!USBPORT_Core_iIrpCsqCompleteDoneTransfer+0x33b
8d952c54 90fa2d18 864b2028 864b2a98 864b2002 USBPORT!USBPORT_Core_UsbIocDpc_Worker+0xbc
8d952c78 82ab33b5 864b2aa4 864b2002 00000000 USBPORT!USBPORT_Xdpc_Worker+0x173
8d952cd4 82ab3218 8d936120 8d93b800 00000000 nt!KiExecuteAllDpcs+0xf9
8d952d20 82ab3038 00000000 0000000e 00000000 nt!KiRetireDpcList+0xd5
8d952d24 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x38

Debug session time: Sun Jan 10 04:06:19.856 2010 (GMT+0)
System Uptime: 0 days 0:23:05.651

1: kd> !analyze -v

PAGE_FAULT_IN_NONPAGED_AREA (50)

PROCESS_NAME:  FlightSimulatorB.exe

CURRENT_IRQL:  0

TRAP_FRAME:  a127fa30 -- (.trap 0xffffffffa127fa30)
ErrCode = 00000000
eax=a127fec8 ebx=00000000 ecx=00000011 edx=86488ba0 esi=86488b78 edi=00000000
eip=8b83b87d esp=a127faa4 ebp=a127fab8 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
fltmgr!TreeFindNodeOrParent+0×9:
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0×764c9) (8b4974c9)[ebp],al ss:0010:2c716f81=??

Resetting default scope

MISALIGNED_IP:
fltmgr!TreeFindNodeOrParent+9
8b83b87d 0885c974498b    or      byte ptr mcupdate_GenuineIntel!_NULL_IMPORT_DESCRIPTOR <PERF> (mcupdate_GenuineIntel+0x764c9) (8b4974c9)[ebp],al

STACK_TEXT: 
a127fa18 82a8d5f8 00000000 8b497414 00000000 nt!MmAccessFault+0x106
a127fa18 8b83b87d 00000000 8b497414 00000000 nt!KiTrap0E+0xdc
a127fab8 8b834340 86488ba4 86e5e458 00000000 fltmgr!TreeFindNodeOrParent+0x9
a127faf8 8b83440a 86488b78 86e5e458 00000000 fltmgr!GetContextFromStreamList+0x50
a127fb14 8b86c6da 86e5e458 86488b78 a127fb40 fltmgr!FltGetStreamContext+0x34
a127fb44 8b866b35 87f30718 a127fb98 a127fba8 fileinfo!FIStreamGet+0x36
a127fbac 8b833aeb 87f30718 a127fbcc a127fbf8 fileinfo!FIPreReadWriteCallback+0xf1
a127fc18 8b83617b a127fc54 85cfd738 a127fcac fltmgr!FltpPerformPreCallbacks+0x34d
a127fc30 8b848c37 0027fc54 8b848ad4 00000000 fltmgr!FltpPassThroughFastIo+0x3d
a127fc74 82c96b32 85cfd738 a127fcb4 00001000 fltmgr!FltpFastIoRead+0x163
a127fd08 82a8a42a 86e484c0 00000000 00000000 nt!NtReadFile+0x2d5
a127fd08 775864f4 86e484c0 00000000 00000000 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0202fc8c 00000000 00000000 00000000 00000000 0x775864f4

IMAGE_NAME:  hardware

1: kd> u fltmgr!TreeFindNodeOrParent
fltmgr!TreeFindNodeOrParent:
8b83b874 8bff            mov     edi,edi
8b83b876 55              push    ebp
8b83b877 8bec            mov     ebp,esp
8b83b879 8b4508          mov     eax,dword ptr [ebp+8]
8b83b87c 8b08            mov     ecx,dword ptr [eax]

8b83b87e 85c9            test    ecx,ecx
8b83b880 7449            je      fltmgr!TreeFindNodeOrParent+0×57 (8b83b8cb)
8b83b882 8b5510          mov     edx,dword ptr [ebp+10h]

1: kd> ub 8b834340
fltmgr!GetContextFromStreamList+0x37:
8b834327 8bcb            mov     ecx,ebx
8b834329 ff15a4d0838b    call    dword ptr [fltmgr!_imp_ExfAcquirePushLockShared (8b83d0a4)]
8b83432f 33db            xor     ebx,ebx
8b834331 895dfc          mov     dword ptr [ebp-4],ebx
8b834334 ff7510          push    dword ptr [ebp+10h]
8b834337 ff750c          push    dword ptr [ebp+0Ch]
8b83433a 57              push    edi
8b83433b e896750000      call    fltmgr!TreeLookup (8b83b8d6)

1: kd> uf 8b83b8d6
fltmgr!TreeLookup:
8b83b8d6 8bff            mov     edi,edi
8b83b8d8 55              push    ebp
8b83b8d9 8bec            mov     ebp,esp
8b83b8db 8d4510          lea     eax,[ebp+10h]
8b83b8de 50              push    eax
8b83b8df ff7510          push    dword ptr [ebp+10h]
8b83b8e2 ff750c          push    dword ptr [ebp+0Ch]
8b83b8e5 ff7508          push    dword ptr [ebp+8]
8b83b8e8 e887ffffff      call    fltmgr!TreeFindNodeOrParent (8b83b874)
8b83b8ed 48              dec     eax
8b83b8ee f7d8            neg     eax
8b83b8f0 1bc0            sbb     eax,eax
8b83b8f2 f7d0            not     eax
8b83b8f4 234510          and     eax,dword ptr [ebp+10h]
8b83b8f7 5d              pop     ebp
8b83b8f8 c20c00          ret     0Ch

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Memory Dump Analysis Anthology, Volume 4

Thursday, February 11th, 2010

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in July 2009 - January 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The fourth volume features:

- 13 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- New appendixes

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 4
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 410 pages
  • Publisher: Opentask (30 March 2010)
  • ISBN-13: 978-1-906717-86-5
  • Hardcover: 410 pages
  • Publisher: Opentask (30 April 2010)
  • ISBN-13: 978-1-906717-87-2

Back cover features memory space art image: Internal Process Combustion.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Anthology, Volume 3

Sunday, December 20th, 2009

“Memory dumps are facts.”

I’m very excited to announce that Volume 3 is available in paperback, hardcover and digital editions:

Memory Dump Analysis Anthology, Volume 3

Table of Contents

In two weeks paperback edition should also appear on Amazon and other bookstores. Amazon hardcover edition is planned to be available in January 2010.

The amount of information was so voluminous that I had to split the originally planned volume into two. Volume 4 should appear by the middle of February together with Color Supplement for Volumes 1-4. 

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump and Minidumps

Wednesday, September 30th, 2009

Welcome to Physicalist Art that has its foundation in Physicalism. The first physicalist composition was on display today and I took a picture of it (weather condition was not good):

 

Material: blue agate

It was originally called “Blue in a gate: memory dump and minidumps”. I plan to reinstall it again with more elaborate surroundings.

- Dmitry Vostokov @ DumpAnalysis.org -

Forthcoming Memory Dump Analysis Anthology, Volume 3

Saturday, September 26th, 2009

This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in October 2008 - June 2009 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The third volume features:

- 15 new crash dump analysis patterns
- 29 new pattern interaction case studies
- Trace analysis patterns
- Updated checklist
- Fully cross-referenced with Volume 1 and Volume 2
- New appendixes

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 3
  • Author: Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 404 pages
  • Publisher: Opentask (20 December 2009)
  • ISBN-13: 978-1-906717-43-8
  • Hardcover: 404 pages
  • Publisher: Opentask (30 January 2010)
  • ISBN-13: 978-1-906717-44-5

Back cover features 3D computer memory visualization image.

- Dmitry Vostokov @ DumpAnalysis.org -

Reconstructing Blue Screen of Death

Friday, August 7th, 2009

While I was listening to Klaus Schulze In Blue album a colleague sent me the link to a tool that reconstructs blue screens from minidumps (small memory dumps):

BlueScreenView (written by Nir Sofer)

I immediately downloaded it at it works even with kernel dumps but without pointing to a module that triggered the bugcheck (it shows modules for minidumps):

It ignores memory dumps and minidumps from x64 Windows so the next version I hope should do it :-)

PS. Long time ago I was thinking about writing a kernel driver that saves BSOD screen and embeds it in a memory dump.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.94

Saturday, May 23rd, 2009

Universal memory dumps come from astronomical observations:

Dumps “are meaningless without” symbols “to interpret them.”

Raymond Arthur Lyttleton, Quoted in Astroparticle Physics, page 50

- Dmitry Vostokov @ DumpAnalysis.org -