Software Diagnostics Library

Got it in the post yesterday shipped via courier from Amazon:

Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (PRO-Developer)

I originally expected it to be a paperback book but as a publisher myself I welcome the final MS Press decision to make it a hardcover. It is more solid. I plan to read it from cover to cover again, starting from tomorrow. I would like to thank authors for including the reference to www.dumpanalysis.org in the conclusion part of the last chapter 14 dedicated to crash dump analysis. This came as a nice surprise for me and I’m going to install a special plaque in the office and put its picture later on my blog.

- Dmitry Vostokov @ DumpAnalysis.org -

One of the common mistakes that I observe is to habitually stick to certain WinDbg commands to recognize patterns. One example is !locks command used to find out any wait chains and deadlock conditions among threads. Recently a service process was reported to be hang and !locks command showed no blocked threads:

0:000> !locks
CritSec +18caf94 at 018CAF94
LockCount          -2
RecursionCount     1
OwningThread       58e8
EntryCount         0
ContentionCount    0
*** Locked

CritSec +18cc7c4 at 018CC7C4
LockCount          -2
RecursionCount     1
OwningThread       58e8
EntryCount         0
ContentionCount    0
*** Locked

The number of threads waiting for the lock is 0 (this calculation is explained in the MSDN article): 

0:000> ? ((-1) - (-2)) >> 2
Evaluate expression: 0 = 00000000

In the past, for that hang sevice memory dumps, !locks command always showed LockCount values corresponding to several waiting threads. Therefore, an engineer assumed that the dump was taken at some random time, not at the time the service was hanging, and asked for a new right dump. The mistake here is that the engineer didn’t look at the corresponding thread stack trace that shows the characteristic pattern of the blocked thread waiting for a reply from an LRPC call:

0:000> ~~[58e8]kc 100

ntdll!KiFastSystemCallRet
ntdll!NtRequestWaitReplyPort
RPCRT4!LRPC_CCALL::SendReceive
RPCRT4!I_RpcSendReceive
RPCRT4!NdrSendReceive
RPCRT4!NdrClientCall2
ServiceA!foo
[…]
ServiceA!bar
RPCRT4!NdrStubCall2
RPCRT4!NdrServerCall2
RPCRT4!DispatchToStubInCNoAvrf
RPCRT4!RPC_INTERFACE::DispatchToStubWorker
RPCRT4!RPC_INTERFACE::DispatchToStub
RPCRT4!RPC_INTERFACE::DispatchToStubWithObject
RPCRT4!LRPC_SCALL::DealWithRequestMessage
RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest
RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls
RPCRT4!RecvLotsaCallsWrapper
RPCRT4!BaseCachedThreadRoutine
RPCRT4!ThreadStartRoutine
kernel32!BaseThreadStart

We don’t see other blocked threads and wait chains because the dump was saved as soon as the freezing condition was detected: the service didn’t allow a user connection to proceed. If more users tried to connect we would have seen critical section wait chains that are absent in this dump.

To prevent such mistakes checklists are indispensable. For one example, see Crash Dump Analysis Checklist. You can also order it in print:

WinDbg: A Reference Poster and Learning Cards

- Dmitry Vostokov @ DumpAnalysis.org -

Crash

The sudden disappearance of a program, service or system from observation.

References: Crashes explained, The difference between crashes and hangs explained

Synonyms: stop.

Antonyms: live.

Also: hang.

- Dmitry Vostokov @ DumpAnalysis.org -

This is the first draft entry for the forthcoming Dictionary of Debugging (ISBN: 978-1906717353). The entry format is not final and more information will be added to individual entries in the future.

Hang

The perceived absence of response from a present program, service or system, either visual or functional.

References: Hangs explained, The difference between crashes and hangs explained

Synonyms: freeze, stop, halt.

Antonyms: live.

Similar: sluggish.

Also: crash.

- Dmitry Vostokov @ DumpAnalysis.org -

“The real” debugging “begins only after hitting” a button.

Anonymous

- Dmitry Vostokov @ DumpAnalysis.org -

Looking at pattern cooperation studies it is easy to see that some patterns precede others, for example, heap corruption might be blocked by a hard error or a message box and therefore block other threads, creating conditions for another pattern to appear, wait chains. Blocked threads may block other coupled processes creating inter-process wait chains. Successive patterns reach the climax at the end and the system is no longer able to generate any other patterns.

The goal here is to find patterns that most likely happen in succession and another pattern series that are less likely to effect other abnormal conditions. Such pattern sequences can help in troubleshooting and finding root causes.

- Dmitry Vostokov @ DumpAnalysis.org - 

Just noticed that RichardS from Australia (nickname: rsayad1) was outraged after reading Dumps, Bugs and Debugging Forensics book and burnt it in his fireplace. I applied my analytical and forensic skills and figured out his name: Richard Sayad. The only excuse for him is that crash dumps is so hot topic that he rushed to buy the book without even looking inside it on Amazon.

What do you think and what is your opinion about this book? It is basically composed from the following material:

Cartoons

The first 64 bugtations

For the book all cartoons were edited, polished and most of them became full color in the book version.

- Dmitry Vostokov @ DumpAnalysis.org - 

Following the success of Windows Debugging: Practical Foundations the following title will be published this summer:

Windows Device Drivers: Practical Foundations (ISBN: 978-0955832840)

Table of contents will be posted later.

Other planned titles:

X64 Windows Debugging: Practical Foundations (ISBN: 978-1906717568) 

Windows Multithreading: Practical Foundations (ISBN: 978-1906717742)

 Like Windows Debugging book, these forthcoming titles are based on my seminars. 

- Dmitry Vostokov @ DumpAnalysis.org -

This is a kind of a “faultomorphism”, a fault, a crash point and stack trace shape preserving map between two platforms (such as 32-bit and 64-bit). This new word was derived from the concatenation of platform and morphism. Here is an example:

; 64-bit crash dump

0: kd> r
Last set context:
rax=0000000063537852 rbx=0000000000000000 rcx=0000000000000009
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffffadf262760da rsp=fffffadf15973968 rbp=0000000070537852
 r8=fffffadf31614b00  r9=fffffadffe9fa7b0 r10=000000000000000a
r11=fffffadf31614bf0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 efl=00010206
rdbss!RxIsThisACscAgentOpen+0×30:
fffffadf`262760da f3a6 repe cmps byte ptr [rsi],byte ptr [rdi]

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffffadf`15973968 fffffadf`2629e768 rdbss!RxIsThisACscAgentOpen+0x30
fffffadf`15973970 fffffadf`262988f5 rdbss!RxInitializeVNetRootParameters+0x31d
fffffadf`159739f0 fffffadf`2629bcfd rdbss!RxFindOrConstructVirtualNetRoot+0x180
fffffadf`15973ad0 fffffadf`26297a6c rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223
fffffadf`15973b70 fffffadf`26272a77 rdbss!RxCommonCreate+0x470
fffffadf`15973c80 fffffadf`261be3e8 rdbss!RxFsdCommonDispatch+0x51c
fffffadf`15973d80 fffffadf`29314db3 mrxsmb!MRxSmbFsdDispatch+0x211
[...]

; 32-bit crash dump

0: kd> r
eax=00000000 ebx=b6a23a80 ecx=00000009 edx=00000000 esi=00000008 edi=b6a23a80
eip=b6a23a5f esp=b3ce800c ebp=b3ce801c iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
rdbss!RxIsThisACscAgentOpen+0×38:
b6a23a5f f3a6 repe cmps byte ptr [esi],byte ptr es:[edi]

0: kd> kL 100
b3ce801c b6a2b431 rdbss!RxIsThisACscAgentOpen+0x38
b3ce803c b6a2bbf7 rdbss!RxInitializeVNetRootParameters+0x282
b3ce809c b6a2e6cd rdbss!RxFindOrConstructVirtualNetRoot+0xdc
b3ce80d0 b6a2ae15 rdbss!RxCanonicalizeNameAndObtainNetRoot+0x197
b3ce8134 b6a20d51 rdbss!RxCommonCreate+0x2c3
b3ce81cc b6a2acc2 rdbss!RxFsdCommonDispatch+0x353
b3ce81f4 b69ac317 rdbss!RxFsdDispatch+0xda
b3ce8214 804e13d9 mrxsmb!MRxSmbFsdDispatch+0x134
[...]

We can see that stack traces are almost the same, function offsets are very close and faulted instruction is the same up to an opcode. Not to mention that bugchecks are identical:

RDR_FILE_SYSTEM (27)
    If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the
    exception record and context record. Do a .cxr on the 3rd parameter and then kb to
    obtain a more informative stack trace.
    The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined
    as follows:
     RDBSS_BUG_CHECK_CACHESUP  = 0xca550000,
     RDBSS_BUG_CHECK_CLEANUP   = 0xc1ee0000,
     RDBSS_BUG_CHECK_CLOSE     = 0xc10e0000,
     RDBSS_BUG_CHECK_NTEXCEPT  = 0xbaad0000

Therefore, we can also say that these crashes are platformorphic. Obviously, this stems from the fact that source code was identical or almost identical for both platforms.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes we see a functional activity in a trace and / or see basic facts. Then we might want to find a correlation between that activity or facts in another part of the trace. If that intra-correlation fits into our problem description we may claim a possible explanation or, if we are lucky, we have just found, an inference to the best explanation, as philosophers of science like to say. Here is an example, but this time using Citrix WindowHistory tracing tool. A third-party application was frequently loosing the focus and the suspicion was on a terminal services client process. It was found that the following WindowHistory trace fragment corresponds to that application:

Handle: 00050586 Class: "Application A Class" Title: ""
     Title changed at 15:52:4:3 to "Application A"
     Title changed at 15:52:10:212 to "Application A - File1"
[...]
   Process ID: 89c
   Thread ID: d6c
[...]
   Visible: true
   Window placement command: SW_SHOWNORMAL
        Placement changed at 15:54:57:506 to SW_SHOWMINIMIZED
        Placement changed at 15:55:2:139 to SW_SHOWNORMAL
   Foreground: false
        Foreground changed at 15:52:4:3 to true
        Foreground changed at 15:53:4:625 to false
        Foreground changed at 15:53:42:564 to true
        Foreground changed at 15:53:44:498 to false
        Foreground changed at 15:53:44:498 to true
        Foreground changed at 15:53:44:592 to false
        Foreground changed at 15:53:45:887 to true
        Foreground changed at 15:53:47:244 to false
        Foreground changed at 15:53:47:244 to true
        Foreground changed at 15:53:47:353 to false
        Foreground changed at 15:54:26:416 to true
        Foreground changed at 15:54:27:55 to false
        Foreground changed at 15:54:27:55 to true
        Foreground changed at 15:54:27:180 to false
        Foreground changed at 15:54:28:428 to true
        Foreground changed at 15:54:28:771 to false
        Foreground changed at 15:54:28:865 to true
        Foreground changed at 15:54:29:99 to false
        Foreground changed at 15:54:30:877 to true
        Foreground changed at 15:54:57:521 to false
        Foreground changed at 15:55:2:76 to true
        Foreground changed at 15:57:3:378 to false
        Foreground changed at 15:57:11:396 to true
        Foreground changed at 15:57:29:601 to false
        Foreground changed at 15:57:39:803 to true
        Foreground changed at 15:58:54:41 to false
        Foreground changed at 15:59:8:96 to true
        Foreground changed at 16:1:19:478 to false
        Foreground changed at 16:1:27:527 to true
        Foreground changed at 16:1:39:914 to false
        Foreground changed at 16:2:0:515 to true
        Foreground changed at 16:7:14:628 to false
        Foreground changed at 16:7:24:246 to true
        Foreground changed at 16:9:53:523 to false
        Foreground changed at 16:10:15:919 to true
        Foreground changed at 16:10:31:426 to false
        Foreground changed at 16:11:12:818 to true
        Foreground changed at 16:11:59:538 to false
        Foreground changed at 16:12:39:456 to true
        Foreground changed at 16:13:6:364 to false

Corresponding terminal services client window trace fragment doesn’t have any foreground changes but another application main window has lots of them:

Handle: 000D0540 Class: "Application B Class" Title: "Application B"
[...]
   Process ID: 3ac
   Thread ID: bd4
[...]
   Foreground: false
        Foreground changed at 15:50:36:972 to true
        Foreground changed at 15:50:53:732 to false
        Foreground changed at 15:50:53:732 to true
        Foreground changed at 15:50:53:826 to false
        Foreground changed at 15:51:51:352 to true
        Foreground changed at 15:51:53:941 to false
        Foreground changed at 15:53:8:135 to true
        Foreground changed at 15:53:8:182 to false
        Foreground changed at 15:53:10:178 to true
        Foreground changed at 15:53:13:938 to false
        Foreground changed at 15:53:30:443 to true
        Foreground changed at 15:53:31:20 to false
        Foreground changed at 15:53:31:20 to true
        Foreground changed at 15:53:31:129 to false
        Foreground changed at 15:53:34:78 to true
        Foreground changed at 15:53:34:795 to false
        Foreground changed at 15:53:34:795 to true
        Foreground changed at 15:53:34:873 to false
        Foreground changed at 15:53:36:901 to true
        Foreground changed at 15:53:42:502 to false
        Foreground changed at 15:53:42:502 to true
        Foreground changed at 15:53:42:564 to false
        Foreground changed at 15:57:3:425 to true
        Foreground changed at 15:57:4:595 to false
        Foreground changed at 15:57:10:507 to true
        Foreground changed at 15:57:11:318 to false
        Foreground changed at 15:57:29:632 to true
        Foreground changed at 15:57:31:67 to false
        Foreground changed at 15:57:32:721 to true
        Foreground changed at 15:57:33:844 to false
        Foreground changed at 15:58:54:88 to true
        Foreground changed at 15:58:56:178 to false
        Foreground changed at 15:59:6:505 to true
        Foreground changed at 15:59:7:987 to false
        Foreground changed at 16:1:19:525 to true
        Foreground changed at 16:1:19:961 to false
        Foreground changed at 16:1:26:607 to true
        Foreground changed at 16:1:27:434 to false
        Foreground changed at 16:1:39:914 to true
        Foreground changed at 16:1:39:992 to false
        Foreground changed at 16:1:49:798 to true
        Foreground changed at 16:2:0:437 to false
        Foreground changed at 16:7:14:628 to true
        Foreground changed at 16:7:14:847 to false
        Foreground changed at 16:7:18:76 to true
        Foreground changed at 16:7:24:106 to false
        Foreground changed at 16:9:58:790 to true
        Foreground changed at 16:10:4:16 to false
        Foreground changed at 16:10:4:874 to true
        Foreground changed at 16:10:4:890 to false
        Foreground changed at 16:10:8:634 to true
        Foreground changed at 16:10:15:779 to false
        Foreground changed at 16:10:56:766 to true
        Foreground changed at 16:10:59:402 to false
        Foreground changed at 16:10:59:652 to true
        Foreground changed at 16:10:59:667 to false
        Foreground changed at 16:12:9:397 to true
        Foreground changed at 16:12:39:347 to false
        Foreground changed at 16:13:18:375 to true
        Foreground changed at 16:14:33:656 to false

We can see that most of the time when Application A window looses focus Application B window gets it.

- Dmitry Vostokov @ TraceAnalysis.org -

This is the main topic of the forthcoming next issue of Debugged MZ/PE magazine. The most close term is called “fault injection” but I rediscovered it as a “software defect construction”, “software defect simulation” or “software defect modeling”. The latter term is also used to refer to construction of mathematical models related to software product quality and corresponding statistics but “modeling software defects” seems appropriate subtitle for the magazine front cover picture… Software defect construction is more general term than fault injection. The latter is used for testing but we want to simulate bugs and abnormal system conditions to study debugging and memory dump analysis techniques or to build reproduction environments. I actually recently found and bought the used copy of this book:

Software Fault Injection: Inoculating Programs Against Errors

and plan to write my own book with the following working title later:

Software Defect Construction: Simulation and Modeling of Software Bugs (ISBN: 978-1906717759)

- Dmitry Vostokov @ DumpAnalysis.org -

“Touched by an” exception.

Touched by an Angel

- Dmitry Vostokov @ DumpAnalysis.org -  

“A personal touch in Certification”

Dmitry Vostokov,
Founding Director of
Memory Analysis and Debugging Institute

More details on the previously announced MDA certification. Every certificate will have a separate unique ISBN number and available in bookshops and libraries worldwide. Of course, this comes at a price and it will be announced later. An example of a certificate as a book will be available by the end of July:

Memory Dump Analysis Certificate 0×1 (ISBN: 978-1906717766)

- Dmitry Vostokov @ DumpAnalysis.org -  

The Adelphi Training Center in Dublin, Ireland, is haunted by the ghost of pre-Internet epoch. During one debugging night, when installing a secret service, Drilliam Traceless was murdered by a redundant engineer who was envy to his charisma. He stubbed him in the back while Drilliam was unlocking his mini-computer case. He died whispering: “I’ll be back debugging”. His face sometimes appears on greenish screensavers running on computers located in that center. During morning training sessions, many trainees think this is a kind of the so called Guinness effect.

The story is adapted from The Ghost of Adelphi Theatre. 

- Dmitry Vostokov @ DumpAnalysis.org -

Quantum computation, quantum memory and quantum information are hot topics today. Unfortunately quantum mechanics forbids perfect (ideal) memory dumps due to the so called no cloning theorem. Still it is possible to get inconsistent (imperfect) memory dumps and perfect memory dumps can be made from quantum computer simulators. The analysis of quantum memory snapshots is the domain of Quantum Memoretics.

- Dmitry Vostokov @ DumpAnalysis.org -

Sometimes I’m asked about a broad software engineering book to recommend for general memory dump analysis that covers software architecture, design methods and diagramming languages like UML, programming languages, concurrency, real-time issues and many other topics you need to know to have systems understanding that helps in problem identification and debugging. Here’s the book that I was fortunate to buy 4-5 years ago in a book shop and is a sheer pleasure to read:

Software Engineering for Real-Time Systems

Today I found that there even exists an OMG certification based on it:

http://www.omg.org/ocres/exam-info.htm

I might try later this summer. 

- Dmitry Vostokov @ DumpAnalysis.org -

The following WER dialog appeared after I copied an RTF text from an e-mail client to IE while editing another blog post:

Although the dialog points to html.iec it is good to show the basic techniques for component identification using WinDbg. I took a manual user dump of that unresponsive process while it was showing that dialog above using Task Manager:

The saved process memory dump shows that IE was virtualized 32-bit process running on x64 Windows:

STACK_TEXT: 
00000000`000dee18 00000000`75bbab46 : wow64cpu!TurboDispatchJumpAddressEnd+0xef
00000000`000deec0 00000000`75bba14c : wow64!Wow64SystemServiceEx+0×27a
00000000`000deef0 00000000`778f52d3 : wow64!Wow64LdrpInitialize+0×4b4
00000000`000df450 00000000`778f5363 : ntdll!CsrClientConnectToServer+0×493
00000000`000df700 00000000`778e85ce : ntdll!CsrClientConnectToServer+0×523
00000000`000df7b0 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

However, even after switching to x86 mode, !analyze -v doesn’t report any signs of exception violation, c0000005, seen from WER dialog above:

0:000> .load wow64exts; .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> !analyze -v
[...]

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

STACK_TEXT: 
0027e428 7679e91a ntdll_77a60000!NtWaitForMultipleObjects+0x15
0027e4c4 76a58f76 kernel32!WaitForMultipleObjectsEx+0x11d
0027e518 75ad6071 user32!RealMsgWaitForMultipleObjectsEx+0x14d
WARNING: Frame IP not in any known module. Following frames may be wrong.
0027e538 75ad61f0 ieui+0x6071
0027e560 75ad6196 ieui+0x61f0
0027e59c 7291ffc6 ieui+0x6196
0027e5a8 7290f579 ieframe!BrowserThreadProc+0x3f
0027e5cc 7290f4c7 ieframe!BrowserNewThreadProc+0x7b
0027f63c 728fd1ba ieframe!SHOpenFolderWindow+0x188
0027f86c 013133c3 ieframe!IEWinMain+0x2d9
0027fcb0 0131325a iexplore!wWinMain+0x27b
0027fd44 7680e4a5 iexplore!_initterm_e+0x1b1
0027fd50 77adcfed kernel32!BaseThreadInitThunk+0xe
0027fd90 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23
0027fda8 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1b

We list all threads for any exception processing signs and we find one such thread indeed:

0:002:x86> ~*kc 100

[...]

   2  Id: 404.ba4 Suspend: 0 Teb: 7efaa000 Unfrozen

ntdll_77a60000!NtWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!UnhandledExceptionFilter
ntdll_77a60000!__RtlUserThreadStart
ntdll_77a60000!_EH4_CallFilterFunc
ntdll_77a60000!_except_handler4
ntdll_77a60000!ExecuteHandler2
ntdll_77a60000!ExecuteHandler
ntdll_77a60000!KiUserExceptionDispatcher
WARNING: Frame IP not in any known module. Following frames may be wrong.
0×0
html!LwMultDivRU
html!FMarkListCallback
html!JcCalcFromXaExtents
html!EmitNonBreakingSpace
html!FEmitHtmlFnOtag
html!ConvertRtfToForeign
html!FceRtfToForeign
html!RtfToForeign32
mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml
mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml
mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml
mshtmled!CHTMLEditor::ConvertRTFToHTML
mshtmled!CPasteCommand::PasteFromClipboard
mshtmled!CPasteCommand::PrivateExec
mshtmled!CCommand::Exec
mshtmled!CMshtmlEd::Exec
mshtml!CEditRouter::ExecEditCommand
mshtml!CDoc::ExecHelper
mshtml!CFrameSite::Exec
mshtml!CDoc::RouteCTElement
mshtml!CDoc::ExecHelper
mshtml!CDoc::Exec
mshtml!CDoc::OnCommand
mshtml!CDoc::OnWindowMessage
mshtml!CServer::WndProc
user32!InternalCallWinProc
user32!UserCallWinProcCheckWow
user32!SendMessageWorker
user32!SendMessageW
mshtml!CElement::PerformTA
mshtml!CDoc::PerformTA
mshtml!CDoc::PumpMessage
mshtml!CDoc::DoTranslateAccelerator
mshtml!CServer::TranslateAcceleratorW
mshtml!CDoc::TranslateAcceleratorW
ieframe!CProxyActiveObject::TranslateAcceleratorW
ieframe!CDocObjectView::TranslateAcceleratorW
ieframe!CCommonBrowser::v_MayTranslateAccelerator
ieframe!CShellBrowser2::_MayTranslateAccelerator
ieframe!CShellBrowser2::v_MayTranslateAccelerator
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll_77a60000!__RtlUserThreadStart
ntdll_77a60000!_RtlUserThreadStart

[...]

The same stack fragment with function parameters has a zero for its UnhandledExceptionFilter first parameter, perhaps because of optimized code reusing local variable slots, and we cannot apply .exptr meta-command:

0:002:x86> ~2kv
ChildEBP RetAddr  Args to Child             
0361d938 7679e91a 00000002 0361d988 00000001 ntdll_77a60000!NtWaitForMultipleObjects+0x15
0361d9d4 767949d9 0361d988 0361da24 00000000 kernel32!WaitForMultipleObjectsEx+0x11d
0361d9f0 7684573d 00000002 0361da24 00000000 kernel32!WaitForMultipleObjects+0x18
0361da5c 76845969 0361db2c 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d
0361da70 7681c66f 0361db2c 00000001 35137a6c kernel32!WerpReportFault+0x70
0361dafc 77add03e 00000000 77aaf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5
0361db04 77aaf2d0 00000000 0361fe04 77a8da38 ntdll_77a60000!__RtlUserThreadStart+0×6f
0361db18 77b129b3 00000000 00000000 00000000 ntdll_77a60000!_EH4_CallFilterFunc+0×12
0361db40 77a83099 fffffffe 0361fdf4 0361dc7c ntdll_77a60000!_except_handler4+0×8e
0361db64 77a8306b 0361dc2c 0361fdf4 0361dc7c ntdll_77a60000!ExecuteHandler2+0×26
0361dc14 77a82eff 0161dc2c 0361dc7c 0361dc2c ntdll_77a60000!ExecuteHandler+0×24
0361dc18 0161dc2c 0361dc7c 0361dc2c 0361dc7c ntdll_77a60000!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0361df5c 72333b0e 07feaee4 0000001f 00000000 0×161dc2c
0361df74 72332ba9 00000000 0361dffc 00000000 html!LwMultDivRU+0×4b6
0361dfb8 7233518a 0361dffc 0000000c 0361e074 html!FMarkListCallback+0×56c
0361dfc8 723368b7 0000001f 0361dffc 0000000f html!JcCalcFromXaExtents+0×91
0361e074 723370c5 0000000f 00000000 000007d0 html!EmitNonBreakingSpace+0×445
0361e1a0 723407ff 00000000 0000000f 00000104 html!FEmitHtmlFnOtag+0×17d
0361e1c4 7231c6a8 00000001 00000000 00000000 html!ConvertRtfToForeign+0×105
0361e64c 7231c745 00ed03bc 00000000 00ed0374 html!FceRtfToForeign+0×266

Also the transition from html.iec module to exception processing code is through an invalid address 0×161dc2c so we might guess that this was an instance of wild code pointer or the case of incorrect stack trace. However using techniques to get exception context from hidden exceptions we get the following stack trace:

0:002:x86> .cxr 0361dc7c
eax=0000004e ebx=09d5755f ecx=07feaee4 edx=07feaf08 esi=07feaee4 edi=00000000
eip=7233d50b esp=0361df48 ebp=0361df5c iopl=0  nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
html!FPseudoStyleBis+0×26:
7233d50b 0fb70f          movzx   ecx,word ptr [edi]       ds:002b:00000000=????

0:002:x86> kL 100
ChildEBP RetAddr 
0361df50 7233d597 html!FPseudoStyleBis+0x26
0361df5c 72333b0e html!BisFromLpxszStyle+0x1c
0361df74 72332ba9 html!LwMultDivRU+0x4b6
0361dfb8 7233518a html!FMarkListCallback+0x56c
0361dfc8 723368b7 html!JcCalcFromXaExtents+0x91
0361e074 723370c5 html!EmitNonBreakingSpace+0x445
0361e1a0 723407ff html!FEmitHtmlFnOtag+0x17d
0361e1c4 7231c6a8 html!ConvertRtfToForeign+0x105
0361e64c 7231c745 html!FceRtfToForeign+0x266
0361e674 744e5ad4 html!RtfToForeign32+0x51
0361eabc 744e5c83 mshtmled!CRtfToHtmlConverter::ExternalRtfToInternalHtml+0x163
0361ef10 7449cc15 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x11a
0361ef2c 7449cd81 mshtmled!CRtfToHtmlConverter::StringRtfToStringHtml+0x38
0361ef40 744cdcea mshtmled!CHTMLEditor::ConvertRTFToHTML+0x12
0361efac 744ce392 mshtmled!CPasteCommand::PasteFromClipboard+0x2c0
0361f01c 7448d218 mshtmled!CPasteCommand::PrivateExec+0x47a
0361f040 7448d1ad mshtmled!CCommand::Exec+0x4b
0361f064 74000d14 mshtmled!CMshtmlEd::Exec+0xf9
0361f094 73ff88a8 mshtml!CEditRouter::ExecEditCommand+0xd6
0361f43c 7415eccf mshtml!CDoc::ExecHelper+0x338d
0361f488 73ff8a2f mshtml!CFrameSite::Exec+0x264
0361f4bc 73ff87af mshtml!CDoc::RouteCTElement+0xf1
0361f854 73ff8586 mshtml!CDoc::ExecHelper+0x325e
0361f874 740a0e7b mshtml!CDoc::Exec+0x1e
0361f8ac 7401a708 mshtml!CDoc::OnCommand+0x9c
0361f9c0 73f297e1 mshtml!CDoc::OnWindowMessage+0x841
0361f9ec 76a58807 mshtml!CServer::WndProc+0x78
0361fa18 76a58962 user32!InternalCallWinProc+0x23
0361fa90 76a5c4b6 user32!UserCallWinProcCheckWow+0x109
0361fad4 76a5c517 user32!SendMessageWorker+0x55b
0361faf8 7400fb9b user32!SendMessageW+0x7f
0361fb24 73f68e5a mshtml!CElement::PerformTA+0x71
0361fb44 73f68db9 mshtml!CDoc::PerformTA+0xd8
0361fbc0 73ff381c mshtml!CDoc::PumpMessage+0x8e0
0361fc74 73ff3684 mshtml!CDoc::DoTranslateAccelerator+0x33f
0361fc90 73ff34cc mshtml!CServer::TranslateAcceleratorW+0x56
0361fcb0 7296f550 mshtml!CDoc::TranslateAcceleratorW+0x83
0361fccc 7296f600 ieframe!CProxyActiveObject::TranslateAcceleratorW+0x30
0361fcf0 7296fca1 ieframe!CDocObjectView::TranslateAcceleratorW+0xb1
0361fd10 7296faf4 ieframe!CCommonBrowser::v_MayTranslateAccelerator+0xda
0361fd3c 7296f7b0 ieframe!CShellBrowser2::_MayTranslateAccelerator+0x68
0361fd4c 7296f7f5 ieframe!CShellBrowser2::v_MayTranslateAccelerator+0x15
0361fdb8 7680e4a5 ieframe!CTabWindow::_TabWindowThreadProc+0x264
0361fdc4 77adcfed kernel32!BaseThreadInitThunk+0xe
0361fe04 77add1ff ntdll_77a60000!__RtlUserThreadStart+0x23
0361fe1c 00000000 ntdll_77a60000!_RtlUserThreadStart+0x1b

Then we can use lmv command to see the component version:

0:002:x86> lmv m html
start             end                 module name
722f0000 7236f000   html     # (pdb symbols)          c:\mss\html.pdb\34A9E6645AEA4E6A83EA51D2849C731C1\html.pdb
    Loaded symbol image file: html.iec
    Image path: C:\Windows\System32\html.iec
    Image name: html.iec
    Timestamp:        Tue Mar 03 03:26:33 2009 (49ACA369)
    CheckSum:         00086241
    ImageSize:        0007F000
    File version:     2017.0.0.18226
    Product version:  6.0.6001.18226
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     HTML.IEC
    OriginalFilename: HTML.IEC
    ProductVersion:   6.0.6001.18226
    FileVersion:      2017.0.0.18226 (vistasp1_gdr.090302-1506)
    FileDescription:  Microsoft HTML Converter
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

- Dmitry Vostokov @ DumpAnalysis.org -

DumpAnalysis.org acquires TraceAnalysis.org to complete computer DATA artifact analysis. The domain currently points to Dump Analysis Portal page but this might change in the future.

- Dmitry Vostokov @ DumpAnalysis.org -

Next obvious structural trace analysis pattern is called Circular Trace. Sometimes this information is missing in the problem description or trace metadata doesn’t reflect this. Then circular traces can be detected by trace file size (usually large) and from timestamps, like this 100Mb CDF trace snippet:

No     Module  PID  TID  Date      Time         Statement
[Begin of trace listing]
1      ModuleA 4280 1736 5/28/2009 08:53:50.496 [... Trace statement 1]
2      ModuleB 6212 6216 5/28/2009 08:53:52.876 [... Trace statement 2] 
3      ModuleA 4280 4776 5/28/2009 08:54:13.537 [... Trace statement 3] 
[... Some traced exceptions helpful for analysis ...]
3799   ModuleA 4280 3776 5/28/2009 09:15:00.853 [… Trace statement 3799]
3800   ModuleA 4280 1736 5/27/2009 09:42:12.029 [… Trace statement 3800]
[… Skipped …]
[… Skipped …]
[… Skipped …]
579210 ModuleA 4280 4776 5/28/2009 08:53:35.989 [… Trace statement 579210]
[End of trace listing]

In such traces the analysis region usually can be found at the beginning at the trace because as soon as elusive and hard to reproduce problem happens the trace is stopped.

- Dmitry Vostokov @ TraceAnalysis.org -

DATA (Dump Artifact + Trace Artifact) - > DATA (Dump Analysis + Trace Analysis) examples:

1.  Efficient

- My 64Gb server bluescreens. I set a complete memory dump option in Control Panel.

- A user cannot connect. I started tracing yesterday. Stopped today.

- I analyze all these artifacts every day.

2. Effective

- My 64Gb server bluescreens. I set a kernel memory dump option in Control Panel.

- A user cannot connect. I started tracing, tried to connect, stopped tracing.

- I analyze all these artifacts every day and write articles to reduce DATA load.

- Dmitry Vostokov @ DumpAnalysis.org -