Software Diagnostics Library

An IE window was frozen and user process memory dump files from all IE process instances inside a user session were saved. The first instance revealed a main thread which self-diagnosed a hang tab and was blocked in a window message chain:

0:000> kL
ChildEBP RetAddr 
0012ea84 7e4194be ntdll!KiFastSystemCallRet
0012eac0 7e4292e3 user32!NtUserMessageCall+0xc
0012eae0 3e4171a1 user32!SendMessageW+0×7f
0012eaf4 3e41863f ieframe!CTabWindow::_MakeBlockingCallToHungTabToTriggerNtUserHangDetection+0×11
0012eb00 3e31d261 ieframe!CTabWindow::MarkTabAsHung+0×48
0012eb1c 7e418734 ieframe!FrameTabWndProc+0×5c
0012eb48 7e418816 user32!InternalCallWinProc+0×28
0012ebb0 7e4189cd user32!UserCallWinProcCheckWow+0×150
0012ec10 7e418a10 user32!DispatchMessageWorker+0×306
0012ec20 3e2ed530 user32!DispatchMessageW+0xf
0012ec88 3e204dd9 ieframe!CBrowserFrame::FrameMessagePump+0×3d7
0012ecd0 3e1ea0a7 ieframe!BrowserThreadProc+0xf7
0012ecf0 3e1ea004 ieframe!BrowserNewThreadProc+0×88
0012fd60 3e1e9f26 ieframe!SHOpenFolderWindow+0×10e
0012fd84 3e1e9c75 ieframe!IEWinMainEx+0×1ff
0012fda0 3e1ebf1d ieframe!IEWinMain+0×77
0012fdd8 00402e11 ieframe!LCIEStartAsFrame+0×252
0012ff2c 0040128e iexplore!wWinMain+0×368
0012ffc0 7c817077 iexplore!_initterm_e+0×1b1
0012fff0 00000000 kernel32!BaseProcessStart+0×23

We looked at other IE instances and found the one thread with a blocking module:

0:017> kL 100
ChildEBP RetAddr 
02c34100 7c90df5a ntdll!KiFastSystemCallRet
02c34104 7c8025db ntdll!ZwWaitForSingleObject+0xc
02c34168 7c802542 kernel32!WaitForSingleObjectEx+0xa8
02c3417c 009f0ed9 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
02c34a08 00bc2c9a ModuleA!DllCanUnloadNow+0×6db39
02c3526c 00bc2fa4 ModuleA!DllCanUnloadNow+0×23f8fa
02c35ae0 00f6413c ModuleA!DllCanUnloadNow+0×23fc04
02c363e8 00c761ab ModuleA!DllCanUnloadNow+0×5e0d9c
02c36c74 00c74daa ModuleA!DllCanUnloadNow+0×2f2e0b
02c374e4 3d1a9eb4 ModuleA!DllCanUnloadNow+0×2f1a0a
02c3753c 3d0ed032 mshtml!CView::SetObjectRectsHelper+0×98
02c37578 3cf7e43b mshtml!CView::EndDeferSetObjectRects+0×75
02c375bc 3cf2542d mshtml!CView::EnsureView+0×39f
02c375d8 3cf4072c mshtml!CElement::EnsureRecalcNotify+0×17c
02c37614 3cf406ce mshtml!CElement::get_clientHeight_Logical+0×54
02c37628 3d0822a1 mshtml!CElement::get_clientHeight+0×27
02c37648 3cf8ad53 mshtml!G_LONG+0×7b
02c376bc 3cf96e21 mshtml!CBase::ContextInvokeEx+0×5d1
02c3770c 3cfa2baf mshtml!CElement::ContextInvokeEx+0×9d
02c37738 3cf8a751 mshtml!CElement::VersionedInvokeEx+0×2d
02c37788 3d7c389a mshtml!PlainInvokeEx+0xea
02c377c8 3d7c37e6 jscript!IDispatchExInvokeEx2+0xf8
02c37804 3d7c4d26 jscript!IDispatchExInvokeEx+0×6a
02c378c4 3d7c4c80 jscript!InvokeDispatchEx+0×98
02c378f8 3d7c4996 jscript!VAR::InvokeByName+0×135
02c37a90 3d7c11ab jscript!CScriptRuntime::Run+0×654
02c37b78 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c37bc4 3d7c48ac jscript!ScrFncObj::Call+0×8f
02c37c48 3d7c26c5 jscript!NameTbl::InvokeInternal+0×137
02c37c7c 3d7c2f14 jscript!VAR::InvokeByDispID+0×17c
02c37e18 3d7c11ab jscript!CScriptRuntime::Run+0×29e0
02c37f00 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c37f4c 3d7c48ac jscript!ScrFncObj::Call+0×8f
02c37fd0 3d7c26c5 jscript!NameTbl::InvokeInternal+0×137
02c38004 3d7c4d93 jscript!VAR::InvokeByDispID+0×17c
02c381a0 3d7c11ab jscript!CScriptRuntime::Run+0×2abe
02c38288 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c382d4 3d7c48ac jscript!ScrFncObj::Call+0×8f
02c38358 3d7c26c5 jscript!NameTbl::InvokeInternal+0×137
02c3838c 3d7c4d93 jscript!VAR::InvokeByDispID+0×17c
02c38528 3d7c11ab jscript!CScriptRuntime::Run+0×2abe
02c38610 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c3865c 3d7c2805 jscript!ScrFncObj::Call+0×8f
02c386e0 3d7c26c5 jscript!NameTbl::InvokeInternal+0×2a2
02c38714 3d7c41fc jscript!VAR::InvokeByDispID+0×17c
02c38754 3d7c22c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
02c38790 3d7c2b6d jscript!VAR::InvokeByName+0×170
02c387dc 3d7c4035 jscript!VAR::InvokeDispName+0×7a
02c3880c 3d7c4d93 jscript!VAR::InvokeByDispID+0xce
02c389a8 3d7c11ab jscript!CScriptRuntime::Run+0×2abe
02c38a90 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c38adc 3d7c48ac jscript!ScrFncObj::Call+0×8f
02c38b60 3d7c26c5 jscript!NameTbl::InvokeInternal+0×137
02c38b94 3d7c4d93 jscript!VAR::InvokeByDispID+0×17c
02c38d30 3d7c11ab jscript!CScriptRuntime::Run+0×2abe
02c38e18 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c38e64 3d7c2805 jscript!ScrFncObj::Call+0×8f
02c38ee8 3d7c26c5 jscript!NameTbl::InvokeInternal+0×2a2
02c38f1c 3d7c41fc jscript!VAR::InvokeByDispID+0×17c
02c38f5c 3d7c22c1 jscript!VAR::InvokeJSObj<SYM *>+0xb8
02c38f98 3d7c2b6d jscript!VAR::InvokeByName+0×170
02c38fe4 3d7c4035 jscript!VAR::InvokeDispName+0×7a
02c39014 3d7c2f14 jscript!VAR::InvokeByDispID+0xce
02c391b0 3d7c11ab jscript!CScriptRuntime::Run+0×29e0
02c39298 3d7c10e5 jscript!ScrFncObj::CallWithFrameOnStack+0xff
02c392e4 3d7c0f13 jscript!ScrFncObj::Call+0×8f
02c39360 3d7a3ea3 jscript!CSession::Execute+0×175
02c393ac 3d7a552f jscript!COleScript::ExecutePendingScripts+0×1c0
02c39410 3d7a5345 jscript!COleScript::ParseScriptTextCore+0×29a
02c39438 3ceca304 jscript!COleScript::ParseScriptText+0×30
02c39490 3d0955af mshtml!CScriptCollection::ParseScriptText+0×219
02c3b528 3d07a59c mshtml!CWindow::ExecuteScriptUri+0×19f
02c3b570 3d0958fd mshtml!CWindow::NavigateEx+0×5a
02c3b5dc 3d10a995 mshtml!CDoc::ExecuteScriptUri+0×262
02c3b648 3d056840 mshtml!CWindow::SuperNavigateInternal+0×335
02c3b67c 3e27d357 mshtml!CWindow::SuperNavigate2WithBindFlags+0×29
02c3b70c 3e27d1fb ieframe!CDocObjectHost::_NavigateDocument+0×1d9
02c3c7b0 3e27ab0e ieframe!CDocObjectHost::SetTarget+0×37b
02c3c7e8 3e27a8f1 ieframe!CDocObjectView::CreateViewWindow2+0xea
02c3c820 3e27a22a ieframe!CDocObjectView::CreateViewWindow+0×49
02c3c8dc 3e27a149 ieframe!FileCabinet_CreateViewWindow2+0×29d
02c3c900 3e27a067 ieframe!CBaseBrowser2::_CreateViewWindow+0×2b
02c3c940 3e279f1b ieframe!CBaseBrowser2::_CreateNewShellView+0×1a6
02c3c970 3e279e4e ieframe!CBaseBrowser2::_CreateNewShellViewPidl+0xe1
02c3d9f4 3e27c2dd ieframe!CBaseBrowser2::v_NavigateToPidl+0×2c3
02c3dc44 3e2ad948 ieframe!CBaseBrowser2::_OnGoto+0×2fb
02c3dc58 3e2e8a01 ieframe!CBaseBrowser2::v_WndProc+0×340
02c3dcbc 3e2e894f ieframe!CShellBrowser2::v_WndProc+0×3fe
02c3dce0 7e418734 ieframe!CShellBrowser2::s_WndProc+0xfb
02c3dd0c 7e418816 user32!InternalCallWinProc+0×28
02c3dd74 7e4189cd user32!UserCallWinProcCheckWow+0×150
02c3ddd4 7e418a10 user32!DispatchMessageWorker+0×306
02c3dde4 3e2ec2a5 user32!DispatchMessageW+0xf
02c3feec 3e293357 ieframe!CTabWindow::_TabWindowThreadProc+0×54c
02c3ffa4 3e134435 ieframe!LCIETab_ThreadProc+0×2c1
02c3ffb4 7c80b729 iertutil!CIsoScope::RegisterThread+0xab
02c3ffec 00000000 kernel32!BaseThreadStart+0×37

The ModuleA component was quite ubiquitous and seen in other threads from the same process:

   1  Id: e8c.b5c Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr 
01f9f698 7c90d21a ntdll!KiFastSystemCallRet
01f9f69c 7c8023f1 ntdll!NtDelayExecution+0xc
01f9f6f4 7c802455 kernel32!SleepEx+0x61
01f9f704 009d284a kernel32!Sleep+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
01f9ffb4 7c80b729 ModuleA!DllCanUnloadNow+0×4f4aa
01f9ffec 00000000 kernel32!BaseThreadStart+0×37

  25  Id: e8c.f20 Suspend: 1 Teb: 7ff9c000 Unfrozen
ChildEBP RetAddr 
086acac4 7c90df5a ntdll!KiFastSystemCallRet
086acac8 7c8025db ntdll!ZwWaitForSingleObject+0xc
086acb2c 7c802542 kernel32!WaitForSingleObjectEx+0xa8
086acb40 00fbba3a kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
086ad3c8 00fbc139 ModuleA!DllCanUnloadNow+0×63869a
086adc38 00faba75 ModuleA!DllCanUnloadNow+0×638d99
086ae4c8 00fa0da8 ModuleA!DllCanUnloadNow+0×6286d5
086aed60 00a45331 ModuleA!DllCanUnloadNow+0×61da08
086af6c4 00a44b10 ModuleA!DllCanUnloadNow+0xc1f91
086affb4 7c80b729 ModuleA!DllCanUnloadNow+0xc1770
086affec 00000000 kernel32!BaseThreadStart+0×37

Fortunately we also had a complete memory dump generated shortly after hang and from it we could find dual stack traces from the same processes and find that blocked threads were waiting for named pipes with endpoints on another PC. So we advised to take a complete memory dump from the coupled machine.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is the variant of the general Wait Chain pattern where threads are waiting for named pipes. This is visible when we examining the pending IRP of a blocked thread:

THREAD 88ec9020  Cid 17a0.2034  Teb: 7ffad000 Win32Thread: bc28c6e8 WAIT: (Unknown) UserMode Non-Alertable
    89095f48  Semaphore Limit 0x10000
IRP List:
    89a5a370: (0006,0094) Flags: 00000900  Mdl: 00000000
Not impersonating
DeviceMap                 d6c30c48
Owning Process            88fffd88       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      5632994        Ticks: 2980 (0:00:00:46.562)
Context Switch Count      2269                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×00a262d0
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b204c000 Current b204bc60 Base b204c000 Limit b2048000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr 
b204bc78 80833ec5 nt!KiSwapContext+0×26
b204bca4 80829c14 nt!KiSwapThread+0×2e5
b204bcec 8093b174 nt!KeWaitForSingleObject+0×346
b204bd50 8088b41c nt!NtWaitForSingleObject+0×9a
b204bd50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b204bd64)
058fcabc 7c827d29 ntdll!KiFastSystemCallRet
058fcac0 77e61d1e ntdll!ZwWaitForSingleObject+0xc
058fcb30 77e61c8d kernel32!WaitForSingleObjectEx+0xac
058fcb44 00f98b4a kernel32!WaitForSingleObject+0×12
[…]
058fffec 00000000 kernel32!BaseThreadStart+0×34

0: kd> !irp 89a5a370
Irp is active with 1 stacks 1 is current (= 0×89a5a3e0)
No Mdl: No System Buffer: Thread 88ec9020:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
>[  3, 0]   0  1 89ebee90 891d4f90 00000000-00000000    pending
                 \FileSystem\Npfs
                                   Args: 00000100 00000000 00000000 00000000

0: kd> !fileobj 891d4f90

\ServiceB\SVC

Device Object: 0x89ebee90   \FileSystem\Npfs
Vpb is NULL

Flags:  0x40080
                Named Pipe
                Handle Created

FsContext: 0xdaeca230 FsContext2: 0x8949bdb0
Private Cache Map: 0x00000001
CurrentByteOffset: 0

The pipe chain can also extend from thread to thread and even cross machine boundary. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Resolution rush - The rush of software technical support and maintenance engineers to provide the resolution to a suddenly escalated incident.

Examples: After it crashed 3 times in a row at the customer site our VP was called and we all got the resolution rush.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Working for more than 7 years in technical support environment I found that many support incidents were resolved more easily by abductive reasoning than by induction and deduction practiced by Sherlock Holmes and observed by Dr. Watson. Abduction as a way to build an incident theory to advance in problem resolution was practiced by a USA colleague of Holmes: Philip Marlowe. Because technical support is less detached from customers (”the world”) when compared to software engineering departments I see the way of Marlowe as more natural. Of course, from time to time the way of Holmes is also appropriate. All depends on a support case. I found that abductive reasoning is also appropriate for memory dump and software trace analysis where “leaps of faith” are necessary because of insufficient information. Such leaps of abduction actually happen all the time when analysts give troubleshooting advice based on patterns.

I plan to write more about the 3rd way of reasoning after I finish reading two Raymond Chandler’s novels and a few other inference, causality and explanation books I mention later: The Big Sleep & Farewell, My Lovely (Modern Library).

I’m grateful for Clive Gamble for pointing this way out in his book Archaeology: The Basics

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When loading a process user memory dump we recognized it as abridged and embedded comment pointed to a spiking thread: 

Loading Dump File [ApplicationA_101212_165342.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Comment: '
*** procdump -c 60 -s 5 -n 3 ApplicationA.exe
*** Process exceeded 60% CPU for 5 seconds. Thread consuming CPU: 540 (0×21c)‘

This thread is already default:

0:005> ~
   0  Id: c1c.c20 Suspend: 0 Teb: 7ffdf000 Unfrozen
   1  Id: c1c.c44 Suspend: 0 Teb: 7ffde000 Unfrozen
   2  Id: c1c.d34 Suspend: 0 Teb: 7ffdc000 Unfrozen
   3  Id: c1c.d38 Suspend: 0 Teb: 7ffda000 Unfrozen
   4  Id: c1c.d3c Suspend: 0 Teb: 7ffd9000 Unfrozen
.  5  Id: c1c.21c Suspend: 0 Teb: 7ffd8000 Unfrozen
   6  Id: c1c.1c10 Suspend: 0 Teb: 7ffdd000 Unfrozen
   7  Id: c1c.1678 Suspend: 0 Teb: 7ffd6000 Unfrozen
   8  Id: c1c.cbc Suspend: 0 Teb: 7ffd5000 Unfrozen
   9  Id: c1c.1754 Suspend: 0 Teb: 7ffaf000 Unfrozen
  10  Id: c1c.c40 Suspend: 0 Teb: 7ffad000 Unfrozen
  11  Id: c1c.1d24 Suspend: 0 Teb: 7ffd7000 Unfrozen

The stack trace looks incorrect: 

0:005> kL
ChildEBP RetAddr 
01abc4d8 6efba23d ntdll!KiFastSystemCallRet
WARNING: Stack unwind information not available. Following frames may be wrong.
01abc988 7c820833 ModuleB+0×2a23d
01abcbe4 7c8207f6 kernel32!GetVolumeNameForRoot+0×26
01abcc0c 7c82e6de kernel32!BasepGetVolumeNameForVolumeMountPoint+0×75
01abcc54 6efaf70b kernel32!GetVolumePathNameW+0×18a
01abccdc 6efbd1a6 ModuleB+0×1f70b
01abcce0 00000000 ModuleB+0×2d1a6

However, we see a 3rd party top module and advise to keep an eye on it:

0:005> lmt m ModuleB
start    end        module name
6ef90000 6efff000   ModuleB    Wed Mar 10 20:18:21 2010

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Similar to Blocking Module pattern we also have Top Module pattern where the difference is in stack trace syntax only. A top module is any module we choose that is simply on top of a stack trace. Most of the time it is likely to be a non-OS vendor module. Whether the stack trace is well-formed and semantically sound or incorrect is irrelevant:

0:005> kL
ChildEBP RetAddr 
01abc4d8 6efba23d ntdll!KiFastSystemCallRet
WARNING: Stack unwind information not available. Following frames may be wrong.
01abc988 7c820833 ModuleB+0×2a23d
01abcbe4 7c8207f6 kernel32!GetVolumeNameForRoot+0×26
01abcc0c 7c82e6de kernel32!BasepGetVolumeNameForVolumeMountPoint+0×75
01abcc54 6efaf70b kernel32!GetVolumePathNameW+0×18a
01abccdc 6efbd1a6 ModuleB+0×1f70b
01abcce0 00000000 ModuleB+0×2d1a6

Here we can also check the validity of ModuleB code by backwards disassembly of 6efba23d return address (ub command) unless we have an abridged dump file (minidump) and we need to specify the image file path in WinDbg,

Why a top module is important? In various troubleshooting scenarious we can check the module timestamp (Not My Version pattern) and other useful information (lmv and !lmi WinDbg commands). If we suspect the module belonging to hooksware we can also recommend removing it or its software vendor package for testing purposes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

With the new year starts the new initiative to integrate traditional multidisciplinary debugging approaches and methodologies with multiplatform pattern-driven software problem solving, unified debugging patterns, best practices in memory dump analysis and software tracing, computer security, economics, and the new emerging trends I’m going to write about during this year.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Similar to Google GMMXIe depiction and interpretation I propose another one related to memory centuries that start from 1000 CE (M…):

MMXI

Malware Memory eXception and Injection

or

Monitoring Memory, eXceptions, and Injections

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is an artwork commissioned for the New Year of DeBugging 0×7DB. How many bugs can you count there? Click on the picture to expand instead of using a magnifying glass (as seen on debugging books covers):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The New Year of DeBugging 0×7DB opens the DeBugging Decade. Just a few tips for the beginning:

1: kd>*
1: kd>* Virtual memory search
1: kd>*

1: kd> s-d 0 L?3FFFFFFF 7DB
00000000`777509dc 0000007db 0004c550 0005dd80 0000b610

1: kd>*
1: kd>* Physical memory search for the first 65535 pages
1: kd>*

1: kd> !search 7DB 0 0 FFFF
Searching PFNs in range 0000000000000001 - 000000000000FFFF for [00000000000007DB - 00000000000007DB]

Pfn              Offset   Hit              Va               Pte
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
000000000000041D 00000D10 000000000000075B FFFFF8A000437D10 FFFFF6FC500021B8
 fffff8a00040b000+0x2cd10 : NtFB (LargePool) -- BitmpSup.c
000000000000045F 00000090 000000000000079B FFFFF8A000439090 FFFFF6FC500021C8
 fffff8a00040b000+0x2e090 : NtFB (LargePool) -- BitmpSup.c
000000000000045F 00000B40 00000000000007CB FFFFF8A000439B40 FFFFF6FC500021C8
 fffff8a00040b000+0x2eb40 : NtFB (LargePool) -- BitmpSup.c
000000000000045F 00000D80 00000000000007D3 FFFFF8A000439D80 FFFFF6FC500021C8
 fffff8a00040b000+0x2ed80 : NtFB (LargePool) -- BitmpSup.c
0000000000000460 000000B0 00000000000007D9 FFFFF8A00043A0B0 FFFFF6FC500021D0
 fffff8a00040b000+0x2f0b0 : NtFB (LargePool) -- BitmpSup.c
0000000000000460 000000C8 00000000000007DA FFFFF8A00043A0C8 FFFFF6FC500021D0
 fffff8a00040b000+0x2f0c8 : NtFB (LargePool) -- BitmpSup.c
0000000000000460 000000E0 00000000000007DB FFFFF8A00043A0E0 FFFFF6FC500021D0
 fffff8a00040b000+0x2f0e0 : NtFB (LargePool) -- BitmpSup.c
0000000000000460 00000218 00000000000007DF FFFFF8A00043A218 FFFFF6FC500021D0
 fffff8a00040b000+0x2f218 : NtFB (LargePool) -- BitmpSup.c
0000000000000460 00000950 00000000000007FB FFFFF8A00043A950 FFFFF6FC500021D0
 fffff8a00040b000+0x2f950 : NtFB (LargePool) -- BitmpSup.c
0000000000000494 00000ED8 00000000000006DB FFFFF8A00042EED8 FFFFF6FC50002170
 fffff8a00040b000+0x23ed8 : NtFB (LargePool) -- BitmpSup.c
00000000000004D0 00000708 00000000000005DB FFFFF8A00042A708 FFFFF6FC50002150
 fffff8a00040b000+0x1f708 : NtFB (LargePool) -- BitmpSup.c
0000000000000562 000002A0 0000000000000FDB FFFFF8A00043C2A0 FFFFF6FC500021E0
 fffff8a00040b000+0x312a0 : NtFB (LargePool) -- BitmpSup.c
00000000000005C8 00000020 00000000000003DB FFFFF8A000422020 FFFFF6FC50002110
 fffff8a00040b000+0x17020 : NtFB (LargePool) -- BitmpSup.c
0000000000000A80 00000F58 00000000000017DB FFFFF8A00041AF58 FFFFF6FC500020D0
 fffff8a00040b000+0xff58 : NtFB (LargePool) -- BitmpSup.c
0000000000003411 000008E0 00000000000005DB FFFFFA80000118E0 FFFFF6FD40000088
0000000000003411 00000948 00000000000005DB FFFFFA8000011948 FFFFF6FD40000088
0000000000003413 00000D18 000000000000079B FFFFFA8000013D18 FFFFF6FD40000098
0000000000003415 00000510 000000000000079B FFFFFA8000015510 FFFFF6FD400000A8
0000000000003415 00000518 000000000000075B FFFFFA8000015518 FFFFF6FD400000A8
0000000000003416 00000CB8 00000000000007D9 FFFFFA8000016CB8 FFFFF6FD400000B0
0000000000003431 00000110 00000000000017DB FFFFFA8000031110 FFFFF6FD40000188
0000000000003431 00000D10 0000000000000FDB FFFFFA8000031D10 FFFFF6FD40000188
0000000000003449 00000118 00000000000017DB FFFFFA8000049118 FFFFF6FD40000248
0000000000003466 00000218 00000000000007CB FFFFFA8000066218 FFFFF6FD40000330
0000000000003466 000004B0 00000000000007D9 FFFFFA80000664B0 FFFFF6FD40000330
0000000000003466 00000510 000000000000075B FFFFFA8000066510 FFFFF6FD40000330
0000000000003467 00000110 00000000000007DB FFFFFA8000067110 FFFFF6FD40000338
000000000000346B 00000918 00000000000007DB FFFFFA800006B918 FFFFF6FD40000358
0000000000003473 00000260 00000000000005DB FFFFFA8000073260 FFFFF6FD40000398
000000000000349A 00000A10 00000000000007CB FFFFFA800009AA10 FFFFF6FD400004D0
000000000000349B 00000348 00000000000007FB FFFFFA800009B348 FFFFF6FD400004D8
000000000000352C 00000510 00000000000006DB FFFFFA800012C510 FFFFF6FD40000960
000000000000352C 00000AE0 00000000000007FB FFFFFA800012CAE0 FFFFF6FD40000960
0000000000003563 000001D0 00000000000007DF FFFFFA80001631D0 FFFFF6FD40000B18
000000000000356A 00000938 00000000000007DA FFFFFA800016A938 FFFFF6FD40000B50
0000000000003596 00000D10 00000000000087DB FFFFFA8000196D10 FFFFF6FD40000CB0
0000000000003599 00000D18 00000000000087DB FFFFFA8000199D18 FFFFF6FD40000CC8
00000000000035A8 00000738 00000000000007DA FFFFFA80001A8738 FFFFF6FD40000D40
000000000000369B 00000B30 00000000000027DB FFFFFA800029BB30 FFFFF6FD400014D8
00000000000036FE 00000B90 00000000000007D3 FFFFFA80002FEB90 FFFFF6FD400017F0
0000000000003710 00000D10 00000000000107DB FFFFFA8000310D10 FFFFF6FD40001880
0000000000003747 00000918 00000000000107DB FFFFFA8000347918 FFFFF6FD40001A38
00000000000037B9 000009D8 00000000000007DF FFFFFA80003B99D8 FFFFF6FD40001DC8
000000000000380D 00000640 00000000000003DB FFFFFA800040D640 FFFFF6FD40002068
00000000000038D7 00000870 0000000000000FDB FFFFFA80004D7870 FFFFF6FD400026B8
000000000000391B 00000490 0000000000000FDB FFFFFA800051B490 FFFFF6FD400028D8
0000000000003923 000003E0 000000000000075B FFFFFA80005233E0 FFFFF6FD40002918
000000000000392A 000001F0 00000000000007DB FFFFFA800052A1F0 FFFFF6FD40002950
000000000000393C 00000A30 00000000000007FB FFFFFA800053CA30 FFFFF6FD400029E0
0000000000003951 00000BE0 00000000000007D3 FFFFFA8000551BE0 FFFFF6FD40002A88
0000000000003964 00000F00 00000000000007FB FFFFFA8000564F00 FFFFF6FD40002B20
0000000000003980 000004D0 00000000000007CB FFFFFA80005804D0 FFFFF6FD40002C00
000000000000399D 00000900 00000000000007DA FFFFFA800059D900 FFFFF6FD40002CE8
0000000000003A11 00000910 00000000000207DB FFFFFA8000611910 FFFFF6FD40003088
0000000000003A16 00000118 00000000000207DB FFFFFA8000616118 FFFFF6FD400030B0
0000000000003B3E 00000570 00000000000006DB FFFFFA800073E570 FFFFF6FD400039F0
0000000000003D23 000003D0 00000000000006DB FFFFFA80009233D0 FFFFF6FD40004918
0000000000003D32 00000AF0 00000000000007FB FFFFFA8000932AF0 FFFFF6FD40004990
0000000000003D5C 00000580 00000000000007DB FFFFFA800095C580 FFFFF6FD40004AE0
0000000000003D7B 00000150 00000000000007DA FFFFFA800097B150 FFFFF6FD40004BD8
0000000000003D81 00000D20 00000000000007D9 FFFFFA8000981D20 FFFFF6FD40004C08
0000000000003DB9 000009A0 00000000000007DB FFFFFA80009B99A0 FFFFF6FD40004DC8
0000000000003DE7 00000AB0 00000000000007D3 FFFFFA80009E7AB0 FFFFF6FD40004F38
0000000000003DE8 000001D0 00000000000007D9 FFFFFA80009E81D0 FFFFF6FD40004F40
0000000000003F29 00000410 00000000000006DB FFFFFA8000B29410 FFFFF6FD40005948
0000000000003FA5 00000918 00000000000407DB FFFFFA8000BA5918 FFFFF6FD40005D28
000000000000401C 00000110 00000000000407DB FFFFFA8000C1C110 FFFFF6FD400060E0
000000000000443D 00000580 00000000000007FB FFFFFA800103D580 FFFFF6FD400081E8
0000000000004502 000006B0 000000000000079B FFFFFA80011026B0 FFFFF6FD40008810
000000000000456B 00000BF0 00000000000007DF FFFFFA800116BBF0 FFFFF6FD40008B58
0000000000004577 00000B90 000000000000075B FFFFFA8001177B90 FFFFF6FD40008BB8
00000000000046B7 00000330 00000000000007CB FFFFFA80012B7330 FFFFF6FD400095B8
0000000000004819 00000F90 00000000000007DB FFFFFA8001419F90 FFFFF6FD4000A0C8
0000000000004A62 00000930 00000000000003DB FFFFFA8001662930 FFFFF6FD4000B310
0000000000004AA2 00000C20 00000000000006DB FFFFFA80016A2C20 FFFFF6FD4000B510
0000000000004AB6 00000870 00000000000007FB FFFFFA80016B6870 FFFFF6FD4000B5B0
0000000000004ABB 000007C0 000000000000079B FFFFFA80016BB7C0 FFFFF6FD4000B5D8
0000000000004AD5 000003E0 00000000000007DA FFFFFA80016D53E0 FFFFF6FD4000B6A8
0000000000004BAA 00000560 00000000000007D3 FFFFFA80017AA560 FFFFF6FD4000BD50
0000000000004BE5 00000570 00000000000007CB FFFFFA80017E5570 FFFFF6FD4000BF28
0000000000004C18 00000510 00000000000807DB FFFFFA8001818510 FFFFF6FD4000C0C0
0000000000004C19 00000118 00000000000807DB FFFFFA8001819118 FFFFF6FD4000C0C8
0000000000004C1F 00000530 00000000000107DB FFFFFA800181F530 FFFFF6FD4000C0F8
0000000000004CAB 00000750 000000000000079B FFFFFA80018AB750 FFFFF6FD4000C558
0000000000004CB3 00000AF0 00000000000007D3 FFFFFA80018B3AF0 FFFFF6FD4000C598
0000000000005061 000002E0 000000000000075B FFFFFA8001C612E0 FFFFF6FD4000E308
0000000000005291 00000F80 00000000000006DB FFFFFA8001E91F80 FFFFF6FD4000F488
00000000000052B7 00000E70 00000000000027DB FFFFFA8001EB7E70 FFFFF6FD4000F5B8
0000000000005314 00000630 00000000000007DB FFFFFA8001F14630 FFFFF6FD4000F8A0
000000000000531D 00000630 000000000000079B FFFFFA8001F1D630 FFFFF6FD4000F8E8
0000000000005336 00000230 00000000000005DB FFFFFA8001F36230 FFFFF6FD4000F9B0
000000000000533E 00000660 00000000000006DB FFFFFA8001F3E660 FFFFF6FD4000F9F0
0000000000005343 00000550 0000000000000FDB FFFFFA8001F43550 FFFFF6FD4000FA18
0000000000005384 00000E60 00000000000007DA FFFFFA8001F84E60 FFFFF6FD4000FC20
00000000000053F1 00000820 00000000000007DA FFFFFA8001FF1820 FFFFF6FD4000FF88
00000000000053F1 00000BB0 00000000000003DB FFFFFA8001FF1BB0 FFFFF6FD4000FF88
00000000000053FF 00000C80 00000000000007D9 FFFFFA8001FFFC80 FFFFF6FD4000FFF8
0000000000005428 00000C00 00000000000007DA FFFFFA8002028C00 FFFFF6FD40010140
000000000000542D 00000DC0 00000000000007FB FFFFFA800202DDC0 FFFFF6FD40010168
0000000000005456 00000800 0000000000000FDB FFFFFA8002056800 FFFFF6FD400102B0
0000000000005459 00000CB0 00000000000006DB FFFFFA8002059CB0 FFFFF6FD400102C8
000000000000546B 00000B30 00000000000003DB FFFFFA800206BB30 FFFFF6FD40010358
000000000000547F 00000450 00000000000005DB FFFFFA800207F450 FFFFF6FD400103F8
000000000000549E 00000290 00000000000087DB FFFFFA800209E290 FFFFF6FD400104F0
000000000000551C 00000410 00000000000003DB FFFFFA800211C410 FFFFF6FD400108E0
000000000000563E 00000510 000000000000075B FFFFFA800223E510 FFFFF6FD400111F0
000000000000567C 00000FD0 00000000000007DF FFFFFA800227CFD0 FFFFF6FD400113E0
0000000000005731 000002A0 00000000000017DB FFFFFA80023312A0 FFFFF6FD40011988
000000000000575A 00000400 00000000000007FB FFFFFA800235A400 FFFFF6FD40011AD0
0000000000005847 00000400 00000000000003DB FFFFFA8003047400 FFFFF6FD40018238
000000000000588E 00000290 00000000000007D9 FFFFFA800308E290 FFFFF6FD40018470
0000000000005890 00000C60 000000000000075B FFFFFA8003090C60 FFFFF6FD40018480
0000000000005893 00000F90 00000000000005DB FFFFFA8003093F90 FFFFF6FD40018498
000000000000589B 000006D0 00000000000006DB FFFFFA800309B6D0 FFFFF6FD400184D8
00000000000058A5 000009C0 000000000000079B FFFFFA80030A59C0 FFFFF6FD40018528
00000000000058AA 000008B0 00000000000017DB FFFFFA80030AA8B0 FFFFF6FD40018550
00000000000058BC 00000400 00000000000003DB FFFFFA80030BC400 FFFFF6FD400185E0
0000000000005911 000002A0 00000000000007D3 FFFFFA80031112A0 FFFFF6FD40018888
0000000000005957 00000A40 00000000000005DB FFFFFA8003157A40 FFFFF6FD40018AB8
0000000000005995 00000E40 00000000000003DB FFFFFA8003195E40 FFFFF6FD40018CA8
000000000000599D 00000AF0 0000000000000FDB FFFFFA800319DAF0 FFFFF6FD40018CE8
00000000000059AD 000002A0 00000000000007DF FFFFFA80031AD2A0 FFFFF6FD40018D68
00000000000059B2 00000910 00000000000007DF FFFFFA80031B2910 FFFFF6FD40018D90
00000000000059D0 00000520 00000000000003DB FFFFFA80031D0520 FFFFF6FD40018E80
00000000000059E1 000003B0 00000000000007FB FFFFFA80031E13B0 FFFFF6FD40018F08
00000000000059E7 00000D10 00000000000007D3 FFFFFA80031E7D10 FFFFF6FD40018F38
00000000000059EC 00000690 00000000000007CB FFFFFA80031EC690 FFFFF6FD40018F60
00000000000059FB 000003C0 00000000000005DB FFFFFA80031FB3C0 FFFFF6FD40018FD8
00000000000059FB 00000FF0 00000000000005DB FFFFFA80031FBFF0 FFFFF6FD40018FD8
00000000000059FE 000003C0 00000000000006DB FFFFFA80031FE3C0 FFFFF6FD40018FF0
0000000000005A01 000003C0 00000000000007DA FFFFFA80032013C0 FFFFF6FD40019008
0000000000005A0A 000003F0 00000000000007D9 FFFFFA800320A3F0 FFFFF6FD40019050
0000000000005A0D 00000390 000000000000075B FFFFFA800320D390 FFFFF6FD40019068
0000000000005A0D 00000960 00000000000007FB FFFFFA800320D960 FFFFF6FD40019068
0000000000005A0D 00000F90 000000000000079B FFFFFA800320DF90 FFFFF6FD40019068
0000000000005A0E 000003B0 00000000000007FB FFFFFA800320E3B0 FFFFF6FD40019070
0000000000005A0E 00000B90 00000000000007DB FFFFFA800320EB90 FFFFF6FD40019070
0000000000005A0F 000006D0 00000000000007DF FFFFFA800320F6D0 FFFFF6FD40019078
0000000000005A11 00000080 00000000000003DB FFFFFA8003211080 FFFFF6FD40019088
0000000000005A19 000006F0 0000000000000FDB FFFFFA80032196F0 FFFFF6FD400190C8
0000000000005A31 00000930 00000000000006DB FFFFFA8003231930 FFFFF6FD40019188
0000000000005A47 00000560 00000000000017DB FFFFFA8003247560 FFFFF6FD40019238
0000000000005A7E 000002B0 00000000000017DB FFFFFA800327E2B0 FFFFF6FD400193F0
0000000000005ADC 00000480 00000000000003DB FFFFFA80032DC480 FFFFF6FD400196E0
0000000000005ADC 00000630 0000000000000FDB FFFFFA80032DC630 FFFFF6FD400196E0
0000000000005AF1 00000180 00000000000007D3 FFFFFA80032F1180 FFFFF6FD40019788
0000000000005AF5 00000B00 0000000000000FDB FFFFFA80032F5B00 FFFFF6FD400197A8
0000000000005AFA 00000630 00000000000003DB FFFFFA80032FA630 FFFFF6FD400197D0
0000000000005AFE 00000890 0000000000000FDB FFFFFA80032FE890 FFFFF6FD400197F0
0000000000005AFE 00000D10 00000000000007D9 FFFFFA80032FED10 FFFFF6FD400197F0
0000000000005B12 00000F30 00000000000007FB FFFFFA8003312F30 FFFFF6FD40019890
0000000000005B1F 00000680 00000000000007CB FFFFFA800331F680 FFFFF6FD400198F8
0000000000005B25 00000F20 00000000000003DB FFFFFA8003325F20 FFFFF6FD40019928
0000000000005B25 00000FE0 00000000000007DF FFFFFA8003325FE0 FFFFF6FD40019928
0000000000005B28 000005F0 00000000000007D3 FFFFFA80033285F0 FFFFF6FD40019940
0000000000005B33 00000DB0 000000000000075B FFFFFA8003333DB0 FFFFF6FD40019998
0000000000005B35 000005B0 00000000000006DB FFFFFA80033355B0 FFFFF6FD400199A8
0000000000005B39 00000120 000000000000075B FFFFFA8003339120 FFFFF6FD400199C8
0000000000005B3A 000006B0 00000000000007DB FFFFFA800333A6B0 FFFFF6FD400199D0
0000000000005B47 00000E50 000000000000079B FFFFFA8003347E50 FFFFF6FD40019A38
0000000000005B55 00000FE0 000000000000079B FFFFFA8003355FE0 FFFFF6FD40019AA8
0000000000005B5A 00000390 00000000000006DB FFFFFA800335A390 FFFFF6FD40019AD0
0000000000005B6B 00000310 00000000000006DB FFFFFA800336B310 FFFFF6FD40019B58
0000000000005BF4 00000200 00000000000027DB FFFFFA80033F4200 FFFFF6FD40019FA0
0000000000005CA3 00000250 00000000000007DF FFFFFA80034A3250 FFFFF6FD4001A518
0000000000005CB0 00000F90 00000000000007DA FFFFFA80034B0F90 FFFFF6FD4001A580
0000000000005CC1 00000310 00000000000003DB FFFFFA80034C1310 FFFFF6FD4001A608
0000000000005D06 00000130 00000000000005DB FFFFFA8003506130 FFFFF6FD4001A830
0000000000005D0E 00000170 00000000000003DB FFFFFA800350E170 FFFFF6FD4001A870
0000000000005D2B 00000330 00000000000007DF FFFFFA800352B330 FFFFF6FD4001A958
0000000000005D30 00000EB0 00000000000007D3 FFFFFA8003530EB0 FFFFF6FD4001A980
0000000000005D33 000003D0 00000000000007D9 FFFFFA80035333D0 FFFFF6FD4001A998
0000000000005D38 00000830 00000000000007DB FFFFFA8003538830 FFFFF6FD4001A9C0
0000000000005D3E 00000980 00000000000007CB FFFFFA800353E980 FFFFF6FD4001A9F0
0000000000005D42 00000700 000000000000075B FFFFFA8003542700 FFFFF6FD4001AA10
0000000000005D47 00000E60 00000000000007D3 FFFFFA8003547E60 FFFFF6FD4001AA38
0000000000005D4E 000002B0 00000000000007FB FFFFFA800354E2B0 FFFFF6FD4001AA70
0000000000005D4F 000008D0 00000000000007DF FFFFFA800354F8D0 FFFFF6FD4001AA78
0000000000005D55 00000030 00000000000007DB FFFFFA8003555030 FFFFF6FD4001AAA8
0000000000005D58 000002A0 00000000000007DF FFFFFA80035582A0 FFFFF6FD4001AAC0
0000000000005D67 00000060 00000000000007D3 FFFFFA8003567060 FFFFF6FD4001AB38
0000000000005D81 000006D0 000000000000075B FFFFFA80035816D0 FFFFF6FD4001AC08
0000000000005D86 00000BF0 00000000000006DB FFFFFA8003586BF0 FFFFF6FD4001AC30
0000000000005D87 00000670 00000000000005DB FFFFFA8003587670 FFFFF6FD4001AC38
0000000000005D87 00000700 00000000000007CB FFFFFA8003587700 FFFFF6FD4001AC38
0000000000005D96 00000B50 00000000000003DB FFFFFA8003596B50 FFFFF6FD4001ACB0
0000000000005DA4 00000290 00000000000005DB FFFFFA80035A4290 FFFFF6FD4001AD20
0000000000005DBF 000005C0 00000000000007D3 FFFFFA80035BF5C0 FFFFF6FD4001ADF8
0000000000005DC6 00000F10 00000000000003DB FFFFFA80035C6F10 FFFFF6FD4001AE30
0000000000005DCF 00000B20 00000000000007D3 FFFFFA80035CFB20 FFFFF6FD4001AE78
0000000000005DD6 00000CF0 00000000000007DB FFFFFA80035D6CF0 FFFFF6FD4001AEB0
0000000000005DDC 00000FF0 00000000000007DF FFFFFA80035DCFF0 FFFFF6FD4001AEE0
0000000000005DEF 00000B90 000000000000075B FFFFFA80035EFB90 FFFFF6FD4001AF78
0000000000005DF1 00000CC0 000000000000079B FFFFFA80035F1CC0 FFFFF6FD4001AF88
0000000000005E03 00000F30 00000000000005DB FFFFFA8003603F30 FFFFF6FD4001B018
0000000000005E12 00000F60 00000000000005DB FFFFFA8003612F60 FFFFF6FD4001B090
0000000000005E13 00000890 00000000000003DB FFFFFA8003613890 FFFFF6FD4001B098
0000000000005E18 00000E10 000000000000075B FFFFFA8003618E10 FFFFF6FD4001B0C0
0000000000005E1A 00000100 00000000000007CB FFFFFA800361A100 FFFFF6FD4001B0D0
0000000000005E38 00000310 000000000000075B FFFFFA8003638310 FFFFF6FD4001B1C0
0000000000005E41 00000820 00000000000007DB FFFFFA8003641820 FFFFF6FD4001B208
0000000000005E54 00000C90 00000000000005DB FFFFFA8003654C90 FFFFF6FD4001B2A0
0000000000005E63 000005D0 00000000000007CB FFFFFA80036635D0 FFFFF6FD4001B318
0000000000005E67 00000680 00000000000005DB FFFFFA8003667680 FFFFF6FD4001B338
0000000000005E73 000009E0 000000000000075B FFFFFA80036739E0 FFFFF6FD4001B398
0000000000005E7E 000007E0 00000000000007CB FFFFFA800367E7E0 FFFFF6FD4001B3F0
0000000000005E82 00000C20 00000000000007DF FFFFFA8003682C20 FFFFF6FD4001B410
0000000000005E84 00000360 00000000000007D9 FFFFFA8003684360 FFFFF6FD4001B420
0000000000005E84 00000FC0 00000000000007DB FFFFFA8003684FC0 FFFFF6FD4001B420
0000000000005E85 000002C0 00000000000006DB FFFFFA80036852C0 FFFFF6FD4001B428
0000000000005E8C 00000C70 00000000000007FB FFFFFA800368CC70 FFFFF6FD4001B460
0000000000005E8C 00000F40 00000000000007D3 FFFFFA800368CF40 FFFFF6FD4001B460
0000000000005E90 00000F90 00000000000007DA FFFFFA8003690F90 FFFFF6FD4001B480
0000000000005E91 00000530 00000000000003DB FFFFFA8003691530 FFFFF6FD4001B488
0000000000005E9B 00000190 000000000000079B FFFFFA800369B190 FFFFF6FD4001B4D8
0000000000005E9B 000003A0 000000000000075B FFFFFA800369B3A0 FFFFF6FD4001B4D8
0000000000005E9E 00000460 00000000000007DA FFFFFA800369E460 FFFFF6FD4001B4F0
0000000000005EA7 00000E50 00000000000003DB FFFFFA80036A7E50 FFFFF6FD4001B538
0000000000005EAF 00000080 00000000000005DB FFFFFA80036AF080 FFFFF6FD4001B578
0000000000005EB9 00000250 00000000000007DA FFFFFA80036B9250 FFFFF6FD4001B5C8
0000000000005EBB 000005C0 00000000000003DB FFFFFA80036BB5C0 FFFFF6FD4001B5D8
0000000000005EC4 00000770 00000000000003DB FFFFFA80036C4770 FFFFF6FD4001B620
0000000000005EC5 00000DC0 00000000000007D3 FFFFFA80036C5DC0 FFFFF6FD4001B628
0000000000005ECA 00000E60 00000000000003DB FFFFFA80036CAE60 FFFFF6FD4001B650
0000000000005ECC 000003C0 000000000000075B FFFFFA80036CC3C0 FFFFF6FD4001B660
0000000000005ECD 00000C20 00000000000007DA FFFFFA80036CDC20 FFFFF6FD4001B668
0000000000005EE9 000004F0 00000000000003DB FFFFFA80036E94F0 FFFFF6FD4001B748
0000000000005EF2 000007C0 00000000000006DB FFFFFA80036F27C0 FFFFF6FD4001B790
0000000000005F05 00000E10 00000000000007CB FFFFFA8003705E10 FFFFF6FD4001B828
0000000000005F07 00000A60 00000000000007D9 FFFFFA8003707A60 FFFFF6FD4001B838
0000000000005F08 00000150 00000000000007FB FFFFFA8003708150 FFFFF6FD4001B840
0000000000005F09 00000710 00000000000005DB FFFFFA8003709710 FFFFF6FD4001B848
0000000000005F17 00000600 00000000000005DB FFFFFA8003717600 FFFFF6FD4001B8B8
0000000000005F1D 00000990 00000000000007DF FFFFFA800371D990 FFFFF6FD4001B8E8
0000000000005F2C 00000E40 00000000000007DA FFFFFA800372CE40 FFFFF6FD4001B960
0000000000005F2D 000001D0 00000000000007DF FFFFFA800372D1D0 FFFFF6FD4001B968
0000000000005F2F 00000EA0 00000000000007D9 FFFFFA800372FEA0 FFFFF6FD4001B978
0000000000005F3F 000008F0 00000000000003DB FFFFFA800373F8F0 FFFFF6FD4001B9F8
0000000000005F57 00000860 00000000000006DB FFFFFA8003757860 FFFFF6FD4001BAB8
0000000000005F59 000001B0 00000000000007CB FFFFFA80037591B0 FFFFF6FD4001BAC8
0000000000005F59 00000A50 00000000000005DB FFFFFA8003759A50 FFFFF6FD4001BAC8
0000000000005F60 00000620 00000000000003DB FFFFFA8003760620 FFFFF6FD4001BB00
0000000000005F63 00000AD0 000000000000075B FFFFFA8003763AD0 FFFFF6FD4001BB18
0000000000005F70 00000790 00000000000007DF FFFFFA8003770790 FFFFF6FD4001BB80
0000000000005F77 000001E0 00000000000007CB FFFFFA80037771E0 FFFFF6FD4001BBB8
0000000000005F80 00000B70 000000000000079B FFFFFA8003780B70 FFFFF6FD4001BC00
0000000000005F97 00000070 00000000000007D3 FFFFFA8003797070 FFFFF6FD4001BCB8
0000000000005FDF 00000550 00000000000007CB FFFFFA80037DF550 FFFFF6FD4001BEF8
0000000000005FF4 000000D0 00000000000007D9 FFFFFA80037F40D0 FFFFF6FD4001BFA0
0000000000006008 00000410 00000000000017DB FFFFFA8003808410 FFFFF6FD4001C040
0000000000006035 00000BF0 00000000000027DB FFFFFA8003835BF0 FFFFF6FD4001C1A8
0000000000006099 00000C10 00000000000047DB FFFFFA8003899C10 FFFFF6FD4001C4C8
00000000000060B8 00000518 0000000000000FDB FFFFFA80038B8518 FFFFF6FD4001C5C0
000000000000613B 00000918 00000000000006DB FFFFFA800393B918 FFFFF6FD4001C9D8
0000000000006153 00000798 00000000000007D3 FFFFFA8003953798 FFFFF6FD4001CA98
000000000000617B 00000F30 0000000000000FDB FFFFFA800397BF30 FFFFF6FD4001CBD8
0000000000006192 00000580 00000000000007DA FFFFFA8003992580 FFFFF6FD4001CC90
0000000000006193 00000180 000000000000079B FFFFFA8003993180 FFFFF6FD4001CC98
0000000000006193 00000D50 00000000000007D9 FFFFFA8003993D50 FFFFF6FD4001CC98
0000000000006194 000009B0 00000000000007DB FFFFFA80039949B0 FFFFF6FD4001CCA0
0000000000006195 00000550 000000000000075B FFFFFA8003995550 FFFFF6FD4001CCA8
0000000000006195 00000670 00000000000007DF FFFFFA8003995670 FFFFF6FD4001CCA8
0000000000006196 00000D50 00000000000006DB FFFFFA8003996D50 FFFFF6FD4001CCB0
0000000000006197 000003B0 00000000000007FB FFFFFA80039973B0 FFFFF6FD4001CCB8
000000000000619A 00000830 00000000000007D3 FFFFFA800399A830 FFFFF6FD4001CCD0
000000000000619B 000002B0 00000000000007CB FFFFFA800399B2B0 FFFFF6FD4001CCD8
00000000000061A9 000003E0 00000000000005DB FFFFFA80039A93E0 FFFFF6FD4001CD48
00000000000061A9 00000D70 00000000000003DB FFFFFA80039A9D70 FFFFF6FD4001CD48
00000000000061B7 00000090 000000000000079B FFFFFA80039B7090 FFFFF6FD4001CDB8
00000000000061BF 00000610 00000000000003DB FFFFFA80039BF610 FFFFF6FD4001CDF8
0000000000007D32 00000EC8 00000000000003DB 000000007790BEC8 FFFFF680003BC858
0000000000009513 00000EC8 00000000000003DB 000000007790BEC8 FFFFF680003BC858
000000000000C631 00000EC8 00000000000003DB 000000007790BEC8 FFFFF680003BC858
000000000000EA70 00000E08 00000000000003DB 0000000000000000 FFFFF8A0099382D8
000000000000EBA3 00000AB0 000000000000075B 0000000000000000 FFFFF8A009938E70
000000000000ED2A 00000E20 00000000000007D9 0000000000000000 FFFFF8A009938EA8
000000000000F4B6 00000BB8 00000000000007D9 0000000000000000 FFFFF8A009938F08
000000000000FCB6 000000C0 00000000020007DB 0000000000000000 FFFFF8A009938D08
000000000000FCE6 00000CF0 00000000000005DB 0000000000000000 FFFFF8A009938E88
000000000000FDE6 00000D90 00000000000007D9 000000000CB3CD90 FFFFF680000659E0
Search done.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Easy to remember abbreviation SCP (Software Crash Patterns) consists of 3 practices:

- Scripts 

- Checklists 

- Patterns

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In these post series we are going to discuss the best practices for software tracing implementation including appropriate patterns and their links to software trace analysis patterns. The first one is called Period Timestamp where the start and the end time (and the date if necessary) are recorded in the trace file. This helps in Inter-Correlation and News Value analysis between several different trace types. For example, in one scenario, we had WindowHistory and MessageHistory logs. We identified a problem in the former log as happening at this time:

Handle: 00010196 Class: "ClassA" Title: "TitleA"
   Captured at: 13:36:30:533
[…]

However, when we looked at the latter trace to search for specific window messages posted or sent before that time we saw that the recording started later than the former event:

Start time: 13:36:35:830

Period timestamps are necessary to distinguish Incomplete History from Truncated Trace where in the former case the absence of expected trace message is due to some problem.

From a unified debugging patterns perspective we have this sequence fragment:

Implementation Patterns: Period Timestamp

Usage Patterns: Trace Simultaneously

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Paraphrasing the title of Philip K. Dick’s novel Do Androids Dream of Electric Sheep? I’d like to tell the dream I had a few nights ago after starting my work on a computational security novel Session Zero (ISBN: 978-1908043092). I was in the hall of a building where Russian and American spies frequently intersect. Apparently, I was a technician there and everyone passing by was complaining about difficulties being a spy. I was listening and telling everyone that I wasn’t a spy, just a technician. Suddenly a thought came to my mind that if I were a real spy masquerading as a technician it would have made make a good novel plot… Then I met colleagues from one of my previous companies and I woke up.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

… the functions you meet on the way up, … you’re going to meet the same functions on the way down. Not always…

Jackie Gleason

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As a part of my efforts to unify malware and forensic analysis with memory dump and software trace analysis from behavioral and structural patterns perspective I created this domain name. The word victimware was borrowed and extended from its previous limited use:

Software Victimology (Part 1)
Software Victimology (Part 2)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

News Value is a pattern that assigns relative importance to software traces for problem solving purposes especially when related to problem description, recent incidents and timestamps of other supporting artifacts (memory dumps, other traces, etc.). For example, in one scenario, an ETW trace was provided with 3 additional log files:

# Source PID  TID  Date       Time         Message 
0 Header 1260 1728 12/14/2010 06:48:56.289 ?????  
[…] 
215301 Unknown 640 808 12/14/2010 07:22:57.508 ?????  Unknown( 16): GUID=[…] (No Format Information found). 

// LogA
05/11/10 18:28:15.1562 : Service() - entry
[...]
14/12/10 10:31:58.0381 : Notification: sleep
* Start of new log *
14/12/10 10:34:38.4687 : Service() - entry
[…]
14/12/10 11:53:35.2729 : Service.CleanUp complete
* Start of new log *
14/12/10 11:56:11.7031 : Service() - entry
[…]
14/12/10 15:25:23.3004 : Notification: sleep

// LogB
[   1] 12/14 10:34:29:890   Entry: ctor
[…]
[   2] 12/14 11:53:30:866   Exit: COMServer.Server.DeleteObject

// LogC
[   1] 12/14 11:56:03:359   Entry: ctor
[…]
[  20] 12/14 15:30:20:110   Exit: Kernel32.Buffer.Release

From the description of the problem we expected LogB and LogC to be logs from two subsequent process executions where the first launch fails (LogB) and the second launch succeeds (LogC). Looking at their start and end times we see that they make sense from the problem description perspective but we have to dismiss ETW trace and most of LogA as recorded earlier and having no value for Inter-Correlation analysis of the more recent logs. We also see that portions of LogA overlap with LogB and LogC and therefore having analysis value for us.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Borrowing routine activity theory (RAT) from criminology I would like to introduce the similar approach to abnormal software behavior with patterning activities that adds additional unmotivated offenders to combine malware (software rats) with unintentional ordinary common bugware:

The application of RAT to software can be metaphorically named as Function Activity Theory (FAT).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The following pattern is useful for inconsistent dumps or incomplete supporting information: Environment Hint. It is mostly environment variable information for troubleshooting suggestions such as product elimination for testing purposes and / or necessary upgrade, for example:

0: kd> !peb
PEB at 7ffd7000
    InheritedAddressSpace:    No
    ReadImageFileExecOptions: Yes
    BeingDebugged:            No
    ImageBaseAddress:         01000000
    Ldr                       7c8897e0
    Ldr.Initialized:          Yes
    Ldr.InInitializationOrderModuleList: 00081f18 . 000f9e88
    Ldr.InLoadOrderModuleList:           00081eb0 . 000f9e78
    Ldr.InMemoryOrderModuleList:         00081eb8 . 000f9e80
            Base TimeStamp                     Module
         1000000 45d6a03c Feb 17 06:27:08 2007 C:\WINNT\system32\svchost.exe
        7c800000 49900d60 Feb 09 11:02:56 2009 C:\WINNT\system32\ntdll.dll
[...]
    SubSystemData:     00000000
    ProcessHeap:       00080000
    ProcessParameters: 00020000
    WindowTitle:  'C:\WINNT\system32\svchost.exe'
    ImageFile:    'C:\WINNT\system32\svchost.exe'
    CommandLine:  'C:\WINNT\system32\svchost.exe -k rpcss'
    DllPath:      [...]
    Environment:  00010000
        ALLUSERSPROFILE=C:\Documents and Settings\All Users
[...]
        PROTECTIONDIR=C:\Documents and Settings\All Users\Application Data\3rdPartyAntivirus\Protection
[…] 
       Path= […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m very excited to announce that 2 my tools Repair Clipboard Chain 2.0.1, StressPrinters 1.3.2 for 32-bit and 64-bit Platforms and Selected Citrix Troubleshooting Tools presentation are in top 3 list:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Golden bug - A software defect if fixed entails a promotion and pay rise prospects.

Examples:

- You are a Principal now!!!
- Oh, thanks to that persistent golden bug nobody could fix until I was assigned…

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -