Software Diagnostics Library

This is an example of a complex bugtation:

Bugteriology is the study of bugteria. “It comprises the identification, classification and characterization of” bugterial “species.” Bugteria “are identified by their properties, for example their looks, what” memory dumps “they can” appear in “or not” appear in, “what” bugs “they require for growth, what” effects “they produce, etc. To study morphology, that is the” phenotype “of” bugteria, “a” debugger “is used.”

Virtual Museum of Bacteria, Bacteriology: the study of bacteria

- Dmitry Vostokov @ DumpAnalysis.org -

I continued thinking about bugteria in memory dumps all the day yesterday and came to the conclusion that the study of crash dump analysis patterns needs its own name and the obvious choice was Bugteriology:

Bugteriology is the study of crash dump analysis patterns (bugteria). Its main subject is identification, classification and characterization of such patterns found in memory dumps (bugterial species).

I initially registered a domain for this purpose (later abandoned) pointing to crash dump analysis and debugging portal but I want to think through this idea and perhaps make it a subdomain of dumpanalysis.org with a page for easy online pattern classification and make it also an online supplement to forthcoming encyclopedia of crash dump analysis patterns.

- Dmitry Vostokov @ DumpAnalysis.org -

Yesterday was one of those days when I was in a good mood thinking about bugs. Suddenly a thought stroke me about the similar sounding words bacterium and bugterium (perhaps because I’m currently reading a theoretical biology book, Essays on Life Itself). I admit that it might be sounding the same only for a non-native English ear though. So the new definition was born:

Bugterium (pl. bugteria) - an instance of a memory dump analysis pattern found in a crash (memory, core) dump file.

Why a bugterium and not a cdarium? The motivation (with a hindsight) lies in the complexity of debugging (and life forms). While a bug is a complex thing (and a beast) and it takes sometimes days or weeks to chase and fix (kill) the one, a bugterium (bacterium) is of relatively smaller complexity and can be easily identified and dealt with by component removal or upgrade (massively killed). From software support perspective remember this bugtation No.14:

Crash dump analysis ”is anticipated with” joy, “performed with” eagerness, “and bragged about forever.”

Although the perceived simplicity of crash dump analysis is deceptive (bugtation No.2):

“It requires a very unusual mind to undertake the analysis of the obvious” crash.

Alfred North Whitehead, Science and the Modern World

- Dmitry Vostokov @ DumpAnalysis.org -

Now it is the turn for the yet another pattern of a deadlock variety involving mixed objects in kernel space. Previously we discussed deadlock patterns involving critical sections in user space, executive resources in kernel space, mixed objects in user space and LPC.

Let’s look at a complete manual dump file from a hanging system:

0: kd> !analyze -v

NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction.  The hardware supplier should
be called.
Arguments:
Arg1: 004f4454
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Here we have problems reading all executive resource locks:

3: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks....

Resource @ nt!CmpRegistryLock (0x808a48c0)    Shared 36 owning threads
    Contention Count = 48
     Threads: 86aecae0-01<*> 8b76db40-01<*> 8b76ddb0-01<*> 89773020-01<*>
              87222db0-01<*> 87024ba8-01<*> 89a324f0-01<*> 86b4e298-01<*>
              87925b40-01<*> 86b4db40-01<*> 8701f738-01<*> 86ffb198-01<*>
              86b492f0-01<*> 8701bad8-01<*> 86ae2db0-01<*> 86c85db0-01<*>
              86a9ddb0-01<*> 86a86db0-01<*> 86aa7db0-01<*> 86a9f5c0-01<*>
              86c5adb0-01<*> 8767ba38-01<*> 86afedb0-01<*> 89877960-01<*>
              8772cdb0-01<*> 87348628-01<*> 874d6748-01<*> 872365e0-01<*>
              87263970-01<*> 873bf020-01<*> 86c13db0-01<*> 893dcdb0-01<*>
              86afa020-01<*> 878e5020-01<*> 874959f8-01<*> 86b2dc70-01<*>
KD: Scanning for held locks…Error 1 in reading nt!_ERESOURCE.SystemResourcesList.Flink @ f76ee2a0

This is probably because the dump was truncated:

Loading Dump File [MEMORY.DMP]
Kernel Complete Dump File: Full address space is available

WARNING: Dump file has been truncated.  Data may be missing.

However looking at the resource 808a48c0 closely we see that it is owned by the thread 86aecae0 (Cid 2810.2910) which is blocked on a mutant owned by the thread 86dcf3a8:

3: kd> !locks -v 0x808a48c0

Resource @ nt!CmpRegistryLock (0x808a48c0)    Shared 36 owning threads
    Contention Count = 48
     Threads: 86aecae0-01<*>

     THREAD 86aecae0  Cid 2810.2910  Teb: 7ffdd000 Win32Thread: bc54ab88 WAIT: (Unknown) KernelMode Non-Alertable
         86dda264  Mutant - owning thread 86dcf3a8
     Not impersonating
     DeviceMap                 da534618
     Owning Process            86f30b70       Image:         ApplicationA.exe
     Wait Start TickCount      1074481        Ticks: 51601 (0:00:13:26.265)
     Context Switch Count      9860                 LargeStack
     UserTime                  00:00:01.125
     KernelTime                00:00:00.890
     Win32 Start Address 0×300019f0
     Start Address kernel32!BaseProcessStartThunk (0×7c8217f8)
     Stack Init b5342000 Current b5341150 Base b5342000 Limit b533d000 Call 0
     Priority 12 BasePriority 10 PriorityDecrement 0
     ChildEBP RetAddr 
     b5341168 80833465 nt!KiSwapContext+0×26
     b5341194 80829a62 nt!KiSwapThread+0×2e5
     b53411dc b91f4c08 nt!KeWaitForSingleObject+0×346
WARNING: Stack unwind information not available. Following frames may be wrong.
     b5341200 b91ee770 driverA+0xec08
     b5341658 b91e9ca7 driverA+0×8770
     b5341af0 8088978c driverA+0×3ca7
     b5341af0 8082f829 nt!KiFastCallEntry+0xfc
     b5341b7c 808ce716 nt!ZwSetInformationFile+0×11
     b5341bbc 808dd8d8 nt!CmpDoFileSetSize+0×5e
     b5341bd4 808bd798 nt!CmpFileSetSize+0×16
     b5341bf4 808be23f nt!HvpGrowLog1+0×52
     b5341c18 808bfc6b nt!HvMarkDirty+0×453
     b5341c40 808c3fd4 nt!HvMarkCellDirty+0×255
     b5341cb4 808b7e2f nt!CmSetValueKey+0×390
     b5341d44 8088978c nt!NtSetValueKey+0×241
     b5341d44 7c9485ec nt!KiFastCallEntry+0xfc
     0013f5fc 00000000 ntdll!KiFastSystemCallRet

8b76db40-01<*>

     THREAD 8b76db40  Cid 0004.00c8  Teb: 00000000 Win32Thread: 00000000 GATEWAIT
     Not impersonating
     DeviceMap                 d6600900
     Owning Process            8b7772a8       Image:         System
     Wait Start TickCount      1074667        Ticks: 51415 (0:00:13:23.359)
     Context Switch Count      65106            
     UserTime                  00:00:00.000
     KernelTime                00:00:00.781
     Start Address nt!ExpWorkerThread (0x80880352)
     Stack Init bae35000 Current bae34c68 Base bae35000 Limit bae32000 Call 0
     Priority 12 BasePriority 12 PriorityDecrement 0
     ChildEBP RetAddr 
     bae34c80 80833465 nt!KiSwapContext+0x26
     bae34cac 8082ffc0 nt!KiSwapThread+0x2e5
     bae34cd4 8087d6f6 nt!KeWaitForGate+0x152
     dbba6d78 00000000 nt!ExfAcquirePushLockExclusive+0x112

[...]

A reminder about Cid. It is the so called Client id composed from Process id and Thread id (Pid.Tid). Also a mutant is just another name for a mutex object which has an ownership semantics:

0: kd> dt _KMUTANT 86dda264
nt!_KMUTANT
   +0x000 Header           : _DISPATCHER_HEADER
   +0x010 MutantListEntry  : _LIST_ENTRY [ 0x86dcf3a8 - 0x86dcf3a8 ]
   +0×018 OwnerThread      : 86dcf3a8 _KTHREAD
   +0×01c Abandoned        : 0 ”
   +0×01d ApcDisable       : 0×1 ”

Now we look at that thread 86dcf3a8 and see that it belongs to ApplicationB (Cid 25a0.14b8):

3: kd> !thread 86dcf3a8
THREAD 86dcf3a8  Cid 25a0.14b8  Teb: 7ffa9000 Win32Thread: bc3e0d20 WAIT: (Unknown) UserMode Non-Alertable
    8708b888  Thread
    86dcf420  NotificationTimer
Not impersonating
DeviceMap                 da534618
Owning Process            87272d88       Image:         ApplicationB.exe
Wait Start TickCount      1126054        Ticks: 28 (0:00:00:00.437)
Context Switch Count      2291                 LargeStack
UserTime                  00:00:00.078
KernelTime                00:00:00.218
Win32 Start Address msvcrt!_endthreadex (0×77b9b4bc)
Start Address kernel32!BaseThreadStartThunk (0×7c8217ec)
Stack Init b550a000 Current b5509c60 Base b550a000 Limit b5507000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
b5509c78 80833465 86dcf3a8 86dcf450 00000003 nt!KiSwapContext+0×26
b5509ca4 80829a62 00000000 b5509d14 00000000 nt!KiSwapThread+0×2e5
b5509cec 80938d0c 8708b888 00000006 00000001 nt!KeWaitForSingleObject+0×346
b5509d50 8088978c 00000960 00000000 b5509d14 nt!NtWaitForSingleObject+0×9a
b5509d50 7c9485ec 00000960 00000000 b5509d14 nt!KiFastCallEntry+0xfc
WARNING: Stack unwind information not available. Following frames may be wrong.
0454f3cc 00000000 00000000 00000000 00000000 ntdll!KiFastSystemCallRet

We see that it is waiting on 8708b888 object which is a thread itself and it is waiting on the same mutant 86dda264 owned by the thread 86dcf3a8 (Cid 25a0.14b8):

3: kd> !thread 8708b888
THREAD 8708b888  Cid 25a0.1cb0  Teb: 7ffa6000 Win32Thread: bc3ecb20 WAIT: (Unknown) KernelMode Non-Alertable
    86dda264  Mutant - owning thread 86dcf3a8
Not impersonating
DeviceMap                 da534618
Owning Process            87272d88       Image:         ApplicationB.exe
Wait Start TickCount      1070470        Ticks: 55612 (0:00:14:28.937)
Context Switch Count      11                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address dll!_beginthread (0×1b1122a9)
Start Address kernel32!BaseThreadStartThunk (0×7c8217ec)
Stack Init b4d12000 Current b4d117fc Base b4d12000 Limit b4d0f000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
b4d11814 80833465 8708b888 8708b930 00000003 nt!KiSwapContext+0×26
b4d11840 80829a62 0000096c b4d118c4 b91e8f08 nt!KiSwapThread+0×2e5
b4d11888 b91f4c08 86dda264 00000006 00000000 nt!KeWaitForSingleObject+0×346
WARNING: Stack unwind information not available. Following frames may be wrong.
b4d118ac b91ee818 86dda260 b4d11d64 86dda000 DriverA+0xec08
b4d11d04 b91e8f58 000025a0 0000096c b4d11d64 DriverA+0×8818
b4d11d58 8088978c 0000096c 0567f974 7c9485ec DriverA+0×2f58
b4d11d58 7c9485ec 0000096c 0567f974 7c9485ec nt!KiFastCallEntry+0xfc
0567f974 30cba6ad 0000096c 00000000 00000003 ntdll!KiFastSystemCallRet

We can summarize our findings on the following wait chain diagram:

Looking from the component-object relationship perspective it is DriverA.sys that is waiting on the mutant 86dda264 although both blocked threads B and C belong to ApplicationB process.

- Dmitry Vostokov @ DumpAnalysis.org -

[Software] Defects “have a character of their own, but they also partake of” a program “character;” programs “have a character of” their “own, but” they “also partake of the world’s character.”

Oliver Wolf Sacks, Awakenings

- Dmitry Vostokov @ DumpAnalysis.org -

“It takes a wise” engineer “to know when not to” debug.

Baltasar Gracián, The Art of Worldly Wisdom

- Dmitry Vostokov @ DumpAnalysis.org -

Just realized that yesterday I wrote 100^th crash dump analysis pattern. Today I’m going to write 101^st. Just to remind that the fully classified color catalog of them is planned to be published:

Forthcoming CDAP Encyclopedia

More details will be announced closer to that date.

- Dmitry Vostokov @ DumpAnalysis.org -

This is a very simple pattern I planned to write about long time ago. It is called C++ Exception. It is similar to Managed Code Exception and can be manifested by the same RaiseException call on top of the stack (red). However it is called by Visual C runtime (I consider Microsoft C/C++ implementation here, msvcrt.dll, magenta). The typical example of it might be checking the validity of a C++ stream operator data format (blue):

STACK_TEXT: 
09d6f264 78007108 KERNEL32!RaiseException+0×56
09d6f2a4 677f2a88 msvcrt!_CxxThrowException+0×34
09d6f2bc 6759afff DLL!MyInputStream::operator>>+0×34

Also, some Visual C++ STL implementations check for out of bounds or invalid parameters and call unhandled exception filter directly, for example:

STACK_TEXT: 
0012d2e8 7c90e9ab ntdll!KiFastSystemCallRet
0012d2ec 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc
0012d388 7c80a075 kernel32!WaitForMultipleObjectsEx+0x12c
0012d3a4 6945763c kernel32!WaitForMultipleObjects+0x18
0012dd38 694582b1 faultrep!StartDWException+0x5df
0012edac 7c8633b1 faultrep!ReportFault+0x533
0012f44c 004409b3 kernel32!UnhandledExceptionFilter+0x587
0012f784 00440a1b Application!_invoke_watson+0xc4
0012f79c 00406f4f Application!_invalid_parameter_noinfo+0xc
0012f7a0 0040566b Application!std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::operator[]+0×12

The latter example also shows how an unhandled exception filter in an application itself calls a postmortem debugger specified by AeDebug registry key (see also Who calls the postmortem debugger? post for detailed explanations).

- Dmitry Vostokov @ DumpAnalysis.org -

Trace “back a little to” debug “further.”

John Clarke (1596-1658), Proverbs: English and Latine

- Dmitry Vostokov @ DumpAnalysis.org -

Paperback edition of Memory Dump Analysis Anthology, Volume 2 is finally available on Amazon and Barnes & Noble. Search Inside is also available on Amazon. In addition, I updated the list of recommended books:

Listmania! Crash Dump Analysis and Debugging

Hardcover edition will be available on Amazon and B&N in 2-3 weeks.

- Dmitry Vostokov @ DumpAnalysis.org -

“The moment you have worked out” a code fix , “start checking it - it probably isn’t right.”

Edmund Callis Berkeley, Computers and Automation magazine, Right Answers: A Short Guide for Obtaining Them

- Dmitry Vostokov @ DumpAnalysis.org -

This one is called No Question and is summarized by the following dialog that I observed many times:

As an engineer proceeds with analysis of a problem a phone rings and an irritated customer (internal or external) asks “What is the answer?”, an engineer glancing over the problem description replies, “What is the question?”.

This immediately came to my mind when I found this famous dialog:

“As the poet Gertrude Stein lay on her deathbed, her partner, Alice B. Toklas, leaned over and whispered, ‘What is the answer, Gertrude?’. Replied Stein, ‘What’s the question?’”

- Dmitry Vostokov @ DumpAnalysis.org -

Before the advent of virtual machines to Wintel platforms people had to take pictures of bluescreens or even write all data down by hand to fax it:

How easy it is today! You just do Print Screen.

- Dmitry Vostokov @ DumpAnalysis.org -

The following bugtation is quite wise and dedicated to beginners learning WinDbg (see Common Mistakes and Coincidental Symbolic Information for some examples).

“You rule the” debugger, “not the” debugger “you”.

John Dryden, The Hind and the Panther

- Dmitry Vostokov @ DumpAnalysis.org -

Almost 2 years have passed since I wrote the first post about crash dump analysis patterns: Multiple Exceptions. Today I write about multiple exceptions or faults in kernel mode. Here I distinguish multiple exceptions from nested exceptions. The latter ones in kernel result in double faults, see for example, Stack Overflow pattern. At the first glance it looked like the dump was saved manually:

0: kd> !analyze -v
[...]
MANUALLY_INITIATED_CRASH (e2)
The user manually initiated this crash dump.
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000
[...]

However further down in analysis report there was the presence of page fault:

TRAP_FRAME:  a38df520 -- (.trap 0xffffffffa38df520)
ErrCode = 00000002
eax=b6d9220f ebx=b6ab4ffb ecx=00000304 edx=eaf2fdea esi=b6d9214c edi=b6ab8189
eip=bfa10e6e esp=a38df594 ebp=a38df5ac iopl=0  nv up ei ng nz ac po cy
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000  efl=00010293
driver+0x2ae6e:
bfa10e6e 895304   mov     dword ptr [ebx+4],edx ds:0023:b6ab4fff=????????
Resetting default scope

STACK_TEXT: 
a38df410 b48aa532 000000e2 00000000 00000000 nt!KeBugCheckEx+0x1b
a38df440 b48a9d2c 000eba28 9282c8c6 00000000 i8042prt!I8xProcessCrashDump+0x256
a38df488 80839595 89d0c008 8a0eb970 0001000a i8042prt!I8042KeyboardInterruptService+0x225
a38df488 80836bfa 89d0c008 8a0eb970 0001000a nt!KiInterruptDispatch+0x49
a38df520 bfa10e6e badb0d00 eaf2fdea 8867cbe8 nt!KiTrap0E+0xbc
WARNING: Stack unwind information not available. Following frames may be wrong.
a38df5ac bfa22461 b6ab423b 000003dc 00000007 driver+0×2ae6e
[…]

Looking at b6ab4fff address shows that it crosses page boundary, see Data Alignment pattern.

We also see that this thread was running and consumed too much kernel time, see Spiking Thread pattern:

0: kd> !thread
THREAD 88e686d8  Cid 1e48.1f7c  Teb: 7ffdf000 Win32Thread: b669de70 RUNNING on processor 0
Not impersonating
DeviceMap                 dc971120
Owning Process            889e0d88       Image:         ProcessA.EXE
Wait Start TickCount      9231345        Ticks: 0
Context Switch Count      2196221                 LargeStack
UserTime                  00:00:35.562
KernelTime                04:51:23.656
[…]

Intrigued, we see another running thread on the second processor:

0: kd> !running
     Prcb      Current   Next   
  0  ffdff120  88e686d8            ................
  1  f7727120  88bd33f8            …………….

0: kd> !thread 88bd33f8
THREAD 88bd33f8  Cid 2fdc.27f0  Teb: 7ffdf000 Win32Thread: b6640ab8 RUNNING on processor 1
Not impersonating
DeviceMap                 d7a13b40
Owning Process            89e45200       Image:         ProcessA.EXE
Wait Start TickCount      9231345        Ticks: 0
Context Switch Count      2324364                 LargeStack
UserTime                  00:00:21.171
KernelTime                05:02:09.500
Win32 Start Address ProcessA (0×30001e28)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ac5e7bd0 Current ac5e7078 Base ac5e8000 Limit ac5e1000 Call ac5e7bd8
Priority 6 BasePriority 6 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
ac5e7150 bfa10e6e badb0d00 dbeaffdb 89a793d8 nt!KiTrap0E+0xbc (FPO: [0,0] TrapFrame @ ac5e7150)
WARNING: Stack unwind information not available. Following frames may be wrong.
ac5e71dc bfa22461 b701f15f ffffff24 00000007 driver+0×2ae6e
[…]

We see it is spiking CPU too and we detect a possible loop in page fault handler:

0: kd> .thread 88bd33f8
Implicit thread is now 88bd33f8

0: kd> ~1s

1: kd> r
eax=fffff81c ebx=ac5e71dc ecx=88bd33f8 edx=dbeaffdb esi=b6f81168 edi=b701fffe
eip=80836bfa esp=ac5e7150 ebp=ac5e7150 iopl=0 nv up ei pl nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000  efl=00000206
nt!KiTrap0E+0xbc:
80836bfa 0f84e5010000  je  nt!KiTrap0E+0×2a7 (80836de5) [br=0]

When looking at raw stack we see that the loop happened after processing this exception:

1: kd> .trap ac5e7150
ErrCode = 00000002
eax=b6f8122b ebx=b701fffa ecx=fffffe4c edx=dbeaffdb esi=b6f81168 edi=b70201a0
eip=bfa10e6e esp=ac5e71c4 ebp=ac5e71dc iopl=0 nv up ei ng nz ac po cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010293
driver+0×2ae6e:
bfa10e6e 895304 mov dword ptr [ebx+4],edx ds:0023:b701fffe=????????

The address crosses page boundary too:

1: kd> !pte b701fffe
               VA b701fffe
PDE at   C0300B70        PTE at C02DC07C
contains 642CF863      contains 2F336863
pfn 642cf ---DA--KWEV    pfn 2f336 ---DA--KWEV

1: kd> !pte b701fffe+3
               VA b7020001
PDE at   C0300B70        PTE at C02DC080
contains 642CF863      contains 00000080
pfn 642cf ---DA--KWEV                           not valid
                       DemandZero
                       Protect: 4 - ReadWrite
 
This is because trap processing code is found below the current ESP value and also 3rd-party virtual block drivers which I guess were trying to satisfy page fault (the latter not shown in the raw stack fragment here):

1: kd> dds esp-1000 esp
[...]
ac5e6f78  00000002
ac5e6f7c  899c05b0
ac5e6f80  88bd33f8
ac5e6f84  00000010
ac5e6f88  ac5e702c
ac5e6f8c  808457ff nt!KeContextFromKframes+0x9b
ac5e6f90  00000023
ac5e6f94  f7727120
ac5e6f98  00000000
ac5e6f9c  808458fd nt!KeContextFromKframes+0x2bc
ac5e6fa0  ac5e70dc
ac5e6fa4  1f840a42
ac5e6fa8  00000000
ac5e6fac  f7727000
ac5e6fb0  00000000
ac5e6fb4  f7727a7c
ac5e6fb8  ac5e6fd4
ac5e6fbc  808398d4 nt!KiDispatchInterrupt+0xd8
ac5e6fc0  00000000
ac5e6fc4  80a801ae hal!HalpDispatchSoftwareInterrupt+0x5e
ac5e6fc8  ac5e700c
ac5e6fcc  ac5e7001
ac5e6fd0  00000002
ac5e6fd4  ac5e6ff0
ac5e6fd8  80a80397 hal!HalpCheckForSoftwareInterrupt+0x3f
ac5e6fdc  00000002
ac5e6fe0  ac5e700c
ac5e6fe4  ac5e700c
ac5e6fe8  ac5e70b0
ac5e6fec  00000001
ac5e6ff0  f772f120
ac5e6ff4  88bd33f8
ac5e6ff8  00000002
ac5e6ffc  ac5e700c
ac5e7000  8a0a88a0
ac5e7004  88bd33f8
ac5e7008  f7727002
ac5e700c  80a8057e hal!HalEndSystemInterrupt+0x6e
ac5e7010  88bd33f8
ac5e7014  f7727002
ac5e7018  00000002
ac5e701c  ac5e702c
ac5e7020  80a80456 hal!KfLowerIrql+0x62
ac5e7024  f7727000
ac5e7028  0000bb40
ac5e702c  ac5e70ac
ac5e7030  808093eb nt!KiSaveProcessorState+0x20
ac5e7034  ac5e70dc
ac5e7038  00000000
ac5e703c  808093f0 nt!KiSaveProcessorState+0x25
ac5e7040  f772713c
ac5e7044  8087dcbd nt!KiFreezeTargetExecution+0x6a
ac5e7048  ac5e70dc
ac5e704c  00000000
ac5e7050  f7727120
ac5e7054  00000000
ac5e7058  80a7e501 hal!KeAcquireQueuedSpinLockRaiseToSynch+0x21
ac5e705c  88bd3401
ac5e7060  ac5e7070
ac5e7064  80a80456 hal!KfLowerIrql+0x62
ac5e7068  80a7e530 hal!KeReleaseInStackQueuedSpinLock
ac5e706c  88bd3401
ac5e7070  ac5e70b0
ac5e7074  80a7e56d hal!KeReleaseQueuedSpinLock+0x2d
ac5e7078  80823822 nt!KiDeliverApc+0x1cc
ac5e707c  00000000
ac5e7080  ac806e00
ac5e7084  00000200
ac5e7088  00000000
ac5e708c  88bd343c
ac5e7090  00000001
ac5e7094  ac5e7934
ac5e7098  89e45200
ac5e709c  809282c8 nt!CmpPostApc
ac5e70a0  00000000
ac5e70a4  0000010c
ac5e70a8  1d01f008
ac5e70ac  ac5e70dc
ac5e70b0  80837c86 nt!KiIpiServiceRoutine+0x8b
ac5e70b4  ac5e70dc
ac5e70b8  00000000
ac5e70bc  80836bfa nt!KiTrap0E+0xbc
ac5e70c0  b6f81168
ac5e70c4  ac5e7150
ac5e70c8  80a7d8fc hal!HalpIpiHandler+0xcc
ac5e70cc  ac5e70dc
ac5e70d0  00000000
ac5e70d4  80a80300 hal!HalpLowerIrqlHardwareInterrupts+0x10c
ac5e70d8  000000e1
ac5e70dc  ac5e7150
ac5e70e0  80836bfa nt!KiTrap0E+0xbc
ac5e70e4  badb0d00
ac5e70e8  dbeaffdb
ac5e70ec  ac5e70fc
ac5e70f0  80a80456 hal!KfLowerIrql+0x62
ac5e70f4  2f336801
ac5e70f8  ac806e00
ac5e70fc  ac5e7138
ac5e7100  8081a2bf nt!MmAccessFault+0x558
ac5e7104  b701fffe
ac5e7108  00000000
ac5e710c  00000000
ac5e7110  00000023
ac5e7114  00000023
ac5e7118  dbeaffdb
ac5e711c  88bd33f8
ac5e7120  fffff81c
ac5e7124  00000000
ac5e7128  ac5e72b0
ac5e712c  00000030
ac5e7130  b701fffe
ac5e7134  b6f81168
ac5e7138  ac5e71dc
ac5e713c  ac5e7150
ac5e7140  00000000
ac5e7144  80836bfa nt!KiTrap0E+0xbc
ac5e7148  00000008
ac5e714c  00000206
ac5e7150  ac5e71dc

What we may guess here is the fact that two page faults happened simultaneously or nearly at the same time and one of them possibly during the attempt to satisfy it and this resulted in two processors looping. The whole system was hang and the usual keyboard method via Scroll Lock was used to generate the manual dump.

- Dmitry Vostokov @ DumpAnalysis.org -

Although complex OS and computer systems are designed to never stop they actually do:

“The only sure thing about” computation “is that it will” stop.

Bret Harte, The Outcasts of Poker Flat, The Luck of Roaring Camp

- Dmitry Vostokov @ DumpAnalysis.org -

I’ve just passed 5 year mark in my Citrix career and I say those years were really exciting! Incidentally at the same time, on 14th of October 2003 I started doing true memory dump analysis in WinDbg-sense of it despite the fact that I’d been doing debugging since 1987 and encountered the first symbolic assembly language debugger on PDP-11 RSX-11M Russian clone in 1988. Also 5 years ago I made my transition from software engineering world to software technical support and started my journey to become a manager.

I’ve put a timeline on a portal accessible from the main page or from the following link:

http://www.dumpanalysis.org/Timeline

The timeline will help me in the future to write my memoirs (Crash Dump: A Software Engineering Autobiography, ISBN: 978-1906717193).

Finally, to record the glorious Citrixofication of the world by various Citrixware, Opentask decided to publish a book where I plan to be an editor of various contributors and probably write a chapter or two myself:

Discovering Citrixware (ISBN: 978-1906717278) 

It is not about troubleshooting but more in a sense of discovery. More details will be announced later.

- Dmitry Vostokov @ DumpAnalysis.org -

“… those who have a natural talent for” troubleshooting “are generally quick at every other kind of knowledge …” debugging “is a kind of knowledge in which the best natures should be trained, and which must not be given up.”

Plato, The Republic

- Dmitry Vostokov @ DumpAnalysis.org -

Here is a bugtation to accompany my post about Debugging Architects:

“His brain, trained by long years of high” design “and plain” architecture, “had become too subtle, too refined an instrument for” debugging.

Stephen Leacock, Literary Lapses

- Dmitry Vostokov @ DumpAnalysis.org -

Seems railroad to it was a success: just got this message in my e-mail:

“Congratulations on passing your recent Microsoft Certification exam, inspiring confidence for your employer, your peers, and yourself with a widely-recognized validation of your skills on Microsoft technology.“

Because I haven’t done any exam since Windows Internals beta I assumed that I passed it and I was right! After registering at Microsoft certification site as MCP I was able to build my logo:

Here is the link to Exam 70-660 information and required skills:

http://www.microsoft.com/learning/en/us/Exams/70-660.aspx

- Dmitry Vostokov @ DumpAnalysis.org -