Software Diagnostics Library

This new pattern is called Blocked Queue and we provide an example of an ALPC port here. If we see an LPC/ALPC wait chain endpoint or just have a message address (and optionally a port address) we can check the port queue length, for example, for a frozen system we have this (WinDbg output was trimmed to save space and paper):

THREAD fffffa8009db7160  Cid 03b0.2ec0  Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa8009db7520  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8a00dbc6650 : queued at port fffffa800577ee60 : owned by process fffffa80056ddb30
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8005691b30       Image:         ServiceA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      39742808       Ticks: 3469954 (0:15:02:11.629)
Context Switch Count      9            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×0000000076cd8e70
Stack Init fffff8800bf60db0 Current fffff8800bf60620
Base fffff8800bf61000 Limit fffff8800bf5b000 Call 0
Priority 10 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`0bf60660 fffff800`016de992 nt!KiSwapContext+0×7a
fffff880`0bf607a0 fffff800`016e0cff nt!KiCommitThreadWait+0×1d2
fffff880`0bf60830 fffff800`016f5d1f nt!KeWaitForSingleObject+0×19f
fffff880`0bf608d0 fffff800`019ddac6 nt!AlpcpSignalAndWait+0×8f
fffff880`0bf60980 fffff800`019dba50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`0bf609e0 fffff800`019d8fcb nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`0bf60b00 fffff800`016d6993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`0bf60bb0 00000000`76d105aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0bf60c20)
00000000`01efe638 000007fe`fec0aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01efe640 000007fe`fecacb64 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`01efe700 000007fe`fecacd55 RPCRT4!NdrpClientCall3+0×244
00000000`01efe9c0 000007fe`fcbf18a1 RPCRT4!NdrClientCall3+0xf2
[…]

0: kd> !alpc /m fffff8a00dbc6650
Message @ fffff8a00dbc6650
  MessageID             : 0x0720 (1824)
  CallbackID            : 0x257C575 (39306613)
  SequenceNumber        : 0x00000002 (2)
  Type                  : LPC_REQUEST
  DataLength            : 0x0044 (68)
  TotalLength           : 0x006C (108)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8006a4bb10 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa8009db7160
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa800577ee60 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa80056ddb30 (ServiceB.exe)
  ServerThread          : fffffa8007ead4d0
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 0000000002a60f40
  ServerContext         : 0000000000000000
  PortContext           : 000000000227a370
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

0: kd> !alpc /p fffffa800577ee60
Port @ fffffa800577ee60
  Type                      : ALPC_CONNECTION_PORT
  CommunicationInfo         : fffff8a0022435d0
    ConnectionPort          : fffffa800577ee60
    ClientCommunicationPort : 0000000000000000
    ServerCommunicationPort : 0000000000000000
  OwnerProcess              : fffffa80056ddb30 (ServiceB.exe)
  SequenceNo                : 0×0000481A (18458)
  CompletionPort            : fffffa8005728e80
  CompletionList            : 0000000000000000
  MessageZone               : 0000000000000000
  ConnectionPending         : No
  ConnectionRefused         : No
  Disconnected              : No
  Closed                    : No
  FlushOnClose              : Yes
  ReturnExtendedInfo        : No
  Waitable                  : No
  Security                  : Static
  Wow64CompletionList       : No

  Main queue is empty.

  Large message queue is empty.

  Pending queue has 698 message(s)

    fffff8a002355aa0 00000404 0000000000001344:0000000000001358 0000000000000000 fffffa8004c0cb60 LPC_REQUEST
    fffff8a00a52f030 00000644 0000000000001078:00000000000024c0 0000000000000000 fffffa80072f1b60 LPC_REQUEST
    fffff8a00abb5030 000007a8 000000000000103c:000000000000050c 0000000000000000 fffffa800725b580 LPC_REQUEST
    fffff8a00239cab0 000000b8 0000000000000480:00000000000015f8 0000000000000000 fffffa80077f0b60 LPC_REQUEST
    fffff8a00ac81a90 00000a18 00000000000028ac:0000000000001e54 0000000000000000 fffffa8007fba060 LPC_CANCELED
    fffff8a005879140 00000f80 0000000000001260:0000000000000730 fffffa8006432060 fffffa8006b18060 LPC_REQUEST
    fffff8a013720d00 00000c6c 0000000000003764:00000000000032a8 0000000000000000 fffffa8006b00a60 LPC_CANCELED
    fffff8a00ac82660 00000810 0000000000003af4:0000000000002a98 0000000000000000 fffffa80068c0b60 LPC_CANCELED
    fffff8a00bdeca50 00000ec8 000000000000233c:00000000000013f8 0000000000000000 fffffa80079455b0 LPC_CANCELED
    fffff8a00b662830 000005cc 00000000000005e4:0000000000000e0c fffffa800791a7a0 fffffa8007376580 LPC_REQUEST
    fffff8a003d57150 00000f08 0000000000002678:0000000000003e0c 0000000000000000 fffffa8007e4a870 LPC_CANCELED
    fffff8a00cd08830 00000750 0000000000003408:0000000000003adc 0000000000000000 fffffa8008631b60 LPC_CANCELED
    fffff8a01855b2f0 000004f4 0000000000002c74:0000000000002d00 0000000000000000 fffffa800746b890 LPC_CANCELED
    fffff8a00da0d0b0 00000db0 0000000000001a34:0000000000002d80 0000000000000000 fffffa800aff4b60 LPC_CANCELED
    fffff8a00eddb030 0000059c 0000000000003f34:0000000000003c8c 0000000000000000 fffffa8008f96060 LPC_CANCELED
    fffff8a017a14d00 00000920 0000000000003850:0000000000002588 0000000000000000 fffffa8009f66060 LPC_CANCELED
    fffff8a01792d030 000007f8 0000000000003844:00000000000028d0 0000000000000000 fffffa800ad56260 LPC_CANCELED
    fffff8a00f8d6ae0 00000f30 000000000000239c:0000000000001694 0000000000000000 fffffa8008b86060 LPC_CANCELED
    fffff8a01395ab80 00000cdc 0000000000003630:00000000000018f8 0000000000000000 fffffa8005bc0770 LPC_CANCELED
    fffff8a0166ff800 00000984 00000000000005e4:00000000000025f4 fffffa8009718910 fffffa8008cbfb60 LPC_REQUEST
    fffff8a012b9f5a0 00000ac8 0000000000002d34:0000000000001b24 0000000000000000 fffffa8009cd8410 LPC_CANCELED
    fffff8a014313830 00000afc 00000000000005e4:00000000000023bc fffffa80073f0230 fffffa80054d7060 LPC_REQUEST
    fffff8a00a34a6b0 00000ca8 0000000000002534:0000000000002dd0 0000000000000000 fffffa80064c3980 LPC_CANCELED
[...]
    fffff8a00ad8f610 00000e64 0000000000003714:00000000000030b8 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a015720710 00001594 0000000000003638:00000000000029b8 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a009bac560 00001508 0000000000003994:0000000000001aac 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a00b6e78f0 00001574 0000000000002938:0000000000001998 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a00b5716b0 00001570 0000000000002938:0000000000001698 0000000000000000 fffffa800a3b8620 LPC_REQUEST
    fffff8a018531d00 00000db8 00000000000016d8:00000000000031c4 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a01112f410 000014b0 0000000000001b6c:0000000000001618 0000000000000000 fffffa800a3b8620 LPC_CANCELED

  Canceled queue is empty.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

As the Year of Dump Analysis 0×7DA (2010) comes closer to the end and the DeBugging decade starts 0×7DB (2011) soon we organize Debugging Joke Competition with the results announced on the 1st of January, 2011 (if Internet works). Please send your jokes using this contact form:

http://www.dumpanalysis.org/contact

Winners get signed (by Dr. DebugLove) copies of Dr. Debugalov book and the forthcoming full color coffee table book Spikes, Hangs, Crashes, Leaks and Dumps of Imagination: The Art of the Debugging Art.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Just came up with this one for a starter:

Q. Why is the execution of this program so stable? A. Because there is a breakpoint at every instruction.

For those from countries in the past socialist camp like Soviet Union it might appear bugtated from a joke I heard from one Moscow State University mathematics professor when I was a student:

“Q. Why is the Communist Party course always straight? A. Because there is an inflection at every point.”

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In order to promote memory dump worldview and associated philosophy of memoidealism we have founded a society with a mission to teach memory dump analysis to everyone.

Dmitry Vostokov
Director of Studies

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memorianity provides a foundation for moral conduct, individual and social character development because it is based on a revelation that everything is saved. The theistic variation of this memory religion has also an organic and harmonious notion of Memory Deity. May you be memorized.

Dmitry Vostokov
Memoriarch

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One of the most common patterns is Crash Signature. It consists of a set of attributes derivable from saved execution context for exceptions, faults and traps. For example, on x64 Windows it is usually RIP and RSP addresses. For x86 it is usually EIP, ESP and EBP. It can also include the application module name.

0:009> !analyze -v

[...]

FAILURE_BUCKET_ID:  SOFTWARE_NX_FAULT_c0000005_ApplicationA.exe!Unknown

BUCKET_ID:  APPLICATION_FAULT_SOFTWARE_NX_FAULT_STACK_CORRUPTION_BAD_IP_ApplicationA+2d560

[...]

0:009> kL
ChildEBP RetAddr 
0354f270 75bc0962 ntdll!NtWaitForMultipleObjects+0x15
0354f30c 7651162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0354f354 76511921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0354f370 76539b0d kernel32!WaitForMultipleObjects+0x18
0354f3dc 76539baa kernel32!WerpReportFaultInternal+0x186
0354f3f0 765398d8 kernel32!WerpReportFault+0x70
0354f400 76539855 kernel32!BasepReportFault+0x20
0354f48c 77750727 kernel32!UnhandledExceptionFilter+0x1af
0354f494 77750604 ntdll!__RtlUserThreadStart+0x62
0354f4a8 777504a9 ntdll!_EH4_CallFilterFunc+0x12
0354f4d0 777387b9 ntdll!_except_handler4+0x8e
0354f4f4 7773878b ntdll!ExecuteHandler2+0x26
0354f5a4 776f010f ntdll!ExecuteHandler+0x24
0354f5a4 0354f958 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 0×354f958
00000000 00000000 0×2ff0340

0:009> kv
ChildEBP RetAddr  Args to Child             
[...]
0354f5a4 0354f958 0154f5bc 0354f60c 0354f5bc ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0354f60c)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 00000000 00000000 00000000 0×354f958
00000000 00000000 00000000 00000000 00000000 0×2ff0340

0:009> .cxr 0354f60c
eax=80010105 ebx=0354f924 ecx=00000003 edx=0000ffff esi=00d7dce0 edi=00d7e0c8
eip=0354f958 esp=0354f8f4 ebp=0354f908 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
0354f958 64f9            stc

0:009> !address 0354f958
 TEB 7efdd000 in range 7efdb000 7efde000
 TEB 7efda000 in range 7efd8000 7efdb000
 TEB 7efd7000 in range 7efd5000 7efd8000
 TEB 7efaf000 in range 7efad000 7efb0000
 TEB 7efac000 in range 7efaa000 7efad000
 TEB 7efa9000 in range 7efa7000 7efaa000
 TEB 7efa6000 in range 7efa4000 7efa7000
 TEB 7efa3000 in range 7efa1000 7efa4000
 TEB 7ef9f000 in range 7ef9d000 7efa0000
 TEB 7ef9c000 in range 7ef9a000 7ef9d000
 TEB 7ef99000 in range 7ef97000 7ef9a000
 ProcessParametrs 007714b0 in range 00770000 00870000
 Environment 007707f0 in range 00770000 00870000
    03450000 : 0354d000 - 00003000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageStack
                    Pid.Tid  1ea0.12dc

0:009> !address 02ff0340
 TEB 7efdd000 in range 7efdb000 7efde000
 TEB 7efda000 in range 7efd8000 7efdb000
 TEB 7efd7000 in range 7efd5000 7efd8000
 TEB 7efaf000 in range 7efad000 7efb0000
 TEB 7efac000 in range 7efaa000 7efad000
 TEB 7efa9000 in range 7efa7000 7efaa000
 TEB 7efa6000 in range 7efa4000 7efa7000
 TEB 7efa3000 in range 7efa1000 7efa4000
 TEB 7ef9f000 in range 7ef9d000 7efa0000
 TEB 7ef9c000 in range 7ef9a000 7ef9d000
 TEB 7ef99000 in range 7ef97000 7ef9a000
 ProcessParametrs 007714b0 in range 00770000 00870000
 Environment 007707f0 in range 00770000 00870000
    02fc0000 : 02fc0000 - 00043000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   00d70000

Stack trace may or may not be included here and it might be incorrect, heuristic and not fully discernible automatically (requires raw stack semantic analysis) like in the example above. In some cases exception information might not be valid though, for example, in the case of laterally damaged or truncated memory dump files.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 4 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4

It features:

- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art

Its table of contents is available here:

http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Where’s God in Memorianity (memory religion)? It borrows Samuel Alexander’s notion of the emergence of God as a new synthesis from patterns. In Alexander’s philosophy the basis of nature is space-time continuum with point-instants. In memoidealism (which is a metaphysical foundation of Memorianity) Memory is the basis of everything and in its memuonic formulation memuons play the role of memory-instants. Their patterns also involve emergents on every hierarchy level with Mind emerging at some memory organization level and then out of mind level we have emerging Deity.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Activity Region pattern highlights “mechanical” and syntactical aspects of trace analysis whereas Focus of Tracing brings attention to changing semantics of trace message flow, for example in Citrix terminal services environment, from logon messages during session initialization to LHC database search. Here is a graphical illustration of this pattern where tracing focus region spans 3 regions of activity:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memorized states and events look different from varying memory perspectives, for example, some historical event looks different to and interpreted differently by people of varying memory backgrounds. Memuonic version of memoidealist philosophy explains differences in memory perspectives by different memorizing orderings of memuons respective to some base memuon.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I have discovered rules that make it possible to devise a memory dump and software trace analysis equivalent of the Periodic Table of Elements in Chemistry. It allows prediction of abnormal software behaviour and structural defects and what patterns to look for after deploying software and collecting its artifacts. More on this is in the next part of these series.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are two trace reading practices with techniques borrowed from structuralist and post-structuralist narratology:

1. Close reading

- emphasizes structural patterns

- looks at a software trace as a unity of messages

- searches for similarities, repetitions and contrasts

- reveals code reflections in message texts

2. Deconstructive reading

- reveals subconscious exposed in message texts

- searches for conflicting and absent messages

- looks at a software trace as a disunity of messages from conflicting components

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a kernel mode counterpart of Divide by Zero pattern in user mode. It manifests under different bugchecks, for example:

1: kd> !analyze -v

[...]

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind that the kernel isn't allowed to have/catch (bound trap) or that is always instant death (double fault).  The first number in the bugcheck params is the number of the trap (8 = double fault, etc) Consult an Intel x86 family manual to learn more about what these traps are. Here is a *portion* of those codes:
If kv shows a taskGate
        use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
        use .trap on that value
Else
        .trap on the appropriate frame will show where the trap was taken
        (on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000000, EXCEPTION_DIVIDED_BY_ZERO
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

[...]

TRAP_FRAME:  a8954c8c -- (.trap 0xffffffffa8954c8c)
ErrCode = 00000000
eax=ffffffff ebx=00000000 ecx=00000005 edx=00000000 esi=00000000 edi=00000000
eip=975c42cd esp=a8954d00 ebp=a8954d4c iopl=0 nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000  efl=00010246
win32k!NtGdiEnumObjects+0xc6:
975c42cd f7f6            div     eax,esi
Resetting default scope

PROCESS_NAME:  Application.EXE

[...]

STACK_TEXT: 
a8954c2c 81ac2b76 0000007f 5317512a 975c42cd nt!KeBugCheck+0x14
a8954c80 81899808 a8954c8c a8954d4c 975c42cd nt!Ki386CheckDivideByZeroTrap+0×44
a8954c80 975c42cd a8954c8c a8954d4c 975c42cd nt!KiTrap00+0×88
a8954d4c 81898a7a 062102ce 00000001 00000000 Driver!EnumObjects+0xc6
a8954d4c 77a59a94 062102ce 00000001 00000000 nt!KiFastCallEntry+0×12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012ca70 00000000 00000000 00000000 00000000 0×77a59a94

0: kd> !analyze -v

[...]

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000094, Exception code that caused the bugcheck
Arg2: fffff9600025ba6d, Address of the exception record for the exception that caused the bugcheck
Arg3: fffff8800ac361d0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

[...]

EXCEPTION_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION}  Integer division by zero.

FAULTING_IP:
Driver!EnumObjects+e9
fffff960`0025ba6d f7f6            div     eax,esi

CONTEXT:  fffff8800ac361d0 -- (.cxr 0xfffff8800ac361d0)
rax=00000000ffffffff rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff9600025ba6d rsp=fffff8800ac36ba0 rbp=fffff8800ac36ca0
 r8=0000000000000000  r9=0000000000000000 r10=0000000005892f18
r11=fffff900c28379e0 r12=0000000000000000 r13=0000000000000002
r14=0000000000000001 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b  efl=00010286
Driver!EnumObjects+0xe9:
fffff960`0025ba6d f7f6            div     eax,esi
Resetting default scope

[...]

STACK_TEXT:
fffff880`0ac36ba0 fffff800`01682993 Driver!EnumObjects+0xe9
fffff880`0ac36c20 00000000`748a1b3a nt!KiSystemServiceCopyEnd+0x13
00000000`001cdf08 00000000`00000000 0x748a1b3a

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

On the value of study and perseverance. It all started with dumb 0xc0000005 (resulted in a dump) and ended up with 5 volumes of Summa Memorianica (Memory Dump Analysis Anthology):

The Dumb 0x.

Albertus Magnus said of Thomas Aquinas

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This case study centers on 3 process dump files (two first chance exception and one second chance exception). To recall the difference between them please read first chance exceptions explained series. When we get first and second chance exception dumps together we usually open a second chance exception dump first. However, in this case, the second chance exception dump had an incorrect stack trace:

(f54.b80): Access violation - code c0000005 (!!! second chance !!!)
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

0:000> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

The default analysis command detected stack overflow pattern: 

0:000> !analyze -v

[...]

FAULTING_IP:
ntdll!RtlDispatchException+8
7c92a978 56              push    esi

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c92a978 (ntdll!RtlDispatchException+0x00000008)
   ExceptionCode: c00000fd (Stack overflow)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: 00032fc0

DEFAULT_BUCKET_ID:  STACK_OVERFLOW

ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.

[...]

Indeed ESP was outside the stack region and that happened during unhandled exception processing:

0:000> r esp
esp=00030e4c

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        000310c4
    StackBase:            00130000
    StackLimit:           00031000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             00000f54 . 00000b80
    RpcHandle:            00000000
    Tls Storage:          001537a8
    PEB Address:          7ffdb000
    LastErrorValue:       2
    LastStatusValue:      c000000f
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dps esp l100
00030e4c  ????????
00030e50  ????????
[...]
00030ff8  ????????
00030ffc  ????????
00031000  00000000
00031004  00000000
00031008  00000000
0003100c  00000000
00031010  00000000
00031014  00000000
00031018  00000000
0003101c  00000000
00031020  00000000
00031024  7c910323 ntdll!RtlpImageNtHeader+0x56
00031028  004054e8 Application+0x54e8
0003102c  00400000 Application
00031030  00400000 Application
00031034  00400100 Application+0x100
00031038  00031028
0003103c  7e390000 USER32!_imp__GetClipRgn <PERF> (USER32+0x0)
00031040  00031060
00031044  7c910385 ntdll!RtlImageDirectoryEntryToData+0x57
00031048  00400000 Application
0003104c  00000001
00031050  0000000e
00031054  00031084
00031058  7c910323 ntdll!RtlpImageNtHeader+0x56
0003105c  004054e8 Application+0x54e8
00031060  7c900000 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0x0)
00031064  0012ff00
00031068  7c9000d0 ntdll!RtlDosPathSeperatorsString <PERF> (ntdll+0xd0)
0003106c  0003105c
00031070  0000000e
00031074  00114e88
00031078  7c90e920 ntdll!_except_handler3
0003107c  7c910328 ntdll!`string'+0x4
00031080  ffffffff
00031084  7c910323 ntdll!RtlpImageNtHeader+0x56
00031088  7c935f1c ntdll!RtlLookupFunctionTable+0xc5
0003108c  7c92ab3a ntdll!RtlLookupFunctionTable+0xf2
00031090  7c97e178 ntdll!LdrpLoaderLock
00031094  000310c4
00031098  7c809ad8 kernel32!_except_handler3
0003109c  7c833fd9 kernel32!UnhandledExceptionFilter+0xf
000310a0  7c834aa8 kernel32!`string’+0×1c
000310a4  000310d0
000310a8  0040550c Application+0×550c
000310ac  000310b4
000310b0  7c9032a8 ntdll!ExecuteHandler2+0×26
000310b4  00031198
000310b8  0012ffb4
000310bc  000311ac 

Before we tried to reconstruct the stack trace we opened the earlier first-chance exception dump file:

0:000> .opendump 1stchance.dmp

Loading Dump File [1stchance.dmp]
User Dump File: Only application data is available

Opened '1stchance.dmp'

||0:0:000> g

(f54.b80): Stack overflow - code c00000fd (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0003332c ebx=00033040 ecx=00033054 edx=7c90e514 esi=000333a8 edi=00000000
eip=7c92a978 esp=00032fc4 ebp=00033028 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
ntdll!RtlDispatchException+0×8:
7c92a978 56              push    esi

Here we were able to get stack trace from the saved nested exception: 

||1:1:020> kL 1000
ChildEBP RetAddr 
00033028 7c90e48a ntdll!RtlDispatchException+0x8
00033028 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033390 7c90e48a ntdll!RtlDispatchException+0x133
00033390 7c95019e ntdll!KiUserExceptionDispatcher+0xe
000336f8 7c90e48a ntdll!RtlDispatchException+0x133
000336f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033a60 7c90e48a ntdll!RtlDispatchException+0x133
00033a60 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00033dc8 7c90e48a ntdll!RtlDispatchException+0x133
00033dc8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034130 7c90e48a ntdll!RtlDispatchException+0x133
00034130 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034498 7c90e48a ntdll!RtlDispatchException+0x133
00034498 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034800 7c90e48a ntdll!RtlDispatchException+0x133
00034800 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034b68 7c90e48a ntdll!RtlDispatchException+0x133
00034b68 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00034ed0 7c90e48a ntdll!RtlDispatchException+0x133
00034ed0 7c95019e ntdll!KiUserExceptionDispatcher+0xe
[...]
001143f8 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114760 7c90e48a ntdll!RtlDispatchException+0x133
00114760 7c95019e ntdll!KiUserExceptionDispatcher+0xe
00114ac8 7c90e48a ntdll!RtlDispatchException+0x133
00114ac8 7c7e2afb ntdll!KiUserExceptionDispatcher+0xe
00114e30 0057ad17 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
00114e54 0098ff95 Application+0x17ad17
[...]
00121fd8 7e398734 Application+0x313be
00122004 7e398816 USER32!InternalCallWinProc+0x28
0012206c 7e3a8ea0 USER32!UserCallWinProcCheckWow+0x150
001220c0 7e3aacd1 USER32!DispatchClientMessage+0xa3
001220f0 7c90e473 USER32!__fnINSTRING+0x37
0012212c 7e3993e9 ntdll!KiUserCallbackDispatcher+0x13
00122158 7e3aa43b USER32!NtUserPeekMessage+0xc
00122184 004794d9 USER32!PeekMessageA+0xeb
00122234 00461667 Application+0x794d9
[...]
0012ffc0 7c7e7077 Application+0x60610b
0012fff0 00000000 kernel32!BaseProcessStart+0x23

This all pointed to a problem exception handler:

||1:1:020> !analyze -v

[...]

CONTEXT:  00114b10 -- (.cxr 0x114b10)
eax=00114de0 ebx=0eedfade ecx=00000000 edx=001537a8 esi=00114e88 edi=00000007
eip=7c7e2afb esp=00114ddc ebp=00114e30 iopl=0 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200202
kernel32!RaiseException+0x53:
7c7e2afb 5e              pop     esi
Resetting default scope

[...]

||1:1:020> .cxr 0x114b10

||1:1:020> kv 1
ChildEBP RetAddr  Args to Child             
00114e30 0057ad17 0eedfade 00000001 00000007 kernel32!RaiseException+0×53

Being curious we also opened the second first chance exception dump and it pointed to the expected crash point (the same as seen in the second chance exception crash dump)

||1:1:020> .opendump 1stchance2.dmp

Loading Dump File [1stchance2.dmp]
User Dump File: Only application data is available

Opened '1stchance2.dmp'

||1:1:020> g

[...]

(f54.b80): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000248 ebx=00000000 ecx=004054e8 edx=7c9032bc esi=00000000 edi=00000000
eip=7c7d24f0 esp=00030e4c ebp=000310a4 iopl=0 nv up ei pl nz ac po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000  efl=00200212
kernel32!_SEH_prolog+0×1a:
7c7d24f0 53              push    ebx

||2:2:040> kL
ChildEBP RetAddr 
000310a4 00000000 kernel32!_SEH_prolog+0x1a

We found the similar past issue for a different process name but our main process module information included the same vendor name so it was easy to contact the corresponding vendor.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If you develop and debug user space applications (and/or doing crash dump analysis in user space) or specialize in user space security and you want to understand Windows kernel dumps and device drivers better (and probably start writing your own kernel tools) or understand malware rootkits better here is the reading list I found the most effective over the last 7 years:

0.0. Read and re-read Windows Internals book in parallel while reading all other books. I read all editions by the way. It will show you the big picture and useful WinDbg commands and techniques but you need to read device driver books to fill the gaps and be confident in kernel space:

0.1. Start with The Windows 2000 Device Driver Book: A Guide for Programmers. This short book will show you the basics and you can start writing your drivers and kernel tools immediately.

0.2. Next read Windows NT Device Driver Development book to consolidate your knowledge. This book has been reprinted by OSR (I own the original New Riders Press edition):

0.3. Don’t stop here. Read Developing Windows NT Device Drivers: A Programmer’s Handbook. This is the very good book explaining everything in great detail and good pictures. You will finally understand various buffering methods.

0.4. Continue with WDM drivers and modern presentation: Programming the Microsoft Windows Driver Model. Must read even if your drivers are not WDM.

0.5. Finally read Developing Drivers with the Windows Driver Foundation book. It also covers ETW (event tracing for Windows), WinDbg extensions, PREfast and static driver verifier.

0.6. There is a forthcoming book Windows 7 Device Driver at the time of this writing that also covers WDF so you might want to start with #0.6 and continue with #0.5 as a reference:

Additional reading (not including DDK Help which you will use anyway) can be done in parallel after finishing “Windows NT Device Driver Development” book:

1.1. OSR NT Insider articles. I have their full printed collection 1996 - 2006 plus all the latest issues (looks like print editions are discontinued and the new ones are only digital):

http://www.osronline.com/

1.2. Windows NT File System Internals reprinted by OSR (I have the original O’Reilly edition):

1.3. Windows NT/2000 Native API Reference is fun to browse occasionally and indispensable if you don’t have access to Windows source code:

1.4. Rootkits: Subverting the Windows Kernel book will show you Windows kernel from the hacker perspective. In addition you will find the overview of kernel areas not covered in other books.

1.5. The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System is another excellent book that is up to date and explains kernel staff from ab initio. I’m reading it at the time of this writing and recommend it to read first or in parallel to all other books:

Of course, you must know C language and its idioms really well. Really know it down to assembly language level! I’ll publish other reading lists soon including reverse engineering classics. Stay tuned.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have very similar abnormal software behaviour dispositions (like crashes with similar stack traces) for different applications or services. In such cases, we should also check application or service vendor and copyright in the output of lmv command. Similar to Template Module Same Vendor pattern can be useful to relate such different incidents. Usually, in the same company, code and people reuse tends to distribute code fragments and code construction styles across different product lines, and software defects might surface in different images. For example:

0:000> lmv m ApplicationA
start    end        module name
00400000 00d99000   ApplicationA   (deferred)
[...]
Image name: ApplicationA.exe
Timestamp:        [...]
CheckSum:         00000000
[...]
CompanyName:      CompanyA
ProductName:      CompanyA Application
LegalCopyright:   Copyright (c) CompanyA
[…]

0:000> lmv m ApplicationB
start    end        module name
00400000 019d0000   ApplicationB  C (no symbols)
Image name: ApplicationB.exe
[...]
CompanyName:      CompanyA
ProductName:      ApplicationB
LegalCopyright:   Copyright (c) CompanyA
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Having spent 16 years in software engineering I ventured into software support in 2003 (with 8th year started at the time of this writing). Now it is time for the next gradual shift into software security (the domain I previously had exposure to but not as a primary focus):

The title of this post is borrowed from the book I read from cover to cover long time ago and recently put on my desk again:

Journey of the Software Professional: The Sociology of Software Development

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Previously announced DSS has got its logo and now affiliated with DA+TA Facebook group:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Motivated by the existence of London School of Economics (LSE) I just founded DSS. The program to be communicated soon and includes general memory dump and software trace analysis as a foundation for security. I like the name very much because of its additional meaning:

DUmps Binary Logs INternals

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -