Archive for August, 2011

Crash Dump Analysis Patterns (Part 147)

Tuesday, August 30th, 2011

This is a new pattern about activation contexts. Here we have software exceptions STATUS_SXS_*, for example:

STATUS_SXS_EARLY_DEACTIVATION 0xC015000F
STATUS_SXS_INVALID_DEACTIVATION 0xC0150010

0:000> !analyze -v

[...]

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 77a54441 (ntdll!RtlDeactivateActivationContext+0x00000154)
   ExceptionCode: c015000f
  ExceptionFlags: 00000000
NumberParameters: 3
   Parameter[0]: 00000000
   Parameter[1]: 0056cbe8
   Parameter[2]: 0056cc18

EXCEPTION_CODE: (NTSTATUS) 0xc015000f - The activation context being deactivated is not the most recently activated one.

CONTEXT:  003df6c8 -- (.cxr 0x3df6c8)
eax=003df9bc ebx=13050002 ecx=00000000 edx=00000000 esi=0056cbe8 edi=0056cc18
eip=77a54441 esp=003df9b0 ebp=003dfa0c iopl=0  nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00000206
ntdll!RtlDeactivateActivationContext+0x154:
77a54441 8b36 mov    esi,dword ptr [esi]  ds:002b:0056cbe8=0056cbb8
Resetting default scope

STACK_TEXT: 
003dfa0c 755aa138 005507d0 13050002 003dfa7c ntdll!RtlDeactivateActivationContext+0×154
003dfa1c 002b1235 00000000 13050002 3a92c68c kernel32!DeactivateActCtx+0×31
003dfa7c 002b13b5 00000001 01f01e98 01f01ec8 TestActCtx!wmain+0×225
003dfac4 75593677 7efde000 003dfb10 77a09f02 TestActCtx!__tmainCRTStartup+0xfa
003dfad0 77a09f02 7efde000 7e35c89d 00000000 kernel32!BaseThreadInitThunk+0xe
003dfb10 77a09ed5 002b140c 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
003dfb28 00000000 002b140c 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

The ReactOS code for RtlDeactivateActivationContext function suggests the following line of inquiry:

0:000> dt _TEB
TestActCtx!_TEB
   +0x000 NtTib            : _NT_TIB
   +0x01c EnvironmentPointer : Ptr32 Void
   +0x020 ClientId         : _CLIENT_ID
   +0x028 ActiveRpcHandle  : Ptr32 Void
   +0x02c ThreadLocalStoragePointer : Ptr32 Void
   +0x030 ProcessEnvironmentBlock : Ptr32 _PEB
   +0x034 LastErrorValue   : Uint4B
   +0x038 CountOfOwnedCriticalSections : Uint4B
   +0x03c CsrClientThread  : Ptr32 Void
   +0x040 Win32ThreadInfo  : Ptr32 Void
   +0x044 User32Reserved   : [26] Uint4B
   +0x0ac UserReserved     : [5] Uint4B
   +0x0c0 WOW32Reserved    : Ptr32 Void
   +0x0c4 CurrentLocale    : Uint4B
   +0x0c8 FpSoftwareStatusRegister : Uint4B
   +0x0cc SystemReserved1  : [54] Ptr32 Void
   +0x1a4 ExceptionCode    : Int4B
   +0×1a8 ActivationContextStack : _ACTIVATION_CONTEXT_STACK
   +0×1bc SpareBytes1      : [24] UChar
   +0×1d4 GdiTebBatch      : _GDI_TEB_BATCH
   +0×6b4 RealClientId     : _CLIENT_ID
   +0×6bc GdiCachedProcessHandle : Ptr32 Void
   +0×6c0 GdiClientPID     : Uint4B
   +0×6c4 GdiClientTID     : Uint4B
   +0×6c8 GdiThreadLocalInfo : Ptr32 Void
   +0×6cc Win32ClientInfo  : [62] Uint4B
   +0×7c4 glDispatchTable  : [233] Ptr32 Void
   +0xb68 glReserved1      : [29] Uint4B
   +0xbdc glReserved2      : Ptr32 Void
   +0xbe0 glSectionInfo    : Ptr32 Void
   +0xbe4 glSection        : Ptr32 Void
   +0xbe8 glTable          : Ptr32 Void
   +0xbec glCurrentRC      : Ptr32 Void
   +0xbf0 glContext        : Ptr32 Void
   +0xbf4 LastStatusValue  : Uint4B
   +0xbf8 StaticUnicodeString : _UNICODE_STRING
   +0xc00 StaticUnicodeBuffer : [261] Wchar
   +0xe0c DeallocationStack : Ptr32 Void
   +0xe10 TlsSlots         : [64] Ptr32 Void
   +0xf10 TlsLinks         : _LIST_ENTRY
   +0xf18 Vdm              : Ptr32 Void
   +0xf1c ReservedForNtRpc : Ptr32 Void
   +0xf20 DbgSsReserved    : [2] Ptr32 Void
   +0xf28 HardErrorMode    : Uint4B
   +0xf2c Instrumentation  : [16] Ptr32 Void
   +0xf6c WinSockData      : Ptr32 Void
   +0xf70 GdiBatchCount    : Uint4B
   +0xf74 InDbgPrint       : UChar
   +0xf75 FreeStackOnTermination : UChar
   +0xf76 HasFiberData     : UChar
   +0xf77 IdealProcessor   : UChar
   +0xf78 Spare3           : Uint4B
   +0xf7c ReservedForPerf  : Ptr32 Void
   +0xf80 ReservedForOle   : Ptr32 Void
   +0xf84 WaitingOnLoaderLock : Uint4B
   +0xf88 Wx86Thread       : _Wx86ThreadState
   +0xf94 TlsExpansionSlots : Ptr32 Ptr32 Void
   +0xf98 ImpersonationLocale : Uint4B
   +0xf9c IsImpersonating  : Uint4B
   +0xfa0 NlsCache         : Ptr32 Void
   +0xfa4 pShimData        : Ptr32 Void
   +0xfa8 HeapVirtualAffinity : Uint4B
   +0xfac CurrentTransactionHandle : Ptr32 Void
   +0xfb0 ActiveFrame      : Ptr32 _TEB_ACTIVE_FRAME
   +0xfb4 FlsData          : Ptr32 Void

0:000> dt _ACTIVATION_CONTEXT_STACK
TestActCtx!_ACTIVATION_CONTEXT_STACK
   +0x000 Flags            : Uint4B
   +0x004 NextCookieSequenceNumber : Uint4B
   +0x008 ActiveFrame      : Ptr32 _RTL_ACTIVATION_CONTEXT_STACK_FRAME
   +0x00c FrameListCache   : _LIST_ENTRY

0:000> dt _RTL_ACTIVATION_CONTEXT_STACK_FRAME
ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
   +0x000 Previous         : Ptr32 _RTL_ACTIVATION_CONTEXT_STACK_FRAME
   +0x004 ActivationContext : Ptr32 _ACTIVATION_CONTEXT
   +0x008 Flags            : Uint4B

0:000> dd 0056cc18 l4
0056cc18  0056cbe8 0056ca6c 00000028 13050003

0:000> dd 0056cbe8
0056cbe8  0056cbb8 0056c934 00000028 13050002
0056cbf8  00000000 00000000 00000000 00000000
0056cc08  00000000 00000000 00000000 00000000
0056cc18  0056cbe8 0056ca6c 00000028 13050003
0056cc28  00000000 00000000 00000000 00000000
0056cc38  00000000 00000000 00000000 00000000
0056cc48  00000000 00000000 0000000c 00000000
0056cc58  00000000 00000000 00000000 00000000

0:000> dd 0056cbb8
0056cbb8  00000000 0056c7fc 00000028 13050001
0056cbc8  00000000 00000000 00000000 00000000
0056cbd8  00000000 00000000 00000000 00000000
0056cbe8  0056cbb8 0056c934 00000028 13050002
0056cbf8  00000000 00000000 00000000 00000000
0056cc08  00000000 00000000 00000000 00000000
0056cc18  0056cbe8 0056ca6c 00000028 13050003
0056cc28  00000000 00000000 00000000 00000000
 

We see that a different cookie was found on top of the thread activation stack and the code raised the runtime exception.

For this pattern I have also created a modeling application and present its source code with additional memory dump analysis in a separate post.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Icons for Memory Dump Analysis Patterns (Part 97)

Monday, August 29th, 2011

Today we introduce an icon for Multiple Exceptions (managed space) pattern:

B/W

Color

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

5 Years of Blogging!

Monday, August 29th, 2011

This month marks 5 years of blogging that resulted in more than 2,500 posts across 8 blogs. I would like to thank everyone for their continuing support! The major milestones are marked on the updated timeline.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Abscess (Debugging Slang, Part 27)

Thursday, August 25th, 2011

Abscess - A Bad Service and Customer Experience Short Story. Can also be an abbreviation for a bad support …

Examples: After combating an abscess a customer decided to take a social media approach. An abscess process.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Exam and Certification: Basic and Intermediate Memory Dump Analysis (Windows track)

Monday, August 15th, 2011

Memory Dump Analysis Services announces development of the first memory dump analysis certification and is looking for volunteers to participate in its beta program. Please visit its website for further details.

Source: http://www.dumpanalysis.com/anon-beta-exam-mda-bi-w

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Advanced Windows Memory Dump Analysis with Data Structures Training Course

Sunday, August 14th, 2011

Due to the need to extend existing basic and intermediate Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises advanced training course. Here is the description and registration information:

Learn how to navigate through memory dump space and Windows data structures to troubleshoot and debug complex software incidents. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose structural and behavioral patterns in 32-bit and 64-bit process, kernel and complete memory dumps.

Advanced Windows Memory Dump Analysis Logo

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

  1. A full transcript in PDF format (retail price $200)
  2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
  3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic and intermediate level Windows memory dump analysis: ability to list processors, processes, threads, modules, apply symbols, walk through stack traces and raw stack data, diagnose patterns such as heap corruption, CPU spike, memory and handle leaks, access violation, stack overflow, critical section and resource wait chains and deadlocks. If you don’t feel comfortable with prerequisites then Accelerated Windows Memory Dump Analysis training is recommended to take (or purchase a corresponding book) before attending this course.

Audience: Software developers, software technical support and escalation engineers.

Session: December 9, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/3788047691824598784

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg Command on Certificate

Saturday, August 13th, 2011

Finally you can even learn a WinDbg command from a certificate. Memory Dump Analysis Services has created a certificate with dv WinDbg command on it:

Source: http://www.dumpanalysis.com/sample-certificate

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Accelerated .NET Memory Dump Analysis Training Course

Thursday, August 11th, 2011

Due to popular demand and the need to extend existing Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises the new training course. Here is the description and registration information:

Learn how to analyze .NET application and service crashes and freezes, navigate through memory dump space (managed and unmanaged code) and diagnose corruption, leaks, CPU spikes, blocked threads, deadlocks, wait chains, resource contention, and much more. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose patterns in 32-bit and 64-bit process memory dumps.

Accelerated .NET Memory Dump Analysis Logo

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

  1. A full transcript in PDF format (retail price $200)
  2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
  3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic .NET programming and debugging.

Audience: Software developers, software technical support and escalation engineers.

Session: October 28, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/423991811034037760

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Pleasing the WinDbg SOS Extension

Wednesday, August 10th, 2011

Recently when looking at one of process memory dumps with CLR runtime I got this message when trying to load SOS extension from the appropriate Framework location:

0:022> .load [..]\Microsoft.NET\Framework\v2.0.50727\sos.dll
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------

So I literally did what was asked by saving a new dump from the dump:

0:022> .dump /ma c:\userdumps\dump-as-asked.dmp
Creating c:\userdumps\dump-as-asked.dmp - mini user dump
Dump successfully written

I got a slightly bigger dump than the original and after that SOS didn’t complain when I opened the new dump and loaded the extension.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

What is to be done?

Tuesday, August 9th, 2011

I showed this artwork to many people and they responded that they didn’t understand or hope one day they would understand. Only one guy (a professional manager) responded positively with understanding. What I think is that real art can be interpreted in many ways. So I kindly await your criticism.

PS. This Computicart (Computical Art) work was inspired by a pet parrot at home. I’ve been observing its behaviour for more than 6 months and tried to discern a few patterns. Its name is KiKi (not related to Ki* functions).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Will To Be Memorized

Tuesday, August 9th, 2011

After reading many history books I found that the real historical, sociological and psychological force (often seen unconscious from outside) is the will to be memorized (saved in memories, in memory dumps, also called as the will to be remembered) and not the will to power (Nietzsche). As often with the will it is also linked with vices and virtues. For the vice side we site Michael Burleigh“Concern with posterity was apparent among the Nazi leadership, […]” and what we see now “… B-movie villains were self-consciously assigning themselves parts within an A-movie which runs and runs” today (documentaries, books, magazines, TV and the so called ‘Hitler industry’ “by the way of continuity”). As a more recent example, the recent tragedy in Norway clearly shows the desire of the protagonist to be memorized (masqueraded as a political agenda).

This is not to say that the will to power is not important, it is important by the way that by being memorized you actually exert more power in the future. Memorianity (memory religion) and its philosophical foundation (memoidealism) promote the will to be memorized as a virtue if all your deeds are seen as a virtue (see Memorianity and Morality).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

New Book: Accelerated Windows Memory Dump Analysis

Sunday, August 7th, 2011

During the previous several months many people expressed their interest in the training (the next one is scheduled for November) but its time was not suitable due to the very different geographic time zones. So I have decided to publish this training in book format (currently in PDF) and make it available in paperback on Amazon and B&N later. Book details:

  • Title: Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes
  • Description: The full transcript of Memory Dump Analysis Services Training with 21 step-by-step exercises, notes, source code of specially created modeling applications and selected Q&A. Covers about 50 crash dump analysis patterns from process, kernel and complete memory dumps.
  • Authors: Dmitry Vostokov, Memory Dump Analysis Services
  • Publisher: OpenTask (August 2011)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 360 pages
  • ISBN-13: 978-1908043290

Table of Contents

Now available for sale in PDF format from Memory Dump Analysis Services.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Accelerated Windows Memory Dump Analysis Training Course (November)

Saturday, August 6th, 2011

Due to popular demand (the previous training was fully booked) Memory Dump Analysis Services scheduled the next training sessions.

Learn how to analyze application, service and system crashes and freezes, navigate through memory dump space and diagnose heap corruption, memory leaks, CPU spikes, blocked threads, deadlocks, wait chains, and much more. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of more than 20 practical step-by-step exercises using WinDbg highlighting more than 50 patterns diagnosed in 32-bit and 64-bit process, kernel and complete memory dumps.

Public preview (selected slides) of the previous training

Accelerated Windows Memory Dump Analysis Logo

Memory Dump Analysis Services organizes a training course.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of 4 two-hour sessions (2 hours every day). When you finish the training you additionally get:

  1. A full transcript in PDF format (retail price $200)
  2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
  3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic Windows troubleshooting

Session 1: November 1, 2011 4:00 PM - 6:00 PM GMT
Session 2: November 2, 2011 4:00 PM - 6:00 PM GMT
Session 3: November 3, 2011 4:00 PM - 6:00 PM GMT
Session 4: November 4, 2011 4:00 PM - 6:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now.

If scheduled dates or time are not suitable for you Memory Dump Analysis Services offers the same training in book format

Training testimonials:

I would like to thank you and recommend your training. I think that the “Accelerated Windows Memory Dump Analysis” training is a pin-point, well taught training. I think it’s the leading training in the dump analysis area and I’ve enjoyed it, the books and materials are very detailed and well written and Dmitry answered all of the needed question. In addition after the training Dmitry sent a PDF with written answers and more information about the questions that were asked. I will give this training 5/5. Thank you Dmitry.

Yaniv Miron, Security Researcher, IL.Hack 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finction (Debugging Slang, Part 26)

Friday, August 5th, 2011

Finction (or simply Finc) - A function with a flaw, having abnormal behaviour, a candidate for a fix or axing.

Examples: After delivering a hotfix with axed code they were embarrassed to see that finction again on a stack trace.

Etymology: Derives from “function”. Inspired by MDAA V3 errata.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Bugtation No.145

Wednesday, August 3rd, 2011

Never thought that I would one day bugtate Bill Gates but found his quote in Knuth’s The Art of Computer Programming, Volume 1:

Memory Dumps “have” not “changed in the past two decades.”

Bill Gates

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -