Software Diagnostics Library

For some time I was struggling with finding a good name for memory dump and software trace analysis activities. The name Memoretics I use for the science of memory dump analysis (that also incorporates software traces) seems not so good to describe the whole practical activity that should be transparent to everyone in IT. Fortunately, I timely understood that all these activities constitute the essence of software diagnostics that previously lacked any solid foundation. Thus, Software Diagnostics Institute was reborn from the previous Crash Dump Analysis Portal. This institute does pure and applied research and scientific activities and in recent years was funded mainly from OpenTask publisher and recently from Memory Dump Analysis Services. The latter company also recognized that the broadening of its commercial activities requires a new name. So, Software Diagnostics Services was reborn:

The First Comprehensive Software Diagnostics Service

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The new 6th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about memory dump and software trace analysis, software troubleshooting and debugging written in November 2010 - October 2011 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The sixth volume features:

• 56 new crash dump analysis patterns including 14 new .NET memory dump analysis patterns
• 4 new pattern interaction case studies
• 11 new trace analysis patterns
• New Debugware pattern
• Introduction to UI problem analysis patterns
• Introduction to intelligence analysis patterns
• Introduction to unified debugging pattern language
• Introduction to generative debugging, metadefect template library and DNA of software behaviour
• The new school of debugging and trends
• .NET memory dump analysis checklist
• Software trace analysis checklist
• Introduction to close and deconstructive readings of a software trace
• Memory dump analysis compass
• Computical and Stack Trace Art
• The abductive reasoning of Philip Marlowe
• Orbifold memory space and cloud computing
• Memory worldview
• Interpretation of cyberspace
• Relationship of memory dumps to religion
• Fully cross-referenced with Volume 1, Volume 2, Volume 3, Volume 4, and Volume 5

Product information:

• Title: Memory Dump Analysis Anthology, Volume 6
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 300 pages
• Publisher: Opentask (December 2011)
• ISBN-13: 978-1-908043-19-1

• Hardcover: 300 pages
• Publisher: Opentask (January 2012)
• ISBN-13: 978-1-908043-20-7

Back cover features 3d memory space visualization image created with ParaView.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to popular demand and the need to extend existing Accelerated Windows Memory Dump Analysis training Memory Dump Analysis Services organises the new training course. Here is the description and registration information:

Learn how to analyze .NET application and service crashes and freezes, navigate through memory dump space (managed and unmanaged code) and diagnose corruption, leaks, CPU spikes, blocked threads, deadlocks, wait chains, resource contention, and much more. We use a unique and innovative pattern-driven analysis approach to speed up the learning curve. The training consists of practical step-by-step exercises using WinDbg to diagnose patterns in 32-bit and 64-bit process memory dumps.

If you are registered you are allowed to optionally submit your memory dumps before the training. This will allow us in addition to the carefully constructed problems tailor extra examples to the needs of the attendees.

The training consists of one four-hour session and additional homework exercises. When you finish the training you additionally get:

1. A full transcript in PDF format (retail price $200)
2. 5 volumes of Memory Dump Analysis Anthology in PDF format (retail price $100)
3. A personalized attendance certificate with unique CID (PDF format)

Prerequisites: Basic .NET programming and debugging.

Audience: Software developers, software technical support and escalation engineers.

Session: October 28, 2011 4:00 PM - 8:00 PM GMT

Price: 210 USD

Space is limited.
Reserve your remote training seat now at:
https://student.gototraining.com/24s4l/register/423991811034037760

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

To generate crash dumps for my recent presentation The Old New Crash: Cloud Memory Dump Analysis I created the tool similar to TestWER. It is a .NET worker service role for Windows Azure that does a NULL pointer assignment. Here is a source code from WorkerRole.cs (I left EnableCollection twice just in case :-)):

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.Diagnostics.Management;

namespace TestWAERWorker
{
    public class WorkerRole : RoleEntryPoint
    {
        public override void Run()
        {
            Trace.WriteLine("TestWAERWorker entry point called", "Information");

            Thread.Sleep(60000);

            unsafe
            {
                int* p = null;
                *p = 1;
            };

            while (true)
            {
                Thread.Sleep(10000);
                Trace.WriteLine("Working", "Information");
            }
        }

        public override bool OnStart()
        {
            // Set the maximum number of concurrent connections
            ServicePointManager.DefaultConnectionLimit = 12;

            CrashDumps.EnableCollection(true);

            DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();
            string conn_str = RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString");
            CloudStorageAccount account = CloudStorageAccount.Parse(conn_str);
            config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
            DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(account, config);

            CrashDumps.EnableCollection(true);

            return base.OnStart();
        }
    }
}

The following forum discussion really helped me to get things right: http://stackoverflow.com/questions/5727692/azure-crash-dumps-not-appearing. So I’d like to thank Oliver Bock here.

If you want to deploy it to Windows Azure then don’t forget to add your storage account name and its key in the service configuration file. For your development environment you will need to replace the whole value with “UseDevelopmentStorage=true”. The full case study with detailed steps is forthcoming.

You can download the package from here: TestWAER.zip

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After getting my hands on crash dumps from Windows Azure cloud platform I discerned a few patterns and the first one is just called Cloud Environment. The pattern covers both development (emulator, if it exists) and real (staging and deployment) environments. This pattern is best diagnosed by looking at specific infrastructure modules:

0:016> lm m Wa*
start             end                 module name
00000000`00b00000 00000000`00b0c000   WaWorkerHost
00000000`74fb0000 00000000`74fbd000   WaRuntimeProxy

0:016> lm m *Azure*
start             end                 module name
00000000`57cd0000 00000000`57d26000   Microsoft_WindowsAzure_StorageClient
00000000`58820000 00000000`5886c000   Microsoft_WindowsAzure_Diagnostics
00000000`5c750000 00000000`5c764000   Microsoft_WindowsAzure_ServiceRuntime

Development platform can be distinguished for now by looking at ntdll version: 

0:016> lmv m ntdll
start             end                 module name
00000000`76de0000 00000000`76f5f000   ntdll
    Loaded symbol image file: ntdll.dll
    Image path: D:\Windows\System32\ntdll.dll
    Image name: ntdll.dll
    Timestamp:        Fri May 13 21:45:21 2011 (4DCD9861)
    CheckSum:         00188814
    ImageSize:        0017F000
    File version:     6.0.6002.18446
    Product version:  6.0.6002.18446
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   6.0.6002.18446
    FileVersion:      6.0.6002.18446 (rd_os_v1.110513-1321)
    FileDescription:  NT Layer DLL
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

0:016> lmv m ntdll
start             end                 module name
00000000`775a0000 00000000`7774b000   ntdll
    Loaded symbol image file: ntdll.dll
    Image path: C:\Windows\System32\ntdll.dll
    Image name: ntdll.dll
    Timestamp:        Tue Jul 14 02:32:27 2009 (4A5BE02B)
    CheckSum:         001B1CB5
    ImageSize:        001AB000
    File version:     6.1.7600.16385
    Product version:  6.1.7600.16385
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   6.1.7600.16385
    FileVersion:      6.1.7600.16385 (win7_rtm.090713-1255)
    FileDescription:  NT Layer DLL
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

We see that real Windows Azure is currently RD OS V1 (at least in a datacenter chosen for West European affinity during deployment).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -