Archive for March, 2011

Crash Dump Analysis Patterns (Part 131)

Monday, March 7th, 2011

In certain software behavior scenarios such as a memory leak when we see top modules calling OS API functions we might suspect them having defects. However, this might not be the case and these modules were used from Directing Module  keeping references or handles preventing top modules from freeing memory or releasing resources.

For example, a memory dump from a process had 2 growing heap segments and one of them had this recurrent stack trace saved in a user mode stack trace database:

38D2CE78: 02ba8 . 02ba8 [07] - busy (2b90), tail fill
Stack trace (38101) at 83e390:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F

337d0572: ModuleA!XHeapAlloc+0×00000115
338809e2: ModuleA!Execute+0×000002CD

488b3fc1: ModuleB!Execute+0×000000D3
679b8c64: ModuleC!ExecuteByHandle+0×00000074
67d241cb: ModuleD!Query+0×0000016B
67ba2ed4: ModuleE!Browse+0×000000E4
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406


Initially we suspected ModuleA but found a different recurrent stack trace corresponding to another growing segment:

40C81688: 000c8 . 00058 [07] - busy (40), tail fill
Stack trace (38136) at 83f6a4:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F
7c3416b3: msvcr71!_heap_alloc+0×000000E0
7c3416db: msvcr71!_nh_malloc+0×00000010

67745875: ModuleX!BufAllocate+0×00000015
6775085e: ModuleY!QueryAttribute+0×0000008E
677502b5: ModuleY!Query+0×00000015
67ba2f19: ModuleE!Browser+0×00000129
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406

From the common stack trace fragment (highlighted in blue) we transferred our investigation to ModuleE and indeed the similar software incident (as the latter trace) was found in our troubleshooting database.

- Dmitry Vostokov @ + -

The Church of Memory Analysis

Friday, March 4th, 2011

This is another name for Memorianity (Memory Religion) that incorporates M->analysis techniques. I’m working now on the full statement of creed to be published soon. May you be memorized.

Dmitry Vostokov

- Dmitry Vostokov @ Memory Religion Portal -


Thursday, March 3rd, 2011

Memory->analysis or M->analysis in short is a new pattern-driven psychotherapeutic method based on Unified Memory Theory (another name for Memory Worldview). It has a logo as well:

The name was inspired by M-theory and unified field theories. Please also see my interpretation of M-theory. Another name variant is M-analysis which can be used as a shortcut to M->analysis.

- Dmitry Vostokov @ + -

Stack Overflow Patterns

Wednesday, March 2nd, 2011

A page to reference all different kinds of stack overflow is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ + -

Crash Dump Analysis Patterns (Part 16c)

Tuesday, March 1st, 2011

Stack Overflow pattern variants in user and kernel mode are ISA (Instruction Set Architecture) and processor architecture oriented. Another pattern variant is software stack implementations where push and pop operations check  stack ADT preconditions and throw a software exception (overflow or underflow) or call an assertion mechanism to display an error message. For the latter example, we look at a bugcheck for the specific stack implementation on Windows: IRP stack locations array. For a graphical reminder on how driver-to-driver communication is implemented by an IRP stack corresponding to a driver stack please refer to UML diagram no. 3 in the old post about using UML for describing device driver design. The following WinDbg command output is from a kernel memory dump:

0: kd> !analyze -v
A higher level driver has attempted to call a lower level driver through the IoCallDriver() interface, but there are no more stack locations in the packet, hence, the lower level driver would not be able to access its parameters, as there are no parameters for it. This is a disasterous situation, since the higher level driver "thinks" it has filled in the parameters for the lower level driver (something it MUST do before it calls it), but since there is no stack location for the latter driver, the former has written off of the end of the packet.  This means that some other memory has probably been trashed at this point.
Arg1: fffffa800500c9e0, Address of the IRP
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffff880`01fe2338 fffff800`016d7732 nt!KeBugCheckEx
fffff880`01fe2340 fffff800`01754f27 nt!KiBugCheck3+0x12
fffff880`01fe2370 fffff880`0177e271 nt! ?? ::FNODOBFM::`string’+0×3f31b
fffff880`01fe23a0 fffff880`0177c138 DriverA!CallProvider+0×161
fffff880`01fe2cb0 fffff800`0197a7c6 nt!ExpWorkerThread+0×111
fffff880`01fe2d40 fffff800`016b5c26 nt!PspSystemThreadStartup+0×5a
fffff880`01fe2d80 00000000`00000000 nt!KxStartSystemThread+0×16

0: kd> !irp fffffa800500c9e0
Irp is active with 1 stacks 0 is current (= 0xfffffa8006c2e960)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  4, 0]   0 e0 fffffa8004045c50 fffffa8006c2e960 fffff88005a04460-fffffa8005b9c370 Success Error Cancel
        \DriverA DriverB!CompleteRoutine
   Args: 00000008 00000000 00000000 00000000

0: kd> ub fffff880`0177e271
fffff880`0177e24e mov     qword ptr [r11-10h],rax
fffff880`0177e252 mov     qword ptr [r11-8],r12
fffff880`0177e256 mov     byte ptr [r11-45h],0E0h
fffff880`0177e25b mov     rcx,qword ptr [rdi+40h]
fffff880`0177e25f call    qword ptr [DriverA!_imp_IoGetAttachedDevice (fffff880`017790b0)]
fffff880`0177e265 mov     rdx,rbp
fffff880`0177e268 mov     rcx,rax
fffff880`0177e26b call    qword ptr [DriverA!_imp_IofCallDriver (fffff880`01779068)]

- Dmitry Vostokov @ + -