Archive for the ‘Assembly Language’ Category

Crash Dump Analysis Patterns (Part 24, Linux)

Saturday, December 19th, 2015

This is a Linux variant of Coincidental Symbolic Information pattern previously described for Mac OS X and Windows platforms. The idea is the same: to disassemble the address to see if the preceding instruction is a call. If it is indeed then most likely the symbolic address is a return address from past Execution Residue:

(gdb) x/i 0x4005e6
0x4005e6 <_Z6work_3v+9>: pop    %rbp

(gdb) disassemble 0x4005e6
Dump of assembler code for function _Z6work_3v:
0x00000000004005dd <+0>: push   %rbp
0x00000000004005de <+1>: mov    %rsp,%rbp
0x00000000004005e1 <+4>: callq  0×4005d2 <_Z6work_4v>
0×00000000004005e6 <+9>: pop    %rbp
0×00000000004005e7 <+10>: retq
End of assembler dump.

(gdb) x/4i 0x49c740-4
0x49c73c: add    %al,(%rax)
0x49c73e: add    %al,(%rax)
0×49c740 <default_attr>: add    %al,(%rax)
0×49c742 <default_attr+2>: add    %al,(%rax)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 2, Linux)

Saturday, December 19th, 2015

This is a Linux variant of Dynamic Memory Corruption (process heap) pattern previously described for Mac OS X and Windows platforms.

The corruption may be internal for heap structures with subsequent memory access violation:

(gdb) bt
#0  0×000000000041482e in _int_malloc ()
#1  0×0000000000416d88 in malloc ()
#2  0×00000000004005dc in proc ()
#3  0×00000000004006ee in bar_three ()
#4  0×00000000004006fe in foo_three ()
#5  0×0000000000400716 in thread_three ()
#6  0×0000000000401760 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#7  0×0000000000432609 in clone ()
#8  0×0000000000000000 in ?? ()

(gdb) x/i $rip
=> 0x41482e <_int_malloc+622>: mov    %rbx,0×10(%r12)

(gdb) x $r12+0x10
0x21687371: Cannot access memory at address 0x21687371

(gdb) p (char[4])0x21687371
$1 = "qsh!"

Or it may be detected with a diagnostic message (similar to double free):

(gdb) bt
#0  0×000000000043ef65 in raise ()
#1  0×0000000000409fc0 in abort ()
#2  0×000000000040bf5b in __libc_message ()
#3  0×0000000000412042 in malloc_printerr ()

#4  0×0000000000416c27 in free ()
#5  0×0000000000400586 in proc ()
#6  0×000000000040067e in bar_four ()
#7  0×000000000040068e in foo_four ()
#8  0×00000000004006a6 in thread_four ()
#9  0×00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#10 0×0000000000432589 in clone ()
#11 0×0000000000000000 in ?? ()

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 235, Linux)

Friday, December 18th, 2015

We first introduced Critical Region pattern in Accelerated Mac OS X Core Dump Analysis training but didn’t submit the pattern itself to the catalog at that time.

A critical region is usually a region of code protected by synchronization objects such as critical sections and mutexes. However, Critical Region analysis pattern is about identifying code regions “sandwiched” between contending function calls (which may or may not involve synchronization objects and corresponding synchronization calls such as identified in Contention patterns), and then identifying any possible shared data referenced by such code regions:

(gdb) thread apply all bt

Thread 6 (Thread 0x7f2665377700 (LWP 17000)):
#0  0x00000000004151a1 in _int_malloc ()
#1  0x0000000000416cf8 in malloc ()
#2  0x00000000004005a4 in proc ()
#3  0x0000000000400604 in bar_two ()
#4  0x0000000000400614 in foo_two ()
#5  0x000000000040062c in thread_two ()
#6  0x00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#7  0x0000000000432589 in clone ()
#8  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f2664b76700 (LWP 17001)):
#0  __lll_unlock_wake_private ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:343
#1  0×000000000041886d in _L_unlock_9670 ()
#2  0×0000000000416d22 in malloc ()
#3  0×00000000004005a4 in proc ()

#4  0×0000000000400641 in bar_three ()
#5  0×0000000000400651 in foo_three ()
#6  0×0000000000400669 in thread_three ()
#7  0×00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#8  0×0000000000432589 in clone ()
#9  0×0000000000000000 in ?? ()

Thread 4 (Thread 0x7f2665b78700 (LWP 16999)):
#0  __lll_lock_wait_private ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
#1  0×0000000000418836 in _L_lock_9558 ()
#2  0×0000000000416c1c in free ()
#3  0×0000000000400586 in proc ()

#4  0×00000000004005c7 in bar_one ()
#5  0×00000000004005d7 in foo_one ()
#6  0×00000000004005ef in thread_one ()
#7  0×00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#8  0×0000000000432589 in clone ()
#9  0×0000000000000000 in ?? ()

Thread 3 (Thread 0x1ab1860 (LWP 16998)):
#0  0x000000000042fed1 in nanosleep ()
#1  0x000000000042fda0 in sleep ()
#2  0x000000000040078a in main ()

Thread 2 (Thread 0x7f2663b74700 (LWP 17003)):
#0  __lll_lock_wait_private ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
#1  0x0000000000418836 in _L_lock_9558 ()
#2  0x0000000000416c1c in free ()
#3  0x0000000000400586 in proc ()
#4  0x00000000004006bb in bar_five ()
#5  0x00000000004006cb in foo_five ()
#6  0x00000000004006e3 in thread_five ()
#7  0x00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#8  0x0000000000432589 in clone ()
#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f2664375700 (LWP 17002)):
#0  0x000000000043ef65 in raise ()
#1  0x0000000000409fc0 in abort ()
#2  0x000000000040bf5b in __libc_message ()
#3  0x0000000000412042 in malloc_printerr ()
#4  0x0000000000416c27 in free ()
#5  0x0000000000400586 in proc ()
#6  0x000000000040067e in bar_four ()
#7  0x000000000040068e in foo_four ()
#8  0x00000000004006a6 in thread_four ()
#9  0x00000000004016c0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#10 0x0000000000432589 in clone ()
#11 0x0000000000000000 in ?? ()

From threads #4 and #5 we can identify one such a region with a shared buffer 0×6b8fc0 which may further point to heap entries.

(gdb) disassemble proc
Dump of assembler code for function proc:
0x00000000004004f0 <+0>: push   %rbp
0x00000000004004f1 <+1>: mov    %rsp,%rbp
0x00000000004004f4 <+4>: push   %rbx
0x00000000004004f5 <+5>: sub    $0x18,%rsp
0x00000000004004f9 <+9>: callq  0x40ac70 <rand>
0x00000000004004fe <+14>: mov    %eax,%ecx
0x0000000000400500 <+16>: mov    $0x68db8bad,%edx
0x0000000000400505 <+21>: mov    %ecx,%eax
0x0000000000400507 <+23>: imul   %edx
0x0000000000400509 <+25>: sar    $0xc,%edx
0x000000000040050c <+28>: mov    %ecx,%eax
0x000000000040050e <+30>: sar    $0x1f,%eax
0x0000000000400511 <+33>: mov    %edx,%ebx
0x0000000000400513 <+35>: sub    %eax,%ebx
0x0000000000400515 <+37>: mov    %ebx,%eax
0x0000000000400517 <+39>: mov    %eax,-0x14(%rbp)
0x000000000040051a <+42>: mov    -0x14(%rbp),%eax
0x000000000040051d <+45>: imul   $0x2710,%eax,%eax
0x0000000000400523 <+51>: mov    %ecx,%edx
0x0000000000400525 <+53>: sub    %eax,%edx
0x0000000000400527 <+55>: mov    %edx,%eax
0x0000000000400529 <+57>: mov    %eax,-0x14(%rbp)
0x000000000040052c <+60>: callq  0x40ac70 <rand>
0x0000000000400531 <+65>: mov    %eax,%ecx
0x0000000000400533 <+67>: mov    $0x68db8bad,%edx
0x0000000000400538 <+72>: mov    %ecx,%eax
0x000000000040053a <+74>: imul   %edx
0x000000000040053c <+76>: sar    $0xc,%edx
0x000000000040053f <+79>: mov    %ecx,%eax
0x0000000000400541 <+81>: sar    $0x1f,%eax
0x0000000000400544 <+84>: mov    %edx,%ebx
0x0000000000400546 <+86>: sub    %eax,%ebx
0x0000000000400548 <+88>: mov    %ebx,%eax
0x000000000040054a <+90>: mov    %eax,-0x18(%rbp)
0x000000000040054d <+93>: mov    -0x18(%rbp),%eax
0x0000000000400550 <+96>: imul   $0x2710,%eax,%eax
0x0000000000400556 <+102>: mov    %ecx,%edx
0x0000000000400558 <+104>: sub    %eax,%edx
0x000000000040055a <+106>: mov    %edx,%eax
0x000000000040055c <+108>: mov    %eax,-0x18(%rbp)
0x000000000040055f <+111>: mov    -0x14(%rbp),%eax
0x0000000000400562 <+114>: cltq
0x0000000000400564 <+116>: mov    0x6b8fc0(,%rax,8),%rax
0x000000000040056c <+124>: test   %rax,%rax
0x000000000040056f <+127>: je     0x400597 <proc+167>
0x0000000000400571 <+129>: mov    -0x14(%rbp),%eax
0x0000000000400574 <+132>: cltq
0x0000000000400576 <+134>: mov    0x6b8fc0(,%rax,8),%rax
0x000000000040057e <+142>: mov    %rax,%rdi
0x0000000000400581 <+145>: callq  0x416bc0 <free>
0×0000000000400586 <+150>: mov    -0×14(%rbp),%eax
0×0000000000400589 <+153>: cltq
0×000000000040058b <+155>: movq   $0×0,0×6b8fc0(,%rax,8)
0×0000000000400597 <+167>: mov    -0×18(%rbp),%eax
0×000000000040059a <+170>: cltq
0×000000000040059c <+172>: mov    %rax,%rdi

0×000000000040059f <+175>: callq  0×416c90 <malloc>
0×00000000004005a4 <+180>: mov    %rax,%rdx
0×00000000004005a7 <+183>: mov    -0×14(%rbp),%eax
0×00000000004005aa <+186>: cltq
0×00000000004005ac <+188>: mov    %rdx,0×6b8fc0(,%rax,8)
0×00000000004005b4 <+196>: jmpq   0×4004f9 <proc+9>
End of assembler dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 6b, Linux)

Friday, December 18th, 2015

This is a Linux variant of NULL Pointer (data) pattern previously described for Mac OS X and Windows platforms:

(gdb) bt
#0  0×0000000000400500 in procA ()
#1  0×000000000040057a in bar_two ()
#2  0×000000000040058a in foo_two ()
#3  0×00000000004005a2 in thread_two ()
#4  0×0000000000401630 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#5  0×00000000004324e9 in clone ()
#6  0×0000000000000000 in ?? ()

(gdb) x/i 0x400500
=> 0x400500 <procA+16>: movl   $0x1,(%rax)

(gdb) info r $rax
rax            0×0 0

(gdb) x $rax
0×0: Cannot access memory at address 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 6a, Linux)

Friday, December 18th, 2015

This is a Linux variant of NULL Pointer (code) pattern previously described for Mac OS X and Windows platforms:

(gdb) bt
#0  0×0000000000000000 in ?? ()
#1  0×0000000000400531 in procB ()
#2  0×00000000004005f8 in bar_four ()
#3  0×0000000000400608 in foo_four ()
#4  0×0000000000400620 in thread_four ()
#5  0×0000000000401630 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6  0×00000000004324e9 in clone ()
#7  0×0000000000000000 in ?? ()

(gdb) disassemble procB
Dump of assembler code for function procB:
0x0000000000400516 <+0>: push   %rbp
0x0000000000400517 <+1>: mov    %rsp,%rbp
0x000000000040051a <+4>: sub    $0x10,%rsp
0x000000000040051e <+8>: movq   $0x0,-0x8(%rbp)
0x0000000000400526 <+16>: mov    -0x8(%rbp),%rdx
0x000000000040052a <+20>: mov    $0x0,%eax
0x000000000040052f <+25>: callq  *%rdx
0×0000000000400531 <+27>: leaveq
0×0000000000400532 <+28>: retq
End of assembler dump.

(gdb) info r rdx
rdx            0×0 0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 78a, Linux)

Tuesday, December 15th, 2015

This is a Linux variant of Divide by Zero (user mode) pattern previously described for Mac OS X and Windows platforms:

GNU gdb (GDB)
[...]
Program terminated with signal 8, Arithmetic exception.
#0 0×000000000040056f in procD ()

(gdb) x/i $rip
=> 0x40056f <procD+18>: idivl -0×8(%rbp)

(gdb) info r $rax
rax 0x1 1

(gdb) x/w $rbp-0x8
0x7f0f6806bd28: 0×00000000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 14, Linux)

Monday, December 14th, 2015

This is a variant of Spiking Thread pattern previously described for Mac OS X and Windows platforms:

(gdb) info threads
Id Target Id Frame
6 LWP 3712 0x00000000004329d1 in nanosleep ()
5 LWP 3717 0×00000000004007a3 in isnan ()
4 LWP 3716 0×00000000004329d1 in nanosleep ()
3 LWP 3715 0×00000000004329d1 in nanosleep ()
2 LWP 3714 0×00000000004329d1 in nanosleep ()
* 1 LWP 3713 0×00000000004329d1 in nanosleep ()

We notice a non-waiting thread and switch to it:

(gdb) thread 5
[Switching to thread 5 (LWP 3717)]
#0 0x00000000004007a3 in isnan ()

(gdb) bt
#0 0×00000000004007a3 in isnan ()
#1 0×0000000000400743 in sqrt ()
#2 0×0000000000400528 in procB ()
#3 0×0000000000400639 in bar_five ()
#4 0×0000000000400649 in foo_five ()
#5 0×0000000000400661 in thread_five ()
#6 0×0000000000403e30 in start_thread ()
#7 0×0000000000435089 in clone ()
#8 0×0000000000000000 in ?? ()

If we disassemble the return address for procB function to come back from sqrt call we see an infinite loop:

(gdb) disassemble 0x400528
Dump of assembler code for function procB:
0x0000000000400500 <+0>: push   %rbp
0x0000000000400501 <+1>: mov    %rsp,%rbp
0x0000000000400504 <+4>: sub    $0x20,%rsp
0x0000000000400508 <+8>: movabs $0x3fd5555555555555,%rax
0x0000000000400512 <+18>: mov    %rax,-0x8(%rbp)
0×0000000000400516 <+22>: mov    -0×8(%rbp),%rax
0×000000000040051a <+26>: mov    %rax,-0×18(%rbp)
0×000000000040051e <+30>: movsd  -0×18(%rbp),%xmm0
0×0000000000400523 <+35>: callq  0×400710 <sqrt>
0×0000000000400528 <+40>: movsd  %xmm0,-0×18(%rbp)
0×000000000040052d <+45>: mov    -0×18(%rbp),%rax
0×0000000000400531 <+49>: mov    %rax,-0×8(%rbp)
0×0000000000400535 <+53>: jmp    0×400516 <procB+22>
End of assembler dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 210)

Monday, September 8th, 2014

Here we provide another variant of a general Wait Chain pattern related to RtlAcquireResourceShared and RtlAcquireResourceExclusive calls:

THREAD fffffa8052d66060  Cid 03c0.3240  Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa804a79ad50  Semaphore Limit 0x7fffffff
Impersonation token:  fffff8a01b19d060 (Level Impersonation)
DeviceMap                 fffff8a0035276c0
Owning Process            fffffa804a16b260       Image:         lsm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      73343513       Ticks: 1460259 (0:06:20:16.546)
Context Switch Count      17             IdealProcessor: 1
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0x000000007735fbf0)
Stack Init fffff8800e870db0 Current fffff8800e870900
Base fffff8800e871000 Limit fffff8800e86b000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a
fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f
fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde
fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0e870c20)
00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa
00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61
00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21
00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202
00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65
00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a
00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d
00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149
00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d
00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b
00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8
00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

These functions are undocumented but ReactOS source code shows they all take a pointer to RTL_RESOURCE structure which has handles to a shared and exclusive semaphores:

RTL_CRITICAL_SECTION Lock
HANDLE SharedSemaphore
ULONG SharedWaiters
HANDLE ExclusiveSemaphore
ULONG ExclusiveWaiters
LONG NumberActive
HANDLE OwningThread
ULONG TimeoutBoost
PVOID DebugInfo

To double check that we disassemble RtlAcquireResourceShared and check the return address from NtWaitForSingleObject call (00000000`773470b4):

0: kd> .thread /r /p fffffa8052d66060
Implicit thread is now fffffa80`52d66060
Implicit process is now fffffa80`4a16b260
Loading User Symbols
..........................................

0: kd> uf ntdll!RtlAcquireResourceShared
[...]
ntdll!RtlAcquireResourceShared+0xc2:
00000000`773470a6 488b4b28 mov rcx,qword ptr [rbx+28h]
00000000`773470aa 4c8bc6 mov r8,rsi
00000000`773470ad 33d2 xor edx,edx
00000000`773470af e83ca20400 call ntdll!NtWaitForSingleObject (00000000`773912f0)
00000000`773470b4 3d02010000 cmp eax,102h
00000000`773470b9 0f8402800600 je ntdll! ?? ::FNODOBFM::`string’+0×12629 (00000000`773af0c1)
[…]
ntdll!RtlAcquireResourceShared:
00000000`77352af0 48895c2420 mov qword ptr [rsp+20h],rbx
00000000`77352af5 57 push rdi
00000000`77352af6 4883ec30 sub rsp,30h
00000000`77352afa 448b4944 mov r9d,dword ptr [rcx+44h]
00000000`77352afe 0fb6fa movzx edi,dl
00000000`77352b01 488bd9 mov rbx,rcx
00000000`77352b04 4585c9 test r9d,r9d
00000000`77352b07 0f88a7000000 js ntdll!RtlAcquireResourceShared+0×65 (00000000`77352bb4)
[…]

We see the handle is taken from [RBX+28] and we see that RBX was saved at the function prologue and then the value of RCX was assigned to RBX. RCX as the first calling convention parameter should be a pointer to RTL_RESOURCE which has RTL_CRITICAL_SECTION as the first member and its size is 0×28:

0: kd> dt ntdll!_RTL_CRITICAL_SECTION
ntdll!_RTL_CRITICAL_SECTION
+0x000 DebugInfo        : Ptr64 _RTL_CRITICAL_SECTION_DEBUG
+0x008 LockCount        : Int4B
+0x00c RecursionCount   : Int4B
+0x010 OwningThread     : Ptr64 Void
+0x018 LockSemaphore    : Ptr64 Void
+0x020 SpinCount        : Uint8B

Therefore [RBX+28] contains SharedSemaphore field which is assigned to RCX as a first parameter to NtWaitForSingleObject. The similar fragment of RtlAcquireResourceExclusive has [RBX+36] which 0×10 further than 0×28 and corresponds to ExclusiveSemaphore handle field:

ntdll!RtlAcquireResourceExclusive+0xd2:
00000000`770c2a12 488b4b38        mov     rcx,qword ptr [rbx+38h]
00000000`770c2a16 4c8bc6          mov     r8,rsi
00000000`770c2a19 33d2            xor     edx,edx
00000000`770c2a1b e8d0e80400      call    ntdll!NtWaitForSingleObject (00000000`771112f0)
00000000`770c2a20 3d02010000      cmp     eax,102h
00000000`770c2a25 0f8401c60600    je      ntdll! ?? ::FNODOBFM::`string’+0×12591 (00000000`7712f02c)

So we just need to know the vale of RBX and dump the structure to find OwningThread field. We can either calculate it from RSP or use /c switch with .frame command:

0: kd> kn
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP          RetAddr           Call Site
00 fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a
01 fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
02 fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f
03 fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde
04 fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13
05 00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa
06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
07 00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61
08 00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21
09 00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202
0a 00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65
0b 00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a
0c 00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d
0d 00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14
0e 00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
0f 00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149
10 00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d
11 00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
12 00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
13 00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b
14 00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8
15 00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
16 00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .frame /c 6
06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
rax=0000000000000000 rbx=00000000023ac128 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000077472410 rdi=0000000000000001
rip=00000000773470b4 rsp=00000000022ae6d0 rbp=0000000000000000
r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=0000000000000000 r12=29406b2a1a85bd43 r13=0000000000000009
r14=000000000000000c r15=00000000022aef20
iopl=0         nv up di pl nz na pe nc
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000
ntdll!RtlAcquireResourceShared+0xd0:
00000000`773470b4 3d02010000      cmp     eax,102h

0: kd> dp rbx+28 L10
00000000`023ac150  00000000`00001244 00000000`000001b5
00000000`023ac160  00000000`00000f3c ffffffff`00000000
00000000`023ac170  00000000`000021a0 00000000`00000000
00000000`023ac180  00000000`02735fc0 00000000`00000001
00000000`023ac190  00000000`00000000 01cf07ac`9fa06d27
00000000`023ac1a0  00000000`00000000 00000000`00000000
00000000`023ac1b0  ffffffff`ffffffff 00000000`00000000
00000000`023ac1c0  00000000`00000000 00000000`00000000

We check all these handles (OwnerThread seems comes earlier with NumberActive field missing but that could just differences between the old x86 structure implemented in ReactOS and x64 Windows):

0: kd> !handle 00000000`00001244

PROCESS fffffa804a16b260
SessionId: 0  Cid: 03c0    Peb: 7fffffdc000  ParentCid: 0350
DirBase: 195950000  ObjectTable: fffff8a0032424e0  HandleCount: 5252.
Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

1244: Object: fffffa804a79ad50  GrantedAccess: 00100003 Entry: fffff8a022b39910
Object: fffffa804a79ad50  Type: (fffffa8048fc8790) Semaphore
ObjectHeader: fffffa804a79ad20 (new version)
HandleCount: 1  PointerCount: 438

0: kd> !handle 00000000`00000f3c

PROCESS fffffa804a16b260
SessionId: 0  Cid: 03c0    Peb: 7fffffdc000  ParentCid: 0350
DirBase: 195950000  ObjectTable: fffff8a0032424e0  HandleCount: 5252.
Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

0f3c: Object: fffffa804fa81f60  GrantedAccess: 00100003 Entry: fffff8a02cd3ecf0
Object: fffffa804fa81f60  Type: (fffffa8048fc8790) Semaphore
ObjectHeader: fffffa804fa81f30 (new version)
HandleCount: 1  PointerCount: 1

0: kd> !thread -t 00000000`000021a0 3f
THREAD fffffa804d5d51b0  Cid 03c0.21a0  Teb: 000007fffff9c000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
fffffa804d5d5578  Semaphore Limit 0×1
Waiting for reply to ALPC Message fffff8a02c9a9500 : queued at port fffffa804ac4e7d0 : owned by process fffffa804adc8730
Not impersonating
DeviceMap                 fffff8a0000088c0
Owning Process            fffffa804a16b260       Image:         lsm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      73337319       Ticks: 1466453 (0:06:21:53.328)
Context Switch Count      69             IdealProcessor: 1
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0×000000007735fbf0)
Stack Init fffff8800aa1fdb0 Current fffff8800aa1f600
Base fffff8800aa20000 Limit fffff8800aa1a000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

Child-SP          RetAddr           Call Site
fffff880`0aa1f640 fffff800`01c76972 nt!KiSwapContext+0x7a
fffff880`0aa1f780 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
fffff880`0aa1f810 fffff800`01ca25af nt!KeWaitForSingleObject+0x19f
fffff880`0aa1f8b0 fffff800`01f968b6 nt!AlpcpSignalAndWait+0x8f
fffff880`0aa1f960 fffff800`01f95fb0 nt!AlpcpReceiveSynchronousReply+0x46
fffff880`0aa1f9c0 fffff800`01f93dab nt!AlpcpProcessSynchronousRequest+0x33d
fffff880`0aa1fb00 fffff800`01c801d3 nt!NtAlpcSendWaitReceivePort+0x1ab
fffff880`0aa1fbb0 00000000`77391b0a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0aa1fc20)
00000000`01dddb48 000007fe`fd8c8306 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01dddb50 000007fe`fd8c2a02 RPCRT4!LRPC_CCALL::SendReceive+0x156
00000000`01dddc10 000007fe`ff5b28c0 RPCRT4!I_RpcSendReceive+0x42
00000000`01dddc40 000007fe`ff5b282f ole32!ThreadSendReceive+0x40 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 5003]
00000000`01dddc90 000007fe`ff5b265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4454]
00000000`01dddd30 000007fe`ff46daaa ole32!CRpcChannelBuffer::SendReceive2+0x11b [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4074]
00000000`01dddef0 000007fe`ff46da0c ole32!CAptRpcChnl::SendReceive+0x52 [d:\w7rtm\com\ole32\com\dcomrem\callctrl.cxx @ 603]
00000000`01dddfc0 000007fe`ff5b205d ole32!CCtxComChnl::SendReceive+0x68 [d:\w7rtm\com\ole32\com\dcomrem\ctxchnl.cxx @ 734]
00000000`01dde070 000007fe`fd96b949 ole32!NdrExtpProxySendReceive+0x45 [d:\w7rtm\com\rpc\ndrole\proxy.cxx @ 1932]
00000000`01dde0a0 000007fe`ff5b21d0 RPCRT4!NdrpClientCall3+0x2e2
00000000`01dde360 000007fe`ff46d8a2 ole32!ObjectStublessClient+0x11d [d:\w7rtm\com\rpc\ndrole\amd64\stblsclt.cxx @ 621]
00000000`01dde6f0 00000000`ff417d26 ole32!ObjectStubless+0x42 [d:\w7rtm\com\rpc\ndrole\amd64\stubless.asm @ 117]
00000000`01dde740 00000000`ff4186ba lsm!CTSSession::Disconnect+0x3a5
00000000`01dde810 000007fe`fd8bff85 lsm!RpcDisconnect+0x15e
00000000`01dde850 000007fe`fd96b68e RPCRT4!Invoke+0x65
00000000`01dde8a0 000007fe`fd8a92e0 RPCRT4!Ndr64StubWorker+0x61b
00000000`01ddee60 000007fe`fd8b3254 RPCRT4!NdrServerCallAll+0x40
00000000`01ddeeb0 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0x14
00000000`01ddeee0 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x146
00000000`01ddf000 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0x149
00000000`01ddf0e0 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0x20d
00000000`01ddf210 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0x3bf
00000000`01ddf350 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
00000000`01ddf3e0 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0x26b
00000000`01ddf470 00000000`7713652d ntdll!TppWorkerThread+0x3f8
00000000`01ddf770 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
00000000`01ddf7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

We see the wait chain continues with waiting for an ALPC request.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 203)

Saturday, December 7th, 2013

Sometimes we look at a stack trace collection or it’s predicate subset and recognize that one of parameters is actually the same structure address or handle. We call this pattern Shared Structure. In x64 case we may possibly see it from the return address backwards disassembly (ub WinDbg command) but in x86 case most of the time we can spot that directly from the verbose stack trace, like in the snippet below (unless a parameter memory slot was reused):

THREAD 830f9990 Cid 0428.0e94 Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
[...]
ChildEBP RetAddr  Args to Child
0031f74c 7784b071 00000000 00000000 7ffdb000 ntdll!RtlpWaitOnCriticalSection+0x154
0031f774 00a91150 00a9b7a8 00000000 00a91452 ntdll!RtlEnterCriticalSection+0×152
WARNING: Stack unwind information not available. Following frames may be wrong.
0031f7c8 76113833 7ffdb000 0031f814 7784a9bd Application+0×1150
0031f7d4 7784a9bd 7ffdb000 003114bf 00000000 kernel32!BaseThreadInitThunk+0xe
0031f814 00000000 00a914a9 7ffdb000 00000000 ntdll!_RtlUserThreadStart+0×23

THREAD 886ee030 Cid 0428.0ef4 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
[...]
ChildEBP RetAddr  Args to Child
0098fcb8 77f881b1 00000000 00000000 001614a0 ntdll!RtlpUnWaitCriticalSection+0x1b
0098fce0 00a9102e 00a9b7a8 00000000 00000000 ntdll!RtlEnterCriticalSection+0×152
WARNING: Stack unwind information not available. Following frames may be wrong.
0098fd28 00a91275 0098fd3c 76113833 001614a0 Application+0×102e
0098fd30 76113833 001614a0 0098fd7c 7784a9bd Application+0×1275
0098fd3c 7784a9bd 001614a0 009811d7 00000000 kernel32!BaseThreadInitThunk+0xe
0098fd7c 00000000 00a911ff 001614a0 00000000 ntdll!_RtlUserThreadStart+0×23

In case of multiple exceptions or even a single exception on one thread involving invalid access to a structure field the reference to the same structure on a different thread may point to possible synchronization problems.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 201)

Monday, November 4th, 2013

Sometimes there are similar crashes in multiplatform products where only some potion of Crash Signature is similar. We call such a pattern Crash Signature Invariant, for example:

x86: cmp dword ptr [eax], 1
x64: cmp dword ptr [r10]. 1

One crash dump had the following condensed stack trace: 

0: kd> kc
DriverA
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServiceCopyEnd

with the following faulting instruction:

DriverA+0x1234:
cmp     dword ptr [r11],1 ds:002b:00000000`00000000=????????

A search for DriverA led to this x86 crash analysed some time ago:

0: kd> kc
DriverA
nt!IopfCallDriver
win32k!GreDeviceIoControl
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!xxxWrapRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServicePostCall

0: kd> r
DtiverA+0x1423:
cmp     dword ptr [ecx],1    ds:0023:00000000=????????

We see common function names on both stack traces and overall flow is the same (only 3 functions are omitted in x64 trace); we see the same NULL pointer dereference for the same comparison instruction with the same comparison operand, #1.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Malware Analysis Patterns (Part 21)

Saturday, February 9th, 2013

Hooksware pattern originally came from memory dump analysis pattern catalog and is too general for malware analysis pattern catalog. So we decided to factor out 3 separate patterns. The first one is called Patched Code and includes cases such as in-place patching:

0:004> u ntdll!ZwQueryDirectoryFile
ntdll!ZwQueryDirectoryFile:
77814db4 b8da000000      mov     eax,0DAh
77814db9 bae8af0500      mov     edx,5AFE8h
77814dbe ff12            call    dword ptr [edx]
77814dc0 c22c00          ret     2Ch
77814dc3 90              nop
ntdll!NtQueryDirectoryObject:
77814dc4 b8db000000      mov     eax,0DBh
77814dc9 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
77814dce ff12            call    dword ptr [edx]

and detour patching:

0:004> u wininet!InternetReadFile
wininet!InternetReadFile:
7758654b e98044ac88      jmp     0004a9d0
77586550 83ec24          sub     esp,24h
77586553 53              push    ebx
77586554 56              push    esi
77586555 57              push    edi
77586556 33ff            xor     edi,edi
77586558 393db8116277    cmp     dword ptr [wininet!GlobalDataInitialized (776211b8)],edi
7758655e 897df4          mov     dword ptr [ebp-0Ch],edi

In case of WinDbg such pattern is usually detected on the crash spot such as from RIP Stack Trace or from !chkimg command output.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Malware Analysis Patterns (Part 11)

Sunday, January 20th, 2013

Injected code address may not be in the address ranges of loaded modules. In such cases, in the execution call history we would see plain EIP and RIP return addresses on stack traces. We call this pattern RIP Stack Trace partly because we have seen these addresses after something had gone wrong and a process crashed:

0:005> k
ChildEBP RetAddr
02aec974 77655620 ntdll!KiFastSystemCallRet
02aec978 77683c62 ntdll!NtWaitForSingleObject+0xc
02aec9fc 77683d4b ntdll!RtlReportExceptionEx+0x14b
02aeca3c 7769fa87 ntdll!RtlReportException+0x3c
02aeca50 7769fb0d ntdll!RtlpTerminateFailureFilter+0x14
02aeca5c 775f9bdc ntdll!RtlReportCriticalFailure+0x6b
02aeca70 775f4067 ntdll!_EH4_CallFilterFunc+0x12
02aeca98 77655f79 ntdll!_except_handler4+0x8e
02aecabc 77655f4b ntdll!ExecuteHandler2+0x26
02aecb6c 77655dd7 ntdll!ExecuteHandler+0x24
02aecb6c 7769faf8 ntdll!KiUserExceptionDispatcher+0xf
02aecee0 776a0704 ntdll!RtlReportCriticalFailure+0x5b
02aecef0 776a07f2 ntdll!RtlpReportHeapFailure+0x21
02aecf24 7766b1a5 ntdll!RtlpLogHeapFailure+0xa1
02aecf6c 7765730a ntdll!RtlpCoalesceFreeBlocks+0x4b9
02aed064 77657545 ntdll!RtlpFreeHeap+0x1e2
02aed080 75e47e4b ntdll!RtlFreeHeap+0x14e
02aed0c8 77037277 kernel32!GlobalFree+0x47
02aed0dc 774b4a1f ole32!ReleaseStgMedium+0x124
02aed0f0 77517feb urlmon!ReleaseBindInfo+0x4c
02aed100 774d9a87 urlmon!CINet::ReleaseCNetObjects+0x3d
02aed118 774d93f0 urlmon!CINetHttp::OnWininetRequestHandleClosing+0x60
02aed12c 76432078 urlmon!CINet::CINetCallback+0x2de
02aed274 76438f5d wininet!InternetIndicateStatus+0xfc
02aed2a4 7643937a wininet!HANDLE_OBJECT::~HANDLE_OBJECT+0xc9
02aed2c0 7643916b wininet!INTERNET_CONNECT_HANDLE_OBJECT::~INTERNET_CONNECT_HANDLE_OBJECT+0x209
02aed2cc 76438d5e wininet!HTTP_REQUEST_HANDLE_OBJECT::`vector deleting destructor'+0xd
02aed2dc 76434e72 wininet!HANDLE_OBJECT::Dereference+0x22
02aed2e8 76439419 wininet!DereferenceObject+0x21
02aed310 76439114 wininet!_InternetCloseHandle+0x9d
02aed330 0004aaaf wininet!InternetCloseHandle+0x11e
WARNING: Frame IP not in any known module. Following frames may be wrong.
02aed33c 774c5d25 0×4aaaf
02aed358 774c5d95 urlmon!CINet::TerminateRequest+0×82
02aed364 774c5d7c urlmon!CINet::MyUnlockRequest+0×10
02aed370 774c5d63 urlmon!CINetProtImpl::UnlockRequest+0×10
02aed37c 774c5d49 urlmon!CINetEmbdFilter::UnlockRequest+0×11
02aed388 774b743d urlmon!CINet::UnlockRequest+0×13
02aed394 774b73e1 urlmon!COInetProt::UnlockRequest+0×11
02aed3a8 774b7530 urlmon!CTransaction::UnlockRequest+0×36
02aed3b4 774b74e0 urlmon!CTransData::~CTransData+0×3a
02aed3c0 774b74c9 urlmon!CTransData::`scalar deleting destructor’+0xd
02aed3d8 774e221f urlmon!CTransData::Release+0×25
02aed3e0 774b6d0a urlmon!CReadOnlyStreamDirect::~CReadOnlyStreamDirect+0×1a
02aed3ec 774b7319 urlmon!CReadOnlyStreamDirect::`vector deleting destructor’+0xd
02aed404 774b72be urlmon!CReadOnlyStreamDirect::Release+0×25
02aed410 774b71f4 urlmon!CBinding::~CBinding+0xb9
02aed41c 774b71dd urlmon!CBinding::`scalar deleting destructor’+0xd
02aed434 6b20b0e8 urlmon!CBinding::Release+0×25
02aed448 6b20b0ba mshtml!ATL::AtlComPtrAssign+0×2b
02aed458 6b20b8de mshtml!ATL::CComPtr<IBindCallbackInternal>::operator=+0×15
02aed464 6b20b8aa mshtml!CBindingXSSFilter::TearDown+0×2b
02aed46c 6b20b887 mshtml!BindingXSSFilter_TearDown+0×19
02aed478 6b0da61a mshtml!CStreamProxy::Passivate+0×12
02aed484 6b0ddf3a mshtml!CBaseFT::Release+0×1d
02aed4ac 6b0e0b70 mshtml!CDwnBindData::TerminateBind+0×11d
02aed4b8 6b11a2a9 mshtml!CDwnBindData::TerminateOnApt+0×14
02aed4ec 6b105066 mshtml!GlobalWndOnMethodCall+0xfb
02aed50c 7742fd72 mshtml!GlobalWndProc+0×183
02aed538 7742fe4a user32!InternalCallWinProc+0×23
02aed5b0 7743018d user32!UserCallWinProcCheckWow+0×14b
02aed614 7743022b user32!DispatchMessageWorker+0×322
02aed624 6ecac1d5 user32!DispatchMessageW+0xf
02aef72c 6ec5337e ieframe!CTabWindow::_TabWindowThreadProc+0×54c
02aef7e4 760f426d ieframe!LCIETab_ThreadProc+0×2c1
02aef7f4 75e4d0e9 iertutil!CIsoScope::RegisterThread+0xab
02aef800 776319bb kernel32!BaseThreadInitThunk+0xe
02aef840 7763198e ntdll!__RtlUserThreadStart+0×23
02aef858 00000000 ntdll!_RtlUserThreadStart+0×1b

However, such addresses need to be checked whether they belong to .NET CLR JIT code.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Optimization Patterns

Friday, November 16th, 2012

A page to reference all different kinds of optimization patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Improbable Occurrences (Part 1)

Saturday, October 27th, 2012

I was analyzing a raw thread stack when came upon this symbolic address which I thought was coincidental:

363b0030  77777777 advapi32!LsaEnumerateAccountRights+0×56

Forward disasssembly makes sense, isn’t it? And every instruction seems have a purpose :-)

0:000> u 77777777
advapi32!LsaEnumerateAccountRights+0×56:
77777777 a4              movs    byte ptr es:[edi],byte ptr [esi]
77777778 fc              cld
77777779 ffc3            inc     ebx
7777777b 8b65e8          mov     esp,dword ptr [ebp-18h]
7777777e ff75e0          push    dword ptr [ebp-20h]
77777781 ff15e4187377    call    dword ptr [advapi32!_imp__I_RpcMapWin32Status (777318e4)]
77777787 50              push    eax
77777788 e8c6f6fbff      call    advapi32!LsapApiReturnResult (77736e53)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 23a, Mac OS X)

Tuesday, May 29th, 2012

This is a Mac OS X / GDB counterpart to Double Free (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e13f84c in free ()
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)

(gdb) x/2i 0x00000001035a8ef4-8
0x1035a8eec : mov -0×20(%rbp),%edi
0×1035a8eef : callq 0×1035a8f06

(gdb) frame 3
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)
at .../DoubleFree/main.c:23
23 free(p2);
Current language: auto; currently minimal

(gdb) x/g $rbp-0x20
0x7fff631a7ae0: 0x00007fe6a8801400

(gdb) x/2w 0x00007fe6a8801400
0x7fe6a8801400: 0x00000000 0xb0000000

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    free(p2);

    free(p1);

    free(p2);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 14, Mac OS X)

Wednesday, May 9th, 2012

This is a Mac OS X / GDB counterpart to Spiking Thread pattern previously described for Windows platforms:

(gdb) info threads
4 0×00007fff85b542df in sqrt$fenv_access_off ()
3 0×00007fff8616ee42 in __semwait_signal ()
2 0×00007fff8616ee42 in __semwait_signal ()
* 1 0×00007fff8616ee42 in __semwait_signal ()

We notice a non-waiting thread and switch to it:

(gdb) thread 4
[Switching to thread 4 (core thread 3)]
0x00007fff85b542df in sqrt$fenv_access_off ()

(gdb) bt
#0  0x00007fff85b542df in sqrt$fenv_access_off ()
#1  0×000000010cc85dc9 in thread_three (arg=0×7fff6c884ac0)
#2  0×00007fff8fac68bf in _pthread_start ()
#3  0×00007fff8fac9b75 in thread_start ()

If we disassemble the return address for thread_three function to come back from sqrt call we see an infinite loop:

(gdb) disass 0x000000010cc85dc9
Dump of assembler code for function thread_three:
0x000000010cc85db0 <thread_three+0>: push   %rbp
0×000000010cc85db1 <thread_three+1>: mov    %rsp,%rbp
0×000000010cc85db4 <thread_three+4>: sub    $0×10,%rsp
0×000000010cc85db8 <thread_three+8>: mov    %rdi,-0×10(%rbp)
0×000000010cc85dbc <thread_three+12>: mov    -0×10(%rbp),%ax
0×000000010cc85dc0 <thread_three+16>: movsd  (%rax),%xmm0
0×000000010cc85dc4 <thread_three+20>: callq  0×10cc85eac <dyld_stub_sqrt>
0×000000010cc85dc9 <thread_three+25>: mov    -0×10(%rbp),%rax
0×000000010cc85dcd <thread_three+29>: movsd  %xmm0,(%rax)
0×000000010cc85dd1 <thread_three+33>: jmpq   0×10cc85dbc <thread_three+12>
End of assembler dump.

Here’s the source code of the modeling application:

void * thread_one (void *arg)

{

    while (1)

    {

       sleep (1);

    }

 

    return 0;

}

 

void * thread_two (void *arg)

{

    while (1)

    {

        sleep (2);

    }

 

    return 0;

}

 

void * thread_three (void *arg)

{

    while (1)

    {

        *(double*)arg=sqrt(*(double *)arg);

    }

 

    return 0;

}

 

int main(int argc, const char * argv[])

{

    pthread_t threadID_one, threadID_two, threadID_three;

 

    double result = 0xffffffff;

 

    pthread_create (&threadID_one, NULL, thread_one, NULL);

    pthread_create (&threadID_two, NULL, thread_two, NULL);

    pthread_create (&threadID_three, NULL, thread_three,

       &result);

 

    pthread_join(threadID_three, NULL);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6a, Mac OS X)

Thursday, May 3rd, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (code) pattern previously described for Windows platforms:

(gdb) bt
#0 0×0000000000000000 in ?? ()
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) disass 0×000000010e8cce73-3 0×000000010e8cce73
Dump of assembler code from 0×10e8cce70 to 0×10e8cce73:
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) info r rdi
rdi 0x7fff6e4cbac0 140735043910336

(gdb) x/2 0x7fff6e4cbac0
0x7fff6e4cbac0: 0x0000000a 0×00000000

(gdb) p/x *($rdi+8)
$7 = 0×0

(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) ptype MYSTRUCT
type = struct _MyStruct_tag {
int data;
PFUNC pfunc;
}

(gdb) print {MYSTRUCT}0×7fff6e4cbac0
$2 = {data = 10, pfunc = 0}

Here’s the source code of the modeling application:

typedef void (*PFUNC)(void);

 

typedef struct _MyStruct_tag

{

    int   data;

    PFUNC pfunc;

} MYSTRUCT;

 

void bar(MYSTRUCT *ps)

{

    ps->pfunc();

}

 

void foo(MYSTRUCT *ps)

{

    bar(ps);

}

 

int main(int argc, const char * argv[])

{

    MYSTRUCT pstruct = {10, NULL};

 

    foo(&pstruct);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Forthcoming 2nd edition of Memory Dump Analysis Anthology, Volume 1

Sunday, April 15th, 2012

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

  • Title: Memory Dump Analysis Anthology, Volume 1
  • Author: Dmitry Vostokov
  • Publisher: OpenTask (Summer 2012)
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 800 pages
  • ISBN-13: 978-1-908043-35-1
  • Hardcover: 800 pages
  • ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Learning x86/x64 Assembly Language in the Context of Windows Debugging

Thursday, April 12th, 2012

Due to many questions on recommended books to learn assembly language asked during Accelerated Windows Memory Dump Analysis training sessions we provide these references:

Windows Debugging: Practical Foundations
x64 Windows Debugging: Practical Foundations

Each book can be read independently although some platform-independent content overlaps. x64 bit book focuses on 64-bit only.

We believe these books provide all necessary motivation, context and practical foundation for other in-depth assembly language textbooks on the market.

I’m also working on the similar book for x64 Mac OS X.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 6b, Mac OS X)

Sunday, March 25th, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (data) pattern previously described for Windows platforms:

(gdb) bt
#0  0×000000010d3b0e90 in bar () at main.c:15
#1  0×000000010d3b0ea9 in foo () at main.c:20
#2  0×000000010d3b0ec4 in main (argc=1,
argv=0×7fff6cfafbf8) at main.c:25

(gdb) disassemble
Dump of assembler code for function bar:
0x000000010d3b0e80 <bar+0>: push   %rbp
0×000000010d3b0e81 <bar+1>: mov    %rsp,%rbp
0×000000010d3b0e84 <bar+4>: movq   $0×0,-0×8(%rbp)
0×000000010d3b0e8c <bar+12>: mov    -0×8(%rbp),%rax
0×000000010d3b0e90 <bar+16>: movl   $0×1,(%rax)
0×000000010d3b0e96 <bar+22>: pop    %bp
0×000000010d3b0e97 <bar+23>: retq
End of assembler dump.

(gdb) p/x $rax
$1 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis