Software Diagnostics Library » Crash Dump Analysis

Crash Dump Analysis Patterns (Part 178)

Wednesday, June 27th, 2012

One of the frequent problems is an access violation at an address that belongs to Unloaded Module. Here’s an example that recently happened on our machine during an auto-update of the popular software package so we immediately attached a debugger after seeing WER dialog box:

0:000> ~*k

. 0 Id: bc8.bcc Suspend: 1 Teb: 7efdd000 Unfrozen ChildEBP RetAddr 0035f1c4 771a0bdd ntdll!ZwWaitForMultipleObjects+0x15 0035f260 75771a2c KERNELBASE!WaitForMultipleObjectsEx+0x100 0035f2a8 75774208 kernel32!WaitForMultipleObjectsExImplementation+0xe0 0035f2c4 757980a4 kernel32!WaitForMultipleObjects+0x18 0035f330 75797f63 kernel32!WerpReportFaultInternal+0x186 0035f344 75797858 kernel32!WerpReportFault+0x70 0035f354 757977d7 kernel32!BasepReportFault+0x20 0035f3e0 77ec74df kernel32!UnhandledExceptionFilter+0x1af 0035f3e8 77ec73bc ntdll!__RtlUserThreadStart+0x62 0035f3fc 77ec7261 ntdll!_EH4_CallFilterFunc+0x12 0035f424 77eab459 ntdll!_except_handler4+0x8e 0035f448 77eab42b ntdll!ExecuteHandler2+0x26 0035f46c 77eab3ce ntdll!ExecuteHandler+0x24 0035f4f8 77e60133 ntdll!RtlDispatchException+0x127 0035f4f8 73eb2200 ntdll!KiUserExceptionDispatcher+0xf WARNING: Frame IP not in any known module. Following frames may be wrong. 0035f844 76e462fa <Unloaded_fpb.tmp>+0×12200 0035f870 76e46d3a USER32!InternalCallWinProc+0×23 0035f8e8 76e4965e USER32!UserCallWinProcCheckWow+0×109 0035f92c 76e496c5 USER32!SendMessageWorker+0×581 0035f950 7269c05c USER32!SendMessageW+0×7f 0035f9ec 7270be62 comctl32!CCSendNotify+0xc19 0035fa28 75f6f52a comctl32!SendNotify+0×36 0035fa4c 75f61d66 SHELL32!SetAppStartingCursor+0×6d 0035fa64 75f61ee2 SHELL32!CShellExecute::ExecuteNormal+0×16 0035fa78 75f61e70 SHELL32!ShellExecuteNormal+0×33 0035fa90 75f53cd0 SHELL32!ShellExecuteExW+0×62 0035fae4 003e2211 SHELL32!ShellExecuteW+0×77 0035fbc4 77e838be InstallFlashPlayer+0×2211 0035fcb4 77e83492 ntdll!RtlpFreeHeap+0xbb1 0035fcd4 757714dd ntdll!RtlFreeHeap+0×142 0035fce8 003f0324 kernel32!HeapFree+0×14 0035fd80 003f0241 InstallFlashPlayer+0×10324 0035fe10 7577339a InstallFlashPlayer+0×10241 0035fe1c 77e89ef2 kernel32!BaseThreadInitThunk+0xe 0035fe5c 77e89ec5 ntdll!__RtlUserThreadStart+0×70 0035fe74 00000000 ntdll!_RtlUserThreadStart+0×1b

1 Id: bc8.6b0 Suspend: 2 Teb: 7efda000 Unfrozen ChildEBP RetAddr 03e1f9e0 77ea2f51 ntdll!ZwWaitForMultipleObjects+0x15 03e1fb74 7577339a ntdll!TppWaiterpThread+0x33d 03e1fb80 77e89ef2 kernel32!BaseThreadInitThunk+0xe 03e1fbc0 77e89ec5 ntdll!__RtlUserThreadStart+0x70 03e1fbd8 00000000 ntdll!_RtlUserThreadStart+0x1b

2 Id: bc8.8dc Suspend: 2 Teb: 7efd7000 Unfrozen ChildEBP RetAddr 03f5fd50 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12 03f5feb0 7577339a ntdll!TppWorkerThread+0x216 03f5febc 77e89ef2 kernel32!BaseThreadInitThunk+0xe 03f5fefc 77e89ec5 ntdll!__RtlUserThreadStart+0x70 03f5ff14 00000000 ntdll!_RtlUserThreadStart+0x1b

3 Id: bc8.944 Suspend: 2 Teb: 7efaf000 Unfrozen ChildEBP RetAddr 0416f8b4 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12 0416fa14 7577339a ntdll!TppWorkerThread+0x216 0416fa20 77e89ef2 kernel32!BaseThreadInitThunk+0xe 0416fa60 77e89ec5 ntdll!__RtlUserThreadStart+0x70 0416fa78 00000000 ntdll!_RtlUserThreadStart+0x1b

Exception thread shows fpb.tmp module as unloaded:

0:000> lmv m fpb.tmp start end module name

Unloaded modules: 00cb0000 00d5a000 fpb.tmp Timestamp: Fri Jun 01 02:56:00 2012 (4FC82130) Checksum: 000B0CD5 ImageSize: 000AA000 73ea0000 73f15000 fpb.tmp Timestamp: Fri Jun 01 02:49:25 2012 (4FC81FA5) Checksum: 0007A7CE ImageSize: 00075000

We change the exception thread context to get registers at the time of the exception:

0:000> kv ChildEBP RetAddr Args to Child 0035f1c4 771a0bdd 00000002 0035f214 00000001 ntdll!ZwWaitForMultipleObjects+0x15 0035f260 75771a2c 0035f214 0035f288 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 0035f2a8 75774208 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 0035f2c4 757980a4 00000002 0035f2f8 00000000 kernel32!WaitForMultipleObjects+0x18 0035f330 75797f63 0035f410 00000001 00000001 kernel32!WerpReportFaultInternal+0x186 0035f344 75797858 0035f410 00000001 0035f3e0 kernel32!WerpReportFault+0x70 0035f354 757977d7 0035f410 00000001 658587c7 kernel32!BasepReportFault+0x20 0035f3e0 77ec74df 00000000 77ec73bc 00000000 kernel32!UnhandledExceptionFilter+0x1af 0035f3e8 77ec73bc 00000000 0035fe5c 77e7c530 ntdll!__RtlUserThreadStart+0x62 0035f3fc 77ec7261 00000000 00000000 00000000 ntdll!_EH4_CallFilterFunc+0x12 0035f424 77eab459 fffffffe 0035fe4c 0035f560 ntdll!_except_handler4+0x8e 0035f448 77eab42b 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler2+0x26 0035f46c 77eab3ce 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler+0x24 0035f4f8 77e60133 0135f510 0035f560 0035f510 ntdll!RtlDispatchException+0x127 0035f4f8 73eb2200 0135f510 0035f560 0035f510 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0035f560) WARNING: Frame IP not in any known module. Following frames may be wrong. 0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200 0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23 0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109 0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581 0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f 0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19 0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36 0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d 0035fa64 75f61ee2 0035faa4 00001500 0035faa4 SHELL32!CShellExecute::ExecuteNormal+0×16 0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33 0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62 0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77 0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211 0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1 0035fcd4 757714dd 00da0000 00000000 00da2320 ntdll!RtlFreeHeap+0×142 0035fce8 003f0324 00da0000 00000000 003f0343 kernel32!HeapFree+0×14 0035fd80 003f0241 003e0000 00000000 010d3135 InstallFlashPlayer+0×10324 0035fe10 7577339a 7efde000 0035fe5c 77e89ef2 InstallFlashPlayer+0×10241 0035fe1c 77e89ef2 7efde000 77c46545 00000000 kernel32!BaseThreadInitThunk+0xe 0035fe5c 77e89ec5 003f02ac 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70 0035fe74 00000000 003f02ac 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> .cxr 0035f560 eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200 edi=00000000 eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210206 <Unloaded_fpb.tmp>+0×12200: 73eb2200 ?? ???

Then we double check that a window procedure was indeed called from that module range:

0:000> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child WARNING: Frame IP not in any known module. Following frames may be wrong. 0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200 0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23 0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109 0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581 0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f 0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19 0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36 0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d 0035fa64 75f61ee2 0035faa4 00001500 0035faa4 SHELL32!CShellExecute::ExecuteNormal+0×16 0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33 0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62 0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77 0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211 0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1 00da15a0 00000000 00da1780 02971450 003e1000 ntdll!RtlFreeHeap+0×142

0:000> ub 76e462fa USER32!InternalCallWinProc+0×6: 76e462dd 68cdabbadc push 0DCBAABCDh 76e462e2 56 push esi 76e462e3 ff7518 push dword ptr [ebp+18h] 76e462e6 ff7514 push dword ptr [ebp+14h] 76e462e9 ff7510 push dword ptr [ebp+10h] 76e462ec ff750c push dword ptr [ebp+0Ch] 76e462ef 64800dca0f000001 or byte ptr fs:[0FCAh],1 76e462f7 ff5508 call dword ptr [ebp+8]

We now get a memory value pointed to by EBP+8 address:

0:000> r Last set context: eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200 edi=00000000 eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210206 <Unloaded_fpb.tmp>+0×12200: 73eb2200 ?? ???

0:000> dp 0035f870+8 l1 0035f878 73eb2200

0:000> dd 73eb2200 73eb2200 ???????? ???????? ???????? ???????? 73eb2210 ???????? ???????? ???????? ???????? 73eb2220 ???????? ???????? ???????? ???????? 73eb2230 ???????? ???????? ???????? ???????? 73eb2240 ???????? ???????? ???????? ???????? 73eb2250 ???????? ???????? ???????? ???????? 73eb2260 ???????? ???????? ???????? ???????? 73eb2270 ???????? ???????? ???????? ????????

The value is indeed belongs to unloaded fpb.tmp module address range:

0:000> ln 73eb2200 (73eb2200) <Unloaded_fpb.tmp>+0×12200

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | 2 Comments »

Individual and Enterprise Software Diagnostics Certifications

Monday, June 18th, 2012

Memory Dump Analysis Services will be administering certifications developed by Software Diagnostics Institute for memory dump and software trace analysis:

Software Diagnostics Maturity Enterprise Certification
Memory Dump Analysis Certification is available this September

Debugging TV Frames episode 0×10 contains some background information.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Announcements, Certification, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Software Diagnostics, Software Diagnostics Institute, Software Diagnostics Patterns, Software Engineering, Software Technical Support, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Software Diagnostics Institute

Tuesday, June 12th, 2012

DumpAnalysis.org portal has been reorganized to Software Diagnostics Institute to reflect the nature of its research activities. More updates later on.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Announcements, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Software Behavior Patterns, Software Diagnostics, Software Diagnostics Institute, Software Diagnostics Patterns, Software Engineering, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 177)

Saturday, June 9th, 2012

Stack Trace Change is an important pattern for differential memory dump analysis, for example, when memory dumps were generated before and after a problem such as a CPU spike or hang. In the example below we have a normal expected thread stack trace from a memory dump saved before an application was reported unresponsive and another different thread stack trace after:

3 Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen ChildEBP RetAddr 037dfadc 75210bdd ntdll!ZwWaitForMultipleObjects+0x15 037dfb78 75791a2c KERNELBASE!WaitForMultipleObjectsEx+0x100 037dfbc0 7511086a kernel32!WaitForMultipleObjectsExImplementation+0xe0 037dfc14 00d17c1d user32!RealMsgWaitForMultipleObjectsEx+0x14d 037dfc3c 00ce161d ApplicationA!MsgWaitForMultipleObjects+0x2d 037dfc60 00cdc757 ApplicationA!WaitForSignal+0x1d 037dfc80 00cdaaf6 ApplicationA!WorkLoop+0x57 037dfca4 7579339a ApplicationA!ThreadStart+0x26 037dfcb0 77699ef2 kernel32!BaseThreadInitThunk+0xe 037dfcf0 77699ec5 ntdll!__RtlUserThreadStart+0x70 037dfd08 00000000 ntdll!_RtlUserThreadStart+0x1b

3 Id: 24b8.24e4 Suspend: 0 Teb: 7efa1000 Unfrozen ChildEBP RetAddr 037df38c 752131bb ntdll!ZwDelayExecution+0x15 037df3f4 75213a8b KERNELBASE!SleepEx+0x65 037df404 00d1670b KERNELBASE!Sleep+0xf 037df40c 00d350ef ApplicationA!Sleep+0xb 037df430 6a868aab ApplicationA!PutData+0xbf 037df444 6a8662ec ModuleA!OutputData+0x1b 037df464 00d351de ModuleA!ProcessData+0x16c 037df4a4 00ca8cb4 ApplicationA!SendData+0xbe [...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 176)

Wednesday, June 6th, 2012

Sometimes, when an application is sluggish, periodically consumes CPU, it is possible to create a set of consecutive memory dumps of the same process to see the temporal development of any thread CPU consumption and figure out potential Spike Interval(s). For example, the following diagram was plotted from !runaway WinDbg command output for thread #1:

The 3rd and the 5th user process memory dumps in addition to increased CPU consumption also have corresponding non-waiting stack trace frames caught while executing some CPU instructions in ModuleA (not preempted with saved context). The first memory dump (yellow bar) with 437 ms user time spent out of 629 ms elapsed time also has a non-waiting stack trace but we consider it a normal application startup CPU consumption spike.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

7 Habits of Highly Effective Diagnosticians (Part 0)

Tuesday, June 5th, 2012

Motivated by 7 Habits of Highly Effective Debuggers I would like to reflect on a distinction between diagnostics and problem solving as separate processes (although highly related). First, we reverse the precept from that article because stories such as software logs and traces are of primary importance to software diagnostics (and not only). And without diagnostics there is no effective debugging (treatment, problem solving, etc.)

The Principle Precept of Diagnostics

Stories NOT Statistics secure certainty.

Next parts will be about actual habits so please stay tuned. I would try to finish this list before the forthcoming Webinar on software diagnostics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in 7 Habits, Core Dump Analysis, Crash Dump Analysis, Escalation Engineering, Software Diagnostics, Software Engineering, Software Narratology, Software Problem Solving, Software Technical Support, Software Trace Analysis | No Comments »

Crash Dump Analysis Patterns (Part 23a, Mac OS X)

Tuesday, May 29th, 2012

This is a Mac OS X / GDB counterpart to Double Free (process heap) pattern previously described for Windows platforms:

(gdb) bt #0 0x00007fff8479582a in __kill () #1 0x00007fff8e0e0a9c in abort () #2 0x00007fff8e13f84c in free () #3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)

(gdb) x/2i 0x00000001035a8ef4-8 0x1035a8eec : mov -0×20(%rbp),%edi 0×1035a8eef : callq 0×1035a8f06

(gdb) frame 3 #3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20) at .../DoubleFree/main.c:23 23 free(p2); Current language: auto; currently minimal

(gdb) x/g $rbp-0x20 0x7fff631a7ae0: 0x00007fe6a8801400

(gdb) x/2w 0x00007fe6a8801400 0x7fe6a8801400: 0x00000000 0xb0000000

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

char *p1 = (char *) malloc (1024);

printf(“p1 = %p\n”, p1);

char *p2 = (char *) malloc (1024);

printf(“p2 = %p\n”, p2);

free(p2);

free(p1);

free(p2);

return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Posted in Assembly Language, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, GDB for WinDbg Users, Mac Crash Corner, Mac OS X, Software Defect Construction, x64 Mac OS X | No Comments »

Crash Dump Analysis Patterns (Part 2, Mac OS X)

Sunday, May 27th, 2012

This is a Mac OS X / GDB counterpart to Dynamic Memory Corruption (process heap) pattern previously described for Windows platforms:

(gdb) bt #0 0x00007fff8479582a in __kill () #1 0x00007fff8e0e0a9c in abort () #2 0x00007fff8e1024ac in szone_error () #3 0x00007fff8e1024e8 in free_list_checksum_botch () #4 0x00007fff8e102a7b in small_free_list_remove_ptr () #5 0x00007fff8e106bf7 in szone_free_definite_size () #6 0x00007fff8e13f789 in free () #7 0x000000010afafe23 in main (argc=1, argv=0x7fff6abaeb08)

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

char *p1 = (char *) malloc (1024);

printf(“p1 = %p\n”, p1);

char *p2 = (char *) malloc (1024);

printf(“p2 = %p\n”, p2);

char *p3 = (char *) malloc (1024);

printf(“p3 = %p\n”, p3);

char *p4 = (char *) malloc (1024);

printf(“p4 = %p\n”, p4);

char *p5 = (char *) malloc (1024);

printf(“p5 = %p\n”, p5);

char *p6 = (char *) malloc (1024);

printf(“p6 = %p\n”, p6);

char *p7 = (char *) malloc (1024);

printf(“p7 = %p\n”, p7);

free(p6);

free(p4);

free(p2);

printf(“Hello Crash!\n”);

strcpy(p2, “Hello Crash!”);

strcpy(p4, “Hello Crash!”);

strcpy(p6, “Hello Crash!”);

p2 = (char *) malloc (512);

printf(“p2 = %p\n”, p2);

p4 = (char *) malloc (1024);

printf(“p4 = %p\n”, p4);

p6 = (char *) malloc (512);

printf(“p6 = %p\n”, p6);

free (p7);

free (p6);

free (p5);

free (p4);

free (p3);

free (p2);

free (p1);

return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, GDB for WinDbg Users, Mac Crash Corner, Mac OS X, Software Defect Construction, x64 Mac OS X | No Comments »

Crash Dump Analysis Patterns (Part 175)

Wednesday, May 23rd, 2012

Stored Exception pattern is mostly useful when an exception thread is not present like in this rare example:

ERROR: Unable to find system thread 9B7E ERROR: The thread being debugged has either exited or cannot be accessed ERROR: Many commands will not work properly This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. ERROR: Exception C0000005 occurred on unknown thread 9B7E (95f4.9b7e): Access violation - code c0000005 (first/second chance not available)

.ecxr will not work here but the exception record is available via .exr command:

0:???> .exr -1 ExceptionAddress: 08a9ae18 (DllB.dll+0x001cae18) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000001 NumberParameters: 1 Parameter[0]: 00000008

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, WinDbg Tips and Tricks | No Comments »

Crash Dump Analysis Patterns (Part 174)

Sunday, May 20th, 2012

Activity Resonance pattern is observed when two products from different vendors compete in some functional domain such malware detection. In the example below ApplicationA and AVDriverA modules belong to Vendor A and AV-B module belongs to Vendor B. Both threads are spiking threads blocking all other activity in the system:

0: kd> !running

System Processors: (0000000000000003) Idle Processors: (0000000000000000) (0000000000000000) (0000000000000000) (0000000000000000)

Prcbs Current Next 0 fffff80001845e80 fffffa8004350060 ................ 1 fffff880009c4180 fffffa80028e7060 ................

0: kd> !thread fffffa8004350060 ff THREAD fffffa8004350060 Cid 14424.14b34 Teb: 000000007efdb000 Win32Thread: fffff900c1d32c30 RUNNING on processor 0 Not impersonating DeviceMap fffff8a00148fe80 Owning Process fffffa8003d6cb30 Image: ApplicationA.exe Attached Process N/A Image: N/A Wait Start TickCount 10568630 Ticks: 0 Context Switch Count 345 LargeStack UserTime 00:02:21.360 KernelTime 01:09:32.130 Win32 Start Address ApplicationA!mainCRTStartup (0×0000000000404c1b) Stack Init fffff88006c71db0 Current fffff88006c71670 Base fffff88006c72000 Limit fffff88006c6a000 Call 0 Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`06c70ec0 fffff880`0197d53c AVDriverA+0×15d69 fffff880`06c70f10 fffff880`01988556 AVDriverA+0×1453c fffff880`06c70fd0 fffff880`019886a8 AVDriverA+0×1f556 fffff880`06c71000 fffff800`0198ebfd AVDriverA+0×1f6a8 fffff880`06c71060 fffff800`019bf4f2 nt! ?? ::NNGAKEGL::`string’+0×2a6fd fffff880`06c711e0 fffff800`019c3385 nt!PspCreateThread+0×246 fffff880`06c71460 fffff800`016d28d3 nt!NtCreateThreadEx+0×25d fffff880`06c71bb0 00000000`76e61d9a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`06c71c20) 00000000`0008e178 00000000`74990411 ntdll!ZwCreateThreadEx+0xa 00000000`0008e180 00000000`7497cf87 wow64!whNtCreateThreadEx+0×815 00000000`0008e350 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7 00000000`0008ec10 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d 00000000`0008ecd0 00000000`7497c549 wow64!RunCpuSimulation+0xa 00000000`0008ed20 00000000`76e54956 wow64!Wow64LdrpInitialize+0×429 00000000`0008f270 00000000`76e51a17 ntdll!LdrpInitializeProcess+0×17e4 00000000`0008f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29220 00000000`0008f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

0: kd> !thread fffffa80028e7060 ff THREAD fffffa80028e7060 Cid 0dc4.0e5c Teb: 000000007efa4000 Win32Thread: 0000000000000000 RUNNING on processor 1 Not impersonating DeviceMap fffff8a000008b30 Owning Process fffffa8002817060 Image: AV-B.exe Attached Process N/A Image: N/A Wait Start TickCount 10568617 Ticks: 13 (0:00:00:00.203) Context Switch Count 1763138 UserTime 00:04:26.765 KernelTime 03:09:31.140 Win32 Start Address AV-B (0×00000000004289f2) Stack Init fffff88003b88db0 Current fffff88003b88900 Base fffff88003b89000 Limit fffff88003b83000 Call 0 Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`03b88660 fffff800`019919a9 nt!ObReferenceObjectSafe+0xf fffff880`03b88690 fffff800`01991201 nt!PsGetNextProcess+0×81 fffff880`03b886e0 fffff800`019dcef6 nt!ExpGetProcessInformation+0×774 fffff880`03b88830 fffff800`019dd949 nt!ExpQuerySystemInformation+0xfb4 fffff880`03b88be0 fffff800`016d28d3 nt!NtQuerySystemInformation+0×4d fffff880`03b88c20 00000000`76e6167a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`03b88c20) 00000000`0118e708 00000000`74987da7 ntdll!NtQuerySystemInformation+0xa 00000000`0118e710 00000000`74988636 wow64!whNT32QuerySystemProcessInformationEx+0×93 00000000`0118e760 00000000`7498a0e9 wow64!whNtQuerySystemInformation_SpecialQueryCase+0×466 00000000`0118e800 00000000`7497cf87 wow64!whNtQuerySystemInformation+0xf1 00000000`0118e840 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7 00000000`0118f100 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d 00000000`0118f1c0 00000000`7497c549 wow64!RunCpuSimulation+0xa 00000000`0118f210 00000000`76e8e707 wow64!Wow64LdrpInitialize+0×429 00000000`0118f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29364 00000000`0118f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Victimware, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 173)

Sunday, May 20th, 2012

Value Adding Process is a frequently observed pattern in terminal services environments when you see one or several process names listed in each session but not necessarily required. They are usually running to provide some user experience enhancements. In such cases if observed functional problems correspond to the purpose of running additional processes we might want to eliminate them for testing and troubleshooting purposes.

0: kd> !sprocess 12 Dumping Session 12

_MM_SESSION_SPACE fffff8800e5d5000 _MMSESSION fffff8800e5d5b40 PROCESS fffffa8008d50b30 SessionId: 12 Cid: 0b04 Peb: 7fffffdc000 ParentCid: 1478 DirBase: 6bb77000 ObjectTable: fffff8a003f280b0 HandleCount: 158. Image: csrss.exe

PROCESS fffffa80030c7060 SessionId: 12 Cid: 1a48 Peb: 7fffffd8000 ParentCid: 1478 DirBase: 0a33c000 ObjectTable: fffff8a003c46c00 HandleCount: 179. Image: winlogon.exe

PROCESS fffffa8008250b30 SessionId: 12 Cid: 18c8 Peb: 7fffffdf000 ParentCid: 1a48 DirBase: 0350d000 ObjectTable: fffff8a0025b6840 HandleCount: 226. Image: LogonUI.exe

PROCESS fffffa8008b00530 SessionId: 12 Cid: 1508 Peb: 7fffffdf000 ParentCid: 02f0 DirBase: 02f65000 ObjectTable: fffff8a003b7e530 HandleCount: 197. Image: ExcitingFeatureX.exe

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 20d)

Saturday, May 19th, 2012

This is a specialization of Insufficient Memory (kernel pool) pattern called Memory Leak (I/O completion packets). The currently unique diagnostics this pattern provides in comparison with other kernel pool tags is that the pool allocation entries show the leaking process:

0: kd> !poolused 3 Sorting by NonPaged Pool Consumed

Pool Used: NonPaged Paged Tag Allocs Frees Diff Used Allocs Frees Diff Used Icp 1294074 42875 1251199 96642976 0 0 0 0 I/O completion packets queue on a completion ports […]

0: kd> !poolfind Icp

Scanning large pool allocation table for Tag: Icp (fffffa8013e00000 : fffffa8014100000)

*fffffa800e188260 size: 50 previous size: 40 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e1882e0 size: 50 previous size: 30 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e188330 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e188380 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e1883d0 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e188420 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e188470 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40 *fffffa800e1884c0 size: 50 previous size: 50 (Allocated) Icp Process: fffffa800899dc40

0: kd> !process fffffa800899dc40 1 PROCESS fffffa800899dc40 SessionId: 0 Cid: 43a4 Peb: 7efdf000 ParentCid: 0412 DirBase: 09d6b000 ObjectTable: fffff8a0046c8c10 HandleCount: 1068. Image: ServiceA.exe […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 172)

Saturday, May 19th, 2012

Recently I observed a few occurrences of a rare No Current Thread pattern in a large set of process memory dumps:

0:???> k WARNING: The debugger does not have a current process or thread WARNING: Many commands will not work ^ Illegal thread error in ‘k’

0:???> ~ WARNING: The debugger does not have a current process or thread WARNING: Many commands will not work 0 Id: 95f4.6780 Suspend: 1 Teb: 7efdd000 Unfrozen

Setting a current thread helps:

0:???> ~0s WARNING: The debugger does not have a current process or thread WARNING: Many commands will not work eax=037d0010 ebx=0002bda0 ecx=03b1a010 edx=00000007 esi=037d0010 edi=03b069fc eip=0397939f esp=0018fd98 ebp=0018fdd8 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200202 DllA+0×939f: 0397939f 8b10 mov edx,dword ptr [eax] ds:002b:037d0010=03b1a010

0:000> k ChildEBP RetAddr WARNING: Stack unwind information not available. Following frames may be wrong. 0018fdd8 03975257 DllA+0x939f 0018fdf8 03975577 DllA+0x5257 0018fe58 772bb9a0 DllA+0x5577 0018fe78 772d9b96 ntdll!LdrpCallInitRoutine+0x14 0018ff1c 772d9a38 ntdll!LdrShutdownProcess+0x1aa 0018ff30 752279f4 ntdll!RtlExitUserProcess+0x74 0018ff44 0040625d kernel32!ExitProcessStub+0x12 0018ff5c 012528e5 Application+0x625d 0018ff88 7522339a Application!foo+0xdc88f1 0018ff94 772bbf42 kernel32!BaseThreadInitThunk+0xe 0018ffd4 772bbf15 ntdll!__RtlUserThreadStart+0x70 0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

However, EIP of the new current thread doesn’t point to any access violation and the dereferenced address is valid:

0:000> !address 037d0010 Usage: <unclassified> Allocation Base: 037d0000 Base Address: 037d0000 End Address: 038dd000 Region Size: 0010d000 Type: 00020000 MEM_PRIVATE State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE

Also, if we inspect the raw stack data we won’t find any hidden exceptions there. So we conclude that the missing thread was exceptional. Indeed, there is a saved exception context in the process memory dump:

0:000> .exr -1 ExceptionAddress: 08a9ae18 (<Unloaded_DllB.dll>+0x001cae18) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000001 NumberParameters: 1 Parameter[0]: 00000008

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 14, Mac OS X)

Wednesday, May 9th, 2012

This is a Mac OS X / GDB counterpart to Spiking Thread pattern previously described for Windows platforms:

(gdb) info threads 4 0×00007fff85b542df in sqrt$fenv_access_off () 3 0×00007fff8616ee42 in __semwait_signal () 2 0×00007fff8616ee42 in __semwait_signal () * 1 0×00007fff8616ee42 in __semwait_signal ()

We notice a non-waiting thread and switch to it:

(gdb) thread 4 [Switching to thread 4 (core thread 3)] 0x00007fff85b542df in sqrt$fenv_access_off ()

(gdb) bt #0 0x00007fff85b542df in sqrt$fenv_access_off () #1 0×000000010cc85dc9 in thread_three (arg=0×7fff6c884ac0) #2 0×00007fff8fac68bf in _pthread_start () #3 0×00007fff8fac9b75 in thread_start ()

If we disassemble the return address for thread_three function to come back from sqrt call we see an infinite loop:

(gdb) disass 0x000000010cc85dc9 Dump of assembler code for function thread_three: 0x000000010cc85db0 <thread_three+0>: push %rbp 0×000000010cc85db1 <thread_three+1>: mov %rsp,%rbp 0×000000010cc85db4 <thread_three+4>: sub $0×10,%rsp 0×000000010cc85db8 <thread_three+8>: mov %rdi,-0×10(%rbp) 0×000000010cc85dbc <thread_three+12>: mov -0×10(%rbp),%ax 0×000000010cc85dc0 <thread_three+16>: movsd (%rax),%xmm0 0×000000010cc85dc4 <thread_three+20>: callq 0×10cc85eac <dyld_stub_sqrt> 0×000000010cc85dc9 <thread_three+25>: mov -0×10(%rbp),%rax 0×000000010cc85dcd <thread_three+29>: movsd %xmm0,(%rax) 0×000000010cc85dd1 <thread_three+33>: jmpq 0×10cc85dbc <thread_three+12> End of assembler dump.

Here’s the source code of the modeling application:

void * thread_one (void *arg)

{

while (1)

{

sleep (1);

}

return 0;

}

void * thread_two (void *arg)

{

while (1)

{

sleep (2);

}

return 0;

}

void * thread_three (void *arg)

{

while (1)

{

*(double*)arg=sqrt(*(double *)arg);

}

return 0;

}

int main(int argc, const char * argv[])

{

pthread_t threadID_one, threadID_two, threadID_three;

double result = 0xffffffff;

pthread_create (&threadID_one, NULL, thread_one, NULL);

pthread_create (&threadID_two, NULL, thread_two, NULL);

pthread_create (&threadID_three, NULL, thread_three,

&result);

pthread_join(threadID_three, NULL);

return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Posted in Assembly Language, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, GDB for WinDbg Users, Mac Crash Corner, Mac OS X, Software Defect Construction, x64 Mac OS X | No Comments »

Crash Dump Analysis Patterns (Part 6a, Mac OS X)

Thursday, May 3rd, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (code) pattern previously described for Windows platforms:

(gdb) bt #0 0×0000000000000000 in ?? () #1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0) #2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0) #3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) disass 0×000000010e8cce73-3 0×000000010e8cce73 Dump of assembler code from 0×10e8cce70 to 0×10e8cce73: 0×000000010e8cce70 : callq *0×8(%rdi) End of assembler dump.

(gdb) info r rdi rdi 0x7fff6e4cbac0 140735043910336

(gdb) x/2 0x7fff6e4cbac0 0x7fff6e4cbac0: 0x0000000a 0×00000000

(gdb) p/x *($rdi+8) $7 = 0×0

(gdb) bt #0 0x0000000000000000 in ?? () #1 0x000000010e8cce73 in bar (ps=0×7fff6e4cbac0) #2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0) #3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) ptype MYSTRUCT type = struct _MyStruct_tag { int data; PFUNC pfunc; }

(gdb) print {MYSTRUCT}0×7fff6e4cbac0 $2 = {data = 10, pfunc = 0}

Here’s the source code of the modeling application:

typedef void (*PFUNC)(void);

typedef struct _MyStruct_tag

{

int data;

PFUNC pfunc;

} MYSTRUCT;

void bar(MYSTRUCT *ps)

{

ps->pfunc();

}

void foo(MYSTRUCT *ps)

{

bar(ps);

}

int main(int argc, const char * argv[])

{

MYSTRUCT pstruct = {10, NULL};

foo(&pstruct);

return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Posted in Assembly Language, Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, GDB for WinDbg Users, Mac Crash Corner, Mac OS X, Software Defect Construction, x64 Mac OS X | No Comments »

GDB for WinDbg Users (Part 8)

Monday, April 30th, 2012

As we started providing memory dump analysis pattern examples for Mac OS X we resume our table of command correspondence between WinDbg and GDB providing some corrections on the way. For example, in the previous version of table we omitted a correspondence to ub WinDbg command. Now we provide such an equivalent:

(gdb) bt [...] #1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0) […]

(gdb) disas 0×000000010e8cce73-10 0×000000010e8cce73 Dump of assembler code from 0×10e8cce69 to 0×10e8cce73: 0×000000010e8cce69 : mov %edi,-0×8(%rbp) 0×000000010e8cce6c : mov -0×8(%rbp),%rdi 0×000000010e8cce70 : callq *0×8(%rdi) End of assembler dump.

Please note that the beginning of assembly will be dependent on how good we guessed the offset:

(gdb) disas 0x000000010e8cce73-0×10 0×000000010e8cce73 Dump of assembler code from 0×10e8cce63 to 0×10e8cce73: 0×000000010e8cce63 : in $0×48,%eax 0×000000010e8cce65 : sub $0×10,%esp 0×000000010e8cce68 : mov %rdi,-0×8(%rbp) 0×000000010e8cce6c : mov -0×8(%rbp),%rdi 0×000000010e8cce70 : callq *0×8(%rdi) End of assembler dump.

(gdb) disas 0x000000010e8cce73-0×13 0×000000010e8cce73 Dump of assembler code from 0×10e8cce60 to 0×10e8cce73: 0×000000010e8cce60 : push %rbp 0×000000010e8cce61 : mov %rsp,%rbp 0×000000010e8cce64 : sub $0×10,%rsp 0×000000010e8cce68 : mov %rdi,-0×8(%rbp) 0×000000010e8cce6c : mov -0×8(%rbp),%rdi 0×000000010e8cce70 : callq *0×8(%rdi) End of assembler dump.

However, we can ignore that because our goal is to check whether a CPU instruction before a return address is a call.

Additional commands we add are x/<N>bc for db (WinDbg), thread <N> for ~<N>s (WinDbg, process dumps), maintenance info sections for for !address (WinDbg), add-symbol-file for .reload (WinDbg), info r for r (WinDbg).

Action                      | GDB                 | WinDbg
----------------------------------------------------------------
Start the process           | run                 | g
Exit                        | (q)uit              | q
Disassemble (forward)       | (disas)semble       | uf, u
Disassemble N instructions  | x/<N>i              | -
Disassemble (backward)      | disas <a-o> <a>     | ub
Stack trace                 | backtrace (bt)      | k
Full stack trace            | bt full             | kv
Stack trace with parameters | bt full             | kP
Partial trace (innermost)   | bt <N>              | k <N>
Partial trace (outermost)   | bt -<N>             | -
Stack trace for all threads | thread apply all bt | ~*k
Breakpoint                  | break               | bp
Frame numbers               | any bt command      | kn
Select frame                | frame               | .frame
Display parameters          | info args           | dv /t /i /V
Display locals              | info locals         | dv /t /i /V
Dump byte char array        | x/<N>bc             | db
Switch to thread            | thread <N>          | ~<N>s
Sections/regions            | maint info sections | !address
Load symbol file            | add-symbol-file     | .reload
CPU registers               | i(nfo) r            | r

Now an advertisement command:

(gdb) info training (gdb) Accelerated Mac OS X Core Dump Analysis training

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Debugging, GDB for WinDbg Users, Linux Crash Corner, Mac Crash Corner, Mac OS X, WinDbg for GDB Users | 1 Comment »

Crash Dump Analysis Patterns (Part 53c)

Saturday, April 28th, 2012

This is a special variant of Blocked Thread pattern where we have a timeout value so a thread is potentially blocked only temporarily. For example, this main thread is blocked waiting for beep sound to finish after a minute:

0:000> kvL ChildEBP RetAddr Args to Child 0291f354 7c90d21a 7c8023f1 00000001 0291f388 ntdll!KiFastSystemCallRet 0291f358 7c8023f1 00000001 0291f388 7c90d27e ntdll!NtDelayExecution+0xc 0291f3b0 7c837beb 0000ea60 00000001 00000004 kernel32!SleepEx+0×61 0291f404 004952a2 00000370 0000ea60 004d6ae2 kernel32!Beep+0×1b3 0291f410 004d6ae2 00000370 0000ea60 004d6ed4 Application!DoBeep+0×16 […] 0291ffec 00000000 0045aad0 00e470a0 00000000 kernel32!BaseThreadStart+0×37

0:000> ? ea60/0n1000 Evaluate expression: 60 = 0000003c

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Software Behavior Pattern Prediction

Saturday, April 28th, 2012

Sometimes I hear voices saying that Linux, FreeBSD, and Mac OS X core dumps are uninteresting. This is not true. If you haven’t seen anything interesting there it just simply means you have only encountered a limited amount of abnormal software behaviour. The widespread usage of Windows OS means that most patterns have been diagnosed and described first and other OS are waiting their turn.

My goal is to have a pattern catalog with examples from different OS. For example, currently, all Mac OS X patterns I provide are just examples to existing Windows pattern names. All OS share the same structure and behavior, for example, structural memory analysis patterns and the same computational model. Although structural patterns are different from behavioral patterns I also plan to expand the structural list significantly especially in relation to forthcoming Windows malware analysis training. Regarding behavioral patterns it is possible to model and predict specific pattern examples for another OS by using already existing catalog.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Linux Crash Corner, Mac Crash Corner, Mac OS X, Malware Analysis, Malware Patterns, Pattern Models, Pattern Prediction, Pattern-Driven Debugging, Pattern-Driven Software Support, Software Behavior DNA, Software Behavior Patterns, Software Behavioral Genome, Software Diagnostics | No Comments »

Crash Dump Analysis Patterns (Part 171)

Saturday, April 28th, 2012

This is an example of Punctuated Memory Leak pattern somewhat similar to a large block allocation leak for process heap (see a modeling example). An application has some functionality and after each command its commited memory was increasing by 50 - 60 Mb. 3 process dumps were taken with one before failures and then after each failure:

// Before failures

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 267 76c50000 ( 1.856 Gb) 92.79% <unclassified> 270 4d6f000 ( 77.434 Mb) 52.45% 3.78% Image 620 31bf000 ( 49.746 Mb) 33.70% 2.43% Stack 60 1400000 ( 20.000 Mb) 13.55% 0.98% ActivationContextData 48 35000 ( 212.000 kb) 0.14% 0.01% NlsTables 1 23000 ( 140.000 kb) 0.09% 0.01% TEB 20 14000 ( 80.000 kb) 0.05% 0.00% CsrSharedMemory 1 5000 ( 20.000 kb) 0.01% 0.00% PEB 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 296 3bca000 ( 59.789 Mb) 40.50% 2.92% MEM_IMAGE 647 340c000 ( 52.047 Mb) 35.26% 2.54% MEM_MAPPED 78 23ca000 ( 35.789 Mb) 24.24% 1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 267 76c50000 ( 1.856 Gb) 92.79% MEM_RESERVE 125 5006000 ( 80.023 Mb) 54.21% 3.91% MEM_COMMIT 896 439a000 ( 67.602 Mb) 45.79% 3.30%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_EXECUTE_READ 125 1f2c000 ( 31.172 Mb) 21.12% 1.52% PAGE_READONLY 363 1ee5000 ( 30.895 Mb) 20.93% 1.51% PAGE_READWRITE 309 4c2000 ( 4.758 Mb) 3.22% 0.23% PAGE_WRITECOPY 43 6a000 ( 424.000 kb) 0.28% 0.02% PAGE_READWRITE|PAGE_GUARD 40 4b000 ( 300.000 kb) 0.20% 0.01% PAGE_EXECUTE_READWRITE 15 11000 ( 68.000 kb) 0.04% 0.00% PAGE_EXECUTE 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 6130000 5fb70000 ( 1.496 Gb) <unclassified> abf000 13d1000 ( 19.816 Mb) Image 75141000 879000 ( 8.473 Mb) Stack 3290000 fd000 (1012.000 kb) ActivationContextData 50000 4000 ( 16.000 kb) NlsTables 7efb0000 23000 ( 140.000 kb) TEB 7ef6f000 1000 ( 4.000 kb) CsrSharedMemory 7efe0000 5000 ( 20.000 kb) PEB 7efde000 1000 ( 4.000 kb)

// After the 1st failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 267 7388c000 ( 1.805 Gb) 90.26% <unclassified> 272 8133000 ( 129.199 Mb) 64.80% 6.31% Image 614 31bf000 ( 49.746 Mb) 24.95% 2.43% Stack 60 1400000 ( 20.000 Mb) 10.03% 0.98% ActivationContextData 48 35000 ( 212.000 kb) 0.10% 0.01% NlsTables 1 23000 ( 140.000 kb) 0.07% 0.01% TEB 20 14000 ( 80.000 kb) 0.04% 0.00% CsrSharedMemory 1 5000 ( 20.000 kb) 0.01% 0.00% PEB 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 297 6f8e000 ( 111.555 Mb) 55.95% 5.45% MEM_IMAGE 642 340c000 ( 52.047 Mb) 26.10% 2.54% MEM_MAPPED 78 23ca000 ( 35.789 Mb) 17.95% 1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 267 7388c000 ( 1.805 Gb) 90.26% MEM_COMMIT 892 775e000 ( 119.367 Mb) 59.87% 5.83% MEM_RESERVE 125 5006000 ( 80.023 Mb) 40.13% 3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 314 38a3000 ( 56.637 Mb) 28.40% 2.77% PAGE_EXECUTE_READ 125 1f2c000 ( 31.172 Mb) 15.63% 1.52% PAGE_READONLY 363 1ee5000 ( 30.895 Mb) 15.49% 1.51% PAGE_WRITECOPY 34 4d000 ( 308.000 kb) 0.15% 0.01% PAGE_READWRITE|PAGE_GUARD 40 4b000 ( 300.000 kb) 0.15% 0.01% PAGE_EXECUTE_READWRITE 15 11000 ( 68.000 kb) 0.03% 0.00% PAGE_EXECUTE 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 94f4000 5c7ac000 ( 1.445 Gb) <unclassified> 6130000 33c4000 ( 51.766 Mb) Image 75141000 879000 ( 8.473 Mb) Stack 3290000 fd000 (1012.000 kb) ActivationContextData 50000 4000 ( 16.000 kb) NlsTables 7efb0000 23000 ( 140.000 kb) TEB 7ef6f000 1000 ( 4.000 kb) CsrSharedMemory 7efe0000 5000 ( 20.000 kb) PEB 7efde000 1000 ( 4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize Type State Protect Usage ------------------------------------------------------------------------------------------- [...] 5e82000 5f70000 ee000 MEM_PRIVATE MEM_RESERVE <unclassified> 6130000 94f4000 33c4000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unclassified> 74220000 74221000 1000 MEM_IMAGE MEM_COMMIT PAGE_READONLY <unclassified> […]

0:000> ? 33c4000/0n1024 Evaluate expression: 53008 = 0000cf10

// After the 2nd failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 268 704c8000 ( 1.755 Gb) 87.74% <unclassified> 273 b4f7000 ( 180.965 Mb) 72.05% 8.84% Image 614 31bf000 ( 49.746 Mb) 19.81% 2.43% Stack 60 1400000 ( 20.000 Mb) 7.96% 0.98% ActivationContextData 48 35000 ( 212.000 kb) 0.08% 0.01% NlsTables 1 23000 ( 140.000 kb) 0.05% 0.01% TEB 20 14000 ( 80.000 kb) 0.03% 0.00% CsrSharedMemory 1 5000 ( 20.000 kb) 0.01% 0.00% PEB 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 298 a352000 ( 163.320 Mb) 65.03% 7.97% MEM_IMAGE 642 340c000 ( 52.047 Mb) 20.72% 2.54% MEM_MAPPED 78 23ca000 ( 35.789 Mb) 14.25% 1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 268 704c8000 ( 1.755 Gb) 87.74% MEM_COMMIT 893 ab22000 ( 171.133 Mb) 68.14% 8.36% MEM_RESERVE 125 5006000 ( 80.023 Mb) 31.86% 3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 315 6c67000 ( 108.402 Mb) 43.16% 5.29% PAGE_EXECUTE_READ 125 1f2c000 ( 31.172 Mb) 12.41% 1.52% PAGE_READONLY 363 1ee5000 ( 30.895 Mb) 12.30% 1.51% PAGE_WRITECOPY 34 4d000 ( 308.000 kb) 0.12% 0.01% PAGE_READWRITE|PAGE_GUARD 40 4b000 ( 300.000 kb) 0.12% 0.01% PAGE_EXECUTE_READWRITE 15 11000 ( 68.000 kb) 0.03% 0.00% PAGE_EXECUTE 1 1000 ( 4.000 kb) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free c8c4000 593dc000 ( 1.394 Gb) <unclassified> 6130000 33c4000 ( 51.766 Mb) Image 75141000 879000 ( 8.473 Mb) Stack 3290000 fd000 (1012.000 kb) ActivationContextData 50000 4000 ( 16.000 kb) NlsTables 7efb0000 23000 ( 140.000 kb) TEB 7ef6f000 1000 ( 4.000 kb) CsrSharedMemory 7efe0000 5000 ( 20.000 kb) PEB 7efde000 1000 ( 4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize Type State Protect Usage ------------------------------------------------------------------------------------------- 5e82000 5f70000 ee000 MEM_PRIVATE MEM_RESERVE <unclassified> 6130000 94f4000 33c4000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unclassified> 9500000 c8c4000 33c4000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unclassified> 74220000 74221000 1000 MEM_IMAGE MEM_COMMIT PAGE_READONLY <unclassified> […]

The name of this pattern comes from the process of discrete large memory allocations that happen after specific actions or events. Between them there is no visible or substantial increase in memory usage.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Debugging | No Comments »

Crash Dump Analysis Patterns (Part 110, Mac OS X)

Sunday, April 22nd, 2012

This is a provisional Mac OS X example of Shared Buffer Overwrite pattern. Originally I wanted to construct a default C runtime heap corruption example using malloc / free functions. Unfortunately I couldn’t get heap corrupted easily as was possible in Windows Visual C++ environment by writing before or after allocated block. Desperately I printed allocated pointers and they all pointed to memory blocks laid out one after another without any headers in between (could be just a default Apple LLVM C runtime implementation and I have to check that with GCC). Therefore, any subsequent reallocation didn’t cause corruption either. So all this naturally fits into shared buffer overwrites or underwrites where corruption is only detectable when the overwritten data is used such as a pointer dereference.

int main(int argc, const char * argv[])

{

char *p1 = (char *) malloc (1024);

strcpy(p1, “Hello World!”);

printf(“p1 = %p\n”, p1);

printf(“*p1 = %s\n”, p1);

char *p2 = (char *) malloc (1024);

strcpy(p2, “Hello World!”);

printf(“p2 = %p\n”, p2);

printf(“*p2 = %s\n”, p2);

char *p3 = (char *) malloc (1024);

strcpy(p3, “Hello World!”);

printf(“p3 = %p\n”, p3);

printf(“*p3 = %s\n”, p3);

strcpy(p2-sizeof(p2), “Hello Crash!”);

strcpy(p3-sizeof(p3), “Hello Crash!”);

p2 = (char *)realloc(p2, 2048);

printf(“p2 = %p\n”, p2);

printf(“*p2 = %s\n”, p2);

char *p4 = (char *) malloc (1024);

strcpy(p4-sizeof(p4), “Hello Crash!”);

printf(“p4 = %p\n”, p4);

printf(“*p4 = %s\n”, p4);

p3 = (char *)realloc(p3, 2048);

printf(“p3 = %p\n”, p3);

printf(“*p3 = %s\n”, p3);

char *p5 = NULL; // to force a core dump

*p5 = 0;

free (p4);

free (p3);

free (p2);

free (p1);

return 0;

}

When we run the program above we get this output:

p1 = 0x7fc6d9000000 *p1 = Hello World! p2 = 0×7fc6d9001400 *p2 = Hello World! p3 = 0×7fc6d9001800 *p3 = Hello World! p2 = 0×7fc6d9001c00 *p2 = ash! p4 = 0×7fc6d9001400 *p4 = ash! p3 = 0×7fc6d9002400 *p3 = ash! Segmentation fault: 11 (core dumped)

Now is GDB output:

(gdb) x/1024bc p1 0x7fc6d9000000: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9000008: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9000010: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ […] 0×7fc6d90003e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90003f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/32bc p1+1024-sizeof(p1) 0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9000400: 42 ‘*’ 112 ‘p’ 51 ‘3′ 32 ‘ ‘ 61 ‘=’ 32 ‘ ‘ 97 ‘a’ 115 ’s’ 0×7fc6d9000408: 104 ‘h’ 33 ‘!’ 10 ‘\n’ 100 ‘d’ 57 ‘9′ 48 ‘0′ 48 ‘0′ 50 ‘2′ 0×7fc6d9000410: 52 ‘4′ 48 ‘0′ 48 ‘0′ 10 ‘\n’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/2048bc p2 0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ […] 0×7fc6d9001fe8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001ff0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001ff8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’ 0×7fc6d9002000: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002008: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ […] 0×7fc6d90023e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90023f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/64bc p2-sizeof(p2) 0×7fc6d9001bf8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c18: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c20: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c28: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001c30: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/64bc p2+2048-sizeof(p2) 0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/1024bc p3 0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ […] 0×7fc6d90027e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90027f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/64bc p3-sizeof(p3) 0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/64bc p3+1024-sizeof(p3) 0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002800: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002808: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9002830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/1024bc p4 0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ […] 0×7fc6d90017e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90017f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’ (gdb) x/64bc p4-sizeof(p4) 0×7fc6d90013f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’ 0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ (gdb) x/64bc p4+1024-sizeof(p4) 0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’ 0×7fc6d9001800: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’ 0×7fc6d9001808: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0×7fc6d9001830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, GDB for WinDbg Users, Mac Crash Corner, Mac OS X | No Comments »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘Crash Dump Analysis’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta