Archive for May, 2012

Notes on Memoidealism (1.13)

Wednesday, May 30th, 2012

Here we try to map memoidealistic epistemological counterparts to Locke’s theory of perception.

1. Ideas of sensation

Sense organs receive memory snapshots from other memories

2. Ideas of reflection

Memory snapshots from mind reflecting snapshots from other memories

3. Substance

Memory as a foundation of Universe

3. Primary qualities

Qualities associated with Memory as a substance

4. Secondary qualities

Qualities associated with memories as parts of Memory

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The Confluence of Computers, Philosophy, and Religion

Wednesday, May 30th, 2012

We extend the old notion of confluence of Theism and Philosophy. The new confluence of Religion, Philosophy, and Computers resuled in revealed Memory Religion (Memorianity):

It’s testament is soon to be available for free download to spread the word of Memory:

Memory Religion: A Core Testament of Memorianity

- Dmitry Vostokov @ MemoryReligion.com -

Crash Dump Analysis Patterns (Part 23a, Mac OS X)

Tuesday, May 29th, 2012

This is a Mac OS X / GDB counterpart to Double Free (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e13f84c in free ()
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)

(gdb) x/2i 0x00000001035a8ef4-8
0x1035a8eec : mov -0×20(%rbp),%edi
0×1035a8eef : callq 0×1035a8f06

(gdb) frame 3
#3 0x00000001035a8ef4 in main (argc=1, argv=0x7fff631a7b20)
at .../DoubleFree/main.c:23
23 free(p2);
Current language: auto; currently minimal

(gdb) x/g $rbp-0x20
0x7fff631a7ae0: 0x00007fe6a8801400

(gdb) x/2w 0x00007fe6a8801400
0x7fe6a8801400: 0x00000000 0xb0000000

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    free(p2);

    free(p1);

    free(p2);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Sorting and Early Greek Philosophers

Tuesday, May 29th, 2012

Who was the first? Anaximander or Anaximenes? Use alphabetical sorting and you find that Anaximenes was after Anaximander.

Fiction for Debugging: The Problem and The Solution

Monday, May 28th, 2012

After writing about music for debugging and founding software narratology I decided to start writing about fiction. The first masterpiece is The Sound and The Fury by William Faulkner. I confess that I’m in love with Folio Society books and when I saw this color version (an original idea by Faulkner now fulfilled by modern printing technology) I immediately recognized its importance for software trace analysis:

http://www.foliosociety.com/book/SAF/sound-and-the-fury

I’m pretty sure Faulkner would have been delighted to see trace analysis patterns and how they may help in writing fiction.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 2, Mac OS X)

Sunday, May 27th, 2012

This is a Mac OS X / GDB counterpart to Dynamic Memory Corruption (process heap) pattern previously described for Windows platforms:

(gdb) bt
#0 0x00007fff8479582a in __kill ()
#1 0x00007fff8e0e0a9c in abort ()
#2 0x00007fff8e1024ac in szone_error ()
#3 0x00007fff8e1024e8 in free_list_checksum_botch ()
#4 0x00007fff8e102a7b in small_free_list_remove_ptr ()
#5 0x00007fff8e106bf7 in szone_free_definite_size ()
#6 0x00007fff8e13f789 in free ()
#7 0x000000010afafe23 in main (argc=1, argv=0x7fff6abaeb08)

Here’s the source code of the modeling application:

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    printf(“p1 = %p\n”, p1);

 

    char *p2 = (char *) malloc (1024);

    printf(“p2 = %p\n”, p2);

 

    char *p3 = (char *) malloc (1024);

    printf(“p3 = %p\n”, p3);

 

    char *p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    char *p5 = (char *) malloc (1024);

    printf(“p5 = %p\n”, p5);

 

    char *p6 = (char *) malloc (1024);

    printf(“p6 = %p\n”, p6);

 

    char *p7 = (char *) malloc (1024);

    printf(“p7 = %p\n”, p7);

 

    free(p6);

    free(p4);

    free(p2);

 

    printf(“Hello Crash!\n”);        

    strcpy(p2, “Hello Crash!”);

    strcpy(p4, “Hello Crash!”);

    strcpy(p6, “Hello Crash!”);

 

    p2 = (char *) malloc (512);

    printf(“p2 = %p\n”, p2);

 

    p4 = (char *) malloc (1024);

    printf(“p4 = %p\n”, p4);

 

    p6 = (char *) malloc (512);

    printf(“p6 = %p\n”, p6);

 

    free (p7);

    free (p6);

    free (p5);

    free (p4);

    free (p3);

    free (p2);

    free (p1);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 175)

Wednesday, May 23rd, 2012

Stored Exception pattern is mostly useful when an exception thread is not present like in this rare example:

ERROR: Unable to find system thread 9B7E
ERROR: The thread being debugged has either exited or cannot be accessed
ERROR: Many commands will not work properly
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
ERROR: Exception C0000005 occurred on unknown thread 9B7E
(95f4.9b7e): Access violation - code c0000005 (first/second chance not available)

.ecxr will not work here but the exception record is available via .exr command:

0:???> .exr -1
ExceptionAddress: 08a9ae18 (DllB.dll+0x001cae18)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000008

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 174)

Sunday, May 20th, 2012

Activity Resonance pattern is observed when two products from different vendors compete in some functional domain such malware detection. In the example below ApplicationA and AVDriverA modules belong to Vendor A and AV-B module belongs to Vendor B. Both threads are spiking threads blocking all other activity in the system:

0: kd> !running

System Processors: (0000000000000003)
Idle Processors: (0000000000000000) (0000000000000000) (0000000000000000) (0000000000000000)

Prcbs             Current           Next
0    fffff80001845e80  fffffa8004350060                    ................
1    fffff880009c4180  fffffa80028e7060                    ................

0: kd> !thread fffffa8004350060 ff
THREAD fffffa8004350060  Cid 14424.14b34  Teb: 000000007efdb000 Win32Thread: fffff900c1d32c30 RUNNING on processor 0
Not impersonating
DeviceMap                 fffff8a00148fe80
Owning Process            fffffa8003d6cb30       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      10568630       Ticks: 0
Context Switch Count      345                 LargeStack
UserTime                  00:02:21.360
KernelTime                01:09:32.130
Win32 Start Address ApplicationA!mainCRTStartup (0×0000000000404c1b)
Stack Init fffff88006c71db0 Current fffff88006c71670
Base fffff88006c72000 Limit fffff88006c6a000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`06c70ec0 fffff880`0197d53c AVDriverA+0×15d69
fffff880`06c70f10 fffff880`01988556 AVDriverA+0×1453c
fffff880`06c70fd0 fffff880`019886a8 AVDriverA+0×1f556
fffff880`06c71000 fffff800`0198ebfd AVDriverA+0×1f6a8

fffff880`06c71060 fffff800`019bf4f2 nt! ?? ::NNGAKEGL::`string’+0×2a6fd
fffff880`06c711e0 fffff800`019c3385 nt!PspCreateThread+0×246
fffff880`06c71460 fffff800`016d28d3 nt!NtCreateThreadEx+0×25d
fffff880`06c71bb0 00000000`76e61d9a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`06c71c20)
00000000`0008e178 00000000`74990411 ntdll!ZwCreateThreadEx+0xa
00000000`0008e180 00000000`7497cf87 wow64!whNtCreateThreadEx+0×815
00000000`0008e350 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7
00000000`0008ec10 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d
00000000`0008ecd0 00000000`7497c549 wow64!RunCpuSimulation+0xa
00000000`0008ed20 00000000`76e54956 wow64!Wow64LdrpInitialize+0×429
00000000`0008f270 00000000`76e51a17 ntdll!LdrpInitializeProcess+0×17e4
00000000`0008f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29220
00000000`0008f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

0: kd> !thread fffffa80028e7060 ff
THREAD fffffa80028e7060  Cid 0dc4.0e5c  Teb: 000000007efa4000 Win32Thread: 0000000000000000 RUNNING on processor 1
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8002817060       Image:         AV-B.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      10568617       Ticks: 13 (0:00:00:00.203)
Context Switch Count      1763138
UserTime                  00:04:26.765
KernelTime                03:09:31.140
Win32 Start Address AV-B (0×00000000004289f2)
Stack Init fffff88003b88db0 Current fffff88003b88900
Base fffff88003b89000 Limit fffff88003b83000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`03b88660 fffff800`019919a9 nt!ObReferenceObjectSafe+0xf
fffff880`03b88690 fffff800`01991201 nt!PsGetNextProcess+0×81
fffff880`03b886e0 fffff800`019dcef6 nt!ExpGetProcessInformation+0×774
fffff880`03b88830 fffff800`019dd949 nt!ExpQuerySystemInformation+0xfb4
fffff880`03b88be0 fffff800`016d28d3 nt!NtQuerySystemInformation+0×4d
fffff880`03b88c20 00000000`76e6167a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`03b88c20)
00000000`0118e708 00000000`74987da7 ntdll!NtQuerySystemInformation+0xa
00000000`0118e710 00000000`74988636 wow64!whNT32QuerySystemProcessInformationEx+0×93
00000000`0118e760 00000000`7498a0e9 wow64!whNtQuerySystemInformation_SpecialQueryCase+0×466
00000000`0118e800 00000000`7497cf87 wow64!whNtQuerySystemInformation+0xf1
00000000`0118e840 00000000`748c2776 wow64!Wow64SystemServiceEx+0xd7
00000000`0118f100 00000000`7497d07e wow64cpu!TurboDispatchJumpAddressEnd+0×2d
00000000`0118f1c0 00000000`7497c549 wow64!RunCpuSimulation+0xa
00000000`0118f210 00000000`76e8e707 wow64!Wow64LdrpInitialize+0×429
00000000`0118f760 00000000`76e3c32e ntdll! ?? ::FNODOBFM::`string’+0×29364
00000000`0118f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 173)

Sunday, May 20th, 2012

Value Adding Process is a frequently observed pattern in terminal services environments when you see one or several process names listed in each session but not necessarily required. They are usually running to provide some user experience enhancements. In such cases if observed functional problems correspond to the purpose of running additional processes we might want to eliminate them for testing and troubleshooting purposes.

0: kd> !sprocess 12
Dumping Session 12

_MM_SESSION_SPACE fffff8800e5d5000
_MMSESSION        fffff8800e5d5b40
PROCESS fffffa8008d50b30
SessionId: 12  Cid: 0b04    Peb: 7fffffdc000  ParentCid: 1478
DirBase: 6bb77000  ObjectTable: fffff8a003f280b0  HandleCount: 158.
Image: csrss.exe

PROCESS fffffa80030c7060
SessionId: 12  Cid: 1a48    Peb: 7fffffd8000  ParentCid: 1478
DirBase: 0a33c000  ObjectTable: fffff8a003c46c00  HandleCount: 179.
Image: winlogon.exe

PROCESS fffffa8008250b30
SessionId: 12  Cid: 18c8    Peb: 7fffffdf000  ParentCid: 1a48
DirBase: 0350d000  ObjectTable: fffff8a0025b6840  HandleCount: 226.
Image: LogonUI.exe

PROCESS fffffa8008b00530
SessionId: 12  Cid: 1508    Peb: 7fffffdf000  ParentCid: 02f0
DirBase: 02f65000  ObjectTable: fffff8a003b7e530  HandleCount: 197.
Image: ExcitingFeatureX.exe

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 20d)

Saturday, May 19th, 2012

This is a specialization of Insufficient Memory (kernel pool) pattern called Memory Leak (I/O completion packets). The currently unique diagnostics this pattern provides in comparison with other kernel pool tags is that the pool allocation entries show the leaking process:

0: kd> !poolused 3
Sorting by  NonPaged Pool Consumed

Pool Used:
NonPaged                    Paged
Tag    Allocs    Frees     Diff     Used   Allocs    Frees     Diff     Used
Icp   1294074    42875  1251199 96642976        0        0        0        0 I/O completion packets queue on a completion ports
[…]

0: kd> !poolfind Icp

Scanning large pool allocation table for Tag: Icp  (fffffa8013e00000 : fffffa8014100000)

*fffffa800e188260 size:   50 previous size:   40  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e1882e0 size:   50 previous size:   30  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e188330 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e188380 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e1883d0 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e188420 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e188470 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40
*fffffa800e1884c0 size:   50 previous size:   50  (Allocated) Icp  Process: fffffa800899dc40

0: kd> !process  fffffa800899dc40 1
PROCESS fffffa800899dc40
SessionId: 0  Cid: 43a4    Peb: 7efdf000  ParentCid: 0412
DirBase: 09d6b000  ObjectTable: fffff8a0046c8c10  HandleCount: 1068.
Image: ServiceA.exe
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 172)

Saturday, May 19th, 2012

Recently I observed a few occurrences of a rare No Current Thread pattern in a large set of process memory dumps:

0:???> k
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
^ Illegal thread error in ‘k’

0:???> ~
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
0  Id: 95f4.6780 Suspend: 1 Teb: 7efdd000 Unfrozen

Setting a current thread helps:

0:???> ~0s
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
eax=037d0010 ebx=0002bda0 ecx=03b1a010 edx=00000007 esi=037d0010 edi=03b069fc
eip=0397939f esp=0018fd98 ebp=0018fdd8 iopl=0  nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b  efl=00200202
DllA+0×939f:
0397939f 8b10 mov edx,dword ptr [eax] ds:002b:037d0010=03b1a010

0:000> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0018fdd8 03975257 DllA+0x939f
0018fdf8 03975577 DllA+0x5257
0018fe58 772bb9a0 DllA+0x5577
0018fe78 772d9b96 ntdll!LdrpCallInitRoutine+0x14
0018ff1c 772d9a38 ntdll!LdrShutdownProcess+0x1aa
0018ff30 752279f4 ntdll!RtlExitUserProcess+0x74
0018ff44 0040625d kernel32!ExitProcessStub+0x12
0018ff5c 012528e5 Application+0x625d
0018ff88 7522339a Application!foo+0xdc88f1
0018ff94 772bbf42 kernel32!BaseThreadInitThunk+0xe
0018ffd4 772bbf15 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

However, EIP of the new current thread doesn’t point to any access violation and the dereferenced address is valid:

0:000> !address 037d0010
Usage:                  <unclassified>
Allocation Base:        037d0000
Base Address:           037d0000
End Address:            038dd000
Region Size:            0010d000
Type:                   00020000 MEM_PRIVATE
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE

Also, if we inspect the raw stack data we won’t find any hidden exceptions there. So we conclude that the missing thread was exceptional. Indeed, there is a saved exception context in the process memory dump:

0:000> .exr -1
ExceptionAddress: 08a9ae18 (<Unloaded_DllB.dll>+0x001cae18)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000008

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Reading Notebook: 17-May-2012

Friday, May 18th, 2012

Comments in italics are mine and express my own views, thoughts and opinions

Mac OS X Internals by A. Singh:

kextstat command (p. 49) - here’s the output from my system:

MacBook-Air:~ DumpAnalysis$ kextstat
Index Refs Address            Size       Wired      Name (Version) <Linked Against>
1   78 0xffffff7f80739000 0x683c     0x683c     com.apple.kpi.bsd (11.3.0)
2    6 0xffffff7f807de000 0x3d0      0x3d0      com.apple.kpi.dsep (11.3.0)
3  104 0xffffff7f80744000 0x1b9d8    0x1b9d8    com.apple.kpi.iokit (11.3.0)
4  109 0xffffff7f8072f000 0x9b54     0x9b54     com.apple.kpi.libkern (11.3.0)
5   93 0xffffff7f80740000 0x88c      0x88c      com.apple.kpi.mach (11.3.0)
6   37 0xffffff7f80760000 0x4938     0x4938     com.apple.kpi.private (11.3.0)
7   53 0xffffff7f80741000 0x22a0     0x22a0     com.apple.kpi.unsupported (11.3.0)
8   19 0xffffff7f80bc6000 0x7000     0x7000     com.apple.iokit.IOACPIFamily (1.4) <7 6 4 3>
9   27 0xffffff7f80765000 0x1e000    0x1e000    com.apple.iokit.IOPCIFamily (2.6.8) <7 6 5 4 3>
10    2 0xffffff7f81ba4000 0x58000    0x58000    com.apple.driver.AppleACPIPlatform (1.4) <9 8 7 6 5 4 3 1>
11    1 0xffffff7f809cc000 0xc000     0xc000     com.apple.driver.AppleKeyStore (28.18) <7 6 5 4 3 1>
12    9 0xffffff7f807e2000 0x25000    0x25000    com.apple.iokit.IOStorageFamily (1.7) <7 6 5 4 3 1>
13    0 0xffffff7f80c4c000 0x19000    0x19000    com.apple.driver.DiskImages (331.3) <12 7 6 5 4 3 1>
14    0 0xffffff7f818e6000 0x2a000    0x2a000    com.apple.driver.AppleIntelCPUPowerManagement (167.3.0) <7 6 5 4 3 1>
15    0 0xffffff7f807df000 0x3000     0x3000     com.apple.security.TMSafetyNet (7) <7 6 5 4 2 1>
16    2 0xffffff7f80846000 0x4000     0x4000     com.apple.kext.AppleMatch (1.0.0d1) <4 1>
17    1 0xffffff7f8084a000 0x11000    0x11000    com.apple.security.sandbox (177.3) <16 7 6 5 4 3 2 1>
18    0 0xffffff7f8085b000 0x5000     0x5000     com.apple.security.quarantine (1.1) <17 16 7 6 5 4 2 1>
19    0 0xffffff7f81c0b000 0x8000     0x8000     com.apple.nke.applicationfirewall (3.2.30) <7 6 5 4 3 1>
20    0 0xffffff7f818e2000 0x3000     0x3000     com.apple.driver.AppleIntelCPUPowerManagementClient (167.3.0) <7 6 5 4 3 1>
21    0 0xffffff7f81b81000 0x3000     0x3000     com.apple.driver.AppleAPIC (1.5) <4 3>
22    3 0xffffff7f80b62000 0x4000     0x4000     com.apple.iokit.IOSMBusFamily (1.1) <5 4 3>
23    0 0xffffff7f81bfc000 0x7000     0x7000     com.apple.driver.AppleACPIEC (1.4) <22 10 8 5 4 3>
24    0 0xffffff7f816da000 0x4000     0x4000     com.apple.driver.AppleSMBIOS (1.7) <7 4 3>
25    0 0xffffff7f81918000 0x3000     0x3000     com.apple.driver.AppleHPET (1.6) <8 7 5 4 3>
26    0 0xffffff7f816ff000 0x7000     0x7000     com.apple.driver.AppleRTC (1.4) <8 5 4 3 1>
27    6 0xffffff7f809d8000 0x6b000    0x6b000    com.apple.iokit.IOHIDFamily (1.7.1) <11 7 6 5 4 3 2 1>
28    0 0xffffff7f81c05000 0x4000     0x4000     com.apple.driver.AppleACPIButtons (1.4) <27 10 8 7 6 5 4 3 1>
29    1 0xffffff7f81b57000 0x4000     0x4000     com.apple.driver.AppleEFIRuntime (1.5.0) <7 6 5 4 3>
30   13 0xffffff7f80783000 0x4f000    0x4f000    com.apple.iokit.IOUSBFamily (4.5.8) <9 7 5 4 3 1>
32    0 0xffffff7f80a8e000 0x17000    0x17000    com.apple.driver.AppleUSBEHCI (4.5.8) <30 9 7 5 4 3 1>
33    2 0xffffff7f80dc8000 0xa000     0xa000     com.apple.iokit.IOAHCIFamily (2.0.7) <5 4 3 1>
34    0 0xffffff7f81b85000 0x18000    0x18000    com.apple.driver.AppleAHCIPort (2.2.0) <33 9 5 4 3 1>
35    0 0xffffff7f816df000 0x8000     0x8000     com.apple.driver.AppleSmartBatteryManager (161.0.0) <22 8 5 4 3 1>
36    0 0xffffff7f81b5b000 0x7000     0x7000     com.apple.driver.AppleEFINVRAM (1.5.0) <29 7 5 4 3>
37    5 0xffffff7f80986000 0x29000    0x29000    com.apple.iokit.IONetworkingFamily (2.0) <7 6 5 4 3 1>
38    1 0xffffff7f80dfb000 0x38000    0x38000    com.apple.iokit.IO80211Family (412.2) <37 7 5 4 3 1>
39    0 0xffffff7f80e33000 0x1e0000   0x1e0000   com.apple.driver.AirPort.Brcm4331 (513.20.19) <38 37 9 7 5 4 3 1>
40    0 0xffffff7f809c9000 0x3000     0x3000     com.apple.iokit.IOUSBUserClient (4.5.8) <30 7 5 4 3 1>
41    0 0xffffff7f80a79000 0x11000    0x11000    com.apple.driver.AppleUSBHub (4.5.0) <30 5 4 3 1>
42    4 0xffffff7f80ab2000 0x9e000    0x9e000    com.apple.iokit.IOThunderboltFamily (1.7.4) <5 4 3 1>
43    0 0xffffff7f8163e000 0x12000    0x12000    com.apple.driver.AppleThunderboltNHI (1.3.2) <42 9 8 5 4 3 1>
44    0 0xffffff7f80dde000 0x15000    0x15000    com.apple.iokit.IOAHCIBlockStorage (2.0.1) <33 12 5 4 3 1>
45    0 0xffffff7f815b2000 0x4000     0x4000     com.apple.driver.XsanFilter (403) <12 5 4 3 1>
46    0 0xffffff7f81342000 0x9000     0x9000     com.apple.BootCache (33) <7 6 5 4 3 1>
47    0 0xffffff7f81b46000 0x5000     0x5000     com.apple.AppleFSCompression.AppleFSCompressionTypeZlib (1.0.0d1) <6 4 3 2 1>
48    0 0xffffff7f81b4d000 0x5000     0x5000     com.apple.AppleFSCompression.AppleFSCompressionTypeDataless (1.0.0d1) <7 6 4 3 2 1>
49    1 0xffffff7f807d2000 0x6000     0x6000     com.apple.driver.AppleUSBComposite (4.5.8) <30 4 3 1>
50    0 0xffffff7f807d8000 0x6000     0x6000     com.apple.driver.AppleUSBMergeNub (4.5.3) <49 30 4 3 1>
51    3 0xffffff7f80a43000 0x8000     0x8000     com.apple.iokit.IOUSBHIDDriver (4.4.5) <30 27 5 4 3 1>
52    0 0xffffff7f815de000 0x4000     0x4000     com.apple.driver.AppleUSBTCKeyboard (225.2) <51 30 27 7 6 5 4 3 1>
55    2 0xffffff7f80cc1000 0x76000    0x76000    com.apple.iokit.IOBluetoothFamily (4.0.3f12) <7 5 4 3 1>
56    1 0xffffff7f80d57000 0xe000     0xe000     com.apple.driver.AppleUSBBluetoothHCIController (4.0.3f12) <55 30 7 5 4 3>
57    0 0xffffff7f80d6d000 0x9000     0x9000     com.apple.driver.BroadcomUSBBluetoothHCIController (4.0.3f12) <56 55 30 5 4 3>
58    0 0xffffff7f81632000 0x4000     0x4000     com.apple.driver.AppleThunderboltPCIDownAdapter (1.2.1) <42 9 4 3>
59    0 0xffffff7f815e7000 0x13000    0x13000    com.apple.driver.AppleUSBMultitouch (227.1) <51 30 27 6 5 4 3 1>
60    1 0xffffff7f81650000 0x8000     0x8000     com.apple.driver.AppleThunderboltDPAdapterFamily (1.5.9) <42 9 8 5 4 3>
61    0 0xffffff7f81658000 0x4000     0x4000     com.apple.driver.AppleThunderboltDPInAdapter (1.5.9) <60 42 9 8 5 4 3>
62    0 0xffffff7f815e3000 0x3000     0x3000     com.apple.driver.AppleUSBTCButtons (225.2) <51 30 27 7 6 5 4 3 1>
64    3 0xffffff7f80861000 0x2b000    0x2b000    com.apple.iokit.IOSCSIArchitectureModelFamily (3.0.3) <5 4 3 1>
65    1 0xffffff7f809b8000 0x11000    0x11000    com.apple.iokit.IOUSBMassStorageClass (3.0.1) <64 30 12 5 4 3 1>
67   14 0xffffff7f80c02000 0x38000    0x38000    com.apple.iokit.IOGraphicsFamily (2.3.2) <9 7 5 4 3>
68    0 0xffffff7f817a8000 0x3a000    0x3a000    com.apple.driver.AppleIntelSNBGraphicsFB (7.1.8) <67 9 8 7 6 5 4 3 1>
72    7 0xffffff7f80c3a000 0x12000    0x12000    com.apple.iokit.IONDRVSupport (2.3.2) <67 9 7 5 4 3>
73    1 0xffffff7f81b1c000 0x3000     0x3000     com.apple.driver.AppleBacklightExpert (1.0.3) <72 67 9 5 4 3>
74    0 0xffffff7f81b71000 0x5000     0x5000     com.apple.driver.AppleBacklight (170.1.9) <73 72 67 9 5 4 3>
75    1 0xffffff7f81b0a000 0x3000     0x3000     com.apple.driver.AppleGraphicsControl (3.0.16) <72 67 9 8 7 5 4 3 1>
77    0 0xffffff7f8179b000 0x3000     0x3000     com.apple.driver.AppleLPC (1.5.3) <9 5 4 3>
78    0 0xffffff7f816c9000 0x3000     0x3000     com.apple.driver.AppleSMBusPCI (1.0.10d0) <9 5 4 3>
79    1 0xffffff7f80bcd000 0x13000    0x13000    com.apple.driver.IOPlatformPluginFamily (4.7.5d4) <8 7 6 5 4 3>
80    3 0xffffff7f80be0000 0xc000     0xc000     com.apple.driver.AppleSMC (3.1.1d8) <8 7 5 4 3>
81    0 0xffffff7f80bec000 0x11000    0x11000    com.apple.driver.ACPI_SMC_PlatformPlugin (4.7.5d4) <80 79 9 8 7 6 5 4 3>
82    0 0xffffff7f81b0d000 0xf000     0xf000     com.apple.driver.ApplePolicyControl (3.0.16) <75 72 67 9 8 7 5 4 3 1>
83    2 0xffffff7f8135c000 0x6000     0x6000     com.apple.kext.OSvKernDSPLib (1.3) <5 4>
84    4 0xffffff7f81362000 0x2a000    0x2a000    com.apple.iokit.IOAudioFamily (1.8.6fc6) <83 5 4 3 1>
85    0 0xffffff7f8138c000 0x4000     0x4000     com.apple.driver.AudioIPCDriver (1.2.2) <84 5 4 3 1>
86    0 0xffffff7f812a6000 0x5000     0x5000     com.apple.Dont_Steal_Mac_OS_X (7.0.0) <80 7 4 3 1>
87    2 0xffffff7f81931000 0xc000     0xc000     com.apple.iokit.IOHDAFamily (2.1.7f9) <5 4 3 1>
88    1 0xffffff7f8196c000 0x1a000    0x1a000    com.apple.driver.AppleHDAController (2.1.7f9) <87 67 9 6 5 4 3 1>
89    1 0xffffff7f80d76000 0x5000     0x5000     com.apple.iokit.IOEthernetAVBController (1.0.0d5) <37 5 4 3 1>
90    0 0xffffff7f80d7b000 0x9000     0x9000     com.apple.iokit.IOAVBFamily (1.0.0d22) <89 37 5 4 3 1>
91    1 0xffffff7f80b66000 0xe000     0xe000     com.apple.iokit.IOSerialFamily (10.0.5) <7 6 5 4 3 1>
92    0 0xffffff7f80d49000 0xe000     0xe000     com.apple.iokit.IOBluetoothSerialManager (4.0.3f12) <91 7 5 4 3 1>
93    0 0xffffff7f816c2000 0x5000     0x5000     com.apple.driver.AppleSMCLMU (2.0.1d2) <80 67 5 4 3>
94    0 0xffffff7f80b50000 0x12000    0x12000    com.apple.iokit.IOSurface (80.0) <7 5 4 3 1>
95    0 0xffffff7f809af000 0x6000     0x6000     com.apple.iokit.IOUserEthernet (1.0.0d1) <37 6 5 4 3 1>
96    0 0xffffff7f817e2000 0xe1000    0xe1000    com.apple.driver.AppleIntelHD3000Graphics (7.1.8) <72 67 9 7 5 4 3 1>
97    1 0xffffff7f816cc000 0xe000     0xe000     com.apple.driver.AppleSMBusController (1.0.10d0) <22 9 8 5 4 3>
98    0 0xffffff7f81afb000 0xb000     0xb000     com.apple.driver.AGPM (100.12.42) <72 67 9 5 4 3>
100    0 0xffffff7f8174b000 0x4000     0x4000     com.apple.driver.ApplePlatformEnabler (2.0.4d2) <7 5 4 3>
101    0 0xffffff7f81392000 0x5000     0x5000     com.apple.driver.AudioAUUC (1.59) <84 67 9 8 7 5 4 3 1>
102    0 0xffffff7f81b77000 0xa000     0xa000     com.apple.driver.AppleAVBAudio (1.0.0d11) <5 4 3 1>
103    0 0xffffff7f8176c000 0xa000     0xa000     com.apple.driver.AppleMCCSControl (1.0.26) <67 9 7 5 4 3 1>
104    0 0xffffff7f81601000 0x5000     0x5000     com.apple.driver.AppleUpstreamUserClient (3.5.9) <67 9 8 7 5 4 3 1>
105    0 0xffffff7f8193d000 0x22000    0x22000    com.apple.driver.AppleMikeyDriver (2.1.7f9) <97 8 5 4 3 1>
106    1 0xffffff7f81986000 0xa4000    0xa4000    com.apple.driver.DspFuncLib (2.1.7f9) <84 83 5 4 3 1>
107    0 0xffffff7f81a2a000 0xaf000    0xaf000    com.apple.driver.AppleHDA (2.1.7f9) <106 88 87 84 72 67 6 5 4 3 1>
109    0 0xffffff7f81761000 0x3000     0x3000     com.apple.driver.AppleMikeyHIDDriver (122) <27 7 4 3 1>
110    1 0xffffff7f8134c000 0x5000     0x5000     com.apple.kext.triggers (1.0) <7 6 5 4 3 1>
111    0 0xffffff7f81351000 0x9000     0x9000     com.apple.filesystems.autofs (3.0) <110 7 6 5 4 3 1>
116    3 0xffffff7f80b8a000 0xd000     0xd000     com.apple.iokit.IOCDStorageFamily (1.7) <12 5 4 3 1>
117    2 0xffffff7f80b97000 0xb000     0xb000     com.apple.iokit.IODVDStorageFamily (1.7) <116 12 5 4 3 1>
118    1 0xffffff7f80ba2000 0xa000     0xa000     com.apple.iokit.IOBDStorageFamily (1.6) <117 116 12 5 4 3 1>
119    0 0xffffff7f80bac000 0x1a000    0x1a000    com.apple.iokit.IOSCSIMultimediaCommandsDevice (3.0.3) <118 117 116 64 12 5 4 3 1>
121    0 0xffffff7f81911000 0x5000     0x5000     com.apple.driver.AppleHWSensor (1.9.4d0) <5 4 3>
122    7 0xffffff7f81c20000 0x46000    0x46000    com.apple.iokit.AppleProfileFamily (85.2) <9 7 6 5 4 3 1>
123    0 0xffffff7f81c66000 0x7000     0x7000     com.apple.driver.AppleIntelProfile (85.2) <122 6 4 3>
124    0 0xffffff7f81c6f000 0x4000     0x4000     com.apple.driver.AppleProfileCallstackAction (85.2) <122 6 5 4 3 1>
125    0 0xffffff7f81c73000 0x3000     0x3000     com.apple.driver.AppleProfileKEventAction (85.2) <122 4 3 1>
126    0 0xffffff7f81c76000 0x4000     0x4000     com.apple.driver.AppleProfileReadCounterAction (85.2) <122 6 4 3>
127    0 0xffffff7f81c7a000 0x3000     0x3000     com.apple.driver.AppleProfileRegisterStateAction (85.2) <122 4 3 1>
128    0 0xffffff7f81c7d000 0x4000     0x4000     com.apple.driver.AppleProfileThreadInfoAction (85.2) <122 6 4 3 1>
129    0 0xffffff7f81c81000 0x4000     0x4000     com.apple.driver.AppleProfileTimestampAction (85.2) <122 5 4 3 1>
130    0 0xffffff7f80807000 0xc000     0xc000     com.apple.nke.ppp (1.7) <7 6 5 4 3 1>
313    0 0xffffff7f808ff000 0x2000     0x2000     com.apple.driver.AppleUSBODD (3.0.1) <65 64 30 12 5 4 3 1>
315    0 0xffffff7f8147b000 0x35000    0x35000    com.apple.filesystems.udf (2.2) <7 5 4 1>

XNU is not a microkernel (p. 50) - Windows Internals book also mentions that about itself at the beginning

u-area (p. 52) - in Windows the equivalent can be TEB and PEB structures

UBC (p. 52) - looks like in Windows we have the same unification of file cache and virtual memory subsystems

Memorandum (Debugging Slang, Part 31)

Thursday, May 10th, 2012

Memorandum - when memory ran dump.

Examples: We got a few memorandums from that market leader.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 14, Mac OS X)

Wednesday, May 9th, 2012

This is a Mac OS X / GDB counterpart to Spiking Thread pattern previously described for Windows platforms:

(gdb) info threads
4 0×00007fff85b542df in sqrt$fenv_access_off ()
3 0×00007fff8616ee42 in __semwait_signal ()
2 0×00007fff8616ee42 in __semwait_signal ()
* 1 0×00007fff8616ee42 in __semwait_signal ()

We notice a non-waiting thread and switch to it:

(gdb) thread 4
[Switching to thread 4 (core thread 3)]
0x00007fff85b542df in sqrt$fenv_access_off ()

(gdb) bt
#0  0x00007fff85b542df in sqrt$fenv_access_off ()
#1  0×000000010cc85dc9 in thread_three (arg=0×7fff6c884ac0)
#2  0×00007fff8fac68bf in _pthread_start ()
#3  0×00007fff8fac9b75 in thread_start ()

If we disassemble the return address for thread_three function to come back from sqrt call we see an infinite loop:

(gdb) disass 0x000000010cc85dc9
Dump of assembler code for function thread_three:
0x000000010cc85db0 <thread_three+0>: push   %rbp
0×000000010cc85db1 <thread_three+1>: mov    %rsp,%rbp
0×000000010cc85db4 <thread_three+4>: sub    $0×10,%rsp
0×000000010cc85db8 <thread_three+8>: mov    %rdi,-0×10(%rbp)
0×000000010cc85dbc <thread_three+12>: mov    -0×10(%rbp),%ax
0×000000010cc85dc0 <thread_three+16>: movsd  (%rax),%xmm0
0×000000010cc85dc4 <thread_three+20>: callq  0×10cc85eac <dyld_stub_sqrt>
0×000000010cc85dc9 <thread_three+25>: mov    -0×10(%rbp),%rax
0×000000010cc85dcd <thread_three+29>: movsd  %xmm0,(%rax)
0×000000010cc85dd1 <thread_three+33>: jmpq   0×10cc85dbc <thread_three+12>
End of assembler dump.

Here’s the source code of the modeling application:

void * thread_one (void *arg)

{

    while (1)

    {

       sleep (1);

    }

 

    return 0;

}

 

void * thread_two (void *arg)

{

    while (1)

    {

        sleep (2);

    }

 

    return 0;

}

 

void * thread_three (void *arg)

{

    while (1)

    {

        *(double*)arg=sqrt(*(double *)arg);

    }

 

    return 0;

}

 

int main(int argc, const char * argv[])

{

    pthread_t threadID_one, threadID_two, threadID_three;

 

    double result = 0xffffffff;

 

    pthread_create (&threadID_one, NULL, thread_one, NULL);

    pthread_create (&threadID_two, NULL, thread_two, NULL);

    pthread_create (&threadID_three, NULL, thread_three,

       &result);

 

    pthread_join(threadID_three, NULL);

 

    return 0;

}

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Crash Dump Analysis Patterns (Part 6a, Mac OS X)

Thursday, May 3rd, 2012

This is a Mac OS X / GDB counterpart to NULL Pointer (code) pattern previously described for Windows platforms:

(gdb) bt
#0 0×0000000000000000 in ?? ()
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) disass 0×000000010e8cce73-3 0×000000010e8cce73
Dump of assembler code from 0×10e8cce70 to 0×10e8cce73:
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) info r rdi
rdi 0x7fff6e4cbac0 140735043910336

(gdb) x/2 0x7fff6e4cbac0
0x7fff6e4cbac0: 0x0000000a 0×00000000

(gdb) p/x *($rdi+8)
$7 = 0×0

(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
#2 0×000000010e8cce95 in foo (ps=0×7fff6e4cbac0)
#3 0×000000010e8cced5 in main (argc=1, argv=0×7fff6e4cbb08)

(gdb) ptype MYSTRUCT
type = struct _MyStruct_tag {
int data;
PFUNC pfunc;
}

(gdb) print {MYSTRUCT}0×7fff6e4cbac0
$2 = {data = 10, pfunc = 0}

Here’s the source code of the modeling application:

typedef void (*PFUNC)(void);

 

typedef struct _MyStruct_tag

{

    int   data;

    PFUNC pfunc;

} MYSTRUCT;

 

void bar(MYSTRUCT *ps)

{

    ps->pfunc();

}

 

void foo(MYSTRUCT *ps)

{

    bar(ps);

}

 

int main(int argc, const char * argv[])

{

    MYSTRUCT pstruct = {10, NULL};

 

    foo(&pstruct);

 

    return 0;

} 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Dump Analysis as a Labour Process

Tuesday, May 1st, 2012

; Composed a verse for today

Labour Day
First of May
Analyze
Today

; Plan to analyze from 32 to 64 dumps

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -