Software Diagnostics Library

It is hard to write a review of this book because I have been reading it sporadically for more than 2 years and just finished today. When I bought it I didn’t know much about Christianity and it various branches (as being educated in secular Soviet Union) so it was difficult reading due to many historical and theological facts. Now I plan to watch DVD series from the same author and already started reading multi-volume sets such as History of the Christian Church (Schaff, Protestant perspective), Studies in Church History (Parsons, Catholic perspective) and waiting for arrival of 9 volumes of Cambridge History of Christianity bought with a great discount from Folio Society.

Just a small note that the last chapters were brief but very enlightening, for example, last pages about the disappearing of Hell and the appearing of burning (cremation).

Christianity: The First Three Thousand Years

PS. Actually learning about Christian faith helped me to deeply understand my own Memory Religion (Memorianity) with its conception of original memory defect: Memory Religion: A Core Testament of Memorianity (with an old original cover below)

- Dmitry Vostokov @ LiterateScientist.com -

As we started providing memory dump analysis pattern examples for Mac OS X we resume our table of command correspondence between WinDbg and GDB providing some corrections on the way. For example, in the previous version of table we omitted a correspondence to ub WinDbg command. Now we provide such an equivalent:

(gdb) bt
[...]
#1 0×000000010e8cce73 in bar (ps=0×7fff6e4cbac0)
[…]

(gdb) disas 0×000000010e8cce73-10 0×000000010e8cce73
Dump of assembler code from 0×10e8cce69 to 0×10e8cce73:
0×000000010e8cce69 : mov %edi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

Please note that the beginning of assembly will be dependent on how good we guessed the offset:

(gdb) disas 0x000000010e8cce73-0×10 0×000000010e8cce73
Dump of assembler code from 0×10e8cce63 to 0×10e8cce73:
0×000000010e8cce63 : in $0×48,%eax
0×000000010e8cce65 : sub $0×10,%esp
0×000000010e8cce68 : mov %rdi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

(gdb) disas 0x000000010e8cce73-0×13 0×000000010e8cce73
Dump of assembler code from 0×10e8cce60 to 0×10e8cce73:
0×000000010e8cce60 : push %rbp
0×000000010e8cce61 : mov %rsp,%rbp
0×000000010e8cce64 : sub $0×10,%rsp
0×000000010e8cce68 : mov %rdi,-0×8(%rbp)
0×000000010e8cce6c : mov -0×8(%rbp),%rdi
0×000000010e8cce70 : callq *0×8(%rdi)
End of assembler dump.

However, we can ignore that because our goal is to check whether a CPU instruction before a return address is a call.

Additional commands we add are x/<N>bc for db (WinDbg), thread <N> for ~<N>s (WinDbg, process dumps), maintenance info sections for for !address (WinDbg), add-symbol-file for .reload (WinDbg), info r for r (WinDbg).

Action                      | GDB                 | WinDbg
----------------------------------------------------------------
Start the process           | run                 | g
Exit                        | (q)uit              | q
Disassemble (forward)       | (disas)semble       | uf, u
Disassemble N instructions  | x/<N>i              | -
Disassemble (backward)      | disas <a-o> <a>     | ub
Stack trace                 | backtrace (bt)      | k
Full stack trace            | bt full             | kv
Stack trace with parameters | bt full             | kP
Partial trace (innermost)   | bt <N>              | k <N>
Partial trace (outermost)   | bt -<N>             | -
Stack trace for all threads | thread apply all bt | ~*k
Breakpoint                  | break               | bp
Frame numbers               | any bt command      | kn
Select frame                | frame               | .frame
Display parameters          | info args           | dv /t /i /V
Display locals              | info locals         | dv /t /i /V
Dump byte char array        | x/<N>bc             | db
Switch to thread            | thread <N>          | ~<N>s
Sections/regions            | maint info sections | !address
Load symbol file            | add-symbol-file     | .reload
CPU registers               | i(nfo) r            | r

Now an advertisement command:

(gdb) info training
(gdb) Accelerated Mac OS X Core Dump Analysis training

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a special variant of Blocked Thread pattern where we have a timeout value so a thread is potentially blocked only temporarily. For example, this main thread is blocked waiting for beep sound to finish after a minute:

0:000> kvL
ChildEBP RetAddr Args to Child
0291f354 7c90d21a 7c8023f1 00000001 0291f388 ntdll!KiFastSystemCallRet
0291f358 7c8023f1 00000001 0291f388 7c90d27e ntdll!NtDelayExecution+0xc
0291f3b0 7c837beb 0000ea60 00000001 00000004 kernel32!SleepEx+0×61
0291f404 004952a2 00000370 0000ea60 004d6ae2 kernel32!Beep+0×1b3
0291f410 004d6ae2 00000370 0000ea60 004d6ed4 Application!DoBeep+0×16
[…]
0291ffec 00000000 0045aad0 00e470a0 00000000 kernel32!BaseThreadStart+0×37

0:000> ? ea60/0n1000
Evaluate expression: 60 = 0000003c

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes I hear voices saying that Linux, FreeBSD, and Mac OS X core dumps are uninteresting. This is not true. If you haven’t seen anything interesting there it just simply means you have only encountered a limited amount of abnormal software behaviour. The widespread usage of Windows OS means that most patterns have been diagnosed and described first and other OS are waiting their turn.

My goal is to have a pattern catalog with examples from different OS. For example, currently, all Mac OS X patterns I provide are just examples to existing Windows pattern names. All OS share the same structure and behavior, for example, structural memory analysis patterns and the same computational model. Although structural patterns are different from behavioral patterns I also plan to expand the structural list significantly especially in relation to forthcoming Windows malware analysis training. Regarding behavioral patterns it is possible to model and predict specific pattern examples for another OS by using already existing catalog.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is an example of Punctuated Memory Leak pattern somewhat similar to a large block allocation leak for process heap (see a modeling example). An application has some functionality and after each command its commited memory was increasing by 50 - 60 Mb. 3 process dumps were taken with one before failures and then after each failure:

// Before failures

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          76c50000 (   1.856 Gb)           92.79%
<unclassified>                          270           4d6f000 (  77.434 Mb)  52.45%    3.78%
Image                                   620           31bf000 (  49.746 Mb)  33.70%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  13.55%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.14%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.09%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.05%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             296           3bca000 (  59.789 Mb)  40.50%    2.92%
MEM_IMAGE                               647           340c000 (  52.047 Mb)  35.26%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  24.24%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          76c50000 (   1.856 Gb)           92.79%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  54.21%    3.91%
MEM_COMMIT                              896           439a000 (  67.602 Mb)  45.79%    3.30%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  21.12%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  20.93%    1.51%
PAGE_READWRITE                          309            4c2000 (   4.758 Mb)   3.22%    0.23%
PAGE_WRITECOPY                           43             6a000 ( 424.000 kb)   0.28%    0.02%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.20%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.04%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         6130000          5fb70000 (   1.496 Gb)
<unclassified>                                abf000           13d1000 (  19.816 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

// After the 1st failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          7388c000 (   1.805 Gb)           90.26%
<unclassified>                          272           8133000 ( 129.199 Mb)  64.80%    6.31%
Image                                   614           31bf000 (  49.746 Mb)  24.95%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  10.03%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.10%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.07%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.04%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             297           6f8e000 ( 111.555 Mb)  55.95%    5.45%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  26.10%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  17.95%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          7388c000 (   1.805 Gb)           90.26%
MEM_COMMIT                              892           775e000 ( 119.367 Mb)  59.87%    5.83%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  40.13%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          314           38a3000 (  56.637 Mb)  28.40%    2.77%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  15.63%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  15.49%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.15%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.15%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         94f4000          5c7ac000 (   1.445 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
[...]
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

0:000> ? 33c4000/0n1024
Evaluate expression: 53008 = 0000cf10

// After the 2nd failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    268          704c8000 (   1.755 Gb)           87.74%
<unclassified>                          273           b4f7000 ( 180.965 Mb)  72.05%    8.84%
Image                                   614           31bf000 (  49.746 Mb)  19.81%    2.43%
Stack                                    60           1400000 (  20.000 Mb)   7.96%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.08%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.05%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.03%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             298           a352000 ( 163.320 Mb)  65.03%    7.97%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  20.72%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  14.25%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                268          704c8000 (   1.755 Gb)           87.74%
MEM_COMMIT                              893           ab22000 ( 171.133 Mb)  68.14%    8.36%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  31.86%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          315           6c67000 ( 108.402 Mb)  43.16%    5.29%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  12.41%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  12.30%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.12%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.12%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         c8c4000          593dc000 (   1.394 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
9500000  c8c4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

The name of this pattern comes from the process of discrete large memory allocations that happen after specific actions or events. Between them there is no visible or substantial increase in memory usage.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The cover of the latest Economist issue arrived today picturing a third industrial revolution prompted me to write about Software as means of production that I was thinking for sometime and even created a Software Generalist Party you are welcome to join. Software generalists are the future driving force of society change and I started working on a work comparable to Marx’s Capital called Software, Volume 1 subtitled as A Critical Analysis of Industrial Production (ISBN: 978-1908043375). It will also include an analysis of new emerging commodities such as memories.

This is a provisional Mac OS X example of Shared Buffer Overwrite pattern. Originally I wanted to construct a default C runtime heap corruption example using malloc / free functions. Unfortunately I couldn’t get heap corrupted easily as was possible in Windows Visual C++ environment by writing before or after allocated block. Desperately I printed allocated pointers and they all pointed to memory blocks laid out one after another without any headers in between (could be just a default Apple LLVM C runtime implementation and I have to check that with GCC). Therefore, any subsequent reallocation didn’t cause corruption either. So all this naturally fits into shared buffer overwrites or underwrites where corruption is only detectable when the overwritten data is used such as a pointer dereference.

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    strcpy(p1, “Hello World!”);

    printf(“p1 = %p\n”, p1);

    printf(“*p1 = %s\n”, p1);

    char *p2 = (char *) malloc (1024);

    strcpy(p2, “Hello World!”);

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

    char *p3 = (char *) malloc (1024);

    strcpy(p3, “Hello World!”);

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

    strcpy(p2-sizeof(p2), “Hello Crash!”);

    strcpy(p3-sizeof(p3), “Hello Crash!”);

    p2 = (char *)realloc(p2, 2048);

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

    char *p4 = (char *) malloc (1024);

    strcpy(p4-sizeof(p4), “Hello Crash!”);

    printf(“p4 = %p\n”, p4);

    printf(“*p4 = %s\n”, p4);

    p3 = (char *)realloc(p3, 2048);

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

    char *p5 = NULL; // to force a core dump

    *p5 = 0;

    free (p4);

    free (p3);

    free (p2);

    free (p1);

    return 0;

}

When we run the program above we get this output:

p1 = 0x7fc6d9000000
*p1 = Hello World!
p2 = 0×7fc6d9001400
*p2 = Hello World!
p3 = 0×7fc6d9001800
*p3 = Hello World!
p2 = 0×7fc6d9001c00
*p2 = ash!
p4 = 0×7fc6d9001400
*p4 = ash!
p3 = 0×7fc6d9002400
*p3 = ash!
Segmentation fault: 11 (core dumped)

Now is GDB output:

(gdb) x/1024bc p1
0x7fc6d9000000: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9000008: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000010: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90003e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/32bc p1+1024-sizeof(p1)
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000400: 42 ‘*’ 112 ‘p’ 51 ‘3′ 32 ‘ ‘ 61 ‘=’ 32 ‘ ‘ 97 ‘a’ 115 ’s’
0×7fc6d9000408: 104 ‘h’ 33 ‘!’ 10 ‘\n’ 100 ‘d’ 57 ‘9′ 48 ‘0′ 48 ‘0′ 50 ‘2′
0×7fc6d9000410: 52 ‘4′ 48 ‘0′ 48 ‘0′ 10 ‘\n’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/2048bc p2
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d9001fe8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9002000: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002008: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90023e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2-sizeof(p2)
0×7fc6d9001bf8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c18: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c20: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c28: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c30: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2+2048-sizeof(p2)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p3
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90027e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3-sizeof(p3)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3+1024-sizeof(p3)
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002800: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002808: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p4
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90017e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
(gdb) x/64bc p4-sizeof(p4)
0×7fc6d90013f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p4+1024-sizeof(p4)
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001800: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001808: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

• Title: Memory Dump Analysis Anthology, Volume 1
• Author: Dmitry Vostokov
• Publisher: OpenTask (Summer 2012)
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 800 pages
• ISBN-13: 978-1-908043-35-1

• Hardcover: 800 pages
• ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In a week this short full color book should appear in online bookstores:

• Title: Software Narratology: An Introduction to the Applied Science of Software Stories
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Description: This is a transcript of Memory Dump Analysis Services Webinar about Software Narratology: an exciting new discipline and a field of research founded by DumpAnalysis.org. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes. Topics also include software narremes and their types, anticipatory software construction and software diagnostics.
• Publisher: OpenTask (April 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 26 pages
• ISBN-13: 978-1908043078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Finally the book has arrived and I plan to continue my close reading with relevant comments pointing to DumpAnalysis.org and any additional experiments if needed, for example, to cover x64 Windows (the new edition is still 32-bit oriented in WinDbg examples).

Windows Internals, Part 1: Covering Windows Server 2008 R2 and Windows 7

Due to many questions on recommended books to learn assembly language asked during Accelerated Windows Memory Dump Analysis training sessions we provide these references:

Windows Debugging: Practical Foundations
x64 Windows Debugging: Practical Foundations

Each book can be read independently although some platform-independent content overlaps. x64 bit book focuses on 64-bit only.

We believe these books provide all necessary motivation, context and practical foundation for other in-depth assembly language textbooks on the market.

I’m also working on the similar book for x64 Mac OS X.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There are many lists for great books but I’d like to share the one that have a more personal touch. These books I wanted to read on the first sight and devoured them when I had an occasion.

First book that made a great impression on me was C Programming Language which I saw and read in 1987 (in Russian translation with very nice hardcover binding). There was a C implementation available on a PDP-11 clone I worked with at that time.

Another book that reinforced my better function prototype reading and pointer declarations was Expert C Programming.

I confess I had a long trouble understanding C++ long back ago despite reading The C++ Programming Language by Bjarne Stroustrup (reading several editions) and Effective C++ series by Scott Meyers partly because of my long practice of Win32 programming using C and partly because I didn’t understand why I needed C++ to do the same. Popular books such as C++ for Real Programmers (in Russian translation) terrified me with their complexity (although I made honest efforts to read them from cover to cover). However, when I saw and started reading the book Advanced CORBA Programming with C++ I really understood why we needed C++ and how it all worked.

After that rereading The C++ Programming Language and (More) Effective C++ books was a great pleasure.

The other part of C++, STL was greatly clarified by reading the following book in 2003 from cover to cover (at that time the first edition): The C++ Standard Library: A Tutorial and Reference.

Inside the C++ Object Model book clarified C++ implementation issues and helped later with crash dump analysis of C++ programs.

I plan to continue this “Great Impression” series with other topics and also to publish a recommended reading list for C++ based on books from my personal library that started growing when I was an employee at Programming Research in 2001 - 2003:

The row of books is all about C++. Except the big thick red book about English language grammar (as I became interested in other grammars than C++) The Cambridge Grammar of the English Language:

Now !ad debugger command: I’m developing a traning course Deep Down C++ with Memory Dump Analysis.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Allocated dynamic memory such as process heap can remain reserved after deallocation and its virtual memory region might become unavailable for usage. One example of this I encountered recently while debugging a .NET service. During a peak usage it reported various out-of-memory events but its managed heap was healthy and didn’t consume much. However, its process heap statistics showed a large reserved heap segment missing in a similar memory dump from a development environment. Remaining allocated entries in that heap segment contained a specific module hint that allowed us to suggest removing a 3rd-party product from a production environment.

In order to provide the proof of that possible scenario of reserved heap regions we created a special modeling application:

int _tmain(int argc, _TCHAR* argv[])
{
  static char *pAlloc[1000000];
  for (int i = 0; i < 1000000; i++)
  {
    pAlloc[i] = (char *)malloc (1000);
  }
  getc(stdin);
  for (int i = 0; i < 1000000; i++)
  {
     free(pAlloc[i]);
  }
  getc(stdin);
  return 0;
}

Here’s the debugging log:

0:001> .symfix c:\mss

0:001> .reload
Reloading current modules
.....

After allocation:

0:001> !heap -s
LFH Key                   : 0x156356e0
Termination on corruption : ENABLED
Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
(k)     (k)    (k)     (k) length      blocks cont. heap
—————————————————————————–
00520000 00000002    1024    112   1024      8     1     1    0      0   LFH
007e0000 00001002 1019328 1012444 1019328    131    68    67    0      0   LFH
—————————————————————————–

0:001> g
(1588.14b0): Break instruction exception - code 80000003 (first chance)
eax=7efda000 ebx=00000000 ecx=00000000 edx=770ff85a esi=00000000 edi=00000000
eip=7707000c esp=00f0f7e4 ebp=00f0f810 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
7707000c cc              int     3

After deallocation:

0:001> !heap -s
LFH Key                   : 0x156356e0
Termination on corruption : ENABLED
Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
(k)     (k)    (k)     (k) length      blocks cont. heap
—————————————————————————–
00520000 00000002    1024    112   1024      8     1     1    0      0   LFH
007e0000 00001002 1019328  73040 1019328  71365   419   165    0      0   LFH
External fragmentation  97 % (419 free blocks)
Virtual address fragmentation  92 % (165 uncommited ranges)
—————————————————————————–

0:001> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                     26          3fbe7000 (1019.902 Mb)           49.80%
<unclassified>                          752          3f8ec000 (1016.922 Mb)  98.92%   49.66%
Image                                    41            76b000 (   7.418 Mb)   0.72%    0.36%
Stack                                     6            200000 (   2.000 Mb)   0.19%    0.10%
MemoryMappedFile                          8            1af000 (   1.684 Mb)   0.16%    0.08%
TEB                                       2              2000 (   8.000 kb)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             734          3f8a2000 (1016.633 Mb)  98.89%   49.64%
MEM_IMAGE                                68            9b8000 (   9.719 Mb)   0.95%    0.47%
MEM_MAPPED                                8            1af000 (   1.684 Mb)   0.16%    0.08%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                 26          3fbe7000 (1019.902 Mb)           49.80%
MEM_RESERVE                             374          3f6e8000 (1014.906 Mb)  98.72%   49.56%
MEM_COMMIT                              436            d21000 (  13.129 Mb)   1.28%    0.64%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          383            725000 (   7.145 Mb)   0.69%    0.35%
PAGE_EXECUTE_READ                        10            414000 (   4.078 Mb)   0.40%    0.20%
PAGE_READONLY                            29            1cd000 (   1.801 Mb)   0.18%    0.09%
PAGE_WRITECOPY                           10             12000 (  72.000 kb)   0.01%    0.00%
PAGE_READWRITE|PAGE_GUARD                 4              9000 (  36.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                        3f0c0000          33050000 ( 816.313 Mb)
<unclassified>                              158a1000            fcf000 (  15.809 Mb)
Image                                        1083000            3d1000 (   3.816 Mb)
Stack                                         200000             fd000 (1012.000 kb)
MemoryMappedFile                            7efe5000             fb000 (1004.000 kb)
TEB                                         7efda000              1000 (   4.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

We see that free memory available for allocation is only 816 Mb.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services organizes a free Webinar on Unified Software Diagnostics (USD) and the new scalable cost-effective software support model called Pattern-Driven Software Support devised to address various shortcomings in existing tiered software support organizations. Examples cover Windows, Mac OS  and Linux.

Date: 22nd of June, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/172771078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Address space-wide search for errors and status codes may show Coincidental Error Code pattern:

0:000> !heap -x -v c0000005
Search VM for address range c0000005 - c0000005 : 028690b8 (c0000005), [...]

0:000> dd 028690b8 l1
028690b8  c0000005

In such cases we need to check whether the addresses belong to volatile regions such as stack because it is possible to have such values as legitimate code and image data:

0:000> !address 028690b8
Usage:                  Image
Allocation Base:        02700000
Base Address:           02869000
End Address:            02874000
Region Size:            0000b000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000002 PAGE_READONLY
More info:              lmv m ModuleA
More info:              !lmi ModuleA
More info:              ln 0×28690b8

0:000> u 028690b8
ModuleA!ComputeB:
028690b8 050000c000      add     eax,0C00000h
[...]

Another example:

0:000> !heap -x -v c0000005
Search VM for address range 00000000c0000005 - 00000000c0000005 : 7feff63ab60 (c0000005),

0:000> !address 7feff63ab60
Usage:                  Image
Allocation Base:        000007fe`ff460000
Base Address:           000007fe`ff635000
End Address:            000007fe`ff63c000
Region Size:            00000000`00007000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE
More info:              lmv m ole32
More info:              !lmi ole32
More info:              ln 0×7feff63ab60

0:000> dp 7feff63ab60
000007fe`ff63ab60  00000000`c0000005 c0000194`00000001
000007fe`ff63ab70  00000001`00000000 00000000`c00000aa
000007fe`ff63ab80  80000002`00000001 00000001`00000000
000007fe`ff63ab90  00000000`c0000096 c000001d`00000001
000007fe`ff63aba0  00000001`00000000 00000000`80000003
000007fe`ff63abb0  c00000fd`00000001 00000001`00000000
000007fe`ff63abc0  00000000`c0000235 c0000006`00000001
000007fe`ff63abd0  00000001`00000000 00000000`c0000420

In the latter case the data structure suggests a table of errors:

0:000> ln 7feff63ab60
(000007fe`ff63ab60)   ole32!gReportedExceptions

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Discovered this forthcoming book and immediately preordered:

Inside Windows Debugging: A Practical Guide to Debugging and Tracing Strategies in Windows

From Safari Books Online table of contents I see it also includes Event Tracing for Windows:

http://my.safaribooksonline.com/book/-/9780735671348

Looking forward to reading it and writing a review.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first Windows pattern called Multiple Exceptions in user mode now has Mac OS X equivalent. In the example below there are 3 threads and two of them experienced NULL Pointer (data) access violation exception:

(gdb) thread apply all bt full

Thread 3 (core thread 2):
#0  0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 2 (core thread 1):
#0  0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 1 (core thread 0):
#0  0x00007fff854e0e42 in __semwait_signal ()
No symbol table info available.
#1  0x00007fff8ababdea in nanosleep ()
No symbol table info available.
#2  0x00007fff8ababc2c in sleep ()
No symbol table info available.
#3  0x00000001062ffec3 in main (argc=1, argv=0x7fff65efeab8)
at main.c:36
threadID_one = (pthread_t) 0×1063b4000
threadID_two = (pthread_t) 0×106581000

(gdb) thread 2
[Switching to thread 2 (core thread 1)]
0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
16    *p = 1;

(gdb) p/x p
$1 = 0×0

(gdb) thread 3
[Switching to thread 3 (core thread 2)]
0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
24    *p = 2;

(gdb) p/x p
$2 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -