Software Diagnostics Library

This is an example of Punctuated Memory Leak pattern somewhat similar to a large block allocation leak for process heap (see a modeling example). An application has some functionality and after each command its commited memory was increasing by 50 - 60 Mb. 3 process dumps were taken with one before failures and then after each failure:

// Before failures

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          76c50000 (   1.856 Gb)           92.79%
<unclassified>                          270           4d6f000 (  77.434 Mb)  52.45%    3.78%
Image                                   620           31bf000 (  49.746 Mb)  33.70%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  13.55%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.14%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.09%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.05%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             296           3bca000 (  59.789 Mb)  40.50%    2.92%
MEM_IMAGE                               647           340c000 (  52.047 Mb)  35.26%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  24.24%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          76c50000 (   1.856 Gb)           92.79%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  54.21%    3.91%
MEM_COMMIT                              896           439a000 (  67.602 Mb)  45.79%    3.30%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  21.12%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  20.93%    1.51%
PAGE_READWRITE                          309            4c2000 (   4.758 Mb)   3.22%    0.23%
PAGE_WRITECOPY                           43             6a000 ( 424.000 kb)   0.28%    0.02%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.20%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.04%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         6130000          5fb70000 (   1.496 Gb)
<unclassified>                                abf000           13d1000 (  19.816 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

// After the 1st failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          7388c000 (   1.805 Gb)           90.26%
<unclassified>                          272           8133000 ( 129.199 Mb)  64.80%    6.31%
Image                                   614           31bf000 (  49.746 Mb)  24.95%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  10.03%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.10%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.07%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.04%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             297           6f8e000 ( 111.555 Mb)  55.95%    5.45%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  26.10%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  17.95%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          7388c000 (   1.805 Gb)           90.26%
MEM_COMMIT                              892           775e000 ( 119.367 Mb)  59.87%    5.83%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  40.13%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          314           38a3000 (  56.637 Mb)  28.40%    2.77%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  15.63%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  15.49%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.15%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.15%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         94f4000          5c7ac000 (   1.445 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
[...]
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

0:000> ? 33c4000/0n1024
Evaluate expression: 53008 = 0000cf10

// After the 2nd failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    268          704c8000 (   1.755 Gb)           87.74%
<unclassified>                          273           b4f7000 ( 180.965 Mb)  72.05%    8.84%
Image                                   614           31bf000 (  49.746 Mb)  19.81%    2.43%
Stack                                    60           1400000 (  20.000 Mb)   7.96%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.08%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.05%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.03%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             298           a352000 ( 163.320 Mb)  65.03%    7.97%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  20.72%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  14.25%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                268          704c8000 (   1.755 Gb)           87.74%
MEM_COMMIT                              893           ab22000 ( 171.133 Mb)  68.14%    8.36%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  31.86%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          315           6c67000 ( 108.402 Mb)  43.16%    5.29%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  12.41%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  12.30%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.12%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.12%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         c8c4000          593dc000 (   1.394 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
9500000  c8c4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

The name of this pattern comes from the process of discrete large memory allocations that happen after specific actions or events. Between them there is no visible or substantial increase in memory usage.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a provisional Mac OS X example of Shared Buffer Overwrite pattern. Originally I wanted to construct a default C runtime heap corruption example using malloc / free functions. Unfortunately I couldn’t get heap corrupted easily as was possible in Windows Visual C++ environment by writing before or after allocated block. Desperately I printed allocated pointers and they all pointed to memory blocks laid out one after another without any headers in between (could be just a default Apple LLVM C runtime implementation and I have to check that with GCC). Therefore, any subsequent reallocation didn’t cause corruption either. So all this naturally fits into shared buffer overwrites or underwrites where corruption is only detectable when the overwritten data is used such as a pointer dereference.

int main(int argc, const char * argv[])

{

    char *p1 = (char *) malloc (1024);

    strcpy(p1, “Hello World!”);

    printf(“p1 = %p\n”, p1);

    printf(“*p1 = %s\n”, p1);

    char *p2 = (char *) malloc (1024);

    strcpy(p2, “Hello World!”);

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

    char *p3 = (char *) malloc (1024);

    strcpy(p3, “Hello World!”);

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

    strcpy(p2-sizeof(p2), “Hello Crash!”);

    strcpy(p3-sizeof(p3), “Hello Crash!”);

    p2 = (char *)realloc(p2, 2048);

    printf(“p2 = %p\n”, p2);

    printf(“*p2 = %s\n”, p2);

    char *p4 = (char *) malloc (1024);

    strcpy(p4-sizeof(p4), “Hello Crash!”);

    printf(“p4 = %p\n”, p4);

    printf(“*p4 = %s\n”, p4);

    p3 = (char *)realloc(p3, 2048);

    printf(“p3 = %p\n”, p3);

    printf(“*p3 = %s\n”, p3);

    char *p5 = NULL; // to force a core dump

    *p5 = 0;

    free (p4);

    free (p3);

    free (p2);

    free (p1);

    return 0;

}

When we run the program above we get this output:

p1 = 0x7fc6d9000000
*p1 = Hello World!
p2 = 0×7fc6d9001400
*p2 = Hello World!
p3 = 0×7fc6d9001800
*p3 = Hello World!
p2 = 0×7fc6d9001c00
*p2 = ash!
p4 = 0×7fc6d9001400
*p4 = ash!
p3 = 0×7fc6d9002400
*p3 = ash!
Segmentation fault: 11 (core dumped)

Now is GDB output:

(gdb) x/1024bc p1
0x7fc6d9000000: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9000008: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000010: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90003e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/32bc p1+1024-sizeof(p1)
0×7fc6d90003f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9000400: 42 ‘*’ 112 ‘p’ 51 ‘3′ 32 ‘ ‘ 61 ‘=’ 32 ‘ ‘ 97 ‘a’ 115 ’s’
0×7fc6d9000408: 104 ‘h’ 33 ‘!’ 10 ‘\n’ 100 ‘d’ 57 ‘9′ 48 ‘0′ 48 ‘0′ 50 ‘2′
0×7fc6d9000410: 52 ‘4′ 48 ‘0′ 48 ‘0′ 10 ‘\n’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/2048bc p2
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d9001fe8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001ff8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9002000: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002008: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90023e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2-sizeof(p2)
0×7fc6d9001bf8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c00: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001c08: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c10: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c18: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c20: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c28: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001c30: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p2+2048-sizeof(p2)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p3
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90027e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3-sizeof(p3)
0×7fc6d90023f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9002408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p3+1024-sizeof(p3)
0×7fc6d90027f8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002800: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002808: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9002830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/1024bc p4
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
[…]
0×7fc6d90017e8: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f0: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
(gdb) x/64bc p4-sizeof(p4)
0×7fc6d90013f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001400: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001408: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001410: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001418: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001420: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001428: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001430: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
(gdb) x/64bc p4+1024-sizeof(p4)
0×7fc6d90017f8: 72 ‘H’ 101 ‘e’ 108 ‘l’ 108 ‘l’ 111 ‘o’ 32 ‘ ‘ 67 ‘C’ 114 ‘r’
0×7fc6d9001800: 97 ‘a’ 115 ’s’ 104 ‘h’ 33 ‘!’ 0 ‘\0′ 32 ‘ ‘ 87 ‘W’ 111 ‘o’
0×7fc6d9001808: 114 ‘r’ 108 ‘l’ 100 ‘d’ 33 ‘!’ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001810: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001818: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001820: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001828: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′
0×7fc6d9001830: 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′ 0 ‘\0′

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After 4 years in print this bestselling title needs an update to address minor changes, include extra examples and reference additional research published in Volumes 2, 3, 4, 5 and 6.

• Title: Memory Dump Analysis Anthology, Volume 1
• Author: Dmitry Vostokov
• Publisher: OpenTask (Summer 2012)
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 800 pages
• ISBN-13: 978-1-908043-35-1

• Hardcover: 800 pages
• ISBN-13: 978-1-908043-36-8

The cover for both paperback and hardcover titles will also have a matte finish. We used A Memory Window artwork for the back cover.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In a week this short full color book should appear in online bookstores:

• Title: Software Narratology: An Introduction to the Applied Science of Software Stories
• Authors: Dmitry Vostokov, Memory Dump Analysis Services
• Description: This is a transcript of Memory Dump Analysis Services Webinar about Software Narratology: an exciting new discipline and a field of research founded by DumpAnalysis.org. When software executes it gives us its stories in the form of UI events, software traces and logs. Such stories can be analyzed for their structure and patterns for troubleshooting, debugging and problem resolution purposes. Topics also include software narremes and their types, anticipatory software construction and software diagnostics.
• Publisher: OpenTask (April 2012)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 26 pages
• ISBN-13: 978-1908043078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Due to many questions on recommended books to learn assembly language asked during Accelerated Windows Memory Dump Analysis training sessions we provide these references:

Windows Debugging: Practical Foundations
x64 Windows Debugging: Practical Foundations

Each book can be read independently although some platform-independent content overlaps. x64 bit book focuses on 64-bit only.

We believe these books provide all necessary motivation, context and practical foundation for other in-depth assembly language textbooks on the market.

I’m also working on the similar book for x64 Mac OS X.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Allocated dynamic memory such as process heap can remain reserved after deallocation and its virtual memory region might become unavailable for usage. One example of this I encountered recently while debugging a .NET service. During a peak usage it reported various out-of-memory events but its managed heap was healthy and didn’t consume much. However, its process heap statistics showed a large reserved heap segment missing in a similar memory dump from a development environment. Remaining allocated entries in that heap segment contained a specific module hint that allowed us to suggest removing a 3rd-party product from a production environment.

In order to provide the proof of that possible scenario of reserved heap regions we created a special modeling application:

int _tmain(int argc, _TCHAR* argv[])
{
  static char *pAlloc[1000000];
  for (int i = 0; i < 1000000; i++)
  {
    pAlloc[i] = (char *)malloc (1000);
  }
  getc(stdin);
  for (int i = 0; i < 1000000; i++)
  {
     free(pAlloc[i]);
  }
  getc(stdin);
  return 0;
}

Here’s the debugging log:

0:001> .symfix c:\mss

0:001> .reload
Reloading current modules
.....

After allocation:

0:001> !heap -s
LFH Key                   : 0x156356e0
Termination on corruption : ENABLED
Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
(k)     (k)    (k)     (k) length      blocks cont. heap
—————————————————————————–
00520000 00000002    1024    112   1024      8     1     1    0      0   LFH
007e0000 00001002 1019328 1012444 1019328    131    68    67    0      0   LFH
—————————————————————————–

0:001> g
(1588.14b0): Break instruction exception - code 80000003 (first chance)
eax=7efda000 ebx=00000000 ecx=00000000 edx=770ff85a esi=00000000 edi=00000000
eip=7707000c esp=00f0f7e4 ebp=00f0f810 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
7707000c cc              int     3

After deallocation:

0:001> !heap -s
LFH Key                   : 0x156356e0
Termination on corruption : ENABLED
Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
(k)     (k)    (k)     (k) length      blocks cont. heap
—————————————————————————–
00520000 00000002    1024    112   1024      8     1     1    0      0   LFH
007e0000 00001002 1019328  73040 1019328  71365   419   165    0      0   LFH
External fragmentation  97 % (419 free blocks)
Virtual address fragmentation  92 % (165 uncommited ranges)
—————————————————————————–

0:001> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                     26          3fbe7000 (1019.902 Mb)           49.80%
<unclassified>                          752          3f8ec000 (1016.922 Mb)  98.92%   49.66%
Image                                    41            76b000 (   7.418 Mb)   0.72%    0.36%
Stack                                     6            200000 (   2.000 Mb)   0.19%    0.10%
MemoryMappedFile                          8            1af000 (   1.684 Mb)   0.16%    0.08%
TEB                                       2              2000 (   8.000 kb)   0.00%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             734          3f8a2000 (1016.633 Mb)  98.89%   49.64%
MEM_IMAGE                                68            9b8000 (   9.719 Mb)   0.95%    0.47%
MEM_MAPPED                                8            1af000 (   1.684 Mb)   0.16%    0.08%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                 26          3fbe7000 (1019.902 Mb)           49.80%
MEM_RESERVE                             374          3f6e8000 (1014.906 Mb)  98.72%   49.56%
MEM_COMMIT                              436            d21000 (  13.129 Mb)   1.28%    0.64%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          383            725000 (   7.145 Mb)   0.69%    0.35%
PAGE_EXECUTE_READ                        10            414000 (   4.078 Mb)   0.40%    0.20%
PAGE_READONLY                            29            1cd000 (   1.801 Mb)   0.18%    0.09%
PAGE_WRITECOPY                           10             12000 (  72.000 kb)   0.01%    0.00%
PAGE_READWRITE|PAGE_GUARD                 4              9000 (  36.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                        3f0c0000          33050000 ( 816.313 Mb)
<unclassified>                              158a1000            fcf000 (  15.809 Mb)
Image                                        1083000            3d1000 (   3.816 Mb)
Stack                                         200000             fd000 (1012.000 kb)
MemoryMappedFile                            7efe5000             fb000 (1004.000 kb)
TEB                                         7efda000              1000 (   4.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

We see that free memory available for allocation is only 816 Mb.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services organizes a free Webinar on Unified Software Diagnostics (USD) and the new scalable cost-effective software support model called Pattern-Driven Software Support devised to address various shortcomings in existing tiered software support organizations. Examples cover Windows, Mac OS  and Linux.

Date: 22nd of June, 2012
Time: 17:00 (BST) 12:00 (EST) 09:00 (PST)
Duration: 60 minutes

Space is limited.
Reserve your Webinar seat now at:
https://www3.gotomeeting.com/register/172771078

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Address space-wide search for errors and status codes may show Coincidental Error Code pattern:

0:000> !heap -x -v c0000005
Search VM for address range c0000005 - c0000005 : 028690b8 (c0000005), [...]

0:000> dd 028690b8 l1
028690b8  c0000005

In such cases we need to check whether the addresses belong to volatile regions such as stack because it is possible to have such values as legitimate code and image data:

0:000> !address 028690b8
Usage:                  Image
Allocation Base:        02700000
Base Address:           02869000
End Address:            02874000
Region Size:            0000b000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000002 PAGE_READONLY
More info:              lmv m ModuleA
More info:              !lmi ModuleA
More info:              ln 0×28690b8

0:000> u 028690b8
ModuleA!ComputeB:
028690b8 050000c000      add     eax,0C00000h
[...]

Another example:

0:000> !heap -x -v c0000005
Search VM for address range 00000000c0000005 - 00000000c0000005 : 7feff63ab60 (c0000005),

0:000> !address 7feff63ab60
Usage:                  Image
Allocation Base:        000007fe`ff460000
Base Address:           000007fe`ff635000
End Address:            000007fe`ff63c000
Region Size:            00000000`00007000
Type:                   01000000 MEM_IMAGE
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE
More info:              lmv m ole32
More info:              !lmi ole32
More info:              ln 0×7feff63ab60

0:000> dp 7feff63ab60
000007fe`ff63ab60  00000000`c0000005 c0000194`00000001
000007fe`ff63ab70  00000001`00000000 00000000`c00000aa
000007fe`ff63ab80  80000002`00000001 00000001`00000000
000007fe`ff63ab90  00000000`c0000096 c000001d`00000001
000007fe`ff63aba0  00000001`00000000 00000000`80000003
000007fe`ff63abb0  c00000fd`00000001 00000001`00000000
000007fe`ff63abc0  00000000`c0000235 c0000006`00000001
000007fe`ff63abd0  00000001`00000000 00000000`c0000420

In the latter case the data structure suggests a table of errors:

0:000> ln 7feff63ab60
(000007fe`ff63ab60)   ole32!gReportedExceptions

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Discovered this forthcoming book and immediately preordered:

Inside Windows Debugging: A Practical Guide to Debugging and Tracing Strategies in Windows

From Safari Books Online table of contents I see it also includes Event Tracing for Windows:

http://my.safaribooksonline.com/book/-/9780735671348

Looking forward to reading it and writing a review.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The first Windows pattern called Multiple Exceptions in user mode now has Mac OS X equivalent. In the example below there are 3 threads and two of them experienced NULL Pointer (data) access violation exception:

(gdb) thread apply all bt full

Thread 3 (core thread 2):
#0  0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 2 (core thread 1):
#0  0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
p = (int *) 0×0
#1  0×00007fff8abf58bf in _pthread_start ()
No symbol table info available.
#2  0×00007fff8abf8b75 in thread_start ()
No symbol table info available.

Thread 1 (core thread 0):
#0  0x00007fff854e0e42 in __semwait_signal ()
No symbol table info available.
#1  0x00007fff8ababdea in nanosleep ()
No symbol table info available.
#2  0x00007fff8ababc2c in sleep ()
No symbol table info available.
#3  0x00000001062ffec3 in main (argc=1, argv=0x7fff65efeab8)
at main.c:36
threadID_one = (pthread_t) 0×1063b4000
threadID_two = (pthread_t) 0×106581000

(gdb) thread 2
[Switching to thread 2 (core thread 1)]
0x00000001062ffe1e in thread_one (arg=0x0)
at main.c:16
16    *p = 1;

(gdb) p/x p
$1 = 0×0

(gdb) thread 3
[Switching to thread 3 (core thread 2)]
0x00000001062ffe4e in thread_two (arg=0x0)
at main.c:24
24    *p = 2;

(gdb) p/x p
$2 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a story I would like to present for a possible discussion. A leading debugging services company A was contacted by a company B to analyze dozens of crash dumps for a modest fee^*. No NDA was signed and the work was done promptly. Unfortunately there was no response to an invoice and after some time a representative from the company A contacted the company B and got an immediate reply that they were bought by a company C. The new invoice was requested and promptly sent by the company A to the company C. More time passed beyond any reasonable time frame and a representative from the company A contacted the company C again and didn’t get an immediate reply as before. After some time out of the blue came a group reply from all high execs saying that the modest fee cannot be paid because it had to be done by the company B that they bought. And by the way all guys from the company B dealing with the company A were no longer with the company C. The company A pointed out that by an implicit agreement an act of nonpayment from the company B due to unforeseen circumstances automatically made all crash dumps submitted to the company A a property of the company A.

Many would say please contact a lawyer but the modest fee doesn’t worth such a contact. When I devise a happy ending I write a second part.

^* an average one day exec salary or less

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The founder of pattern-driven debugging was invited to HQ a few years ago to join the Corporation and work on what would become the killer of current debuggers and disassemblers.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Typology - a logic of typos in software. Typological - a logical typo.

Examples: A typology of defects in source code. An engineer committed a grave typological mistake.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a Mac OS X / GDB counterpart to Stack Trace pattern previously described for Windows platforms. Here we show a stack trace when symbols are not available and also how to apply symbols:

(gdb) bt
#0  0×000000010d3b0e90 in ?? ()
#1  0×000000010d3b0ea9 in ?? ()
#2  0×000000010d3b0ec4 in ?? ()
#3  0×000000010d3b0e74 in ?? ()

(gdb) maintenance info sections
Exec file:
[...]
Core file:
`/cores/core.262', file type mach-o-le.
0×000000010d3b0000->0×000000010d3b1000 at 0×00001000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b1000->0×000000010d3b2000 at 0×00002000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b2000->0×000000010d3b3000 at 0×00003000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b3000->0×000000010d3b4000 at 0×00004000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b4000->0×000000010d3b5000 at 0×00005000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b5000->0×000000010d3b6000 at 0×00006000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3b6000->0×000000010d3cb000 at 0×00007000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cb000->0×000000010d3cc000 at 0×0001c000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cc000->0×000000010d3cd000 at 0×0001d000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3cd000->0×000000010d3e2000 at 0×0001e000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3e2000->0×000000010d3e3000 at 0×00033000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d3e3000->0×000000010d3e4000 at 0×00034000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0×000000010d400000->0×000000010d500000 at 0×00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
[…]

(gdb) add-symbol-file ~/Documents/Work/Test.sym 0×000000010d3b0000
add symbol table from file “/Users/DumpAnalysis/Documents/Work/Test.sym” at
LC_SEGMENT.__TEXT = 0×10d3b0000
(y or n) y
Reading symbols from /Users/DumpAnalysis/Documents/Work/Test.sym…done.

(gdb) bt
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

Users of WinDbg debugger accustomed to full thread stack traces will wonder whether a thread starts from main:

(gdb) where
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25

Of course, not and by default a stack trace is shown starting from main function. You can change this behavior by using the following command:

(gdb) set backtrace past-main

Now we see an additional frame:

(gdb) where
#0  0x000000010d3b0e90 in bar () at main.c:15
#1  0x000000010d3b0ea9 in foo () at main.c:20
#2  0x000000010d3b0ec4 in main (argc=1,
argv=0x7fff6cfafbf8) at main.c:25
#3  0×000000010d3b0e74 in start ()

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

This is a Mac OS X / GDB counterpart to NULL Pointer (data) pattern previously described for Windows platforms:

(gdb) bt
#0  0×000000010d3b0e90 in bar () at main.c:15
#1  0×000000010d3b0ea9 in foo () at main.c:20
#2  0×000000010d3b0ec4 in main (argc=1,
argv=0×7fff6cfafbf8) at main.c:25

(gdb) disassemble
Dump of assembler code for function bar:
0x000000010d3b0e80 <bar+0>: push   %rbp
0×000000010d3b0e81 <bar+1>: mov    %rsp,%rbp
0×000000010d3b0e84 <bar+4>: movq   $0×0,-0×8(%rbp)
0×000000010d3b0e8c <bar+12>: mov    -0×8(%rbp),%rax
0×000000010d3b0e90 <bar+16>: movl   $0×1,(%rax)
0×000000010d3b0e96 <bar+22>: pop    %bp
0×000000010d3b0e97 <bar+23>: retq
End of assembler dump.

(gdb) p/x $rax
$1 = 0×0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Forthcoming Training: Accelerated Mac OS X Core Dump Analysis

The following command is useful for searching a process virtual space for any value references:

!heap -x -v <value> ”will search the entire virtual memory space of the current process for pointers to this” value (from WinDbg help).

Example:

0:000> !heap -x -v 6e412d82
Search VM for address range 6e412d82 - 6e412d82 : 778042bc (6e412d82),

0:000> dp 778042bc l1
778042bc  6e412d82

0:000> !heap -x -v c0000005
Search VM for address range c0000005 - c0000005 : 014df8d0 (c0000005), 014dfe8c (c0000005), 0155d908 (c0000005), 0155dd10 (c0000005), 0155ddc8 (c0000005), 0155dfa8 (c0000005), 0155dff0 (c0000005), 0155ea20 (c0000005), 6d000f9c (c0000005), 70d44054 (c0000005), 725c30d4 (c0000005), 7270d20c (c0000005), 7282ef74 (c0000005), 7449a878 (c0000005), 74511958 (c0000005), 74562ec4 (c0000005), 74563280 (c0000005), 74564fc8 (c0000005), 7456562c (c0000005), 74565748 (c0000005), 745664a8 (c0000005), 74566a30 (c0000005), 74566ad8 (c0000005), 747f6730 (c0000005), 747f682c (c0000005), 74861ef0 (c0000005), 7488743c (c0000005), 748aea68 (c0000005), 748b2830 (c0000005), 748c5118 (c0000005), 74935068 (c0000005), 749412a8 (c0000005), 7495caf0 (c0000005), 74a3a780 (c0000005), 74aa462c (c0000005), 74b19b68 (c0000005), 74b61060 (c0000005), 74b8fb44 (c0000005), 74b9d1c8 (c0000005), 74be1ad8 (c0000005), 74be72c8 (c0000005), 74c14b60 (c0000005), 74c83b84 (c0000005), 74c83b88 (c0000005), 74c83b9c (c0000005), 74c83ba0 (c0000005), 74c83ba4 (c0000005), 74c83ba8 (c0000005), 74c83bac (c0000005), 74c83bb0 (c0000005), 74c83bb4 (c0000005), 74c83bb8 (c0000005), 74c83bbc (c0000005), 74c83bc0 (c0000005), 74c83bc8 (c0000005), 74c83bcc (c0000005), 74c83bd0 (c0000005), 74c83bd4 (c0000005), 74c83bd8 (c0000005), 74c83bdc (c0000005), 74c83be0 (c0000005), 74c83be4 (c0000005), 74c83be8 (c0000005), 74c83bec (c0000005), 74c83bf0 (c0000005), 74c83bf4 (c0000005), 74c83bf8 (c0000005), 74c83bfc (c0000005), 74c83c00 (c0000005), 74c83c04 (c0000005), 74c83c08 (c0000005), 74c83c0c (c0000005), 74c83c10 (c0000005), 74c83c14 (c0000005), 74c83c18 (c0000005), 74c83c1c (c0000005), 74c83c20 (c0000005), 74c83c24 (c0000005), 74c83c28 (c0000005), 74c83c2c (c0000005), 74c83c34 (c0000005), 74c83c38 (c0000005), 74c83c3c (c0000005), 74c8c7ac (c0000005), 75019298 (c0000005), 750ff7b0 (c0000005), 751c1adc (c0000005), 751c2514 (c0000005), 7522c530 (c0000005), 752c311c (c0000005), 752d4734 (c0000005), 752d4ae8 (c0000005), 752d534c (c0000005), 752d7038 (c0000005), 752d7e9c (c0000005), 752eda04 (c0000005), 752edab0 (c0000005), 756d6624 (c0000005), 7571adc0 (c0000005), 7571addc (c0000005), 75723780 (c0000005), 757af774 (c0000005), 759c0f10 (c0000005), 76702360 (c0000005), 76703a30 (c0000005), 76d437ac (c0000005), 76d527ec (c0000005), 76dd0fa4 (c0000005), 77581f2c (c0000005), 777a33c0 (c0000005), 777c8b14 (c0000005),

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Narrative theory distinguishes between frame types such as (Fludernik, McHale, Nelles, Wolf):

- introductory framing (missing end frame) [—————————-

- terminal framing (missing opening frame) —————————-]

- [—————————-]

- interpolated framing [—-[  ]—-[     ]——–]

At the level of the software trace or an adjoint thread as a whole the first 3 types correspond to various types of this pattern Partition: Head, Prologue, Core, Epilogue, Tail where certain parts are missing. The first 2 types can also be instances of Truncated Trace pattern. Interpolated framing can be an instance of multiple discontinuities. All 4 types also correspond to foreground component messages and in general we have multiple Trace Frames as depicted:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is another “blockage” pattern called Blocked DPC. Here we have blocked per-processor Deferred Procedure Call queues because of threads running on processors with IRQL > DISPATCH_LEVEL. For example, on the processor 11 (0×0b):

11: kd> !dpcs
CPU Type      KDPC       Function
3: Normal  : 0x8accacec 0xf710567a DriverA

5: Normal  : 0x89f449e4 0xf595b83a DriverB

7: Normal  : 0x8a63664c 0xf59e3f04 USBPORT!USBPORT_IsrDpc

11: Normal  : 0x8acb2cec 0xf710567a DriverA
11: Normal  : 0x8b5e955c 0xf73484e6 ACPI!ACPIInterruptServiceRoutineDPC

11: kd> !thread
THREAD 89806428  Cid 0934.0944  Teb: 7ffdb000 Win32Thread: bc17dda0 RUNNING on processor b
Not impersonating
DeviceMap                 e1002258
Owning Process            89972290       Image:         ApplicationA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      2863772        Ticks: 368905 (0:01:36:04.140)
Context Switch Count      145085                 LargeStack
UserTime                  00:00:00.015
KernelTime                01:36:04.203
Win32 Start Address MSVCR90!_threadstartex (0×7854345e)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init f3f63000 Current f3f62c4c Base f3f63000 Limit f3f5f000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child
f777d3b0 f3f62d28 00000010 00000000 00000000 hal!KeAcquireInStackQueuedSpinLockRaiseToSynch+0×36
WARNING: Frame IP not in any known module. Following frames may be wrong.
f777d3b4 00000000 00000000 00000000 00000000 0xf3f62d28

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A page to reference all different kinds of patterns related to memory dumps as a whole and their properties is necessary, so I created this post:

• False Positive Dump
• Lateral Damage
• Inconsistent Dump
• Truncated Dump
• Early Crash Dump
• Manual Dump (kernel)
• Manual Dump (process)
• Corrupt Dump
• No Process Dumps
• No System Dumps
• Self-Dump
• Abridged Dump
• Fat Process Dump
• Unsynchronized Dumps
• Quiet Dump
• Late Crash Dump
• Step Dumps
• Tampered Dump
• Evental Dumps
• Clone Dump
• Mirror Dump Set
• Hyperdump
• Dump Context

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -