Crash Dump Analysis Patterns (Part 171)

This is an example of Punctuated Memory Leak pattern somewhat similar to a large block allocation leak for process heap (see a modeling example). An application has some functionality and after each command its commited memory was increasing by 50 - 60 Mb. 3 process dumps were taken with one before failures and then after each failure:

// Before failures

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          76c50000 (   1.856 Gb)           92.79%
<unclassified>                          270           4d6f000 (  77.434 Mb)  52.45%    3.78%
Image                                   620           31bf000 (  49.746 Mb)  33.70%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  13.55%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.14%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.09%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.05%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             296           3bca000 (  59.789 Mb)  40.50%    2.92%
MEM_IMAGE                               647           340c000 (  52.047 Mb)  35.26%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  24.24%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          76c50000 (   1.856 Gb)           92.79%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  54.21%    3.91%
MEM_COMMIT                              896           439a000 (  67.602 Mb)  45.79%    3.30%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  21.12%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  20.93%    1.51%
PAGE_READWRITE                          309            4c2000 (   4.758 Mb)   3.22%    0.23%
PAGE_WRITECOPY                           43             6a000 ( 424.000 kb)   0.28%    0.02%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.20%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.04%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         6130000          5fb70000 (   1.496 Gb)
<unclassified>                                abf000           13d1000 (  19.816 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

// After the 1st failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    267          7388c000 (   1.805 Gb)           90.26%
<unclassified>                          272           8133000 ( 129.199 Mb)  64.80%    6.31%
Image                                   614           31bf000 (  49.746 Mb)  24.95%    2.43%
Stack                                    60           1400000 (  20.000 Mb)  10.03%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.10%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.07%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.04%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             297           6f8e000 ( 111.555 Mb)  55.95%    5.45%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  26.10%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  17.95%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                267          7388c000 (   1.805 Gb)           90.26%
MEM_COMMIT                              892           775e000 ( 119.367 Mb)  59.87%    5.83%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  40.13%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          314           38a3000 (  56.637 Mb)  28.40%    2.77%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  15.63%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  15.49%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.15%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.15%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         94f4000          5c7ac000 (   1.445 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
[...]
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

0:000> ? 33c4000/0n1024
Evaluate expression: 53008 = 0000cf10

// After the 2nd failure

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    268          704c8000 (   1.755 Gb)           87.74%
<unclassified>                          273           b4f7000 ( 180.965 Mb)  72.05%    8.84%
Image                                   614           31bf000 (  49.746 Mb)  19.81%    2.43%
Stack                                    60           1400000 (  20.000 Mb)   7.96%    0.98%
ActivationContextData                    48             35000 ( 212.000 kb)   0.08%    0.01%
NlsTables                                 1             23000 ( 140.000 kb)   0.05%    0.01%
TEB                                      20             14000 (  80.000 kb)   0.03%    0.00%
CsrSharedMemory                           1              5000 (  20.000 kb)   0.01%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             298           a352000 ( 163.320 Mb)  65.03%    7.97%
MEM_IMAGE                               642           340c000 (  52.047 Mb)  20.72%    2.54%
MEM_MAPPED                               78           23ca000 (  35.789 Mb)  14.25%    1.75%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                268          704c8000 (   1.755 Gb)           87.74%
MEM_COMMIT                              893           ab22000 ( 171.133 Mb)  68.14%    8.36%
MEM_RESERVE                             125           5006000 (  80.023 Mb)  31.86%    3.91%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          315           6c67000 ( 108.402 Mb)  43.16%    5.29%
PAGE_EXECUTE_READ                       125           1f2c000 (  31.172 Mb)  12.41%    1.52%
PAGE_READONLY                           363           1ee5000 (  30.895 Mb)  12.30%    1.51%
PAGE_WRITECOPY                           34             4d000 ( 308.000 kb)   0.12%    0.01%
PAGE_READWRITE|PAGE_GUARD                40             4b000 ( 300.000 kb)   0.12%    0.01%
PAGE_EXECUTE_READWRITE                   15             11000 (  68.000 kb)   0.03%    0.00%
PAGE_EXECUTE                              1              1000 (   4.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         c8c4000          593dc000 (   1.394 Gb)
<unclassified>                               6130000           33c4000 (  51.766 Mb)
Image                                       75141000            879000 (   8.473 Mb)
Stack                                        3290000             fd000 (1012.000 kb)
ActivationContextData                          50000              4000 (  16.000 kb)
NlsTables                                   7efb0000             23000 ( 140.000 kb)
TEB                                         7ef6f000              1000 (   4.000 kb)
CsrSharedMemory                             7efe0000              5000 (  20.000 kb)
PEB                                         7efde000              1000 (   4.000 kb)

0:000> !address -f:VAR

BaseAddr EndAddr+1 RgnSize     Type       State                 Protect             Usage
-------------------------------------------------------------------------------------------
5e82000  5f70000    ee000 MEM_PRIVATE MEM_RESERVE                                    <unclassified>
6130000  94f4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>
9500000  c8c4000  33c4000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     <unclassified>

74220000 74221000     1000 MEM_IMAGE   MEM_COMMIT  PAGE_READONLY                      <unclassified>
[…]

The name of this pattern comes from the process of discrete large memory allocations that happen after specific actions or events. Between them there is no visible or substantial increase in memory usage.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Leave a Reply