Archive for June, 2009

LiterateScientist update (June, 2009)

Friday, June 26th, 2009

Monthly summary of my Literate Scientist blog:

Philosophy: The Basics 

Stalin: The Court of the Red Tsar

A History of Russia

Religion: The Basics

- Dmitry Vostokov @ DumpAnalysis.org -

ManagementBits update (May - June, 2009)

Thursday, June 25th, 2009

Monthly summary of my Management Bits and Tips blog:

Local Workplace Guides

On Management Amnesia

Hidden Transcripts

Management Bit and Tip 0×8000

Downfloored or Upfloored? 

- Dmitry Vostokov @ DumpAnalysis.org -

Stack trace collection, message box, hidden exception, nested offender, insufficient memory, C++ exception, heap leak and ubiquitous component: pattern cooperation

Thursday, June 25th, 2009

My IE suddenly showed this message box:

I have never seen such message from IE so I rushed to Task Manager to save a dump. Default analysis (!analyze -v) was not able to find the problem:

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

STACK_TEXT: 
0031e624 77289244 7719c3e4 00000002 0031e678 ntdll!KiFastSystemCallRet
0031e628 7719c3e4 00000002 0031e678 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0031e6c4 76f50208 0031e678 0031e6ec 00000000 kernel32!WaitForMultipleObjectsEx+0x11d
0031e718 70196071 00000028 0031e74c ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x13c
0031e738 701961f0 000004ff ffffffff 00000000 ieui!CoreSC::Wait+0x49
0031e760 70196196 000004ff 00000000 6f16074e ieui!CoreSC::WaitMessage+0x54
0031e76c 6f16074e 000c0ec8 00166038 00000000 ieui!WaitMessageEx+0x33
0031e79c 6f0fffce 00172e10 0031e7cc 6f0ef579 ieframe!CBrowserFrame::FrameMessagePump+0x199
0031e7a8 6f0ef579 00000000 00000000 000c0ec8 ieframe!BrowserThreadProc+0x3f
0031e7cc 6f0ef4c7 1ee1000a 000c0ec8 00000000 ieframe!BrowserNewThreadProc+0x7b
0031f83c 6f0dd1ba 000c0ec8 00000001 00000001 ieframe!SHOpenFolderWindow+0x188
0031fa6c 009033c3 000dd2c8 00000001 00910070 ieframe!IEWinMain+0x2d9
0031feb0 0090325a 00900000 00000000 000c1aa6 iexplore!wWinMain+0x27b
0031ff44 77194911 7ffd7000 0031ff90 7726e4b6 iexplore!_initterm_e+0x1b1
0031ff50 7726e4b6 7ffd7000 722730b1 00000000 kernel32!BaseThreadInitThunk+0xe
0031ff90 7726e489 009030dd 7ffd7000 00000000 ntdll!__RtlUserThreadStart+0x23
0031ffa8 00000000 009030dd 7ffd7000 00000000 ntdll!_RtlUserThreadStart+0x1b

However when browsing through stack trace collection I could spot a thread blocked by a message box, find another hidden exception and from it see the real nested offender that experienced insufficient memory condition resulted in C++ exception. You can see WinDbg output from this dump in the post about Nested Offender pattern (I don’t want to repeat it in this post).

Apart from that, the size of the memory dump, almost 1.8Gb, suggested a memory leak and we clearly see expanded heaps that also suggest the case of a heap leak:

0:087> !heap =s
Index   Address  Name      Debugging options enabled
  1:   000c0000
    Segment at 000c0000 to 001c0000 (00100000 bytes committed)
    Segment at 04510000 to 04610000 (00100000 bytes committed)
    Segment at 063b0000 to 065b0000 (00200000 bytes committed)
    Segment at 05f80000 to 06380000 (00400000 bytes committed)
    Segment at 0a8d0000 to 0b0d0000 (00800000 bytes committed)
    Segment at 0eab0000 to 0fa80000 (00fd0000 bytes committed)
    Segment at 160b0000 to 17080000 (00fd0000 bytes committed)
    Segment at 19020000 to 19ff0000 (00fd0000 bytes committed)
    Segment at 23fe0000 to 24fb0000 (00fd0000 bytes committed)
    Segment at 2ac20000 to 2bbf0000 (00fd0000 bytes committed)
    Segment at 34cc0000 to 35c90000 (00fd0000 bytes committed)
    Segment at 35fc0000 to 36f90000 (00fd0000 bytes committed)
    Segment at 40660000 to 41630000 (00fd0000 bytes committed)
    Segment at 45230000 to 46200000 (00fd0000 bytes committed)
    Segment at 4aed0000 to 4bea0000 (00fd0000 bytes committed)
    Segment at 4ee20000 to 4fdf0000 (00fd0000 bytes committed)
    Segment at 52eb0000 to 53e80000 (00fd0000 bytes committed)
    Segment at 53e80000 to 54e50000 (00fd0000 bytes committed)
    Segment at 575e0000 to 585b0000 (00fd0000 bytes committed)
    Segment at 58cb0000 to 59c80000 (00fd0000 bytes committed)
    Segment at 5ad00000 to 5bcd0000 (00fd0000 bytes committed)
    Segment at 5bcd0000 to 5cca0000 (00fd0000 bytes committed)
    Segment at 5ddb0000 to 5ed80000 (00fd0000 bytes committed)
    Segment at 77490000 to 78460000 (00fd0000 bytes committed)
    Segment at 78460000 to 79430000 (00fd0000 bytes committed)
    Segment at 7c420000 to 7d3f0000 (00fd0000 bytes committed)
    Segment at 7d3f0000 to 7e3c0000 (00fd0000 bytes committed)
    Segment at 7b690000 to 7be78000 (007e8000 bytes committed)
    Segment at 70470000 to 70864000 (003f4000 bytes committed)
    Segment at 72020000 to 72414000 (003f4000 bytes committed)
    Segment at 6ca70000 to 6cc6a000 (001fa000 bytes committed)
    Segment at 6d450000 to 6d64a000 (001fa000 bytes committed)
    Segment at 6c620000 to 6c81a000 (001fa000 bytes committed)
    Segment at 6e1b0000 to 6e3aa000 (001e6000 bytes committed)
    Segment at 701c0000 to 703ba000 (001ee000 bytes committed)
    Segment at 70ab0000 to 70caa000 (001ee000 bytes committed)
    Segment at 71770000 to 7196a000 (001e9000 bytes committed)
    Segment at 68060000 to 6825a000 (001f0000 bytes committed)
    Segment at 72a40000 to 72c3a000 (001ef000 bytes committed)
    Segment at 73170000 to 7336a000 (001f2000 bytes committed)
    Segment at 6d6c0000 to 6dab4000 (003b7000 bytes committed)
    Segment at 7a400000 to 7b3d0000 (00fb1000 bytes committed)
    Segment at 3c480000 to 3c57d000 (000d2000 bytes committed)
    Segment at 2e950000 to 2ea4d000 (000cc000 bytes committed)
    Segment at 7b3d0000 to 7b5ca000 (001c1000 bytes committed)
    Segment at 2ec60000 to 2ed5d000 (000c1000 bytes committed)
    Segment at 31570000 to 3166d000 (000c1000 bytes committed)
    Segment at 43050000 to 4314d000 (000c1000 bytes committed)
    Segment at 48f30000 to 4902d000 (000c1000 bytes committed)
    Segment at 492b0000 to 493ad000 (000c1000 bytes committed)
    Segment at 49bb0000 to 49cad000 (000c1000 bytes committed)
    Segment at 49d50000 to 49e4d000 (000c1000 bytes committed)
    Segment at 4bea0000 to 4bf9d000 (000c1000 bytes committed)
    Segment at 4d140000 to 4d23d000 (000c1000 bytes committed)
    Segment at 55040000 to 5513d000 (000c1000 bytes committed)
    Segment at 55180000 to 5527d000 (000c1000 bytes committed)
    Segment at 555c0000 to 556bd000 (000c1000 bytes committed)
    Segment at 557d0000 to 558cd000 (000c1000 bytes committed)
    Segment at 5a380000 to 5a47d000 (000c1000 bytes committed)
    Segment at 5a980000 to 5aa7d000 (000c1000 bytes committed)
    Segment at 5ab40000 to 5ac3d000 (000c1000 bytes committed)
    Segment at 6dce0000 to 6dddd000 (000c6000 bytes committed)
    Segment at 75680000 to 7577d000 (000c6000 bytes committed)
    Segment at 4d6f0000 to 4d7ed000 (000c1000 bytes committed)
    Segment at 2ca40000 to 2cabf000 (00041000 bytes committed)
    Segment at 4aa30000 to 4aaaf000 (00041000 bytes committed)
    Segment at 67c20000 to 67d1d000 (000c1000 bytes committed)
    Segment at 2e820000 to 2e91d000 (000c1000 bytes committed)
    Segment at 4e680000 to 4e77d000 (000c1000 bytes committed)
    Segment at 5d4c0000 to 5d5bd000 (000c1000 bytes committed)
    Segment at 683a0000 to 6849d000 (000c1000 bytes committed)
    Segment at 6a440000 to 6a53d000 (000c1000 bytes committed)
    Segment at 4d7f0000 to 4d86f000 (00041000 bytes committed)
    Segment at 60380000 to 603ff000 (00041000 bytes committed)
    Segment at 65460000 to 654df000 (00041000 bytes committed)
    Segment at 67fb0000 to 6802f000 (00041000 bytes committed)
    Segment at 684a0000 to 6851f000 (00041000 bytes committed)
    Segment at 6a540000 to 6a5bf000 (00041000 bytes committed)
    Segment at 6ab80000 to 6abff000 (00041000 bytes committed)
    Segment at 6b1e0000 to 6b25f000 (00041000 bytes committed)
    Segment at 6c390000 to 6c40f000 (00041000 bytes committed)
    Segment at 6dee0000 to 6df5f000 (00041000 bytes committed)
    Segment at 6e8d0000 to 6e94f000 (00041000 bytes committed)
    Segment at 6ef90000 to 6f00f000 (00041000 bytes committed)
    Segment at 72420000 to 7249f000 (00041000 bytes committed)
    Segment at 740d0000 to 7414f000 (00041000 bytes committed)
    Segment at 74c30000 to 74caf000 (00041000 bytes committed)
    Segment at 75150000 to 751cf000 (00041000 bytes committed)
    Segment at 7b5d0000 to 7b64f000 (00041000 bytes committed)
    Segment at 09d10000 to 09d51000 (00041000 bytes committed)
    Segment at 0ddb0000 to 0ddf1000 (00041000 bytes committed)
    Segment at 0e810000 to 0e851000 (00041000 bytes committed)
    Segment at 2c580000 to 2c5c1000 (00041000 bytes committed)
    Segment at 2d490000 to 2d4d1000 (00041000 bytes committed)
    Segment at 2ea50000 to 2ea91000 (00041000 bytes committed)
    Segment at 31060000 to 310a1000 (00041000 bytes committed)
    Segment at 322a0000 to 322e1000 (00041000 bytes committed)
    Segment at 323a0000 to 323e1000 (00041000 bytes committed)
    Segment at 32ff0000 to 33031000 (00041000 bytes committed)
    Segment at 33d90000 to 33dd1000 (00041000 bytes committed)
    Segment at 34330000 to 34371000 (00041000 bytes committed)
    Segment at 35c90000 to 35cd1000 (00041000 bytes committed)
    Segment at 37250000 to 37291000 (00041000 bytes committed)
    Segment at 40290000 to 402d1000 (00041000 bytes committed)
    Segment at 42ce0000 to 42d21000 (00041000 bytes committed)
    Segment at 42ff0000 to 43031000 (00041000 bytes committed)
    Segment at 48a40000 to 48a81000 (00041000 bytes committed)
    Segment at 55470000 to 554b1000 (00041000 bytes committed)
    Segment at 55930000 to 55971000 (00041000 bytes committed)
    Segment at 55a80000 to 55ac1000 (00041000 bytes committed)
    Segment at 59e90000 to 59ed1000 (00041000 bytes committed)
    Segment at 59ff0000 to 5a031000 (00041000 bytes committed)
    Segment at 5d730000 to 5d771000 (00041000 bytes committed)
    Segment at 5d810000 to 5d851000 (00041000 bytes committed)
    Segment at 60230000 to 60271000 (00041000 bytes committed)
    Segment at 6a5c0000 to 6a601000 (00041000 bytes committed)
    Segment at 6c5d0000 to 6c611000 (00041000 bytes committed)
    Segment at 6c8e0000 to 6c921000 (00041000 bytes committed)
    Segment at 6cfb0000 to 6cff1000 (00041000 bytes committed)
    Segment at 6dfc0000 to 6e001000 (00041000 bytes committed)
    Segment at 6e730000 to 6e771000 (00041000 bytes committed)
    Segment at 6eec0000 to 6ef01000 (00041000 bytes committed)
    Segment at 700e0000 to 70121000 (00041000 bytes committed)
    Segment at 703c0000 to 70401000 (00041000 bytes committed)
    Segment at 70870000 to 708b1000 (00041000 bytes committed)
    Segment at 70a40000 to 70a81000 (00041000 bytes committed)
    Segment at 71ca0000 to 71ce1000 (00041000 bytes committed)
    Segment at 729e0000 to 72a21000 (00041000 bytes committed)
    Segment at 73070000 to 730b1000 (00041000 bytes committed)
    Segment at 73410000 to 73451000 (00041000 bytes committed)
    Segment at 74a80000 to 74ac1000 (00041000 bytes committed)
    Segment at 75880000 to 758c1000 (00041000 bytes committed)
    Segment at 7f690000 to 7f6d1000 (00041000 bytes committed)
    Segment at 7fd70000 to 7fdb1000 (00041000 bytes committed)

  2:   00010000
    Segment at 00010000 to 00020000 (00003000 bytes committed)
  3:   000a0000
    Segment at 000a0000 to 000b0000 (00010000 bytes committed)
    Segment at 006a0000 to 007a0000 (00014000 bytes committed)
  4:   00020000
    Segment at 00020000 to 00030000 (00010000 bytes committed)
    Segment at 01780000 to 01880000 (00100000 bytes committed)
    Segment at 090b0000 to 092b0000 (00200000 bytes committed)
    Segment at 0cff0000 to 0d3f0000 (00400000 bytes committed)
    Segment at 0d3f0000 to 0dbf0000 (00800000 bytes committed)
    Segment at 10790000 to 11760000 (00fd0000 bytes committed)
    Segment at 143b0000 to 15380000 (00fd0000 bytes committed)
    Segment at 17080000 to 18050000 (00fd0000 bytes committed)
    Segment at 18050000 to 19020000 (00fd0000 bytes committed)
    Segment at 19ff0000 to 1afc0000 (00fd0000 bytes committed)
    Segment at 20be0000 to 21bb0000 (00fd0000 bytes committed)
    Segment at 24fb0000 to 25f80000 (00fd0000 bytes committed)
    Segment at 25f80000 to 26f50000 (00fd0000 bytes committed)
    Segment at 2d6b0000 to 2e680000 (00fd0000 bytes committed)
    Segment at 3a9c0000 to 3b990000 (00fd0000 bytes committed)
    Segment at 3c9c0000 to 3d990000 (00fd0000 bytes committed)
    Segment at 41a50000 to 42a20000 (00fd0000 bytes committed)
    Segment at 44260000 to 45230000 (00fd0000 bytes committed)
    Segment at 46200000 to 471d0000 (00fd0000 bytes committed)
    Segment at 471d0000 to 481a0000 (00fd0000 bytes committed)
    Segment at 4c170000 to 4d140000 (00fd0000 bytes committed)
    Segment at 4fdf0000 to 50dc0000 (00fd0000 bytes committed)
    Segment at 51ee0000 to 52eb0000 (00fd0000 bytes committed)
    Segment at 56400000 to 573d0000 (00fd0000 bytes committed)
    Segment at 68de0000 to 69db0000 (00fd0000 bytes committed)
    Segment at 79430000 to 7a400000 (00fd0000 bytes committed)
    Segment at 7e3c0000 to 7f390000 (00fd0000 bytes committed)
    Segment at 73820000 to 74008000 (007e8000 bytes committed)
    Segment at 6d030000 to 6d424000 (003f4000 bytes committed)
    Segment at 6f680000 to 6fa74000 (003f4000 bytes committed)
    Segment at 6eac0000 to 6eeb4000 (003f4000 bytes committed)
    Segment at 74210000 to 74604000 (003f4000 bytes committed)
    Segment at 7be80000 to 7c274000 (003f4000 bytes committed)
    Segment at 7f7f0000 to 7fbe4000 (003f4000 bytes committed)
    Segment at 6dae0000 to 6dcda000 (001fa000 bytes committed)
    Segment at 6fa80000 to 6fc7a000 (001fa000 bytes committed)
    Segment at 71440000 to 7163a000 (001fa000 bytes committed)
    Segment at 727e0000 to 729da000 (001fa000 bytes committed)
    Segment at 60450000 to 6064a000 (001fa000 bytes committed)
    Segment at 72e50000 to 7304a000 (001fa000 bytes committed)
    Segment at 5d070000 to 5d16d000 (000fd000 bytes committed)
    Segment at 5ac80000 to 5acff000 (00075000 bytes committed)
    Segment at 5d970000 to 5d9ef000 (0007f000 bytes committed)
    Segment at 5dc10000 to 5dc8f000 (0007f000 bytes committed)
    Segment at 5f0e0000 to 5f15f000 (0007f000 bytes committed)
    Segment at 6b0e0000 to 6b1dd000 (000dd000 bytes committed)
    Segment at 494c0000 to 496ba000 (001f2000 bytes committed)
    Segment at 6a240000 to 6a43a000 (001ec000 bytes committed)
    Segment at 376b0000 to 3772f000 (0007f000 bytes committed)

  5:   008e0000
    Segment at 008e0000 to 008f0000 (00010000 bytes committed)
    Segment at 007a0000 to 008a0000 (00100000 bytes committed)
    Segment at 5a6e0000 to 5a8e0000 (00041000 bytes committed)
  6:   00210000
    Segment at 00210000 to 00220000 (00010000 bytes committed)
    Segment at 050f0000 to 051f0000 (00022000 bytes committed)
  7:   00080000
    Segment at 00080000 to 00090000 (00010000 bytes committed)
    Segment at 05d80000 to 05e80000 (00022000 bytes committed)
  8:   018f0000
    Segment at 018f0000 to 01900000 (00010000 bytes committed)
    Segment at 041e0000 to 042e0000 (000fa000 bytes committed)
    Segment at 5eee0000 to 5f0e0000 (00066000 bytes committed)
  9:   001c0000
    Segment at 001c0000 to 001d0000 (00001000 bytes committed)
 10:   018c0000
    Segment at 018c0000 to 018d0000 (00003000 bytes committed)
 11:   01dc0000
    Segment at 01dc0000 to 01e00000 (00032000 bytes committed)
 12:   037c0000
    Segment at 037c0000 to 03800000 (0000a000 bytes committed)
 13:   008c0000
    Segment at 008c0000 to 008d0000 (00006000 bytes committed)
    Segment at 1fc90000 to 1fd90000 (00012000 bytes committed)
 14:   03750000
    Segment at 03750000 to 03760000 (00003000 bytes committed)
 15:   03b20000
    Segment at 03b20000 to 03b60000 (00016000 bytes committed)
 16:   03a40000
    Segment at 03a40000 to 03a80000 (00040000 bytes committed)
    Segment at 0a420000 to 0a520000 (000c6000 bytes committed)
 17:   04050000
    Segment at 04050000 to 04090000 (00040000 bytes committed)
 18:   04340000
    Segment at 04340000 to 04380000 (00040000 bytes committed)
 19:   04500000
    Segment at 04500000 to 04510000 (0000e000 bytes committed)
    Segment at 08b30000 to 08c30000 (000e9000 bytes committed)
    Segment at 2cfe0000 to 2d0e0000 (00022000 bytes committed)
 20:   04800000
    Segment at 04800000 to 04900000 (00100000 bytes committed)
 21:   049c0000
    Segment at 049c0000 to 049d0000 (00010000 bytes committed)
    Segment at 049d0000 to 04ad0000 (00023000 bytes committed)
 22:   04c50000
    Segment at 04c50000 to 04c60000 (00010000 bytes committed)
    Segment at 04ad0000 to 04bd0000 (00100000 bytes committed)
    Segment at 1fa90000 to 1fc90000 (00200000 bytes committed)
    Segment at 49710000 to 49b10000 (003d1000 bytes committed)
 23:   05200000
    Segment at 05200000 to 05300000 (00100000 bytes committed)
 24:   053e0000
    Segment at 053e0000 to 05420000 (0001e000 bytes committed)
 25:   051f0000
    Segment at 051f0000 to 05200000 (00010000 bytes committed)
    Segment at 0a100000 to 0a200000 (00022000 bytes committed)
 26:   040d0000
    Segment at 040d0000 to 040e0000 (00010000 bytes committed)
    Segment at 05570000 to 05670000 (00012000 bytes committed)
 27:   047e0000
    Segment at 047e0000 to 047f0000 (00010000 bytes committed)
    Segment at 05e80000 to 05f80000 (00026000 bytes committed)
 28:   05560000
    Segment at 05560000 to 05570000 (00010000 bytes committed)
    Segment at 05420000 to 05520000 (00032000 bytes committed)
 29:   06b30000
    Segment at 06b30000 to 06b40000 (00010000 bytes committed)
    Segment at 1ff30000 to 20030000 (00080000 bytes committed)
 30:   053b0000
    Segment at 053b0000 to 053c0000 (00010000 bytes committed)
    Segment at 08df0000 to 08ef0000 (00100000 bytes committed)
    Segment at 2bbf0000 to 2bdf0000 (00200000 bytes committed)
    Segment at 316a0000 to 31aa0000 (003f9000 bytes committed)
    Segment at 31aa0000 to 322a0000 (0049c000 bytes committed)
 31:   09460000
    Segment at 09460000 to 094a0000 (0003e000 bytes committed)
    Segment at 14070000 to 14170000 (00100000 bytes committed)
    Segment at 51ce0000 to 51ee0000 (00124000 bytes committed)
 32:   08cf0000
    Segment at 08cf0000 to 08d30000 (00001000 bytes committed)
 33:   09360000
    Segment at 09360000 to 093a0000 (00001000 bytes committed)
 34:   04f40000
    Segment at 04f40000 to 04f80000 (00001000 bytes committed)
 35:   09560000
    Segment at 09560000 to 095a0000 (00001000 bytes committed)
 36:   0b6e0000
    Segment at 0b6e0000 to 0b6f0000 (00010000 bytes committed)
    Segment at 07720000 to 07820000 (000ca000 bytes committed)
 37:   2c410000
    Segment at 2c410000 to 2c420000 (00008000 bytes committed)
 38:   29420000
    Segment at 29420000 to 29460000 (00040000 bytes committed)
 39:   3c120000
    Segment at 3c120000 to 3c130000 (00002000 bytes committed)
 40:   60410000
    Segment at 60410000 to 60450000 (00040000 bytes committed)
 41:   65500000
    Segment at 65500000 to 65510000 (00006000 bytes committed)
 42:   67d60000
    Segment at 67d60000 to 67da0000 (00001000 bytes committed)

I actually wanted to run !heap -s command to get aggregated stats but misprint =s gave me the old format output you can see above that I enjoyed using previous versions of OS and WinDbg. Stats for 2 process heaps:

0:087> !heap -s -h 00020000

[...]

 0: Heap 00020000
   Flags          00001002 - HEAP_GROWABLE
   Reserved memory in segments              424788 (k)
   Commited memory in segments              424532 (k)
   Virtual bytes (correction for large UCR) 424788 (k)
   Free space                               3298 (k) (1858 blocks)

   External fragmentation          0% (1858 free blocks)
   Virtual address fragmentation   0% (49 uncommited ranges)
   Virtual blocks  65 - total 0 KBytes
   Lock contention 18
   Segments        1

   Low fragmentation heap   01780048
       Lock contention        0
       Metadata usage    195584 bytes
       Statistics:
           Segments created     831950
           Segments deleted     826741
           Segments reused           0

[...]

0:087> !heap -s -h 000c0000

[...]

Walking the heap 000c0000 .. 0: Heap 000c0000
   Flags          00000002 - HEAP_GROWABLE
   Reserved memory in segments              479980 (k)
   Commited memory in segments              467984 (k)
   Virtual bytes (correction for large UCR) 479980 (k)
   Free space                               4273 (k) (2000 blocks)

   External fragmentation          0% (2000 free blocks)
   Virtual address fragmentation   2% (134 uncommited ranges)
   Virtual blocks  25 - total 0 KBytes
   Lock contention 9092
   Segments        1

   Low fragmentation heap   000c6a98
       Lock contention        0
       Metadata usage    326656 bytes
       Statistics:
           Segments created    1534237
           Segments deleted    1524948
           Segments reused           0

[...]

Instead of enabling user mode stack trace database or collecting UMDH logs I conjectured a weak link with the found ubiquitous module in the stack trace collection. You can find the discussion and corresponding WinDbg output from the same dump in Ubiquitous Component pattern description. After component removal the problem disappeared. Nice troubleshooting in one iteration.

- Dmitry Vostokov @ DumpAnalysis.org -

Dictionary of Debugging: Crash

Wednesday, June 24th, 2009

Crash

The sudden disappearance of a program, service or system from observation.

References: Crashes explained, The difference between crashes and hangs explained

Synonyms: stop.

Antonyms: live.

Also: hang.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 86)

Wednesday, June 24th, 2009

Sometimes we suspect one component but there is another, Nested Offender, that raised an exception. That exception propagated through exception filters and handlers and prompted the suspected component to respond with a diagnostic message. Here is an example of the thread showing the runtime error message box:

The corresponding stack trace points to MathMLMimer module:

0:087> kL
ChildEBP RetAddr 
5546ddf0 76f50dde ntdll!KiFastSystemCallRet
5546ddf4 76f3b0b2 user32!NtUserWaitMessage+0xc
5546de28 76f3bcda user32!DialogBox2+0x202
5546de50 76f8ccdc user32!InternalDialogBox+0xd0
5546def0 76f8d25e user32!SoftModalMessageBox+0x69f
5546e040 76f8d394 user32!MessageBoxWorker+0x2c7
5546e098 76f8d43e user32!MessageBoxTimeoutW+0x7f
5546e0cc 76f8d5ec user32!MessageBoxTimeoutA+0xa1
5546e0ec 6f245ac7 user32!MessageBoxExA+0x1b
5546e10c 76f8d65e ieframe!Detour_MessageBoxExA+0x2c
5546e128 0c841c28 user32!MessageBoxA+0×45
WARNING: Stack unwind information not available. Following frames may be wrong.
5546e16c 0c83d1b9 MathMLMimer!DllUnregisterServer+0×9437
5546e190 0c83cf72 MathMLMimer!DllUnregisterServer+0×49c8
00000000 00000000 MathMLMimer!DllUnregisterServer+0×4781

0:087> .asm no_code_bytes
Assembly options: no_code_bytes

0:087> ub 0c841c28
MathMLMimer!DllUnregisterServer+0×941d:
0c841c0e push    dword ptr [ebp+10h]
0c841c11 push    dword ptr [ebp+0Ch]
0c841c14 push    dword ptr [ebp+8]
0c841c17 push    dword ptr [ebp-4]
0c841c1a push    dword ptr [MathMLMimer!DllUnregisterServer+0×135ab (0c84bd9c)]
0c841c20 call    MathMLMimer!DllUnregisterServer+0×4aab (0c83d29c)
0c841c25 pop     ecx
0c841c26 call    eax

However, when looking at raw stack data, we see exception processing residue that points to ComponentA that tried to allocate more memory than was available (bad_alloc C++ exception):

0:087> !teb
TEB at 7ff84000
    ExceptionList:        5546e4e8
    StackBase:            55470000
    StackLimit:           55457000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ff84000
    EnvironmentPointer:   00000000
    ClientId:             0000136c . 00001714
    RpcHandle:            00000000
    Tls Storage:          5826eef0
    PEB Address:          7ffd7000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:087> dds 55457000 55470000
55457000  00000000
[...]
5546e694  55470000
5546e698  55457000
5546e69c  00274cb0
5546e6a0  5546e9f4
5546e6a4  772899f7 ntdll!KiUserExceptionDispatcher+0xf
5546e6a8  0046e6b8
5546e6ac  5546e6d8
5546e6b0  5546e6b8
[…]

0:087> .cxr 5546e6d8
eax=5546e9a4 ebx=00001000 ecx=00000003 edx=00000000 esi=689a9c54 edi=6b330000
eip=771942eb esp=5546e9a4 ebp=5546e9f4 iopl=0 nv up ei pl nz ac po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00240212
kernel32!RaiseException+0×58:
771942eb c9              leave

0:087> .exr 5546e6b8
ExceptionAddress: 771942eb (kernel32!RaiseException+0×00000058)
   ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
   Parameter[0]: 19930520
   Parameter[1]: 5546ea3c
   Parameter[2]: 688b7954
  pExceptionObject: 5546ea3c
  _s_ThrowInfo    : 688b7954
  Type            : class std::bad_alloc
  Type            : class std::exception

0:087> kL
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr 
5546e9f4 6882dead kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
5546ea2c 6882a59d ComponentA!DllUnregisterServer+0×1adbe0
5546ea48 6868157b ComponentA!DllUnregisterServer+0×1aa2d0
5546ea74 6869d2c6 ComponentA!DllUnregisterServer+0×12ae
5546ea88 6868a415 ComponentA!DllUnregisterServer+0×1cff9
5546eaa0 685a165e ComponentA!DllUnregisterServer+0xa148
5546eac8 685a9828 ComponentA+0×5165e
5546ebc8 68605fdd ComponentA+0×59828
5546ebf0 6a807aba ComponentA+0xb5fdd
5546ec08 6a807a77 mshtml!CViewDispClient::Invalidate+0×59
5546ec20 00000000 mshtml!CDispRoot::InvalidateRoot+0×1d

0:087> ub 6882dead
ComponentA!DllUnregisterServer+0×1adbc4:
6882de91 je      ComponentA!DllUnregisterServer+0×1adbcd (6882de9a)
6882de93 mov     dword ptr [ebp-0Ch],1994000h
6882de9a lea     eax,[ebp-0Ch]
6882de9d push    eax
6882de9e push    dword ptr [ebp-10h]
6882dea1 push    dword ptr [ebp-1Ch]
6882dea4 push    dword ptr [ebp-20h]
6882dea7 call    dword ptr [ComponentA!DllUnregisterServer+0×1ba0a3 (6883a370)]

It happens that MathMLMimer has an exception filter that shows the runtime error:

0:087> !exchain
5546e4e8: MathMLMimer!DllUnregisterServer+18df (0c83a0d0)
5546e578: kernel32!_except_handler4+0 (7715e289)
5546e5e4: ntdll!ExecuteHandler2+3a (77289bad)
5546faa8: user32!_except_handler4+0 (76f951ba)
5546fb0c: user32!_except_handler4+0 (76f951ba)
5546fbd0: ntdll!_except_handler4+0 (77239834)
Invalid exception stack at ffffffff

Notice the mystical 6882dead return address above. This is just a coincidence, I believe.

Nested Offender is different from Nested Exception pattern. The latter is about an exception handler that experiences or throws another exception.

- Dmitry Vostokov @ DumpAnalysis.org -

Crash Dump Analysis Patterns (Part 85)

Tuesday, June 23rd, 2009

Sometimes we look at a stack trace collection and we see dozens of threads running through some 3rd-party component. We do not normally expect this component to appear or expect only one or two threads running through it. Here is an example from my IE after installing a 3rd-party toolbar that becomes Ubiquitous Component (the component has been renamed not “to awaken a sleeping giant and fill him with a terrible resolve”):

0:087> ~*kc

#  0  Id: 136c.fa0 Suspend: 0 Teb: 7ffdf000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
user32!RealMsgWaitForMultipleObjectsEx
ieui!CoreSC::Wait
ieui!CoreSC::WaitMessage
ieui!WaitMessageEx
ieframe!CBrowserFrame::FrameMessagePump
ieframe!BrowserThreadProc
ieframe!BrowserNewThreadProc
ieframe!SHOpenFolderWindow
ieframe!IEWinMain
iexplore!wWinMain
iexplore!_initterm_e
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   1  Id: 136c.12c0 Suspend: 0 Teb: 7ffdc000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
ntdll!TppWaiterpThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   2  Id: 136c.1120 Suspend: 0 Teb: 7ffda000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
user32!RealMsgWaitForMultipleObjectsEx
ieui!CoreSC::Wait
ieui!CoreSC::xwProcessNL
ieui!GetMessageExA
ieui!ResourceManager::SharedThreadProc
msvcrt!_endthreadex
msvcrt!_endthreadex
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   3  Id: 136c.1588 Suspend: 0 Teb: 7ffd9000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   4  Id: 136c.15a4 Suspend: 0 Teb: 7ffd8000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   5  Id: 136c.111c Suspend: 0 Teb: 7ffd6000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForWorkViaWorkerFactory
ntdll!TppWorkerThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   6  Id: 136c.10a8 Suspend: 0 Teb: 7ffd5000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   7  Id: 136c.fb0 Suspend: 0 Teb: 7ffd3000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   8  Id: 136c.1728 Suspend: 0 Teb: 7ffae000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

   9  Id: 136c.918 Suspend: 0 Teb: 7ffad000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  10  Id: 136c.11c4 Suspend: 0 Teb: 7ffd4000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
mswsock!SockWaitForSingleObject
mswsock!WSPSelect
ws2_32!select
wininet!ICAsyncThread::SelectThread
wininet!ICAsyncThread::SelectThreadWrapper
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  11  Id: 136c.13bc Suspend: 0 Teb: 7ffa8000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  12  Id: 136c.178 Suspend: 0 Teb: 7ffab000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserGetMessage
user32!GetMessageA
winmm!mciwindow
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  13  Id: 136c.1594 Suspend: 0 Teb: 7ffa9000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
wdmaud!CWorker::_ThreadProc
wdmaud!CWorker::_StaticThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  14  Id: 136c.b50 Suspend: 0 Teb: 7ffa0000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  15  Id: 136c.eec Suspend: 0 Teb: 7ff9e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  16  Id: 136c.664 Suspend: 0 Teb: 7ff9d000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  17  Id: 136c.2cc Suspend: 0 Teb: 7ff9b000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  18  Id: 136c.e00 Suspend: 0 Teb: 7ff9c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
mshtml!CTimerMan::ThreadExec
mshtml!CExecFT::ThreadProc
mshtml!CExecFT::StaticThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  19  Id: 136c.620 Suspend: 0 Teb: 7ff95000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  20  Id: 136c.1158 Suspend: 0 Teb: 7ff91000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  21  Id: 136c.10cc Suspend: 0 Teb: 7ff90000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  22  Id: 136c.1264 Suspend: 0 Teb: 7ff8f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  23  Id: 136c.13fc Suspend: 0 Teb: 7ffde000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  24  Id: 136c.914 Suspend: 0 Teb: 7ffdb000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  25  Id: 136c.1194 Suspend: 0 Teb: 7ffac000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  26  Id: 136c.1548 Suspend: 0 Teb: 7ffa6000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  27  Id: 136c.a00 Suspend: 0 Teb: 7ff99000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  28  Id: 136c.1360 Suspend: 0 Teb: 7ff97000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  29  Id: 136c.dc0 Suspend: 0 Teb: 7ff96000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  30  Id: 136c.a80 Suspend: 0 Teb: 7ff94000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  31  Id: 136c.1390 Suspend: 0 Teb: 7ffa1000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
WARNING: Stack unwind information not available. Following frames may be wrong.
Flash10b!DllUnregisterServer
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  32  Id: 136c.123c Suspend: 0 Teb: 7ff9f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
WARNING: Stack unwind information not available. Following frames may be wrong.
Flash10b!DllUnregisterServer
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  33  Id: 136c.1398 Suspend: 0 Teb: 7ff82000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
winmm!timeThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  34  Id: 136c.ca0 Suspend: 0 Teb: 7ff7d000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
user32!RealMsgWaitForMultipleObjectsEx
ieui!CoreSC::Wait
ieui!CoreSC::WaitMessage
ieui!WaitMessageEx
ieframe!CBrowserFrame::FrameMessagePump
ieframe!BrowserThreadProc
ieframe!BrowserNewThreadProc
ieframe!SHOpenFolderWindow
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  35  Id: 136c.135c Suspend: 0 Teb: 7ff7c000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  36  Id: 136c.c34 Suspend: 0 Teb: 7ff7b000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  37  Id: 136c.a08 Suspend: 0 Teb: 7ff7a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  38  Id: 136c.1108 Suspend: 0 Teb: 7ff79000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  39  Id: 136c.c20 Suspend: 0 Teb: 7ff77000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  40  Id: 136c.e48 Suspend: 0 Teb: 7ff74000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  41  Id: 136c.298 Suspend: 0 Teb: 7ff73000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  42  Id: 136c.11c8 Suspend: 0 Teb: 7ff72000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  43  Id: 136c.1180 Suspend: 0 Teb: 7ff68000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  44  Id: 136c.1750 Suspend: 0 Teb: 7ff66000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  45  Id: 136c.16d8 Suspend: 0 Teb: 7ff65000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  46  Id: 136c.15b8 Suspend: 0 Teb: 7ff64000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  47  Id: 136c.b88 Suspend: 0 Teb: 7ff6f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  48  Id: 136c.434 Suspend: 0 Teb: 7ff6e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  49  Id: 136c.16e8 Suspend: 0 Teb: 7ff6d000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  50  Id: 136c.f04 Suspend: 0 Teb: 7ff6c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  51  Id: 136c.128c Suspend: 0 Teb: 7ff67000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  52  Id: 136c.1074 Suspend: 0 Teb: 7ff63000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  53  Id: 136c.2dc Suspend: 0 Teb: 7ff62000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  54  Id: 136c.172c Suspend: 0 Teb: 7ff60000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  55  Id: 136c.1240 Suspend: 0 Teb: 7ff69000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  56  Id: 136c.1604 Suspend: 0 Teb: 7ff61000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  57  Id: 136c.6a4 Suspend: 0 Teb: 7ff5f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  58  Id: 136c.1258 Suspend: 0 Teb: 7ff5e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  59  Id: 136c.8c0 Suspend: 0 Teb: 7ff5b000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  60  Id: 136c.868 Suspend: 0 Teb: 7ff58000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  61  Id: 136c.a54 Suspend: 0 Teb: 7ff57000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  62  Id: 136c.77c Suspend: 0 Teb: 7ff56000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  63  Id: 136c.1290 Suspend: 0 Teb: 7ff59000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  64  Id: 136c.1480 Suspend: 0 Teb: 7ff55000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  65  Id: 136c.1270 Suspend: 0 Teb: 7ff54000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  66  Id: 136c.b8c Suspend: 0 Teb: 7ff53000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  67  Id: 136c.167c Suspend: 0 Teb: 7ff92000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  68  Id: 136c.176c Suspend: 0 Teb: 7ff8e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  69  Id: 136c.80c Suspend: 0 Teb: 7ff8b000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  70  Id: 136c.1570 Suspend: 0 Teb: 7ff8a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  71  Id: 136c.e74 Suspend: 0 Teb: 7ff78000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  72  Id: 136c.1490 Suspend: 0 Teb: 7ff76000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  73  Id: 136c.d28 Suspend: 0 Teb: 7ff6a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  74  Id: 136c.8d8 Suspend: 0 Teb: 7ff5c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  75  Id: 136c.1064 Suspend: 0 Teb: 7ff4a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  76  Id: 136c.1478 Suspend: 0 Teb: 7ff47000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  77  Id: 136c.1470 Suspend: 0 Teb: 7ff44000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  78  Id: 136c.aa0 Suspend: 0 Teb: 7ff43000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  79  Id: 136c.1210 Suspend: 0 Teb: 7ff48000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  80  Id: 136c.954 Suspend: 0 Teb: 7ff46000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  81  Id: 136c.9d4 Suspend: 0 Teb: 7ff45000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  82  Id: 136c.f30 Suspend: 0 Teb: 7ff42000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  83  Id: 136c.cc4 Suspend: 0 Teb: 7ff34000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  84  Id: 136c.1018 Suspend: 0 Teb: 7ff33000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  85  Id: 136c.940 Suspend: 0 Teb: 7ff32000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  86  Id: 136c.bd8 Suspend: 0 Teb: 7ff31000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  87  Id: 136c.1714 Suspend: 0 Teb: 7ff84000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
user32!DialogBox2
user32!InternalDialogBox
user32!SoftModalMessageBox
user32!MessageBoxWorker
user32!MessageBoxTimeoutW
user32!MessageBoxTimeoutA
user32!MessageBoxExA
ieframe!Detour_MessageBoxExA
user32!MessageBoxA
WARNING: Stack unwind information not available. Following frames may be wrong.
MathMLMimer!DllUnregisterServer
MathMLMimer!DllUnregisterServer
MathMLMimer!DllUnregisterServer

  88  Id: 136c.1744 Suspend: 0 Teb: 7ff83000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  89  Id: 136c.9cc Suspend: 0 Teb: 7ff81000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  90  Id: 136c.f68 Suspend: 0 Teb: 7ff5a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  91  Id: 136c.17f4 Suspend: 0 Teb: 7ff49000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  92  Id: 136c.13c Suspend: 0 Teb: 7ff41000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  93  Id: 136c.110 Suspend: 0 Teb: 7ff3f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  94  Id: 136c.8e4 Suspend: 0 Teb: 7ff3e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  95  Id: 136c.fcc Suspend: 0 Teb: 7ff4c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
user32!RealMsgWaitForMultipleObjectsEx
ieui!CoreSC::Wait
ieui!CoreSC::WaitMessage
ieui!WaitMessageEx
ieframe!CBrowserFrame::FrameMessagePump
ieframe!BrowserThreadProc
ieframe!BrowserNewThreadProc
ieframe!SHOpenFolderWindow
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  96  Id: 136c.1378 Suspend: 0 Teb: 7ff4b000 Unfrozen

ntdll!KiFastSystemCallRet
user32!NtUserWaitMessage
ieframe!CTabWindow::_TabWindowThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  97  Id: 136c.8ec Suspend: 0 Teb: 7ff40000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllGetClassObject
ToolbarA!ToolBarMain
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  98  Id: 136c.ac Suspend: 0 Teb: 7ff3d000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

  99  Id: 136c.79c Suspend: 0 Teb: 7ff3c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
WARNING: Stack unwind information not available. Following frames may be wrong.
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow
ToolbarA!DllCanUnloadNow

ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 100  Id: 136c.bd0 Suspend: 0 Teb: 7ff3a000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjects
msiltcfg!WorkerThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 101  Id: 136c.1504 Suspend: 0 Teb: 7ff36000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 102  Id: 136c.5c0 Suspend: 0 Teb: 7ff35000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 103  Id: 136c.17a0 Suspend: 0 Teb: 7ff2d000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 104  Id: 136c.17c4 Suspend: 0 Teb: 7ff2c000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 105  Id: 136c.f08 Suspend: 0 Teb: 7ffaa000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 106  Id: 136c.1268 Suspend: 0 Teb: 7ff28000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 107  Id: 136c.12c8 Suspend: 0 Teb: 7ff27000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 108  Id: 136c.1634 Suspend: 0 Teb: 7ff26000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 109  Id: 136c.120c Suspend: 0 Teb: 7ff21000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 110  Id: 136c.13f4 Suspend: 0 Teb: 7ff20000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 111  Id: 136c.1494 Suspend: 0 Teb: 7ff1f000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 112  Id: 136c.16ec Suspend: 0 Teb: 7ff1e000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 113  Id: 136c.d68 Suspend: 0 Teb: 7ffa4000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
rpcrt4!COMMON_ProcessCalls
rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents
rpcrt4!ProcessIOEventsWrapper
rpcrt4!BaseCachedThreadRoutine
rpcrt4!ThreadStartRoutine
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

 114  Id: 136c.5fc Suspend: 0 Teb: 7ffdd000 Unfrozen

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForWorkViaWorkerFactory
ntdll!TppWorkerThread
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

In contrast, Flash10b module (shown in magenta above) is not ubiquitous, it is present on just 2 threads.

- Dmitry Vostokov @ DumpAnalysis.org -

Dictionary of Debugging: Hang

Tuesday, June 23rd, 2009

This is the first draft entry for the forthcoming Dictionary of Debugging (ISBN: 978-1906717353). The entry format is not final and more information will be added to individual entries in the future.

Hang

The perceived absence of response from a present program, service or system, either visual or functional.

References: Hangs explained, The difference between crashes and hangs explained

Synonyms: freeze, stop, halt.

Antonyms: live.

Similar: sluggish.

Also: crash.

- Dmitry Vostokov @ DumpAnalysis.org -

Review Campaign on Amazon

Monday, June 22nd, 2009

I recently started posting book reviews on Amazon and re-posting them from my blogs. Previously I didn’t do that because I thought that Amazon acquired exclusive rights for submitted reviews but I wanted to include my selected reviews in my own books and on my own websites. Recently I read its small print and found that Amazon rights are non-exclusive. Of course, I take advantage and promote my blogs in book reviews that have the following form:

Review text …

Thanks,
Dmitry Vostokov
Founder of …
[Editor-in-Chief of …]

Software engineering book reviews point to DumpAnalysis Portal and Software Generalist blog, science, history and general non-fiction book reviews point to Literate Scientist blog, management book reviews point to Management Bits and Tips blog. I found already that people Google these blogs to find out who’s that guy and the number of visitors to my other blogs has doubled.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.97

Monday, June 22nd, 2009

“The real” debugging “begins only after hitting” a button.

Anonymous

- Dmitry Vostokov @ DumpAnalysis.org -

Succession of Patterns (Part 1)

Monday, June 22nd, 2009

Looking at pattern cooperation studies it is easy to see that some patterns precede others, for example, heap corruption might be blocked by a hard error or a message box and therefore block other threads, creating conditions for another pattern to appear, wait chains. Blocked threads may block other coupled processes creating inter-process wait chains. Successive patterns reach the climax at the end and the system is no longer able to generate any other patterns.

The goal here is to find patterns that most likely happen in succession and another pattern series that are less likely to effect other abnormal conditions. Such pattern sequences can help in troubleshooting and finding root causes.

- Dmitry Vostokov @ DumpAnalysis.org

Debugalov has been burnt!

Sunday, June 21st, 2009

Just noticed that RichardS from Australia (nickname: rsayad1) was outraged after reading Dumps, Bugs and Debugging Forensics book and burnt it in his fireplace. I applied my analytical and forensic skills and figured out his name: Richard Sayad. The only excuse for him is that crash dumps is so hot topic that he rushed to buy the book without even looking inside it on Amazon.

What do you think and what is your opinion about this book? It is basically composed from the following material:

Cartoons

The first 64 bugtations

For the book all cartoons were edited, polished and most of them became full color in the book version.

- Dmitry Vostokov @ DumpAnalysis.org

Does the size of hardware matter?

Saturday, June 20th, 2009

I was in McDonald’s today with my daughter. This time they popularize Einstein, giving his stature in happy meal packs, telling that the size of his brain was the same as mine. My brain continued to work after meal and I finally understood that the right memory dump is what really matters. Your computer may have 1Tb of memory but if you didn’t get the right dump at the right moment you wasted your time.

- Dmitry Vostokov @ DumpAnalysis.org

Practical Foundations Series

Saturday, June 20th, 2009

Following the success of Windows Debugging: Practical Foundations the following title will be published this summer:

Windows Device Drivers: Practical Foundations (ISBN: 978-0955832840)

Table of contents will be posted later.

Other planned titles:

X64 Windows Debugging: Practical Foundations (ISBN: 978-1906717568) 

Windows Multithreading: Practical Foundations (ISBN: 978-1906717742)

 Like Windows Debugging book, these forthcoming titles are based on my seminars. 

- Dmitry Vostokov @ DumpAnalysis.org -

Null data pointer, pass through functions and platformorphic fault: pattern cooperation

Saturday, June 20th, 2009

We got a bugcheck when accessing a NULL data pointer:

1: kd> r
Last set context:
rax=0000000063537852 rbx=0000000000000000 rcx=0000000000000009
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffffadf262760da rsp=fffffadf15973968 rbp=0000000070537852
 r8=fffffadf31614b00  r9=fffffadffe9fa7b0 r10=000000000000000a
r11=fffffadf31614bf0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 efl=00010206
rdbss!RxIsThisACscAgentOpen+0×30:
fffffadf`262760da f3a6 repe cmps byte ptr [rsi],byte ptr [rdi]

Default analysis via !analyze -v pointed to the first non-MS module DriverA (the identification process is explained here) located on the following stack trace (that also shows exception processing in file system kernel drivers):

1: kd> kc 100
Call Site
nt!KeBugCheckEx
rdbss!RxExceptionFilter+0x15e
rdbss!RxFsdCommonDispatch+0x6d5
nt!_C_specific_handler+0x9b
nt!RtlpExecuteHandlerForException+0xd
nt!RtlDispatchException+0x2c0
nt!KiDispatchException+0xd9
nt!KiExceptionExit
nt!KiPageFault+0x1e1
rdbss!RxIsThisACscAgentOpen+0x30
rdbss!RxInitializeVNetRootParameters+0x31d
rdbss!RxFindOrConstructVirtualNetRoot+0x180
rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223
rdbss!RxCommonCreate+0x470
rdbss!RxFsdCommonDispatch+0x51c
mrxsmb!MRxSmbFsdDispatch+0x211
fltmgr!FltpCreate+0x353
DriverA!DispatchThrough+0×177
DriverB!PassThrough+0×12c
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×3f2
fltmgr!FltpCreate+0×3bd
Mup!DnrRedirectFileOpen+0×791
Mup!DnrNameResolve+0xb19
Mup!DnrStartNameResolution+0×478
Mup!DfsCommonCreate+0×3dc
Mup!DfsFsdCreate+0×10d
Mup!MupCreate+0×125
nt!IopParseDevice+0×1088
nt!ObpLookupObjectName+0×931
nt!ObOpenObjectByName+0×180
nt!IopCreateFile+0×630
nt!IoCreateFileSpecifyDeviceObjectHint+0xb4
fltmgr!FltpExpandFilePathWorker+0×23d
fltmgr!FltpExpandFilePath+0×5b
fltmgr!FltpGetOpenedDestinationFileName+0xd5
fltmgr!FltpGetNormalizedDestinationFileName+0×15
fltmgr!FltGetDestinationFileNameInformation+0×17d
DriverC+0×383e
fltmgr!FltpPerformPreCallbacks+0×3e2
fltmgr!FltpPassThroughInternal+0×40
fltmgr!FltpDispatch+0×102
Mup!DfsCommonSetInformation+0×165
Mup!DfsFsdSetInformation+0×67
nt!NtSetInformationFile+0×916
nt!KiSystemServiceCopyEnd+0×3

Although DriverA function on the stack trace looked like a pass through, DriverA.sys file was removed from the system. Nevertheless, the same pattern continued:

0: kd> kc 100
Call Site
nt!KeBugCheckEx
rdbss!RxExceptionFilter+0x15e
rdbss!RxFsdCommonDispatch+0x6d5
nt!_C_specific_handler+0x9b
nt!RtlpExecuteHandlerForException+0xd
nt!RtlDispatchException+0x2c0
nt!KiDispatchException+0xd9
nt!KiExceptionExit
nt!KiPageFault+0x1e1
rdbss!RxIsThisACscAgentOpen+0x30
rdbss!RxInitializeVNetRootParameters+0x31d
rdbss!RxFindOrConstructVirtualNetRoot+0x180
rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223
rdbss!RxCommonCreate+0x470
rdbss!RxFsdCommonDispatch+0x51c
mrxsmb!MRxSmbFsdDispatch+0x211
fltmgr!FltpCreate+0x353
DriverB!PassThrough+0×12c
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×3f2
fltmgr!FltpCreate+0×3bd
Mup!DnrRedirectFileOpen+0×791
Mup!DnrNameResolve+0xb19
Mup!DnrStartNameResolution+0×478
Mup!DfsCommonCreate+0×3dc
Mup!DfsFsdCreate+0×10d
Mup!MupCreate+0×125
nt!IopParseDevice+0×1088
nt!ObpLookupObjectName+0×931
nt!ObOpenObjectByName+0×180
nt!IopCreateFile+0×630
nt!IoCreateFileSpecifyDeviceObjectHint+0xb4
fltmgr!FltpExpandFilePathWorker+0×23d
fltmgr!FltpExpandFilePath+0×5b
fltmgr!FltpGetOpenedDestinationFileName+0xd5
fltmgr!FltpGetNormalizedDestinationFileName+0×15
fltmgr!FltGetDestinationFileNameInformation+0×17d
DriverC+0×383e
fltmgr!FltpPerformPreCallbacks+0×3e2
fltmgr!FltpPassThroughInternal+0×40
fltmgr!FltpDispatch+0×102
Mup!DfsCommonSetInformation+0×165
Mup!DfsFsdSetInformation+0×67
nt!NtSetInformationFile+0×916
nt!KiSystemServiceCopyEnd+0×3

So it was concluded that the presence of DriverA was irrelevant to the problem. Now DriverB was pointed to by the default analysis as a possible culprit. However, the fault appeared platformorphic: Google search found another similar stack trace shape with the same faulted instruction but without DriverB and DriverC. Therefore the conclusion was that modules DriverA, DriverB and DriverC didn’t have the straightforward contribution to the abnormal system behaviour.

- Dmitry Vostokov @ DumpAnalysis.org -

The Meaning of DUMP

Friday, June 19th, 2009

Following the meaning of DATA and memory dump world view via universal memory dumps I finally deciphered the acronym DUMP:

Digital Universal Memory Phase

This is the view from phase space perspective. From the point of phase space perspective we can also say:

Digital Universal Memory Point

It was the letter P that I was thinking hard about. Fortunately, when I opened Oxford Advanced Learner’s Dictionary on P section, it was “phase” word that grabbed my attention. Familiarity with classical physics and its Hamiltonian formulation provided the necessary glue.

- Dmitry Vostokov @ DumpAnalysis.org -

Dump and Trace Analysis on Facebook

Friday, June 19th, 2009

The following group has been created on Facebook:

DATA (Dump Analysis + Trace Analysis)

Please don’t hesitate to join and spread news about it :-)

- Dmitry Vostokov @ DumpAnalysis.org -

Platformorphism

Thursday, June 18th, 2009

This is a kind of a “faultomorphism”, a fault, a crash point and stack trace shape preserving map between two platforms (such as 32-bit and 64-bit). This new word was derived from the concatenation of platform and morphism. Here is an example:

; 64-bit crash dump

0: kd> r
Last set context:
rax=0000000063537852 rbx=0000000000000000 rcx=0000000000000009
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffffadf262760da rsp=fffffadf15973968 rbp=0000000070537852
 r8=fffffadf31614b00  r9=fffffadffe9fa7b0 r10=000000000000000a
r11=fffffadf31614bf0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 efl=00010206
rdbss!RxIsThisACscAgentOpen+0×30:
fffffadf`262760da f3a6 repe cmps byte ptr [rsi],byte ptr [rdi]

0: kd> kL 100
Child-SP          RetAddr           Call Site
fffffadf`15973968 fffffadf`2629e768 rdbss!RxIsThisACscAgentOpen+0x30
fffffadf`15973970 fffffadf`262988f5 rdbss!RxInitializeVNetRootParameters+0x31d
fffffadf`159739f0 fffffadf`2629bcfd rdbss!RxFindOrConstructVirtualNetRoot+0x180
fffffadf`15973ad0 fffffadf`26297a6c rdbss!RxCanonicalizeNameAndObtainNetRoot+0x223
fffffadf`15973b70 fffffadf`26272a77 rdbss!RxCommonCreate+0x470
fffffadf`15973c80 fffffadf`261be3e8 rdbss!RxFsdCommonDispatch+0x51c
fffffadf`15973d80 fffffadf`29314db3 mrxsmb!MRxSmbFsdDispatch+0x211
[...]

; 32-bit crash dump

0: kd> r
eax=00000000 ebx=b6a23a80 ecx=00000009 edx=00000000 esi=00000008 edi=b6a23a80
eip=b6a23a5f esp=b3ce800c ebp=b3ce801c iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
rdbss!RxIsThisACscAgentOpen+0×38:
b6a23a5f f3a6 repe cmps byte ptr [esi],byte ptr es:[edi]

0: kd> kL 100
b3ce801c b6a2b431 rdbss!RxIsThisACscAgentOpen+0x38
b3ce803c b6a2bbf7 rdbss!RxInitializeVNetRootParameters+0x282
b3ce809c b6a2e6cd rdbss!RxFindOrConstructVirtualNetRoot+0xdc
b3ce80d0 b6a2ae15 rdbss!RxCanonicalizeNameAndObtainNetRoot+0x197
b3ce8134 b6a20d51 rdbss!RxCommonCreate+0x2c3
b3ce81cc b6a2acc2 rdbss!RxFsdCommonDispatch+0x353
b3ce81f4 b69ac317 rdbss!RxFsdDispatch+0xda
b3ce8214 804e13d9 mrxsmb!MRxSmbFsdDispatch+0x134
[...]

We can see that stack traces are almost the same, function offsets are very close and faulted instruction is the same up to an opcode. Not to mention that bugchecks are identical:

RDR_FILE_SYSTEM (27)
    If you see RxExceptionFilter on the stack then the 2nd and 3rd parameters are the
    exception record and context record. Do a .cxr on the 3rd parameter and then kb to
    obtain a more informative stack trace.
    The high 16 bits of the first parameter is the RDBSS bugcheck code, which is defined
    as follows:
     RDBSS_BUG_CHECK_CACHESUP  = 0xca550000,
     RDBSS_BUG_CHECK_CLEANUP   = 0xc1ee0000,
     RDBSS_BUG_CHECK_CLOSE     = 0xc10e0000,
     RDBSS_BUG_CHECK_NTEXCEPT  = 0xbaad0000

Therefore, we can also say that these crashes are platformorphic. Obviously, this stems from the fact that source code was identical or almost identical for both platforms.

- Dmitry Vostokov @ DumpAnalysis.org -

Trace Analysis Patterns (Part 4)

Tuesday, June 16th, 2009

Sometimes we see a functional activity in a trace and / or see basic facts. Then we might want to find a correlation between that activity or facts in another part of the trace. If that intra-correlation fits into our problem description we may claim a possible explanation or, if we are lucky, we have just found, an inference to the best explanation, as philosophers of science like to say. Here is an example, but this time using Citrix WindowHistory tracing tool. A third-party application was frequently loosing the focus and the suspicion was on a terminal services client process. It was found that the following WindowHistory trace fragment corresponds to that application:

Handle: 00050586 Class: "Application A Class" Title: ""
     Title changed at 15:52:4:3 to "Application A"
     Title changed at 15:52:10:212 to "Application A - File1"
[...]
   Process ID: 89c
   Thread ID: d6c
[...]
   Visible: true
   Window placement command: SW_SHOWNORMAL
        Placement changed at 15:54:57:506 to SW_SHOWMINIMIZED
        Placement changed at 15:55:2:139 to SW_SHOWNORMAL
   Foreground: false
        Foreground changed at 15:52:4:3 to true
        Foreground changed at 15:53:4:625 to false
        Foreground changed at 15:53:42:564 to true
        Foreground changed at 15:53:44:498 to false
        Foreground changed at 15:53:44:498 to true
        Foreground changed at 15:53:44:592 to false
        Foreground changed at 15:53:45:887 to true
        Foreground changed at 15:53:47:244 to false
        Foreground changed at 15:53:47:244 to true
        Foreground changed at 15:53:47:353 to false
        Foreground changed at 15:54:26:416 to true
        Foreground changed at 15:54:27:55 to false
        Foreground changed at 15:54:27:55 to true
        Foreground changed at 15:54:27:180 to false
        Foreground changed at 15:54:28:428 to true
        Foreground changed at 15:54:28:771 to false
        Foreground changed at 15:54:28:865 to true
        Foreground changed at 15:54:29:99 to false
        Foreground changed at 15:54:30:877 to true
        Foreground changed at 15:54:57:521 to false
        Foreground changed at 15:55:2:76 to true
        Foreground changed at 15:57:3:378 to false
        Foreground changed at 15:57:11:396 to true
        Foreground changed at 15:57:29:601 to false
        Foreground changed at 15:57:39:803 to true
        Foreground changed at 15:58:54:41 to false
        Foreground changed at 15:59:8:96 to true
        Foreground changed at 16:1:19:478 to false
        Foreground changed at 16:1:27:527 to true
        Foreground changed at 16:1:39:914 to false
        Foreground changed at 16:2:0:515 to true
        Foreground changed at 16:7:14:628 to false
        Foreground changed at 16:7:24:246 to true
        Foreground changed at 16:9:53:523 to false
        Foreground changed at 16:10:15:919 to true
        Foreground changed at 16:10:31:426 to false
        Foreground changed at 16:11:12:818 to true
        Foreground changed at 16:11:59:538 to false
        Foreground changed at 16:12:39:456 to true
        Foreground changed at 16:13:6:364 to false

Corresponding terminal services client window trace fragment doesn’t have any foreground changes but another application main window has lots of them:

Handle: 000D0540 Class: "Application B Class" Title: "Application B"
[...]
   Process ID: 3ac
   Thread ID: bd4
[...]
   Foreground: false
        Foreground changed at 15:50:36:972 to true
        Foreground changed at 15:50:53:732 to false
        Foreground changed at 15:50:53:732 to true
        Foreground changed at 15:50:53:826 to false
        Foreground changed at 15:51:51:352 to true
        Foreground changed at 15:51:53:941 to false
        Foreground changed at 15:53:8:135 to true
        Foreground changed at 15:53:8:182 to false
        Foreground changed at 15:53:10:178 to true
        Foreground changed at 15:53:13:938 to false
        Foreground changed at 15:53:30:443 to true
        Foreground changed at 15:53:31:20 to false
        Foreground changed at 15:53:31:20 to true
        Foreground changed at 15:53:31:129 to false
        Foreground changed at 15:53:34:78 to true
        Foreground changed at 15:53:34:795 to false
        Foreground changed at 15:53:34:795 to true
        Foreground changed at 15:53:34:873 to false
        Foreground changed at 15:53:36:901 to true
        Foreground changed at 15:53:42:502 to false
        Foreground changed at 15:53:42:502 to true
        Foreground changed at 15:53:42:564 to false
        Foreground changed at 15:57:3:425 to true
        Foreground changed at 15:57:4:595 to false
        Foreground changed at 15:57:10:507 to true
        Foreground changed at 15:57:11:318 to false
        Foreground changed at 15:57:29:632 to true
        Foreground changed at 15:57:31:67 to false
        Foreground changed at 15:57:32:721 to true
        Foreground changed at 15:57:33:844 to false
        Foreground changed at 15:58:54:88 to true
        Foreground changed at 15:58:56:178 to false
        Foreground changed at 15:59:6:505 to true
        Foreground changed at 15:59:7:987 to false
        Foreground changed at 16:1:19:525 to true
        Foreground changed at 16:1:19:961 to false
        Foreground changed at 16:1:26:607 to true
        Foreground changed at 16:1:27:434 to false
        Foreground changed at 16:1:39:914 to true
        Foreground changed at 16:1:39:992 to false
        Foreground changed at 16:1:49:798 to true
        Foreground changed at 16:2:0:437 to false
        Foreground changed at 16:7:14:628 to true
        Foreground changed at 16:7:14:847 to false
        Foreground changed at 16:7:18:76 to true
        Foreground changed at 16:7:24:106 to false
        Foreground changed at 16:9:58:790 to true
        Foreground changed at 16:10:4:16 to false
        Foreground changed at 16:10:4:874 to true
        Foreground changed at 16:10:4:890 to false
        Foreground changed at 16:10:8:634 to true
        Foreground changed at 16:10:15:779 to false
        Foreground changed at 16:10:56:766 to true
        Foreground changed at 16:10:59:402 to false
        Foreground changed at 16:10:59:652 to true
        Foreground changed at 16:10:59:667 to false
        Foreground changed at 16:12:9:397 to true
        Foreground changed at 16:12:39:347 to false
        Foreground changed at 16:13:18:375 to true
        Foreground changed at 16:14:33:656 to false

We can see that most of the time when Application A window looses focus Application B window gets it.

- Dmitry Vostokov @ TraceAnalysis.org -

Software Defect Construction

Tuesday, June 16th, 2009

This is the main topic of the forthcoming next issue of Debugged MZ/PE magazine. The most close term is called “fault injection” but I rediscovered it as a “software defect construction”, “software defect simulation” or “software defect modeling”. The latter term is also used to refer to construction of mathematical models related to software product quality and corresponding statistics but “modeling software defects” seems appropriate subtitle for the magazine front cover picture… Software defect construction is more general term than fault injection. The latter is used for testing but we want to simulate bugs and abnormal system conditions to study debugging and memory dump analysis techniques or to build reproduction environments. I actually recently found and bought the used copy of this book:

Software Fault Injection: Inoculating Programs Against Errors

Buy from Amazon

and plan to write my own book with the following working title later:

Software Defect Construction: Simulation and Modeling of Software Bugs (ISBN: 978-1906717759)

- Dmitry Vostokov @ DumpAnalysis.org -

Bugtation No.96

Monday, June 15th, 2009

“Touched by an” exception.

Touched by an Angel

- Dmitry Vostokov @ DumpAnalysis.org -