Archive for November 12th, 2009

Crash Dump Analysis Patterns (Part 91)

Thursday, November 12th, 2009

Sometimes we can observe rare events when abnormal conditions that usually result in a system crash result in a milder problem, for example, a service is unavailable and not affecting other services and users. It was reported that an application was freezing during user session logoff. A complete memory dump was saved at that time and its stack trace collection (!stacks command) shows the following suspicious thread in a user process (all other threads were normally waiting):

0: kd> !stacks
Proc.Thread  .Thread  Ticks   ThreadState Blocker
[...]
                                    [89cfa960 Application.exe]
ea0.001c4c  89a11db0 0499cd1 Blocked    DriverA+0×69db
[…]

0: kd> !thread 89a11db0 16
THREAD 89a11db0  Cid 0ea0.1c4c  Teb: 7ffdf000 Win32Thread: bc347a48 WAIT: (Unknown) KernelMode Non-Alertable
    89b87770  Unknown
    b97004ac  NotificationEvent
IRP List:
    899e2668: (0006,0244) Flags: 00000884  Mdl: 00000000
Not impersonating
DeviceMap                 daf62b28
Owning Process            89cfa960       Image:         Application.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      909331         Ticks: 4824273 (0:20:56:19.265)
Context Switch Count      186                 LargeStack
UserTime                  00:00:00.015
KernelTime                00:00:00.093
*** ERROR: Module load completed but symbols could not be loaded for Application.exe
Win32 Start Address Application (0×00406b2a)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init b60ceb30 Current b60cdf10 Base b60cf000 Limit b60cb000 Call b60ceb34
Priority 10 BasePriority 10 PriorityDecrement 0
ChildEBP RetAddr  Args to Child             
b60cdf28 80833485 89a11db0 00000002 00000000 nt!KiSwapContext+0×26
b60cdf54 808294b9 dc399008 89b87748 b60ce01c nt!KiSwapThread+0×2e5
b60cdf88 b96d69db 00000002 b60cdfbc 00000001 nt!KeWaitForMultipleObjects+0×3d7
WARNING: Stack unwind information not available. Following frames may be wrong.
b60cdfe8 b96d719e 89b87748 dc399008 b60ce01c DriverA+0×69db
[…]

We notice “89b87770  Unknown” and double check what object the thread is waiting for:

0: kd> dp b60cdfbc L00000002
b60cdfbc  89b87770 b97004ac

These are exactly the same objects that are listed in !thread command output. We see that the second one is normal and resides in a nonpaged area:

0: kd> dt _DISPATCHER_HEADER b97004ac
ntdll!_DISPATCHER_HEADER
   +0x000 Type             : 0 ''
   +0x001 Absolute         : 0 ''
   +0x001 NpxIrql          : 0 ''
   +0x002 Size             : 0x4 ''
   +0x002 Hand             : 0x4 ''
   +0x003 Inserted         : 0 ''
   +0x003 DebugActive      : 0 ''
   +0x000 Lock             : 262144
   +0x004 SignalState      : 0
   +0x008 WaitListHead     : _LIST_ENTRY [ 0x89a11e70 - 0x89a11e70 ]

0: kd> !address b97004ac
  a71e3000 - 13e1d000                          
          Usage       KernelSpaceUsageNonPagedSystem

The other looks like an invalid Random Object from the free nonpaged pool entry (it even says about itself that it is bad) that used to belong in the past to Configuration Manager:

0: kd> !pool 89b87770
Pool page 89b87770 region is Nonpaged pool
[...]
 89b87540 size:   98 previous size:   40  (Allocated)  File (Protected)
*89b875d8 size:  260 previous size:   98  (Free)      *CMpa
  Pooltag CMpa : registry post apcs, Binary : nt!cm

 89b87838 size:   28 previous size:  260  (Allocated)  FSfm
[…]

0: kd> dd 89b87770
89b87770  bad0b0b0 00000000 00000000 00000000
89b87780  8a04be01 00000000 89b87788 89b87788
89b87790  00150006 e56c6946 8993e208 89ab96b8
89b877a0  00000000 00000000 bad0b0b0 c0000800
89b877b0  02110004 63426343 88ebbf80 00001000
89b877c0  00199000 00000000 8993e238 88d0d248
89b877d0  0019a000 00000000 00000000 00000000
89b877e0  00000000 00000000 00000000 00000000

0: kd> dt _DISPATCHER_HEADER 89b87770
ntdll!_DISPATCHER_HEADER
   +0×000 Type             : 0xb0 ”
   +0×001 Absolute         : 0xb0 ”
   +0×001 NpxIrql          : 0xb0 ”
   +0×002 Size             : 0xd0 ”
   +0×002 Hand             : 0xd0 ”
   +0×003 Inserted         : 0xba ”
   +0×003 DebugActive      : 0xba ”
   +0×000 Lock             : -1160728400
   +0×004 SignalState      : 0
   +0×008 WaitListHead     : _LIST_ENTRY [ 0×0 - 0×0 ]

Now some counterfactual thinking. One possible scenario after KeWaitForMultipleObjects was called to wait for both objects to become signalled (3rd WAIT_TYPE parameter) the free pool slot was allocated or coalesced with SignalState becoming nonzero by coincidence and other members becoming random values and then the second normal object becomes signalled when another thread sets the notification event…

- Dmitry Vostokov @ DumpAnalysis.org -

BoBo Address (Debugging Slang, Part 4)

Thursday, November 12th, 2009
bobo address - an address a system will definitely crash at, given an opportunity

Examples: bad0b0b0, 00000000

- Dmitry Vostokov @ DumpAnalysis.org -

BAD0B0B0 Address: Childhood Memories

Thursday, November 12th, 2009

The system crashes at it (Google search shows 164,000 hits). Notice the second part “b0b0″ after “bad”. For my eyes (I wear glasses) it is “bobo” – that was my mother telling me when I was a child and I was about to touch a hot or sharp object.

Bad! Bo-bo!

Was the address value designed intentionally or it is just a coincidence?

- Dmitry Vostokov @ DumpAnalysis.org -

The CRC Encyclopedia of Mathematics

Thursday, November 12th, 2009

The CRC Encyclopedia of Mathematics, Third Edition - 3 Volume Set

Buy from Amazon

I spotted this book on Amazon US and was thrilled to have the new edition in 3 volumes for easy handling when reading. I also have the previous edition that is even featured on my own book cover (the picture of my previous library book arrangement, the book is highlighted in white rectangle in the lower right corner):

This is a unique volume that sits between The Princeton Companion to Mathematics (that I’m also reading now) and Encyclopedic Dictionary of Mathematics: The Mathematical Society of Japan (that I also own). In fact after reading 3 volumes from cover to cover I can start with 2 volumes of EDM. There is also Springer Encyclopaedia of Mathematics in 11 volumes with various additional supplement volumes that I plan to own as well and it looks to me on the same level as EDM.

After searching for the best price I ordered a copy from Amazon DE and after my purchase in just a few days the price was up by 50%! I can only explain this that more people tried to purchase after I used twitter to announce this encyclopedia (there were 5 copies available on Amazon DE and in just 2 days only 1 left) or there was a mistake in price.

3 volumes arrived and I immediately started reading them, a few pages from each volume every day using mod 3 reading technique, for example, Wed - Vol I, Thu - Vol II, Fri - Vol III, Mon - Vol I, an so on. I prefer paper books for bulk reading instead of electronic version (in this case corresponding website) although if I’m interested in a specific article or a keyword I go to Wolfram MathWorld website to get the latest update and citations. These paperback volumes are just for day-to-day scheduled reading to get ideas and general mathematical education. This is why I don’t need an Index. For example, just after reading the first pages I got the idea of cubic (qubic) memory representation.

I usually put reviews on Amazon after I finish a book from cover to cover but in this case the review would be waiting for at least a year so I write it now based on my first impressions. After some time I plan to update it.

- Dmitry Vostokov @ LiterateScientist.com -