WinDbg.org has been updated to include a sorted command check list, a link to MSDN help and a link to yet another book related to WinDbg. All changes are highlighted in red on the following page screenshot:
- Dmitry Vostokov @ DumpAnalysis.org -
Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.
The Day of Mirthful License
Modules, products or systems built on December 1 win customers over despite denying the rules of protocol. They can provide impression of simplicity but this is not the case. Their internals can be very complex and their perceived simplicity is the direct consequence of their user interface. Modules are not fully aware of what they are doing and seen as being driven by external components. Modules, products or systems built on this day are very busy with computation and have little time to care about users despite their built-in human-computer interaction. However they strive to calculate the impossible in all domains. They love to interact with other components with opposite behaviour. December 1 components are free modules and exert the full computation capabilities on the right data arrived at the right time. Working too many hours can seriously damage their internals and they may loose touch with their built-in goals. Sometimes December 1 modules, products or systems outrageous behaviour need to be amended to become more tolerable and not to hang. They need to be idle from time to time to avoid burn-out.
DLL, SYS and EXE born on this date:
VERSION.dll Wed Dec 01 01:37:27 1999
nvcoaft51.sys Wed Dec 01 11:55:40 2004
dump_m5289.sys Wed Dec 01 02:49:17 2004
CFGMGR32.DLL Wed Dec 01 15:37:31 1999
MPRAPI.DLL Wed Dec 01 15:37:29 1999
ICMP.DLL Wed Dec 01 15:37:29 1999
RTUTILS.DLL Wed Dec 01 15:37:27 1999
Weaknesses: Misdirected computation, unawareness of environment.
Strengths: Energetic computation, UI extroverts.
Advice: Keep a handle on your desire to debug. Beware of damaging other processes and alienating users with a overly direct debugging approach.
- Dmitry Vostokov @ DumpAnalysis.org -
This December starts with easy and obvious patterns that I forgot to write about. Integer division by zero is one of the most frequent exceptions. It is easily recognizable in process crash dumps by the processor instruction that caused this exception type (DIV or IDIV):
FAULTING_IP:
DLL!FindHighestID+278
1b2713c4 f775e4 div dword ptr [ebp-0×1c]
EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 1b2713c4 (DLL!FindHighestID+0x00000278)
ExceptionCode: c0000094 (Integer divide-by-zero)
ExceptionFlags: 00000000
NumberParameters: 0
or
FAULTING_IP:
Application+263d8
004263d8 f7fe idiv eax,esi
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 004263d8 (Application+0x000263d8)
ExceptionCode: c0000094 (Integer divide-by-zero)
ExceptionFlags: 00000000
NumberParameters: 0
ERROR_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION} Integer division by zero.
- Dmitry Vostokov @ DumpAnalysis.org -
Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.
The Day of Measured Attack
Modules built on November 30 have a built-in capacity for overcoming challenges of hostile environments. They are capable of bringing surprises to security attacks, for example. One can learn a lot about them by studying their traces or doing reverse engineering. November 30 components do their work to the utmost degree of quality with a little waste of CPU and memory. Message boxes they pop up have a subtle sense of thought-provoking humour but it can also be a full blown thigh-slapping. November 30 systems are very defensive when attacked. They are stubbornly resistant to reverse engineering but at the same time very open to honest debugging.
DLL, SYS and EXE born on this date:
tifsfilt.sys Tue Nov 30 07:16:27 2004
alrsvc.dll Tue Nov 30 17:31:14 1999
ntkrpamp.exe Fri Nov 30 14:54:49 2007
Tppwrif.sys Tue Nov 30 02:38:22 2004
Weaknesses: Over-reactive to code and data injection, funny behaviour.
Strengths: Thorough developed, dynamic responsiveness.
Advice: Improvise during troubleshooting and debugging. Admire control vs. spontaneity balance. Laugh at your failures.
- Dmitry Vostokov @ DumpAnalysis.org -
I often hear about cosmic mysteries or influences when problems happen in computer environments. Passing by an astrology section in a local book shop yesterday a revelation came to me that a compile / link time (build time) might influence a component (DLL, EXE, SYS files), product or system behaviour. From now on I’m going to blog about every build date with examples. And as usual, I’m also going to publish a book for this iterative and incremental activity called:
Title: The Secret Language of Build Dates: Unique Astrology Profiles for Every Build of the Year with Advice on Testing, Troubleshooting and Debugging
ISBN: 978-1906717407
Knowing build dates will help you to test, troubleshoot and even debug software in hopeless cases where you don’t know where to start. Astrology will help you to choose a random direction! Finally the output of WinDbg lmv command has more sense to me 
- Dmitry Vostokov @ DumpAnalysis.org -
Memory dump analysis always starts when a user complains. In this case it was a hanging application from a document processing suit. The manual dump was saved:
Loading Dump File [processA.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: 'Userdump generated complete user-mode minidump with Standalone function on SERVER1'
Main thread stack trace shows a virtualized process (WOW64):
0:000> kL
Child-SP RetAddr Call Site
00000000`0016e7b8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0×3a
00000000`0016e860 00000000`6b0097f4 wow64!RunCpuSimulation+0xa
00000000`0016e890 00000000`6b2936a2 wow64!Wow64KiUserCallbackDispatcher+0×114
00000000`0016ebd0 00000000`77ef317f wow64win!whcbfnDWORD+0xc2
00000000`0016ed80 00000000`78b842d9 ntdll!KiUserCallbackDispatcherContinue
00000000`0016ee08 00000000`78b8428e wow64cpu!CpupSyscallStub+0×9
00000000`0016ee10 00000000`00000000 wow64cpu!Thunk0Arg+0×5
Therefore we switch to x86 32-bit mode and get the right thread stack:
0:000> .load wow64exts
0:000> .effmach x86
Effective machine: x86 compatible (x86)
0:000:x86> kv
ChildEBP RetAddr Args to Child
0012dcac 7d948836 009db2c0 00000000 0000004a user32!NtUserMessageCall+0x15
0012dcd0 30059282 000b0296 0000004a 00000000 user32!SendMessageW+0×82
WARNING: Stack unwind information not available. Following frames may be wrong.
0012fef8 3000268e 02110024 30000000 b90fcc31 ApplicationA+0×59282
0012ff30 3000260b 30000000 00000000 0022245d ApplicationA+0×268e
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationA+0×260b
0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28
We see that the main threads is blockedby sending a synchronous message via SendMessage Win32 API function call. The first argument to it is a window handle. In our case it is 000b0296. It is also known that ApplicationA launches another ApplicationB (coupled process) and its manual memory dump was saved too. It is also a virtualized process and its main GUI thread is blocked:
0:000:x86> kv 100
ChildEBP RetAddr Args to Child
0012ce80 7d4e286c 00000003 0012cecc 00000000 ntdll_7d600000!NtWaitForMultipleObjects+0x15
0012cf28 7d4e3e8e 00000003 0012cf6c 00000001 kernel32!WaitForMultipleObjectsEx+0x11a
0012cf44 0cc7c897 00000003 0012cf6c 00000001 kernel32!WaitForMultipleObjects+0×18
WARNING: Stack unwind information not available. Following frames may be wrong.
0012cf74 0cc7c990 ffffffff 0cc74b23 00000001 3rdPartyDLL+0xc897
[…]
0012d814 7d947568 3a0b28d7 000b0296 00000002 user32!InternalCallWinProc+0×28
0012d88c 7d947d93 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114
0012d8e8 7d947e46 009db2c0 00000000 00000002 user32!DispatchClientMessage+0xdf
0012d924 7d61ea0e 0012d93c 00000000 0012d9b8 user32!__fnDWORD+0×2b
0012d958 3a0baf6a 000b0296 02114600 0012d98c ntdll_7d600000!KiUserCallbackDispatcher+0×2e
[…]
0012db28 7d947568 3a0b28d7 000b0296 00000010 user32!InternalCallWinProc+0×28
0012dba0 7d94778d 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114
0012dc18 7d9477d0 0012dc88 00000000 0012dc4c user32!DispatchMessageWorker+0×37b
0012dc28 3a0b89ec 0012dc88 00000000 0219401c user32!DispatchMessageW+0xf
[…]
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationB+0×260b
0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28
We see that it is blocked waiting for synchronization objects after receiving a message to the same window handle 000b0296 sent from ApplicationA:
0:000:x86> dd 0012dc88 l1
00000000`0012dc88 000b0296
DispatchMessage has its first argument as a pointer to an MSG structure with the first field as a window handle (HWND).
Looking at arguments to WaitForMultipleObjects we see that it is waiting for all three objects to be signalled simultaneously:
0012cf44 0cc7c897 00000003 0012cf6c 00000001kernel32!WaitForMultipleObjects+0×18
0:000:x86> dd 0012cf6c l3
00000000`0012cf6c 00001490 0000149c 00001494
0:000:x86> !handle 00001490
Handle 0000000000001490
Type Mutant
0:000:x86> !handle 0000149c
Handle 000000000000149c
Type Event
0:000:x86> !handle 00001494
Handle 0000000000001494
Type Mutant
Because the waiting call was originated from 3rdPartyDLL module it we can recommend to contact its vendor after determining the origin from the output of lmv command.
- Dmitry Vostokov @ DumpAnalysis.org -
Following the release of WinDbg: A Reference Poster and Learning Cards the following book is planned for Windows (GDB), Linux and FreeBSD users:
- Dmitry Vostokov @ DumpAnalysis.org -
Due to some technical difficulties the release of WinDbg: A Reference Poster and Learning Cards has been delayed by 2 weeks. Now I got a proof copy and approved the book distribution on Amazon, B&N and other bookshops worldwide. Hope you will enjoy it and find it useful.
The similar book for GDB will be announced soon.
- Dmitry Vostokov @ DumpAnalysis.org -
I’ve been thinking through the so called First Faults after Dan Skwire, a veteran in mission-critical computer system problem resolution, problem prevention, and system recovery, organized a group on LinkedIn for first fault problem solving activity. He also has a website:
http://www.firstfaultproblemresolution.com/
From my software technical support experience first fault problem resolution is very important on Windows platforms, especially in enterprise terminal service and virtualized environments where hundreds of users can be hosted on just one server. Therefore, proper tools, processes and checklists need to be set up and established for effective and efficient troubleshooting and problem resolution from both engineering and customer relationship managing perspectives. Here crash and hang dump analysis helps immensely, especially memory analysis patterns and fault databases. More on this later with specific examples. I’m also working currently on incorporating first fault problem resolution into VERSION troubleshooting steps and PARTS troubleshooting methodology.
- Dmitry Vostokov @ DumpAnalysis.org -
A pointer “tends to corrupt, and” a direct pointer “corrupts” directly.
John Emerich Edward Dalberg-Acton, 1st Baron Acton, Lord Acton’s dictum
- Dmitry Vostokov @ DumpAnalysis.org -