Archive for November, 2008

LiterateScientist update (November, 2008)

Sunday, November 30th, 2008

Monthly summary of my Literate Scientist blog:

The Third Reich: A New History

Bird Flu: A Virus of Our Own Hatching

Discrete Mathematics 

This month was very busy for me as a publisher, an editor and a writer and I didn’t have much time to read. Hope in December I write more book reviews. 

- Dmitry Vostokov @ -

Build Date: November 30

Sunday, November 30th, 2008

Warning!: This post belongs to Build Date Astrology category. Do not take it seriously.

The Day of Measured Testing

Modules built on November 30 have a built-in capacity for  overcoming challenges of hostile environments. They are capable of bringing surprises to security attacks, for example. One can learn a lot about them by studying their traces or doing reverse engineering. November 30 components do their work to the utmost degree of quality with a little waste of CPU and memory. Message boxes they pop up have a subtle sense of thought-provoking humour but it can also be a full blown thigh-slapping. November 30 systems are very defensive when attacked. They are stubbornly resistant to reverse engineering but at the same time very open to honest debugging. 

DLL, SYS and EXE born on this date: 

tifsfilt.sys Tue Nov 30 07:16:27 2004
alrsvc.dll   Tue Nov 30 17:31:14 1999
ntkrpamp.exe Fri Nov 30 14:54:49 2007
Tppwrif.sys  Tue Nov 30 02:38:22 2004

Weaknesses: Over-reactive to code and data injection, funny behaviour.

Strengths: Thorough developed, dynamic responsiveness.

Advice: Improvise during troubleshooting and debugging. Admire control vs. spontaneity balance. Laugh at your failures.

- Dmitry Vostokov @ -

Introducing Build Date Astrology

Sunday, November 30th, 2008

I often hear about cosmic mysteries or influences when problems happen in computer environments. Passing by an astrology section in a local book shop yesterday a revelation came to me that a compile / link time (build time) might influence a component (DLL, EXE, SYS files), product or system behaviour. From now on I’m going to blog about every build date with examples. And as usual, I’m also going to publish a book for this iterative and incremental activity called:

Title: The Secret Language of Build Dates: Unique Astrology Profiles for Every Build of the Year with Advice on Testing, Troubleshooting and Debugging
ISBN: 978-1906717407

Knowing build dates will help you to test, troubleshoot and even debug software in hopeless cases where you don’t know where to start. Astrology will help you to choose a random direction! Finally the output of WinDbg lmv command has more sense to me :-)

- Dmitry Vostokov @ -

Implausible Debugging Book Titles (Part 1)

Friday, November 28th, 2008

I found the book How to Avoid Huge Ships: And Other Implausibly Titled Books in a local bookshop yesterday and couldn’t stop laughing. So I took some implausible titles and bugtated them into implausible debugging book titles:

  • - Old Bugs and the Men Who Debug Them
  • - How to Avoid Crashes and Hangs or I’ve Never Met a Bug I Liked
  • - Blue Screen: What’s in it for You?
  • - What to Say When You Debug: Powerful New Techniques to Program your Success!
  • - Redmond: The View from Greenland
  • - Fabulous Small Bugs
  • - Better Never to Have Coded: The Harm of Coding
  • - Code for Impact
  • - Whose Bug? The Clash between Software Vendors

- Dmitry Vostokov @ -

WOW64, blocked threads and coupled processes: pattern cooperation

Friday, November 28th, 2008

Memory dump analysis always starts when a user complains. In this case it was a hanging application from a document processing suit. The manual dump was saved:

Loading Dump File [processA.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: 'Userdump generated complete user-mode minidump with Standalone function on SERVER1'

Main thread stack trace shows a virtualized process (WOW64):

0:000> kL
Child-SP          RetAddr           Call Site
00000000`0016e7b8 00000000`6b006a5a wow64cpu!WaitForMultipleObjects32+0×3a
00000000`0016e860 00000000`6b0097f4 wow64!RunCpuSimulation+0xa
00000000`0016e890 00000000`6b2936a2 wow64!Wow64KiUserCallbackDispatcher+0×114
00000000`0016ebd0 00000000`77ef317f wow64win!whcbfnDWORD+0xc2
00000000`0016ed80 00000000`78b842d9 ntdll!KiUserCallbackDispatcherContinue
00000000`0016ee08 00000000`78b8428e wow64cpu!CpupSyscallStub+0×9
00000000`0016ee10 00000000`00000000 wow64cpu!Thunk0Arg+0×5

Therefore we switch to x86 32-bit mode and get the right thread stack:

0:000> .load wow64exts

0:000> .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> kv
ChildEBP          RetAddr           Args to Child                                        
0012dcac 7d948836 009db2c0 00000000 0000004a user32!NtUserMessageCall+0x15
0012dcd0 30059282 000b0296 0000004a 00000000 user32!SendMessageW+0×82
WARNING: Stack unwind information not available. Following frames may be wrong.
0012fef8 3000268e 02110024 30000000 b90fcc31 ApplicationA+0×59282
0012ff30 3000260b 30000000 00000000 0022245d ApplicationA+0×268e
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationA+0×260b
0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28

We see that the main threads is blocked by sending a synchronous message via SendMessage Win32 API function call. The first argument to it is a window handle. In our case it is 000b0296. It is also known that ApplicationA launches another ApplicationB (coupled process) and its manual memory dump was saved too. It is also a virtualized process and its main GUI thread is blocked:

0:000:x86> kv 100
ChildEBP          RetAddr           Args to Child                                        
0012ce80 7d4e286c 00000003 0012cecc 00000000 ntdll_7d600000!NtWaitForMultipleObjects+0x15
0012cf28 7d4e3e8e 00000003 0012cf6c 00000001 kernel32!WaitForMultipleObjectsEx+0x11a
0012cf44 0cc7c897 00000003 0012cf6c 00000001 kernel32!WaitForMultipleObjects+0×18
WARNING: Stack unwind information not available. Following frames may be wrong.
0012cf74 0cc7c990 ffffffff 0cc74b23 00000001 3rdPartyDLL+0xc897
0012d814 7d947568 3a0b28d7 000b0296 00000002 user32!InternalCallWinProc+0×28
0012d88c 7d947d93 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114
0012d8e8 7d947e46 009db2c0 00000000 00000002 user32!DispatchClientMessage+0xdf
0012d924 7d61ea0e 0012d93c 00000000 0012d9b8 user32!__fnDWORD+0×2b
0012d958 3a0baf6a 000b0296 02114600 0012d98c ntdll_7d600000!KiUserCallbackDispatcher+0×2e
0012db28 7d947568 3a0b28d7 000b0296 00000010 user32!InternalCallWinProc+0×28
0012dba0 7d94778d 00000000 3a0b28d7 000b0296 user32!UserCallWinProcCheckWow+0×114
0012dc18 7d9477d0 0012dc88 00000000 0012dc4c user32!DispatchMessageWorker+0×37b
0012dc28 3a0b89ec 0012dc88 00000000 0219401c user32!DispatchMessageW+0xf
0012ffc0 7d4e7d2a 00000000 00000000 7efde000 ApplicationB+0×260b
0012fff0 00000000 30001d28 00000000 00000000 kernel32!BaseProcessStart+0×28

We see that it is blocked waiting for synchronization objects after receiving a message to the same window handle 000b0296 sent from ApplicationA:

0:000:x86> dd 0012dc88 l1
00000000`0012dc88 000b0296

DispatchMessage has its first argument as a pointer to an MSG structure with the first field as a window handle (HWND). 

Looking at arguments to WaitForMultipleObjects we see that it is waiting for all three objects to be signalled simultaneously:

0012cf44 0cc7c897 00000003 0012cf6c 00000001kernel32!WaitForMultipleObjects+0×18

0:000:x86> dd 0012cf6c l3
00000000`0012cf6c  00001490 0000149c 00001494

0:000:x86> !handle 00001490
Handle 0000000000001490
  Type          Mutant

0:000:x86> !handle 0000149c
Handle 000000000000149c
  Type          Event

0:000:x86> !handle 00001494
Handle 0000000000001494
  Type          Mutant

Because the waiting call was originated from 3rdPartyDLL module we can recommend to contact its vendor after determining the origin from the output of lmv command.

- Dmitry Vostokov @ -

GDB and KDB Debuggers book

Friday, November 28th, 2008

Following the release of WinDbg: A Reference Poster and Learning Cards the following book is planned for Windows (GDB), Linux and FreeBSD users:

  • Title: GDB and KDB Debuggers:
    A Reference Poster and Learning Cards
  • Author: Gonçalo Gomes
  • Publisher: Opentask (1 April 2009)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • ISBN-13: 978-1-906717-39-1
  • Paperback: 16 pages

- Dmitry Vostokov @ -

WinDbg poster and cards book is out!

Friday, November 28th, 2008

Due to some technical difficulties the release of WinDbg: A Reference Poster and Learning Cards has been delayed by 2 weeks. Now I got a proof copy and approved the book distribution on Amazon, B&N and other bookshops worldwide. Hope you will enjoy it and find it useful.

The similar book for GDB will be announced soon.

- Dmitry Vostokov @ -

The Importance of First Fault

Thursday, November 27th, 2008

I’ve been thinking through the so called First Faults after Dan Skwire, a veteran in mission-critical computer system  problem resolution, problem prevention, and system recovery, organized a group on LinkedIn for first fault problem solving activity. He also has a website: 

From my software technical support experience first fault problem resolution is very important on Windows platforms, especially in enterprise terminal service and virtualized environments where hundreds of users can be hosted on just one server. Therefore, proper tools, processes and checklists need to be set up and established for effective and efficient troubleshooting and problem resolution from both engineering and customer relationship managing perspectives. Here crash and hang dump analysis helps immensely, especially memory analysis patterns and fault databases. More on this later with specific examples. I’m also working currently on incorporating first fault problem resolution into VERSION troubleshooting steps and PARTS troubleshooting methodology.

- Dmitry Vostokov @ -

Bugtation No.70

Thursday, November 27th, 2008

A pointer “tends to corrupt, and” a direct pointer “corrupts” directly.

John Emerich Edward Dalberg-Acton, 1st Baron Acton, Lord Acton’s dictum

- Dmitry Vostokov @ -

Writing for Debugged! Magazine

Wednesday, November 26th, 2008

If you have an article idea or if you’d like to write an article for Debugged! MZ/PE please use the following contact form:

Author’s guidelines will be published soon.

- Dmitry Vostokov @

Debugged! Magazine

Tuesday, November 25th, 2008

As one of the new initiatives for the Year of Debugging  DumpAnalysis Portal will publish bimonthly full color 16 page publication called:

Debugged! MZ/PE: MagaZine for/from Practicing Engineers
The only serial publication dedicated entirely to Windows® debugging

The first issue is planned for March, 2009 and will have ISBN-13: 978-1-906717-38-4. If it goes well I’m planning to have ISSN number assigned to it too. More details will be announced soon.

- Dmitry Vostokov @

TOC from Dumps, Bugs and Debugging Forensics Book

Tuesday, November 25th, 2008

I’m pleased to announce that OpenTask has submitted the book Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov for printing and here is the link to TOC:

Table of Contents

- Dmitry Vostokov @

Breaking the Bug: Debugging as a Natural Phenomenon

Monday, November 24th, 2008

I was thinking about the universal character of debugging for quite some time and finally the following bugtation provided an inspiration for a new book title to be published during the Year of Debugging:

Title: Breaking the Bug: Debugging as a Natural Phenomenon
ISBN-13: 978-1906717377

More product details will be announced later.

Actually I believe in the mystical nature of various debugging numbers and sequences. For example, the ISBN number of this book ends in 377 which is the octal base equivalent of 0n255 or 0xFF.

- Dmitry Vostokov @

Bugtation No.69

Monday, November 24th, 2008

“Breaking the” Bug: Debugging “as a Natural Phenomenon”

Daniel Clement Dennett, Breaking the Spell: Religion as a Natural Phenomenon

- Dmitry Vostokov @ -

Bugtation No.68

Monday, November 24th, 2008

“Diamonds are forever but bugs are an error.”

Narasimha Vedala, Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov

Santa bug from Narasimha Vedala

- Dmitry Vostokov @ -

DLL Art Book

Monday, November 24th, 2008

Here are product details and covers for previously announced DLL List Landscape book:

  • Title: DLL List Landscape: The Art from Computer Memory Space
  • Author: Dmitry Vostokov
  • Publisher: Opentask (15 December 2008)
  • Language: English
  • Product Dimensions: 21.6 x 21.6
  • ISBN-13: 978-1-906717-36-0
  • Paperback: 16 pages

Front cover:

Back cover:

- Dmitry Vostokov @ -

WinDbg Release

Monday, November 24th, 2008

Thanks to shellexecute I got the news of this release. Remember, you can always access quick download links from

 Dmitry Vostokov @ -

DLL List Landscape

Sunday, November 23rd, 2008

DLL is also a recursive acronym for DLL List Landscape. OpenTask is going to publish soon the new full color book:

Title: DLL List Landscape: The Art from Computer Memory Space
ISBN-13: 978-1-906717-36-0

More details will be announced tomorrow.  

- Dmitry Vostokov @ -

Heap Corruption

Friday, November 21st, 2008

Below is the list of patterns related to process heap corruption:

and two case studies involving heap corruption:

- Dmitry Vostokov @ -

Stack trace collection, hidden exception and NULL code pointer: pattern cooperation

Friday, November 21st, 2008

This is a common pattern cooperation in user dumps coming from x64 environments. Its characteristic feature is stack trace collection that appears to be damaged or corrupt with lots of zeroed call sites and sometimes having UNICODE values in saved RSP:

0:001> ~*kL

   0  Id: 668.684 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`000afcb8 000007ff`7f1ee614 USER32!NtUserWaitMessage+0xa
00000000`000afcc0 000007ff`7f1adf7a SHELL32!CDesktopBrowser::_MessageLoop+0x287
00000000`000afd50 00000001`0001a46c SHELL32!SHDesktopMessageLoop+0x5c
00000000`000afd80 00000001`000231ba Explorer!ExplorerWinMain+0x6f1
00000000`000afec0 00000000`77d5964c Explorer!ModuleEntry+0x1c6
00000000`000aff80 00000000`00000000 kernel32!BaseProcessStart+0x29

   1  Id: 668.20c Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

   2  Id: 668.298 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

   3  Id: 668.f34 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

   4  Id: 668.824 Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

   5  Id: 668.820 Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

   6  Id: 668.914 Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`0342fb98 000007ff`7fd6ff61 ntdll!ZwReplyWaitReceivePortEx+0xa
00000000`0342fba0 000007ff`7fd45369 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x2a5
00000000`0342feb0 000007ff`7fd65996 RPCRT4!RecvLotsaCallsWrapper+0x9
00000000`0342fee0 000007ff`7fd65d51 RPCRT4!BaseCachedThreadRoutine+0xde
00000000`0342ff50 00000000`77d6b6da RPCRT4!ThreadStartRoutine+0x21
00000000`0342ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a

   7  Id: 668.8cc Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0x7

   8  Id: 668.1078 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

#  9  Id: 668.1118 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

  10  Id: 668.574 Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000020 00000000`00000000 0×0

  11  Id: 668.a54 Suspend: 1 Teb: 000007ff`fffa0000 Unfrozen
Child-SP          RetAddr           Call Site
90000000`0000679f 00000000`00000000 0x72505c74`6e656d65

  12  Id: 668.163c Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
Child-SP          RetAddr           Call Site
0000001e`00057000 00000000`00000000 0x575c3a43`00000075

  13  Id: 668.5b0 Suspend: 1 Teb: 000007ff`fff9e000 Unfrozen
Child-SP          RetAddr           Call Site
49575c3a`43000000 00000000`00000000 0x4e49575c`3a430000

  14  Id: 668.4c0 Suspend: 1 Teb: 000007ff`fff9c000 Unfrozen
Child-SP          RetAddr           Call Site
3a430000`00200004 00000000`00000000 0x43000000`1f0001c0

  15  Id: 668.774 Suspend: 1 Teb: 000007ff`fff9a000 Unfrozen
Child-SP          RetAddr           Call Site
00410044`00500050 00000000`00000000 0×6e006f`00690074

  16  Id: 668.17c0 Suspend: 1 Teb: 000007ff`fff98000 Unfrozen
Child-SP          RetAddr           Call Site
005c0029`00360038 00000000`00000000 0×500048`005c0029

However we notice ‘#’ in front of thread 9:

#  9  Id: 668.1118 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0×0

This marks the thread that caught the exception. We can either switch to it by ~9s or using ~#s command:

0:010> ~#s
00000000`00000000 ??              ???

Now we look at thread raw stack data to search for any hidden exceptions and we find one indeed:

0:009> !teb
TEB at 000007fffffa4000
    ExceptionList:        0000000000000000
    StackBase:            0000000003000000
    StackLimit:           0000000002ff2000

    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000007fffffa4000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000668 . 0000000000001118
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000007fffffdb000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:010> dqs 0000000003232000  0000000003240000
00000000`03232000  00000000`00000000
00000000`03232008  00000000`00000000
00000000`03232010  00000000`00000000
00000000`02ffc8c8  00000000`77ef3202 ntdll!KiUserExceptionDispatcher+0×52
00000000`02ffc8d0  fffffa80`07186100
00000000`02ffc8d8  00000000`02ffc8d0
00000000`02ffc8e0  00000000`00000000

0:009> .cxr 00000000`02ffc8d0
rax=0000000000000000 rbx=0000000000000000 rcx=00000000671b4610
rdx=ffffffff9be48728 rsi=0000000000000000 rdi=0000000000000000
rip=0000000000000000 rsp=0000000002ffce68 rbp=00000000671b4610
 r8=0000000002ffccc0  r9=0000000000000000 r10=000068aa62010001
r11=00000000671b4610 r12=0000000000000000 r13=00000000000006a5
r14=00000000671b45d0 r15=7ffffffffffffffd
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00000000`00000000 ??              ???

0:009> kL
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
00000000`02ffce68 00000000`67199841 0×0
00000000`02ffce70 00000000`67193188 DllA!DllUnregisterServer+0×8401

00000000`02ffd230 00000000`67194f93 DllA!DllUnregisterServer+0×1d48
00000000`02ffd350 000007ff`7ca9cae4 DllA!DllUnregisterServer+0×3b53
00000000`02ffd3f0 000007ff`7ca9d867 NETSHELL!GetPrimaryIPv6AddressForAdapter+0×64
00000000`02ffd720 000007ff`7ca1eb4f NETSHELL!CConnectionFolder::GetDetailsOf+0×62a
00000000`02ffdd20 000007ff`7f27f57f NETSHELL!CConnectionFolder::GetDetailsEx+0×21d
00000000`02ffe870 000007ff`7f27eea6 SHELL32!CNameSpaceItemUIProperty::GetPropertyDisplayValue+0xaf
00000000`02ffe910 000007ff`7f1e7bd4 SHELL32!CDetailsSectionInfoTask::RunInitRT+0×213
00000000`02fffc60 000007ff`7cec9c26 SHELL32!CRunnableTask::Run+0×52
00000000`02fffc90 000007ff`7ef773d8 BROWSEUI!CShellTaskScheduler_ThreadProc+0×1be
00000000`02fffd60 00000000`77eea78a SHLWAPI!ExecuteWorkItem+0×28
00000000`02fffd90 00000000`77eea99f ntdll!RtlpWorkerCallout+0×183
00000000`02fffec0 00000000`77eeac75 ntdll!RtlpExecuteWorkerRequest+0×63
00000000`02ffff00 00000000`77d6b6da ntdll!RtlpWorkerThread+0×71
00000000`02ffff80 00000000`00000000 kernel32!BaseThreadStart+0×3a

Checking disassembly we see that DllA module code dereferenced a NULL code pointer:

0:009> ub DllA!DllUnregisterServer+0×8401
00000000`67199824 0100            add     dword ptr [rax],eax
00000000`67199826 00488b          add     byte ptr [rax-75h],cl
00000000`67199829 cdff            int     0FFh
00000000`6719982b 1568ad0100      adc     eax,1AD68h
00000000`67199830 85c0            test    eax,eax
00000000`67199832 0f85d8000000    jne     DllA!DllUnregisterServer+0×84d0 (00000000`67199910)
00000000`67199838 488bcd          mov     rcx,rbp
00000000`6719983b ff1547ad0100    call    qword ptr [DllA!DllUnregisterServer+0×23148 (00000000`671b4588)]

0:009> dq 00000000`671b4588
00000000`671b4588  00000000`00000000 00000000`00000000
00000000`671b4598  000007ff`77317e40 00000000`00000000
00000000`671b45a8  00000000`00000000 00000000`00000000
00000000`671b45b8  00000000`00000000 00000000`00000000
00000000`671b45c8  00000000`00000000 00000000`00000000
00000000`671b45d8  000007ff`77310000 00000000`01b81240
00000000`671b45e8  00000000`00000001 00000000`020c09c0
00000000`671b45f8  00000000`00000001 00000001`00000001

- Dmitry Vostokov @ -