Archive for January, 2007

Venerable NOP instruction and more

Monday, January 29th, 2007

Does NOP instruction have “parameters”? Yes, it does (depends on CPU model). As I promised earlier I continue to update Asmpedia. Today I added ModRegRM table useful for both disassembling and assembling and NOP description together with WinDbg output:

ModRegRM byte (32/64-bit addressing) 

Note: REX prefix information will be added later

NOP instruction

I’ll keep you up with Asmpedia updates on weekly bases.

- Dmitry Vostokov -

Note: 32-bit stack from 64-bit dump

Friday, January 26th, 2007

Just a short note that might be useful if you get a 64-bit dump of a 32-bit process from x64 Windows. As a reminder I wrote already about separate dump types and tools for 32-bit and 64-bit processes.

Since then I’ve been looking for a simple solution in case we get a 64-bit dump so we could look at 32-bit thread stacks, etc. Yesterday I accidentally found wow64exts.dll extension in 64-bit WinDbg. I should have been looked at \winxp folder before :-) 

The whole discussion and solution can be found on Crash Dump Analysis forum:

Now if by accident a customer sends a 64-bit dump of a 32-bit process (some Citrix services like IMA service are still 32-bit in x64 CPS4) then we can look at threads without asking for a new dump and therefore avoid roundtrips and shorten time to problem resolution.

- Dmitry Vostokov -

Crash Dump Analysis Patterns (Part 7)

Wednesday, January 24th, 2007

We have to live with tools that produce inconsistent dumps. For example, LiveKd.exe from which is widely used by Microsoft and Citrix technical support to save complete memory dumps without server reboot. I even wrote an article for Citrix customers:

Using LiveKD to Save a Complete Memory Dump for Session or System Hangs

If you read it you will find an important note which is reproduced here:

LiveKd.exe-generated dumps are always inconsistent and cannot be a reliable source for certain types of dump analysis, for example, looking at resource contention. This is because it takes a considerable amount of time to save a dump on a live system and the system is being changed during that process. The instantaneous traditional CrashOnCtrlScroll method or SystemDump tool always save a reliable and consistent dump because the system is frozen first (any process or kernel activity is disabled), then a dump is saved to a page file.

If you look at such inconsistent dump you will find that many useful kernel structures such as ERESOURCE list (!locks) are broken and even circular referenced and therefore WinDbg commands display “strange” output.

Easy and painless (for customers) dump generation using such “Live” tools means that it is widely used and we have to analyze dumps saved by these tools and sent from customers. This brings us to the next crash dump analysis pattern called “Inconsistent Dump”.

If you have such dump you should look at it in order to extract maximum useful information that helps in identifying the root cause or give you further directions. Not all information is inconsistent in such dumps. For example, drivers, processes, thread stacks and IRP lists can give you some clues about activities. Even some information not visible in consistent dump can surface in inconsistent dump (subject to commands used).

For example, I had a LiveKd dump where I looked at process stacks by running the script I created earlier:

Yet another WinDbg script

and I found that for some processes in addition to their own threads the script lists additional terminated threads that belong to a completely different process (have never seen it in consistent dump):

Process 89d97d88 is not visible in the active process list (script mentioned above or !process 0 0 command). However, if we feed this memory address to !process command (or explore it as _EPROCESS structure, dt command) we get its contents:


What might have happened there: terminated process 89d97d88 was excluded from active processes list but its structure was left in memory and due to inconsistency thread lists were also broken and therefore terminated threads surfaced when listing other processes and their threads. 

I suspected here that winlogon.exe died in session 2 and left empty desktop window which a customer saw and complained about. The only left and visible process from session 2 was csrss.exe. The conclusion was to enable NTSD as a default postmortem debugger to catch winlogon.exe crash when it happens next time.

- Dmitry Vostokov @ -

SystemDump 3.1

Tuesday, January 23rd, 2007

New version of SystemDump for 32-bit and 64-bit platforms has been released. What’s new in this version:

  • Fixed the bug that prevented SystemDump to bugcheck second time after the first bugcheck and reboot happened (the known workaround was to run it twice second time)
  • Easter egg (hold key and click on About button)

The tool can be downloaded from Citrix support web site.

- Dmitry Vostokov -

MessageHistory 2.0

Wednesday, January 17th, 2007

MessageHistory for 32-bit and 64-bit platforms has been extended and improved to make it better for troubleshooting and debugging GUI. What’s new in this version:

  • Added more filtering to reduce log size for default options
  • Shows names for messages sent to the following controls:
    • edit
    • static
    • button
    • listbox
    • combobox
    • scrollbar  
  • Added Spy++-style log for bulk messages sorted by time
  • Easter egg (hold <Shift> key and click on About button) 

It can be downloaded from Citrix support web site.

The picture from my recent presentation shows schematically the difference between sent and posted messages:  

and the following diagram depicts relationship between processes, threads and windows:


- Dmitry Vostokov -


Saturday, January 6th, 2007

As a part of my Master’s thesis I founded Wintel assembly language encyclopedia:

It is based on MediaWiki and I will start populating it from the end of January onwards. Information will be presented from dump analysis and reverse engineering perspective.

Currently I created some entries to test and collect comments, for example:

MOV instruction (x64 opcodes will be added later)

Instruction description will include:

  • definition and examples
  • x86 and x64 differences
  • C-style pseudo-code
  • annotated WinDbg disassembly
  • C/C++ compiler translation examples

Opcodes and mnemonics are cross-referenced, for example:


I use Intel and AMD manuals and disassembly output from WinDbg as reference.

Finally I can fulfill my desire to learn all x86 instructions :-)

Further plans are to start with ARM assembly language as soon as I finish most of Wintel part because I do development for Windows Mobile and I’m interested in low level stuff there.

- Dmitry Vostokov -

WindowHistory Mobile update (version 2.2)

Thursday, January 4th, 2007

Code changes and bug fixes from the latest WindowHistory 3.0 have been integrated. Also users reported that mobile version doesn’t track parent window handle and this has been fixed too.

- Dmitry Vostokov -

Tracing Win32 API while debugging a process

Wednesday, January 3rd, 2007

Load an executable or attach WinDbg to an existing process and use logexts debugging extension (in output below all API parameters and return values are omitted for visual clarity):

0:001> !logexts.loge
0:001> !logc e *
All categories enabled.
0:001> !logo e d
  Debugger            Enabled
  Text file           Disabled
  Verbose log         Enabled
0:001> g
Thrd 7c0 77555B59 BeginPaint( 0x001103AA) ...
Thrd 7c0 77555B65 GetClientRect( 0x001103AA) ...
Thrd 7c0 77555B96 DrawEdge( 0x01010072 ...) ...
Thrd 7c0 77555C8A DrawFrameControl( 0x01010072 ...) ...
Thrd 7c0 77555CE1 EndPaint( 0x001103AA ... ) ...
Thrd 7c0 004165F2 TlsGetValue( 0x00000006) ...
Thrd 7c0 4B8D54B5 CallNextHookEx( ... ) ...
Thrd 7c0 0040D7CC GetMessageW( ... ) ...

You can break in and put a breakpoint at a return address:

0:001> bp 0040D7CC
0:001> g
Thrd 7c0 0040D7CC GetMessageW( ... ) ...
Breakpoint 0 hit
0040d7cc 85c0            test    eax,eax
0:000> u 0040D7C0 0040D7CC
0040d7c0 50              push    eax
0040d7c1 50              push    eax
0040d7c2 8d7730          lea     esi,[edi+30h]
0040d7c5 56              push    esi
0040d7c6 ff15f8434300    call    dword ptr
[ProcessHistory+0x343f8 (004343f8)]
0:000> dd 004343f8
004343f8  3c001950 3c0018c4 3c00193c 3c0014dc
0:000> u 3c001950
3c001950 b889020000      mov     eax,289h
3c001955 e98e410014      jmp     logexts!LogHook
3c00195a b88a020000      mov     eax,28Ah
3c00195f e984410014      jmp     logexts!LogHook
3c001964 b88b020000      mov     eax,28Bh
3c001969 e97a410014      jmp     logexts!LogHook
3c00196e b88c020000      mov     eax,28Ch
3c001973 e970410014      jmp     logexts!LogHook

Here we can see that logexts patches import table.

And you can trace different API categories:

0:001> !logexts.logc
  1 AdvApi32                        Enabled
  2 AtomFunctions                   Enabled
  3 AVIFileExports                  Enabled
  4 Clipboard                       Enabled
  5 ComponentObjectModel            Enabled
  6 DebuggingAndErrorHandling       Enabled
  7 DeviceFunctions                 Enabled
  8 Direct3D                        Enabled
  9 DirectDraw                      Enabled
 10 DirectPlay                      Enabled
 11 DirectSound                     Enabled
 12 GDI                             Enabled
 13 HandleAndObjectFunctions        Enabled
 14 HookingFunctions                Enabled
 15 IOFunctions                     Enabled
 16 MemoryManagementFunctions       Enabled
 17 Multimedia                      Enabled
 18 Printing                        Enabled
 19 ProcessesAndThreads             Enabled
 20 RegistryFunctions               Enabled
 21 Shell                           Enabled
 22 StringManipulation              Enabled
 23 ThreadLocalStorage              Enabled
 24 User32                          Enabled
 25 User32StringExports             Enabled
 26 Version                         Enabled
 27 WinSock2                        Enabled

- Dmitry Vostokov -

WindowHistory 3.0

Monday, January 1st, 2007

WindowHistory tool has been significantly rewritten and improved to make it better for troubleshooting and debugging GUI. What’s new in this version:

  • Real-time support: windows are tracked as they are created and destroyed, their position and size are changed, etc.
  • Dramatically improved speed, no matter how many windows you have in your session WindowHistory is fast and has minimum impact on the system (O(log(n)))
  • Better formatted output
  • Fixed bugs found in previous version
  • Easter egg (hold <Shift> key and click on About button)


It is a native Windows application written in C++/STL/MFC/Win32.

There are two packages: WindowHistory32 and WindowHistory64. Both can be downloaded from Citrix support web site:

To use download, unpack and run WindowHistory(64).exe.

To uninstall just remove files.

Note: although 32-bit version will run on x64 Windows too, real-time support for 64-bit application windows will not be available. For x64 Windows please use WindowHistory64 which correctly handles both 64-bit and 32-bit application windows.

The following UML collaboration diagram depicts schematically how WindowHistory64 gets notifications from 32-bit windows:

If you want to track window messages and processes simultaneously run it with MessageHistory and ProcessHistory tools.

 - Dmitry Vostokov -