Does NOP instruction have “parameters”? Yes, it does (depends on CPU model). As I promised earlier I continue to update Asmpedia. Today I added ModRegRM table useful for both disassembling and assembling and NOP description together with WinDbg output:
ModRegRM byte (32/64-bit addressing)
Note: REX prefix information will be added later
I’ll keep you up with Asmpedia updates on weekly bases.
- Dmitry Vostokov -
Just a short note that might be useful if you get a 64-bit dump of a 32-bit process from x64 Windows. As a reminder I wrote already about separate dump types and tools for 32-bit and 64-bit processes.
Since then I’ve been looking for a simple solution in case we get a 64-bit dump so we could look at 32-bit thread stacks, etc. Yesterday I accidentally found wow64exts.dll extension in 64-bit WinDbg. I should have been looked at \winxp folder before :-)
The whole discussion and solution can be found on Crash Dump Analysis forum:
http://www.dumpanalysis.org/forum/viewtopic.php?t=39
Now if by accident a customer sends a 64-bit dump of a 32-bit process (some Citrix services like IMA service are still 32-bit in x64 CPS4) then we can look at threads without asking for a new dump and therefore avoid roundtrips and shorten time to problem resolution.
- Dmitry Vostokov -
We have to live with tools that produce inconsistent dumps. For example, LiveKd.exe from sysinternals.com which is widely used by Microsoft and Citrix technical support to save complete memory dumps without server reboot. I even wrote an article for Citrix customers:
Using LiveKD to Save a Complete Memory Dump for Session or System Hangs
If you read it you will find an important note which is reproduced here:
LiveKd.exe-generated dumps are always inconsistent and cannot be a reliable source for certain types of dump analysis, for example, looking at resource contention. This is because it takes a considerable amount of time to save a dump on a live system and the system is being changed during that process. The instantaneous traditional CrashOnCtrlScroll method or SystemDump tool always save a reliable and consistent dump because the system is frozen first (any process or kernel activity is disabled), then a dump is saved to a page file.
If you look at such inconsistent dump you will find that many useful kernel structures such as ERESOURCE list (!locks) are broken and even circular referenced and therefore WinDbg commands display “strange” output.
Easy and painless (for customers) dump generation using such “Live” tools means that it is widely used and we have to analyze dumps saved by these tools and sent from customers. This brings us to the next crash dump analysis pattern called “Inconsistent Dump”.
If you have such dump you should look at it in order to extract maximum useful information that helps in identifying the root cause or give you further directions. Not all information is inconsistent in such dumps. For example, drivers, processes, thread stacks and IRP lists can give you some clues about activities. Even some information not visible in consistent dump can surface in inconsistent dump (subject to commands used).
For example, I had a LiveKd dump where I looked at process stacks by running the script I created earlier:
and I found that for some processes in addition to their own threads the script lists additional terminated threads that belong to a completely different process (have never seen it in consistent dump):
Process 89d97d88 is not visible in the active process list (script mentioned above or !process 0 0 command). However, if we feed this memory address to !process command (or explore it as _EPROCESS structure, dt command) we get its contents:
What might have happened there: terminated process 89d97d88 was excluded from active processes list but its structure was left in memory and due to inconsistency thread lists were also broken and therefore terminated threads surfaced when listing other processes and their threads.
I suspected here that winlogon.exe died in session 2 and left empty desktop window which a customer saw and complained about. The only left and visible process from session 2 was csrss.exe. The conclusion was to enable NTSD as a default postmortem debugger to catch winlogon.exe crash when it happens next time.
- Dmitry Vostokov @ DumpAnalysis.org -
New version of SystemDump for 32-bit and 64-bit platforms has been released. What’s new in this version:
The tool can be downloaded from Citrix support web site.
- Dmitry Vostokov -
MessageHistory for 32-bit and 64-bit platforms has been extended and improved to make it better for troubleshooting and debugging GUI. What’s new in this version:
It can be downloaded from Citrix support web site.
The picture from my recent presentation shows schematically the difference between sent and posted messages:
and the following diagram depicts relationship between processes, threads and windows:
- Dmitry Vostokov -
As a part of my Master’s thesis I founded Wintel assembly language encyclopedia: www.asmpedia.org.
It is based on MediaWiki and I will start populating it from the end of January onwards. Information will be presented from dump analysis and reverse engineering perspective.
Currently I created some entries to test and collect comments, for example:
MOV instruction (x64 opcodes will be added later)
Instruction description will include:
Opcodes and mnemonics are cross-referenced, for example:
I use Intel and AMD manuals and disassembly output from WinDbg as reference.
Finally I can fulfill my desire to learn all x86 instructions 
Further plans are to start with ARM assembly language as soon as I finish most of Wintel part because I do development for Windows Mobile and I’m interested in low level stuff there.
- Dmitry Vostokov -
Code changes and bug fixes from the latest WindowHistory 3.0 have been integrated. Also users reported that mobile version doesn’t track parent window handle and this has been fixed too.
- Dmitry Vostokov -
Load an executable or attach WinDbg to an existing process and use logexts debugging extension (in output below all API parameters and return values are omitted for visual clarity):
0:001> !logexts.loge
0:001> !logc e *
All categories enabled.
0:001> !logo e d
Debugger Enabled
Text file Disabled
Verbose log Enabled
0:001> g
Thrd 7c0 77555B59 BeginPaint( 0x001103AA) ...
Thrd 7c0 77555B65 GetClientRect( 0x001103AA) ...
Thrd 7c0 77555B96 DrawEdge( 0x01010072 ...) ...
Thrd 7c0 77555C8A DrawFrameControl( 0x01010072 ...) ...
Thrd 7c0 77555CE1 EndPaint( 0x001103AA ... ) ...
Thrd 7c0 004165F2 TlsGetValue( 0x00000006) ...
Thrd 7c0 4B8D54B5 CallNextHookEx( ... ) ...
Thrd 7c0 0040D7CC GetMessageW( ... ) ...
You can break in and put a breakpoint at a return address:
0:001> bp 0040D7CC
0:001> g
Thrd 7c0 0040D7CC GetMessageW( ... ) ...
Breakpoint 0 hit
ProcessHistory+0xd7cc:
0040d7cc 85c0 test eax,eax
0:000> u 0040D7C0 0040D7CC
ProcessHistory+0xd7c0:
0040d7c0 50 push eax
0040d7c1 50 push eax
0040d7c2 8d7730 lea esi,[edi+30h]
0040d7c5 56 push esi
0040d7c6 ff15f8434300 call dword ptr
[ProcessHistory+0x343f8 (004343f8)]
0:000> dd 004343f8
004343f8 3c001950 3c0018c4 3c00193c 3c0014dc
0:000> u 3c001950
3c001950 b889020000 mov eax,289h
3c001955 e98e410014 jmp logexts!LogHook
(50005ae8)
3c00195a b88a020000 mov eax,28Ah
3c00195f e984410014 jmp logexts!LogHook
(50005ae8)
3c001964 b88b020000 mov eax,28Bh
3c001969 e97a410014 jmp logexts!LogHook
(50005ae8)
3c00196e b88c020000 mov eax,28Ch
3c001973 e970410014 jmp logexts!LogHook
(50005ae8)
Here we can see that logexts patches import table.
And you can trace different API categories:
0:001> !logexts.logc
Categories:
1 AdvApi32 Enabled
2 AtomFunctions Enabled
3 AVIFileExports Enabled
4 Clipboard Enabled
5 ComponentObjectModel Enabled
6 DebuggingAndErrorHandling Enabled
7 DeviceFunctions Enabled
8 Direct3D Enabled
9 DirectDraw Enabled
10 DirectPlay Enabled
11 DirectSound Enabled
12 GDI Enabled
13 HandleAndObjectFunctions Enabled
14 HookingFunctions Enabled
15 IOFunctions Enabled
16 MemoryManagementFunctions Enabled
17 Multimedia Enabled
18 Printing Enabled
19 ProcessesAndThreads Enabled
20 RegistryFunctions Enabled
21 Shell Enabled
22 StringManipulation Enabled
23 ThreadLocalStorage Enabled
24 User32 Enabled
25 User32StringExports Enabled
26 Version Enabled
27 WinSock2 Enabled
- Dmitry Vostokov -
WindowHistory tool has been significantly rewritten and improved to make it better for troubleshooting and debugging GUI. What’s new in this version:
It is a native Windows application written in C++/STL/MFC/Win32.
There are two packages: WindowHistory32 and WindowHistory64. Both can be downloaded from Citrix support web site:
To use download, unpack and run WindowHistory(64).exe.
To uninstall just remove files.
Note: although 32-bit version will run on x64 Windows too, real-time support for 64-bit application windows will not be available. For x64 Windows please use WindowHistory64 which correctly handles both 64-bit and 32-bit application windows.
The following UML collaboration diagram depicts schematically how WindowHistory64 gets notifications from 32-bit windows:
If you want to track window messages and processes simultaneously run it with MessageHistory and ProcessHistory tools.
- Dmitry Vostokov -
Suppose you have 100 - 200 user dumps from various user processes in the system and you want to quickly check their thread stacks, locks, etc. to see something suspicious related to your product or its environment your customers complaining about. It is much easier to collect such information into text files and browse them quickly than open every dump in WinDbg. I used shell script (VBScript) to automate loading dumps into WinDbg and used WinDbg scripts to run complex commands against loaded user dumps. For example, I used the following shell script:
'
' UDumps2Txt.vbs
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set Folder = fso.GetFolder(".")
Set Files = Folder.Files
Set WshShell = CreateObject("WScript.Shell")
For Each File In Files
Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\mss*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\UDmp2Txt.txt;q"" -Q -QS -QY –QSY")
Do While oExec.Status = 0
WScript.Sleep 1000
Loop
Next
'
' UDumps2Txt.vbs: End of File
'
and the following WinDbg script:
$$
$$ UDmp2Txt: Dump information from user dump into log
$$
.logopen /d
!analyze -v
!locks
~*kv
lmv
.logclose
$$
$$ UDmp2Txt: End of File
$$
The following command launches multiple Dmp2Txt conversions:
C:\UserDumps>cscript /nologo c:\scripts\UDumps2Txt.vbs
You can also use CDB from Debugging Tools for Windows (console debugger) instead of WinDbg. I just use WinDbg uniformly instead of using separately CDB for user process dumps and KD for kernel and complete memory dumps.
Now when you have text files you can search for patterns using regular expressions. I will write more about applying them later. There is a very good book about them from practical point of view I read 6 years ago when I needed to understand them beyond wildcards and question marks. Since that time the book has undergone another two editions:
Mastering Regular Expressions, 3rd edition
Or you can process text files further and feed them into your database - part of automated crash dump analysis system.
- Dmitry Vostokov -
I’ve been doing some research in this direction and found so many patents filed, to name a few:
Method and expert system for analysis of crash dumps
System for performing dump analysis
Some companies have their own systems. For example, Microsoft has its own Online Crash Analysis system (OCA) and even promotes its Corporate Error Reporting (CER) tool. CER architecture is described in the following document:
In the next parts I will try to outline different implementation choices for building automated crash dump analysis system and discuss their advantages and disadvantages from expert systems perspective.
- Dmitry Vostokov -
Microsoft describes the reason behind these changes: silent process death if thread stack is corrupt. In Vista such crashes will be reported to MS via Windows Error Reporting mechanism.
Presentation, Reliability and Recovery, slide 42
- Dmitry Vostokov -
Several readers asked me for possibility to be notified by e-mail when I publish a new post and after trying a few e-mail notification plugins for WordPress I finally put Subscribe2 plugin (had to fix its problems with WordPress 2.0.5). If you would like to be notified by e-mail please use Users \ Subscribe link on a side bar.
- Dmitry Vostokov -
I wasn’t satisfied with default Kubrick header and designed my own based on famous BSOD theme. After seeing so many blue screens they became aesthetically pleasant to me If you do crash dump analysis, read, analyze or write assembly language code then you probably like fixed fonts too. I tried many other Wordpress themes but they didn’t look great with my content which was originally tailored for default Wordpress theme and I’m so used to it. Perhaps I need to create a complete brand new BSOD theme for my blog.
- Dmitry Vostokov -
I have been thinking for a while what kind of a marketing card www.dumpanalysis.org should have (which should be useful to its users) and finally came up with the following design which is being printed now:
Front
Backside
I put most used commands (at least by me) and hope the backside of this card will be useful. If you see me in person you have a chance to get this card in hardcopy If after reading this post you got an idea that we need a crash dump analysis and debugging poster (WinDbg related or a general one) then don’t worry and this is being designed now and details will be announced shortly… All suggestions are welcome anyway and if they are genuine and original then full credit will be given.
- Dmitry Vostokov -
WinDbg help states that NMI_HARDWARE_FAILURE (0×80) bugcheck 80 indicates a hardware fault. This description can easily lead to a conclusion that a kernel or complete crash dump you just got from your customer doesn’t worth examining. But hardware malfunction is not always the case especially if your customer mentions that their system was hanging and they forced a manual dump. Here I would advise to check whether they have a special hardware for debugging purposes, for example, a card or an integrated iLO chip (Integrated Lights-Out) for remote server administration. Both can generate NMI (Non Maskable Interrupt) on demand and therefore bugcheck the system. If this is the case then it is worth examining their dump to see why the system was hanging.
- Dmitry Vostokov -
Welcome to the new blog location at dumpanalysis.org/blog/
Its feed address is
http://feeds.feedburner.com/CrashDumpAnalysis
The blog has been moved from its original location at
citrite.org/blogs/dmitryv/
in order to bring all crash dump analysis and debugging information to one place including www.dumpanalysis.org/forum and the forthcoming online encyclopedia about assembly languages:
Thank you and sorry for any inconvenience this might have caused.
Merry Christmas and Happy Debugging in New Year!
- Dmitry Vostokov -
Now it’s time to ”introduce” Invalid Pointer pattern. It’s just a number saved in a register or in a memory location and when we try to interpret it as a memory address itself and follow it (dereference) to fetch memory contents (value) it points to, OS with the help of hardware tells us that the address doesn’t exist or inaccessible due to security restrictions. The following two slides from my old presentation depict the concept of a pointer:
Pointer definition
Pointers depicted
In Windows you have your process memory partitioned into two big regions: kernel space and process space. Space partition is a different concept than execution mode (kernel or user, ring 0 or ring 3) which is a processor state. Code executing in kernel mode (a driver or OS, for example) can access memory that belongs to user space.
Based on this we can make distinction between invalid pointers containing kernel space addresses (start from 0×80000000 on x86, no /3Gb switch) and invalid pointers containing user space addresses (below 0×7FFFFFFF).
On Windows x64 user space addresses are below 0×0000070000000000 and kernel space addresses start from 0xFFFF080000000000.
When you dereference invalid kernel space address you get bugcheck immediately:
UNEXPECTED_KERNEL_MODE_TRAP (7f)
PAGE_FAULT_IN_NONPAGED_AREA (50)
There is no way you can catch it in your code (by using SEH).
However when you dereference user space address the course of action depends on whether your processor is in kernel mode (ring 0) or in user mode (ring 3). In any mode you can catch the exception (by using appropriate SEH handler) or leave this to the operating system or debugger. If there was no component willing to process the exception when it happened in user mode you get your process crash and in kernel mode you get bugchecks:
SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
I summarized all of this on the following diagram:
NULL pointer is a special class of user space pointers. Usually its value is in the range of 0×00000000 - 0×0000FFFF. You can see them used in instructions like
mov esi, dword ptr [ecx+0×10]
and ecx value is 0×00000000 so you try to access the value located at 0×00000010 memory address.
When you get a crash dump and you see an invalid pointer pattern the next step is to interpret the pointer value which should help in understanding possible steps that led to the crash. Pointer value interpretation is the subject of the next part.
- Dmitry Vostokov @ DumpAnalysis.org -
I consider programmers as real programmers only if they attempted to write something like an editor or a word processor. My favorite interview question is “Did you write a word processor?” This probably explains why my team is small
Why? Because writing a word processor shows your determination, persistence and if you are successful - understanding of OO principles. Even Gang of Four in their seminal book
Design Patterns: Elements of Reusable Object-Oriented Software
used word processor as a unified example. I was very satisfied when I read their book - I designed and implemented at least 2 word processors: one in pure C meant to be better than MS Write 3.x and one in Java with syntax highlighting. And I implemented design patterns without knowing they were design patterns. The Java one was better and here is the screen shot showing my editor with C/C++ syntax highlighting implemented in Java 1.0 without any 3rd-party class libraries:
The earlier one (with Windows 3.x look and feel) was written in C and has MDI interface, floating embedded objects and printing support:
- Dmitry Vostokov -
The next pattern I would like to talk about is Optimized Code. If you have such cases you should not trust your crash dump analysis tools like WinDbg. Always suspect that compiler generated code might have been optimized if you see any suspicious or strange behaviour of your tool. Let’s consider this fragment of stack:
Args to Child
77e44c24 000001ac 00000000 ntdll!KiFastSystemCallRet
000001ac 00000000 00000000 ntdll!NtFsControlFile+0xc
00000034 00000bb8 0013e3f4 kernel32!WaitNamedPipeW+0x2c3
0016fc60 00000000 67c14804 MyModule!PipeCreate+0x48
3rd-party function PipeCreate from MyModule opens a named pipe and its first parameter (0016fc60) points to a pipe name L”\\.\pipe\MyPipe”. Inside the source code it calls Win32 API function WaitNamedPipeW (to wait for the pipe to be available for connection) and passes the same pipe name. But we see that the first parameter to WaitNamedPipeW is 00000034 which cannot be the pointer to a valid Unicode string. And the program should have been crashed if 00000034 were a pointer value.
Everything becomes clear if we look at WaitNamedPipeW disassembly (comments are mine):
0:000> uf kernel32!WaitNamedPipeW
mov edi,edi
push ebp
mov ebp,esp
sub esp,50h
push dword ptr [ebp+8] ; Use pipe name
lea eax,[ebp-18h]
push eax
call dword ptr [kernel32!_imp__RtlCreateUnicodeString (77e411c8)]
…
…
…
…
call dword ptr [kernel32!_imp__NtOpenFile (77e41014)]
cmp dword ptr [ebp-4],edi
mov esi,eax
jne kernel32!WaitNamedPipeW+0×1d5 (77e93316)
cmp esi,edi
jl kernel32!WaitNamedPipeW+0×1ef (77e93331)
movzx eax,word ptr [ebp-10h]
mov ecx,dword ptr fs:[18h]
add eax,0Eh
push eax
push dword ptr [kernel32!BaseDllTag (77ecd14c)]
mov dword ptr [ebp+8],eax ; reuse parameter slot
As we know [ebp+8] is the first function parameter in non-FPO calls:
Parameters and Local Variables
And we see it is reused because after we convert LPWSTR to UNICODE_STRING and call NtOpenFile to get a handle we no longer need our parameter slot and the compiler can reuse it to store other information.
There is another compiler optimization we should be aware of and it is called OMAP. It moves the code inside the code section and puts the most frequently accessed code fragments together. In that case if you type in WinDbg, for example,
0:000> uf nt!someFunction
you get different code than if you type (assuming f4794100 is the address of the function you obtained from stack or disassembly)
0:000> uf f4794100
In conclusion the advice is to be alert and conscious during crash dump analysis and inspect any inconsistencies closer.
Happy debugging!
- Dmitry Vostokov @ DumpAnalysis.org -
