Archive for March 27th, 2008

Ceteris Paribus in Comparative Troubleshooting

Thursday, March 27th, 2008

Ceteris Paribus means “with other things [being] the same” (Latin) and when applied to software troubleshooting and debugging means equal environment and configuration. My favorite example is troubleshooting an issue using two Citrix CDF traces (ETW based): one is for the problem and another for the expected behavior. Say we have a terminal services connectivity problem where a published application doesn’t start on the one particular server in Citrix farm. Here Ceteris Paribus means that the application, connection method, configuration, user name, and so on, are all the same for both traces. 

Looks like I have used Latin to obfuscate something obvious but surely many engineers forget it when facing complex issues. This equally applies to debugging as well. 

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Debugging, Software Technical Support, Troubleshooting Methodology | No Comments »

Crash Dump Analysis Patterns (Part 56)

Thursday, March 27th, 2008

The case when a function pointer or a return address becomes a Wild Pointer and EIP or RIP value lies in a valid region of memory the execution path may continue through a region called Wild Code. This might loop on itself or eventually reach non-executable or invalid pages and produce an exception. Local Buffer Overflow might lead to this behavior and also data corruption that overwrites function pointers with valid memory addresses.

My favorite example is when a function pointer points to zeroed pages with EXECUTE page attribute. What will happen next when we dereference it? All zeroes are perfect x86/x64 code:

0:001> dd 0000000`771afdf0
00000000`771afdf0  00000000 00000000 00000000 00000000
00000000`771afe00  00000000 00000000 00000000 00000000
00000000`771afe10  00000000 00000000 00000000 00000000
00000000`771afe20  00000000 00000000 00000000 00000000
00000000`771afe30  00000000 00000000 00000000 00000000
00000000`771afe40  00000000 00000000 00000000 00000000
00000000`771afe50  00000000 00000000 00000000 00000000
00000000`771afe60  00000000 00000000 00000000 00000000

0:001> u
ntdll!DbgUserBreakPoint:
00000000`771afe00 0000    add     byte ptr [rax],al
00000000`771afe02 0000    add     byte ptr [rax],al
00000000`771afe04 0000    add     byte ptr [rax],al
00000000`771afe06 0000    add     byte ptr [rax],al
00000000`771afe08 0000    add     byte ptr [rax],al
00000000`771afe0a 0000    add     byte ptr [rax],al
00000000`771afe0c 0000    add     byte ptr [rax],al
00000000`771afe0e 0000    add     byte ptr [rax],al

Now if RAX points to a valid memory page with WRITE attribute the code will modify the first byte at that address:

0:001> dq @rax
000007ff`fffdc000 00000000`00000000 00000000`035a0000
000007ff`fffdc010 00000000`0359c000 00000000`00000000
000007ff`fffdc020 00000000`00001e00 00000000`00000000
000007ff`fffdc030 000007ff`fffdc000 00000000`00000000
000007ff`fffdc040 00000000`0000142c 00000000`00001504
000007ff`fffdc050 00000000`00000000 00000000`00000000
000007ff`fffdc060 000007ff`fffd8000 00000000`00000000
000007ff`fffdc070 00000000`00000000 00000000`00000000

Therefore the code will be perfectly executed:

0:001> t
ntdll!DbgBreakPoint+0x2:
00000000`771afdf2 0000    add     byte ptr [rax],al ds:000007ff`fffdc000=00

0:001> t
ntdll!DbgBreakPoint+0x4:
00000000`771afdf4 0000    add     byte ptr [rax],al ds:000007ff`fffdc000=00

0:001> t
ntdll!DbgBreakPoint+0x6:
00000000`771afdf6 0000    add     byte ptr [rax],al ds:000007ff`fffdc000=00

0:001> t
ntdll!DbgBreakPoint+0x8:
00000000`771afdf8 0000    add     byte ptr [rax],al ds:000007ff`fffdc000=00

0:001> t
ntdll!DbgBreakPoint+0xa:
00000000`771afdfa 0000    add     byte ptr [rax],al ds:000007ff`fffdc000=00

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in Assembly Language, Crash Dump Analysis, Crash Dump Patterns | 5 Comments »

Crash Dump Analysis AntiPatterns (Part 9)

Thursday, March 27th, 2008

Symbolless Analysis is another anti-pattern when an engineer either in a hurry or due to laziness doesn’t apply proper symbols and relies only on timestamps and module/offsets or trusts what WinDbg says and ignores symbol warnings. I usually apply symbols even in obvious cases and in hard ones strive to apply them until all possibilities are exhausted including search using PDBFinder.

Another weak variant is called Imageless Analysis when an engineer doesn’t specify proper Executable Image Search Path when it is necessary perhaps due to ignorance or just plain laziness again. Please see Minidump Analysis example for proper minidump analysis.

- Dmitry Vostokov @ DumpAnalysis.org -

Posted in AntiPatterns, Crash Dump Analysis, Crash Dump Patterns, Minidump Analysis, Software Technical Support | No Comments »