While reading memory snapshots (dumps) from Clive James during lunch I came across this:
“… beauty” in debugging “begins as consolation for what can’t be” debugged.
- Dmitry Vostokov @ DumpAnalysis.org -
Citrix released the powerful GUI CDF trace analyzer tool that can work with hundred Mb trace files with millions of trace messages:
http://support.citrix.com/article/CTX122741
I use it on day-to-day basis while analyzing CDF traces from terminal services environments and highly recommend. Its ability to quick filter out modules is indispensable when dealing with messages from hundreds of processes and thousands of threads. I haven’t tried it yet with MS TMF files though.
- Dmitry Vostokov @ DumpAnalysis.org -
Memory Space
The linear range of memory addresses. Usually, addresses are numbers from N0 set. For example, [0, 7FFFFFFF] or [0×80000000, 0xFFFFFFFF]. Memory space is divided into memory regions. The memory contents might not be available for specific memory regions of a memory space.
Synonyms:
Antonyms:
Also: memory dump, memory region, user dump, kernel dump, complete dump, physical memory, virtual memory, user space, kernel space.
- Dmitry Vostokov @ DumpAnalysis.org -
Troubleshooting Unit of Work is another pattern frequently used in manual troubleshooting and debugging. This is usually some independent and self-sufficient list of steps to perform to check something from a troubleshooting checklist or a manual and can be implemented as a separate loadable module, a class to reuse or a function to call. Output from such units of work can be stored in a blackboard system or processed by tools implementing Checklist DebugWare pattern. Typical example is an implementation of the following document:
Required Permissions and Rights for the Ctx_CpsvcUser Account
as a tool:
CTX_CpsvcUser Re-creation Tool for 32-Bit and 64-Bit Versions of Presentation Server 4.5
- Dmitry Vostokov @ DumpAnalysis.org -
The notion of transmigration of memories in Memoidealism has its similarity with Orphicism. The notions of limited (memories) and unlimited memory (Memory), and “things are memories” similar to Pythagoreanism view on Limited and Unlimited, and “things are numbers”.
- Dmitry Vostokov @ DumpAnalysis.org -
The purpose of this case study is to show how to choose what to include in a fiber bundle memory dump when x64 complete memory dumps are huge and not an option to deliver:
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 5880464 ( 23521856 Kb)
[…]
The dump we have is a kernel. When we dump all processes and threads and look for “Waiting for ” we find many ALPC wait chains spanning 3 - 4 processes (sometimes semicircular), sometimes originated from processes with missing threads (just one or two present threads when we expect a dozen of them in a normal state):
1: kd> !process fffffa800b834c10
PROCESS fffffa800b834c10
SessionId: 205 Cid: 13c40 Peb: 7fffffdb000 ParentCid: 133c0
DirBase: 13b61d000 ObjectTable: fffff8800c2295b0 HandleCount: 58.
Image: ProcessA.exe
VadRoot fffffa8007d70c00 Vads 121 Clone 0 Private 497. Modified 0. Locked 0.
DeviceMap fffff88000007450
Token fffff8800c695560
ElapsedTime 00:03:42.083
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 65968
QuotaPoolUsage[NonPagedPool] 11520
Working Set Sizes (now,min,max) (1274, 50, 345) (5096KB, 200KB, 1380KB)
PeakWorkingSetSize 1278
VirtualSize 37 Mb
PeakVirtualSize 38 Mb
PageFaultCount 1286
MemoryPriority BACKGROUND
BasePriority 13
CommitCharge 581
THREAD fffffa800b845bb0 Cid 13c40.1332c Teb: 000007fffffde000 Win32Thread: fffff900c0076010 WAIT: (WrLpcReply) UserMode Non-Alertable
fffffa800b845f40 Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff88012527770 : queued at port fffffa80055bca60 : owned by process fffffa80054dfc10
Not impersonating
DeviceMap fffff88000007450
Owning Process fffffa800b834c10 Image: ProcessA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10912787 Ticks: 14208 (0:00:03:42.000)
Context Switch Count 34 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address 0×00000000fff60260
Stack Init fffffa600e8d5db0 Current fffffa600e8d5670
Base fffffa600e8d6000 Limit fffffa600e8ce000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffffa60`0e8d56b0 fffff800`016a36fa nt!KiSwapContext+0×7f
fffffa60`0e8d57f0 fffff800`0169835b nt!KiSwapThread+0×13a
fffffa60`0e8d5860 fffff800`016cd4e2 nt!KeWaitForSingleObject+0×2cb
fffffa60`0e8d58f0 fffff800`01916d14 nt!AlpcpSignalAndWait+0×92
fffffa60`0e8d5980 fffff800`019137a6 nt!AlpcpReceiveSynchronousReply+0×44
fffffa60`0e8d59e0 fffff800`0190330f nt!AlpcpProcessSynchronousRequest+0×24f
fffffa60`0e8d5b00 fffff800`016a0ef3 nt!NtAlpcSendWaitReceivePort+0×19f
fffffa60`0e8d5bb0 00000000`774d756a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`0e8d5c20)
00000000`0026f038 00000000`00000000 0×774d756a
1: kd> !alpc /m fffff88012527770
Message @ fffff88012527770
MessageID : 0x10E8 (4328)
CallbackID : 0xC3416B (12796267)
SequenceNumber : 0x00000002 (2)
Type : LPC_REQUEST
DataLength : 0x0040 (64)
TotalLength : 0x0068 (104)
Canceled : No
Release : No
ReplyWaitReply : No
Continuation : Yes
OwnerPort : fffffa80076e9660 [ALPC_CLIENT_COMMUNICATION_PORT]
WaitingThread : fffffa800b845bb0
QueueType : ALPC_MSGQUEUE_PENDING
QueuePort : fffffa80055bca60 [ALPC_CONNECTION_PORT]
QueuePortOwnerProcess : fffffa80054dfc10 (ProcessB.exe)
ServerThread : fffffa800b711060
QuotaCharged : No
CancelQueuePort : 0000000000000000
CancelSequencePort : 0000000000000000
CancelSequenceNumber : 0×00000000 (0)
ClientContext : 00000000003fcf20
ServerContext : 0000000000000000
PortContext : 00000000029fda00
CancelPortContext : 0000000000000000
SecurityData : 0000000000000000
View : 0000000000000000
1: kd> !thread fffffa800b711060
THREAD fffffa800b711060 Cid 032c.146e8 Teb: 000007fffff7c000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
fffffa800b7113f0 Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8800e401200 : queued at port fffffa8005a32730 : owned by process fffffa8004c39040
Not impersonating
DeviceMap fffff88000007450
Owning Process fffffa80054dfc10 Image: ProcessB.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10916800 Ticks: 10195 (0:00:02:39.296)
Context Switch Count 401
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0×000007fefe647780
Stack Init fffffa6001d33db0 Current fffffa6001d33670
Base fffffa6001d34000 Limit fffffa6001d2e000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 1 IoPriority 2 PagePriority 5
Child-SP RetAddr : Call Site
fffffa60`01d336b0 fffff800`016a36fa : nt!KiSwapContext+0×7f
fffffa60`01d337f0 fffff800`0169835b : nt!KiSwapThread+0×13a
fffffa60`01d33860 fffff800`016cd4e2 : nt!KeWaitForSingleObject+0×2cb
fffffa60`01d338f0 fffff800`01916d14 : nt!AlpcpSignalAndWait+0×92
fffffa60`01d33980 fffff800`019137a6 : nt!AlpcpReceiveSynchronousReply+0×44
fffffa60`01d339e0 fffff800`0190330f : nt!AlpcpProcessSynchronousRequest+0×24f
fffffa60`01d33b00 fffff800`016a0ef3 : nt!NtAlpcSendWaitReceivePort+0×19f
fffffa60`01d33bb0 00000000`774d756a : nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`01d33c20)
00000000`03d8e458 00000000`00000000 : 0×774d756a
1: kd> !alpc /m fffff8800e401200
Message @ fffff8800e401200
MessageID : 0x0BA4 (2980)
CallbackID : 0xC3E68A (12838538)
SequenceNumber : 0x00021911 (137489)
Type : LPC_REQUEST
DataLength : 0x00C0 (192)
TotalLength : 0x00E8 (232)
Canceled : No
Release : No
ReplyWaitReply : No
Continuation : Yes
OwnerPort : fffffa8005b119c0 [ALPC_CLIENT_COMMUNICATION_PORT]
WaitingThread : fffffa800b711060
QueueType : ALPC_MSGQUEUE_PENDING
QueuePort : fffffa8005a32730 [ALPC_CONNECTION_PORT]
QueuePortOwnerProcess : fffffa8004c39040 (ProcessC.exe)
ServerThread : fffffa800a843bb0
QuotaCharged : No
CancelQueuePort : 0000000000000000
CancelSequencePort : 0000000000000000
CancelSequenceNumber : 0×00000000 (0)
ClientContext : 0000000002e2e810
ServerContext : 0000000000000000
PortContext : 00000000002f3eb0
CancelPortContext : 0000000000000000
SecurityData : 0000000000000000
View : 0000000000000000
1: kd> !thread fffffa800a843bb0
THREAD fffffa800a843bb0 Cid 048c.fbec Teb: 000007ffffdaa000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa8006027d80 Semaphore Limit 0x7fffffff
fffffa800a843c68 NotificationTimer
Not impersonating
DeviceMap fffff88001800ba0
Owning Process fffffa8004c39040 Image: ProcessC.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10916801 Ticks: 10194 (0:00:02:39.281)
Context Switch Count 239
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address 0×000007fefe647780
Stack Init fffffa601b280db0 Current fffffa601b280940
Base fffffa601b281000 Limit fffffa601b27b000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Call Site
fffffa60`1b280980 fffff800`016a36fa : nt!KiSwapContext+0×7f
fffffa60`1b280ac0 fffff800`0169835b : nt!KiSwapThread+0×13a
fffffa60`1b280b30 fffff800`019013e8 : nt!KeWaitForSingleObject+0×2cb
fffffa60`1b280bc0 fffff800`016a0ef3 : nt!NtWaitForSingleObject+0×98
fffffa60`1b280c20 00000000`774d6d5a : nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`1b280c20)
00000000`10b7e548 00000000`00000000 : 0×774d6d5a
Some processes designed to be non-interactive have threads that wait for UI messages and therefore could be potential message or dialog box threads waiting for a dismissal and blocking other threads:
THREAD fffffa8005a7aa20 Cid 061c.0778 Teb: 000007fffff9e000 Win32Thread: fffff900c079fd50 WAIT: (WrUserRequest) UserMode Non-Alertable
fffffa8005a7a5a0 SynchronizationEvent
Not impersonating
DeviceMap fffff88000007450
Owning Process fffffa80058f01b0 Image: ProcessD.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10911798 Ticks: 15197 (0:00:03:57.453)
Context Switch Count 88939 LargeStack
UserTime 00:00:00.078
KernelTime 00:00:00.609
Win32 Start Address 0×000007fefa8238a0
Stack Init fffffa60046a8db0 Current fffffa60046a8720
Base fffffa60046a9000 Limit fffffa60046a0000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffffa60`046a8760 fffff800`016a36fa nt!KiSwapContext+0×7f
fffffa60`046a88a0 fffff800`0169835b nt!KiSwapThread+0×13a
fffffa60`046a8910 fffff960`0014c053 nt!KeWaitForSingleObject+0×2cb
fffffa60`046a89a0 fffff960`0014c0ea win32k!xxxRealSleepThread+0×25f
fffffa60`046a8a40 fffff960`0014bb3a win32k!xxxSleepThread+0×56
fffffa60`046a8a70 fffff960`0014bc39 win32k!xxxRealInternalGetMessage+0×72e
fffffa60`046a8b50 fffff960`0014d0d9 win32k!xxxInternalGetMessage+0×35
fffffa60`046a8b90 fffff800`016a0ef3 win32k!NtUserGetMessage+0×79
fffffa60`046a8c20 00000000`773dd58a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`046a8c20)
00000000`03d2f7b8 00000000`00000000 0×773dd58a
We also have more than 30,000 zombie processes including some special ones signifying past faults:
1: kd> !vm
[...]
15714 ProcessE.exe 0 ( 0 Kb)
15650 WerFault.exe 0 ( 0 Kb)
15644 ProcessF.exe 0 ( 0 Kb)
15640 ProcessE.exe 0 ( 0 Kb)
15610 ProcessG.exe 0 ( 0 Kb)
1560c ProcessE.exe 0 ( 0 Kb)
155f8 ProcessH.exe 0 ( 0 Kb)
155e8 ProcessE.exe 0 ( 0 Kb)
155c4 ProcessG.exe 0 ( 0 Kb)
155bc ProcessE.exe 0 ( 0 Kb)
155b8 ProcessH.exe 0 ( 0 Kb)
1559c WerFault.exe 0 ( 0 Kb)
15560 ProcessE.exe 0 ( 0 Kb)
[…]
What we recommend here is to save user dumps of processes A, B, C and D and then force a kernel dump next time the problem surfaces. Also to check WER settings for any recorder faults and, because of the fact the the system is W2K8, configure LocalDumps registry keys to capture full user dumps.
- Dmitry Vostokov @ DumpAnalysis.org -
I plan the following titles to be published in Q4:
- Debugged! MZ/PE: Software Tracing, September, 2009 (ISBN: 978-1906717797)
- Windows Debugging Notebook: Essential Concepts, WinDbg Commands and Tools (ISBN: 978-1906717001)
- Memory Dump Analysis Anthology, Volume 3 (ISBN: 978-1906717438 and 978-1906717445)
- Memory Dump Analysis Anthology: Color Supplement for Volumes 1-3 (ISBN: 978-1906717698)
- First Fault Software Problem Solving: A Guide for Engineers, Managers and Users (ISBN: 978-1906717421) by Dan Skwire
- Crash Dump Analysis for System Administrators and Support Engineers (Windows Edition) (ISBN: 978-1906717025)
The title of the latter book was slightly changed. After some time we realized that the same material is appropriate for support engineers as well.
- Dmitry Vostokov @ DumpAnalysis.org -
Some users and fans were asking why many announced books are delayed and delayed. The answer is very simple: I am a full time employee of Citrix Systems, prioritize my job first and put books and blogs at the end of the priority chain. However, I have now reserved 2-3 hours every evening to finish a few books and my next blog post announces which of them are scheduled for Q4 this Year of Debugging.
- Dmitry Vostokov @ DumpAnalysis.org -
Occasionally I check my books to see how they are positioned on Amazon and noticed that Windows Debugging: Practical Foundations and Memory Dump Analysis Anthology, Volume 1 paperback titles are #1 and #2 bestsellers (at the time of this writing) on Amazon Debugging and Assembly Language Programming bestselling lists:
- Dmitry Vostokov @ DumpAnalysis.org -
“You” run code “from beginning to end. You” debug code “the opposite way. You start with the end, and then you do everything you must to reach” the beginning.
- Dmitry Vostokov @ DumpAnalysis.org -
SecretSES (Secret Software Engineering Society) announces Debugging Spy Network of Memory Analysis Forensics and Intelligence Agents (MAFIA).
Motivation: seeing various nomadic and settled debugging teams I finally decided to unfold my own net.
- Dmitry Vostokov @ DumpAnalysis.org -
Having discussed dereference fixpoints we come back to the quiz code and see what happens when we execute it after compilation as default Debug target with Debug Information Format set to Program Database to avoid extra stack space allocation:
int _tmain(int argc, _TCHAR* argv[])
{
char c;
char* pc = &c;
while(1)
{
*pc = 0;
pc++;
}
return 0;
}
Expecting crashes I created the following key HKEY_LOCAL_MACHINE \ SOFTWARE \ Microsoft \ Windows \ Windows Error Reporting \ LocalDumps with the following values: DumpFolder (REG_EXPAND_SZ) and DumpType (2, Full).
When running the compiled program I noticed that it crashed according to my expectations. The saved dump StackErasure.exe.2096.dmp confirmed that the crash was due to the stack underflow when it hit the base address:
0:000> r
eax=002c0000 ebx=7efde000 ecx=00000001 edx=002c0000 esi=00000000 edi=00000000
eip=00e11039 esp=002bf7c4 ebp=002bf7d4 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
StackErasure!wmain+0x29:
00e11039 c60200 mov byte ptr [edx],0 ds:002b:002c0000=??
0:000> !teb
TEB at 7efdd000
ExceptionList: 002bf810
StackBase: 002c0000
StackLimit: 002be000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7efdd000
EnvironmentPointer: 00000000
ClientId: 00000830 . 00000a78
RpcHandle: 00000000
Tls Storage: 7efdd02c
PEB Address: 7efde000
LastErrorValue: 0
LastStatusValue: 0
Count Owned Locks: 0
HardErrorMode: 0
The loop from source code is highlighted in blue:
0:000> uf wmain
StackErasure!wmain:
00e11010 push ebp
00e11011 mov ebp,esp
00e11013 sub esp,10h
00e11016 mov eax,0CCCCCCCCh
00e1101b mov dword ptr [ebp-10h],eax
00e1101e mov dword ptr [ebp-0Ch],eax
00e11021 mov dword ptr [ebp-8],eax
00e11024 mov dword ptr [ebp-4],eax
00e11027 lea eax,[ebp-5]
00e1102a mov dword ptr [ebp-10h],eax
StackErasure!wmain+0x1d:
00e1102d mov ecx,1
00e11032 test ecx,ecx
00e11034 je StackErasure!wmain+0x37 (00e11047)
StackErasure!wmain+0x26:
00e11036 mov edx,dword ptr [ebp-10h]
00e11039 mov byte ptr [edx],0
00e1103c mov eax,dword ptr [ebp-10h]
00e1103f add eax,1
00e11042 mov dword ptr [ebp-10h],eax
00e11045 jmp StackErasure!wmain+0x1d (00e1102d)
StackErasure!wmain+0x37:
00e11047 xor eax,eax
00e11049 push edx
00e1104a mov ecx,ebp
00e1104c push eax
00e1104d lea edx,[StackErasure!wmainCRTStartup+0x10 (00e11060)]
00e11053 call StackErasure!__tmainCRTStartup+0x50 (00e110c0)
00e11058 pop eax
00e11059 pop edx
00e1105a mov esp,ebp
00e1105c pop ebp
00e1105d ret
We see that our char variable ‘c’ is located at EBP-5 and the pointer ‘pc’ is located at EBP-10 (in another words ‘c’ follows ‘pc’ in memory):
00e11027 lea eax,[ebp-5]
00e1102a mov dword ptr [ebp-10h],eax
Both locations were initialized to 0xCCCCCCCC:
00e11016 mov eax,0CCCCCCCCh
00e1101b mov dword ptr [ebp-10h],eax
00e1101e mov dword ptr [ebp-0Ch],eax
00e11021 mov dword ptr [ebp-8],eax ; this ends with EBP-5
00e11024 mov dword ptr [ebp-4],eax
The memory layout before the start of the loop is depicted on the following diagram in the style of Windows Debugging: Practical Foundations book:
At the crash point we have the following final memory layout:
We can see it from the raw stack:
0:000> db esp
002bf7c4 00 00 2c 00 cc cc cc cc-cc cc cc 00 00 00 00 00
002bf7d4 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf7e4 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf7f4 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf804 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf814 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf824 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
002bf834 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
or in pointer-sized (double word) values where we can see little endian effects (compare 00 00 2c 00 with 002c0000):
0:000> dp esp
002bf7c4 002c0000 cccccccc 00cccccc 00000000
002bf7d4 00000000 00000000 00000000 00000000
002bf7e4 00000000 00000000 00000000 00000000
002bf7f4 00000000 00000000 00000000 00000000
002bf804 00000000 00000000 00000000 00000000
002bf814 00000000 00000000 00000000 00000000
002bf824 00000000 00000000 00000000 00000000
002bf834 00000000 00000000 00000000 00000000
The loop code erases stack starting from EBP-5 until it hits the base address.
Now we change Basic Runtime Checks in Code Generation properties to Default, recompile and launch the project. Suddenly it no longer crashes but loops infinitely (shown in blue):
0:000> bp wmain
0:000> g
[...]
0:000> uf wmain
StackErasure!wmain:
00d01010 push ebp
00d01011 mov ebp,esp
00d01013 sub esp,8
00d01016 lea eax,[ebp-5]
00d01019 mov dword ptr [ebp-4],eax
StackErasure!wmain+0xc:
00d0101c mov ecx,1
00d01021 test ecx,ecx
00d01023 je StackErasure!wmain+0x26 (00d01036)
StackErasure!wmain+0x15:
00d01025 mov edx,dword ptr [ebp-4]
00d01028 mov byte ptr [edx],0
00d0102b mov eax,dword ptr [ebp-4]
00d0102e add eax,1
00d01031 mov dword ptr [ebp-4],eax
00d01034 jmp StackErasure!wmain+0xc (00d0101c)
StackErasure!wmain+0x26:
00d01036 xor eax,eax
00d01038 mov esp,ebp
00d0103a pop ebp
00d0103b ret
At first it looks strange but if we look at the stack memory layout we would see that ‘pc’ pointer follows ‘c’ (the opposite to the memory layout discussed above):
00d01016 lea eax,[ebp-5]
00d01019 mov dword ptr [ebp-4],eax
0:000> dp esp
002dfb90 00d014ee 002dfb93 002dfbe4 00d01186
002dfba0 00000001 00081d70 00081df8 5a16a657
002dfbb0 00000000 00000000 7ffdb000 00000000
002dfbc0 00000000 00000000 00000000 002dfbac
002dfbd0 000001bb 002dfc28 00d06e00 5aed06eb
002dfbe0 00000000 002dfbec 00d0105f 002dfbf8
002dfbf0 77844911 7ffdb000 002dfc38 7791e4b6
002dfc00 7ffdb000 705b3701 00000000 00000000
Therefore, when the pointer at EBP-4 is incremented by 1 during the first loop iteration it becomes a dereference fixpoint:
0:000> bp 00d0101c
0:000> g
Breakpoint 1 hit
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=00081df8 esi=00000000 edi=00000000
eip=00d0101c esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
StackErasure!wmain+0xc:
00d0101c b901000000 mov ecx,1
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=00081df8 esi=00000000 edi=00000000
eip=00d01021 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
StackErasure!wmain+0x11:
00d01021 85c9 test ecx,ecx
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=00081df8 esi=00000000 edi=00000000
eip=00d01023 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x13:
00d01023 7411 je StackErasure!wmain+0x26 (00d01036) [br=0]
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=00081df8 esi=00000000 edi=00000000
eip=00d01025 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x15:
00d01025 8b55fc mov edx,dword ptr [ebp-4] ss:0023:002dfb94=002dfb93
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01028 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x18:
00d01028 c60200 mov byte ptr [edx],0 ds:0023:002dfb93=00
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d0102b esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x1b:
00d0102b 8b45fc mov eax,dword ptr [ebp-4] ss:0023:002dfb94=002dfb93
0:000> t
eax=002dfb93 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d0102e esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x1e:
00d0102e 83c001 add eax,1
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01031 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x21:
00d01031 8945fc mov dword ptr [ebp-4],eax ss:0023:002dfb94=002dfb93
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01034 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x24:
00d01034 ebe6 jmp StackErasure!wmain+0xc (00d0101c)
0:000> dp ebp-4 l1
002dfb94 002dfb94
During the second iteration the assignment of zero to ‘*pc’ clears the first byte of ‘pc’:
0:000> t
Breakpoint 1 hit
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d0101c esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0xc:
00d0101c b901000000 mov ecx,1
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01021 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x11:
00d01021 85c9 test ecx,ecx
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01023 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x13:
00d01023 7411 je StackErasure!wmain+0x26 (00d01036) [br=0]
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb93 esi=00000000 edi=00000000
eip=00d01025 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x15:
00d01025 8b55fc mov edx,dword ptr [ebp-4] ss:0023:002dfb94=002dfb94
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb94 esi=00000000 edi=00000000
eip=00d01028 esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x18:
00d01028 c60200 mov byte ptr [edx],0 ds:0023:002dfb94=94
0:000> t
eax=002dfb94 ebx=7ffdb000 ecx=00000001 edx=002dfb94 esi=00000000 edi=00000000
eip=00d0102b esp=002dfb90 ebp=002dfb98 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
StackErasure!wmain+0x1b:
00d0102b 8b45fc mov eax,dword ptr [ebp-4] ss:0023:002dfb94=002dfb00
0:000> dp esp
002dfb90 00d014ee 002dfb00 002dfbe4 00d01186
002dfba0 00000001 00081d70 00081df8 5a16a657
002dfbb0 00000000 00000000 7ffdb000 00000000
002dfbc0 00000000 00000000 00000000 002dfbac
002dfbd0 000001bb 002dfc28 00d06e00 5aed06eb
002dfbe0 00000000 002dfbec 00d0105f 002dfbf8
002dfbf0 77844911 7ffdb000 002dfc38 7791e4b6
002dfc00 7ffdb000 705b3701 00000000 00000000
The new ‘pc’ pointer points to the following region of the stack:
0:000> dp 002dfb00 l100/4
002dfb00 002dfb0c 00000004 00000000 5c008ede
002dfb10 002dfb28 00d0634a 0008128c 5aed018b
002dfb20 000807f8 7790fb66 00000000 7ffdb000
002dfb30 00000000 002dfb40 00d089a6 00d68ab8
002dfb40 002dfb4c 00d019bc 00000008 002dfb84
002dfb50 00d07520 00d07519 5a16a637 00000000
002dfb60 00000000 7ffdb000 00d02b10 00000004
002dfb70 00000002 002dfbd4 00d06e00 5aed007b
002dfb80 fffffffe 002dfb90 00d0769e 002dfba0
002dfb90 00d014ee 002dfb00 002dfbe4 00d01186
002dfba0 00000001 00081d70 00081df8 5a16a657
002dfbb0 00000000 00000000 7ffdb000 00000000
002dfbc0 00000000 00000000 00000000 002dfbac
002dfbd0 000001bb 002dfc28 00d06e00 5aed06eb
002dfbe0 00000000 002dfbec 00d0105f 002dfbf8
002dfbf0 77844911 7ffdb000 002dfc38 7791e4b6
The loop code now starts clearing this region until the pointer becomes the fixpoint again after successive increments and therefore continues to loop indefinitely:
0:000> bc 0 1
0:000> g
(1238.c9c): Break instruction exception - code 80000003 (first chance)
eax=7ffde000 ebx=00000000 ecx=00000000 edx=7796d094 esi=00000000 edi=00000000
eip=77927dfe esp=00a4ff30 ebp=00a4ff5c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!DbgBreakPoint:
77927dfe cc int 3
0:001> dp 002dfb00 l100/4
002dfb00 0000000c 00000000 00000000 00000000
002dfb10 00000000 00000000 00000000 00000000
002dfb20 00000000 00000000 00000000 00000000
002dfb30 00000000 00000000 00000000 00000000
002dfb40 00000000 00000000 00000000 00000000
002dfb50 00000000 00000000 00000000 00000000
002dfb60 00000000 00000000 00000000 00000000
002dfb70 00000000 00000000 00000000 00000000
002dfb80 00000000 00000000 00000000 00000000
002dfb90 00000000 002dfb1f 002dfbe4 00d01186
002dfba0 00000001 00081d70 00081df8 5a16a657
002dfbb0 00000000 00000000 7ffdb000 00000000
002dfbc0 00000000 00000000 00000000 002dfbac
002dfbd0 000001bb 002dfc28 00d06e00 5aed06eb
002dfbe0 00000000 002dfbec 00d0105f 002dfbf8
002dfbf0 77844911 7ffdb000 002dfc38 7791e4b6
StackErasure that loops indefinitely is available for download.
- Dmitry Vostokov @ DumpAnalysis.org -
Ntdebugging blog has put the link to the video online from Microsoft Global Engineering Conference where I presented the pattern-driven memory dump analysis methodology:
Citrix engineers at Microsoft GEC
Note: you need to open a video link URL from the blog post in Windows Media Player if you don’t have an association for WMV files or save the file.
- Dmitry Vostokov @ DumpAnalysis.org -
Imagine we have the following arrangements in memory:
address: value
where value == address, so we have effectively:
address: address
So when we dereference the address we get the address value. If we name the dereference function as p(address) we get
p(address) = address
That gave me an idea to name after the mathematical notion of a function fixpoint (fixed point).
In C++ we can write the following code to initialize a fixpoint:
void *pc = &pc;
in assembly language:
lea eax, [pc]
mov dword ptr [pc], eax
or using local variables:
lea eax, [ebp-4]
mov dword ptr [ebp-4], eax
Dereference of a fixpoint pointer gives us the same value as its address, for example, using old style conversion:
int *pc = (int *)&pc;
if (pc == (int *)*pc) {
// TRUE
or for C++ purists:
int *pc = reinterpret_cast<int *>(&pc);
if (pc == reinterpret_cast<int *>(*pc)) {
// TRUE
In x86 assembly language we have this comparison:
mov eax,dword ptr [pc]
mov ecx,dword ptr [pc]
cmp ecx,dword ptr [eax]
or using local variables:
mov eax,dword ptr [ebp-4]
mov ecx,dword ptr [ebp-4]
cmp ecx,dword ptr [eax]
Now, having discussed fixpoints, let me ask the question to ponder over this weekend. What would this code do?
int _tmain(int argc, _TCHAR* argv[])
{
char c;
char* pc = &c;
while(1)
{
*pc = 0;
pc++;
}
return 0;
}
Would it produce stack overflow with an exception, or stack underflow with an exception or loop indefinitely? The C++ Standard answer of compiler and platform dependence is not acceptable. I plan to elaborate on this topic on Monday.
The notion of counterfactual debugging (”what if” debugging) was inspired by the so called counterfactual history.
- Dmitry Vostokov @ DumpAnalysis.org -
While finishing Comrades book I started to read this “sequel” to Young Stalin (it was published before the latter book). I’m interested in psychology of a court and think this book is a good supplement to The 48 Laws of Power book that I started reading too. I have also Beria biography on the reading list. Actually I became interested in Stalin epoch after reading a book in Russian 2 years ago with a title that can be translated to English like “Killers of Stalin and Beria”. The main idea of that book were that Beria (and Stalin) wanted to do Perestroika similar to what Gorbachev did and Khrushchev murdered him (and possibly murdered Stalin too) for that. Anyway The Court of the Red Tsar was very smooth and fascinating read, revealing hidden transcripts of Stalin power. At the end the author also mentions the possibility that Beria was a possible precursor to Perestroika but contrary to that Russian book I read before he mentions the hypothesis that Beria himself poisoned Stalin’s wine. The finishing touch of Valechka weeping on Stalin corpse like Russian baba really made me sorrow. I really liked Postscriptum where the fortunes of Stalin’s and other magnates’ relatives, children and grandchildren fortunes after Stalin death up to now was mentioned.
Stalin: The Court of the Red Tsar
- Dmitry Vostokov @ LiterateScientist.com -
It was reported that functionality of some applications was not available until ServiceA was killed and restarted. Complete memory dump was forced. For that memory dump WinDbg !stacks command shows every thread bugchecking the system, for example:
0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[...]
[8ca72020 ApplicationA.exe]
4bc.0004c4 8bf76af0 0000cc4 Blocked nt!KeBugCheckEx+0x1b
4bc.0004d0 8bf53650 000055d Blocked nt!KeBugCheckEx+0x1b
4bc.0004e8 8bf33c88 0000094 Blocked nt!KeBugCheckEx+0x1b
4bc.001ffc 8ba86020 0000eed Blocked nt!KeBugCheckEx+0x1b
4bc.001e30 8b7a0ca8 00003ac Blocked nt!KeBugCheckEx+0x1b
4bc.001934 8b664020 0000eed Blocked nt!KeBugCheckEx+0x1b
4bc.001af0 8ca6d3b0 0000094 Blocked nt!KeBugCheckEx+0x1b
4bc.001cac 8bf41af0 0001412 Blocked nt!KeBugCheckEx+0x1b
4bc.00141c 8b6458d0 0000094 Blocked nt!KeBugCheckEx+0x1b
[...]
Seeing this nonsense I checked that complete dump was truncated by half because page file was 4Gb but the amount of physical memory was 8Gb:
0: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 2096553 ( 8386212 Kb)
Paging File Name paged out
Current: 4195328 Kb Free Space: 4182336 Kb
Minimum: 4195328 Kb Maximum: 4195328 Kb
[…]
However it was possible to get stack trace collection using !process 0 ff command with most data from _ETHREAD structures present but most of kernel and user space stack traces not present in the output. ServiceA has many threads waiting for LPC requests during last 5 minutes (the bugcheck was forced after 3 - 4 minutes the issue manifested itself):
THREAD 8cc9b590 Cid 053c.0668 Teb: 7ffaf000 Win32Thread: b84e5770 WAIT: (Unknown) UserMode Non-Alertable
8cc9b77c Semaphore Limit 0x1
Waiting for reply to LPC MessageId 0005bbdf:
Current LPC port e2cee338
IRP List:
8b6548b8: (0006,0268) Flags: 00000000 Mdl: 00000000
Not impersonating
DeviceMap e1003860
Owning Process 8bf81d88 Image: ServiceA.exe
Attached Process N/A Image: N/A
Wait Start TickCount 193823 Ticks: 13752 (0:00:03:34.875)
Context Switch Count 389 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0×7d1f5e70
Start Address 0×77e617ec
Stack Init b3a44000 Current b3a43c08 Base b3a44000 Limit b3a41000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0
Kernel stack not resident.
The similar waits were found for ApplicationB and ApplicationC processes. Therefore it was advised to limit the amount of physical memory to 4Gb and reproduce the issue or if the liveliness of the system needs to be preserved to manually dump the following processes on the next occasion:
ServiceA.exe
ApplicationB.exe
ApplicationC.exe
- Dmitry Vostokov @ DumpAnalysis.org -
Trace Expert pattern came to my mind when I was writing about software trace patterns. It is a very lightweight expert system relying on trace collector and trace formatter (patterns to be written about soon). It is a module that takes a preformatted software trace message file or a buffer and a set of built in rules and uses simple search (peharps involving regular expressions) to dig out diagnostic information and provide troubleshooting and debugging directions.
This module is schematically depicted on the following UML component diagram:
- Dmitry Vostokov @ DumpAnalysis.org -
Responding to numerous requests and suggestions I plan to make magazine interior excluding promotional vouchers available for free download. If someone needs covers including back covers where I put tips and tables to be used as posters or certification vouchers printed inside then they should buy the magazine from Amazon or other bookshops.
This initiative will be accompanied by a smart marketing trick that I plan to unveil in a few days together with the magazine website.
- Dmitry Vostokov @ DumpAnalysis.org -
Consider this example mapping (taken metaphorically from the mathematical notion of an injection) of one domain of knowledge to another:
This mapping between concepts and ideas was once called “bijectivism” but was trivially described either as one to one mapping between two domains (like physical vs. mathematical) or fusing different concepts together to get another emerging concept. I myself proposed the similar mapping and called it a metaphorical bijection.
Now consider another mapping metaphorically equivalent to a mathematical notion of a surjection where all constituents of the second domain are covered metaphorically by the first domain:
What we strive for is to establish the complete bijective mapping and reorganize our knowledge of both domains to achieve that:
In diagrams above small boxes can represent sets of ideas, methods, etc. or individual ideas, methods, etc. The established metaphorical bijection can divide sets or combine them if needed. There can be several such bijections, of course, and we can use other methods of inquiry (for example, the scientific method) to choose between competing metaphorical bijections.
Useful mnemonic:
BEIS (B=I+S or to BE IS …)
Bijectionism Equals Injection + Surjection
Another mnemonic:
BET (B=T or to BE Transformation…)
Bijectionism Equals Transformation
Note also the second letter of Alef-Beis or Alef-Bet, the letter of Light that has interpretation of Creation in Biblical Hebrew.
More on this later as I need to come back to DebugWare patterns.
- Dmitry Vostokov @ DumpAnalysis.org -
Although the first 2 parameters are passed via registers RCX and RDX they are saved on a stack as the part of a function prolog (as can be seen in many examples from my book x64 Windows Debugging: Practical Foundations):
0:000> uf arithmetic
FunctionParameters!arithmetic [c:\dumps\wdpf-x64\functionparameters\arithmetic.cpp @ 2]:
2 00000001`40001020 mov dword ptr [rsp+10h],edx
2 00000001`40001024 mov dword ptr [rsp+8],ecx
2 00000001`40001028 push rdi
3 00000001`40001029 mov eax,dword ptr [rsp+10h]
3 00000001`4000102d mov ecx,dword ptr [rsp+18h]
3 00000001`40001031 add ecx,eax
3 00000001`40001033 mov eax,ecx
3 00000001`40001035 mov dword ptr [rsp+18h],eax
4 00000001`40001039 mov eax,dword ptr [rsp+10h]
4 00000001`4000103d add eax,1
4 00000001`40001040 mov dword ptr [rsp+10h],eax
5 00000001`40001044 mov eax,dword ptr [rsp+10h]
5 00000001`40001048 imul eax,dword ptr [rsp+18h]
5 00000001`4000104d mov dword ptr [rsp+18h],eax
7 00000001`40001051 mov eax,dword ptr [rsp+18h]
8 00000001`40001055 pop rdi
8 00000001`40001056 ret
Notice that RDI is saved too. This helps us later in a more complex case. If we put a breakpoint at arithmetic entry we see that WinDbg is not able to get parameters from RCX and RDX:
0:000> bp 00000001`40001020
0:000> g
ModLoad: 000007fe`ff4d0000 000007fe`ff5d8000 C:\Windows\system32\ADVAPI32.DLL
ModLoad: 000007fe`fef80000 000007fe`ff0c3000 C:\Windows\system32\RPCRT4.dll
Breakpoint 0 hit
FunctionParameters!arithmetic:
00000001`40001020 89542410 mov dword ptr [rsp+10h],edx ss:00000000`0012fe88=cccccccc
0:000> kv
Child-SP RetAddr : Args to Child : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d
This seems correct approach in general because at the time of any other breakpoint in the middle of the code parameter passing registers could be already overwritten, for example, RCX at 0000000140001031. However, as soon as we execute the first two MOV instruction one by one, parameters appear on kv output one by one too:
0:000> t
ModLoad: 000007fe`fd810000 000007fe`fd845000 C:\Windows\system32\apphelp.dll
FunctionParameters!arithmetic+0x4:
00000001`40001024 894c2408 mov dword ptr [rsp+8],ecx ss:00000000`0012fe80=cccccccc
0:000> kv
Child-SP RetAddr : Args to Child : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`cccccccc cccccccc`00000001 cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic+0×4
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d
0:000> t
FunctionParameters!arithmetic+0x8:
00000001`40001028 57 push rdi
0:000> kv
Child-SP RetAddr : Args to Child : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`00000001 cccccccc`00000001 cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic+0×8
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d
Now we come to the more complex example:
1: kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr : Args to Child : Call Site
fffffa60`166a7020 fffffa60`07e9dbf2 : fffffa80`1dbd8820 00000000`00000000 fffffa80`1ec3b7a8 fffffa60`166a7278 : Driver!DeviceWrite+0xae
fffffa60`166a7050 fffffa60`062ae7cb : 00000000`00000000 fffffa80`1dbd8820 fffffa60`166a7340 fffffa80`1df4f520 : Driver!RawWrite+0×8a
[…]
1: kd> r
Last set context:
rax=000000000083a03b rbx=fffffa801ec3b800 rcx=fffffa8018cdc000
rdx=0000000000000004 rsi=fffffa801ec3b9f0 rdi=0000000005040000
rip=fffffa6007ea006e rsp=fffffa60166a7020 rbp=fffffa801ec3b7a8
r8=fffff6fd400f61e0 r9=000000000083a03b r10=fffffa801ec3b9f8
r11=fffffa801ec3b9f8 r12=fffffa801e7c9000 r13=0000000000000000
r14=000000000038011b r15=fffffa8019891670
iopl=0 nv up ei ng nz na po cy
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010287
Driver!DeviceWrite+0xae:
fffffa60`07ea006e mov rax,qword ptr [rdi+10h] ds:002b:00000000`05040010=????????????????
We know that the first parameter to Write function is a pointer to some structure we want to explore because we see from the disassembly that some member from that structure was used ([rcx+320h]) and it was used as a pointer (assigned to RDI) that was trapping ([rdi+10h]):
1: kd> .asm no_code_bytes
Assembly options: no_code_bytes
1: kd> u Driver!DeviceWrite Driver!DeviceWrite+0xae+10
Driver!DeviceWrite:
fffffa60`07e9ffc0 mov qword ptr [rsp+8],rbx
fffffa60`07e9ffc5 mov qword ptr [rsp+10h],rbp
fffffa60`07e9ffca mov qword ptr [rsp+18h],rsi
fffffa60`07e9ffcf push rdi
fffffa60`07e9ffd0 sub rsp,20h
fffffa60`07e9ffd4 mov rdi,qword ptr [rcx+320h]
[…]
fffffa60`07ea006e mov rax,qword ptr [rdi+10h] ; TRAP
[…]
Unfortunately RCX was not saved on the stack and fffffa80`1dbd8820 from kv was just the value of the saved RBX. This can be double-checked by verifying that parameter+320 doesn’t point to RDI value (05040000) at the time of the trap:
1: kd> dq fffffa80`1dbd8820+320 l1
fffffa80`1dbd8b40 00000000`00020000
Looking at DeviceWrite caller we see that RCX was initialized from RDI:
1: kd> ub fffffa60`07e9dbf2
Driver!RawWrite+0x66:
fffffa60`07e9dbce mov rax,qword ptr [rdi+258h]
fffffa60`07e9dbd5 mov qword ptr [rcx-18h],rax
fffffa60`07e9dbd9 mov rax,qword ptr [rdi+260h]
fffffa60`07e9dbe0 mov qword ptr [rcx-20h],rax
fffffa60`07e9dbe4 mov dword ptr [rdx+10h],ebp
fffffa60`07e9dbe7 mov rdx,rsi
fffffa60`07e9dbea mov rcx,rdi
fffffa60`07e9dbed call Driver!DeviceWrite (fffffa60`07e9ffc0)
We also see that RDI was saved at the function prolog so we can get our real first parameter from the raw stack bearing in mind that 0×20 was subtracted from RSP too:
1: kd> dq esp
fffffa60`166a7020 fffffa80`1ec3b800 00000000`05040000 ; SUB RSP, 20H
fffffa60`166a7030 fffffa60`07edc5dd fffffa60`07ee6f8f ;
fffffa60`166a7040 fffffa80`1ec3b520 fffffa60`07e9dbf2 ; Saved RDI - Return Address
fffffa60`166a7050 fffffa80`1dbd8820 00000000`00000000 ; Saved RBX and RBP
fffffa60`166a7060 fffffa80`1ec3b7a8 fffffa60`166a7278 ; Saved RSI
fffffa60`166a7070 fffffa60`166a7250 fffffa60`01ab5180
fffffa60`166a7080 fffffa80`1e2937c8 fffff800`018a928a
fffffa60`166a7090 00000003`0004000d 00000026`0024000d
We see that saved RDI value +320 points to the right expected address:
1: kd> dq fffffa80`1ec3b520+320 l1
fffffa80`1ec3b840 00000000`05040000
Now we can investigate the structure but this is beyond the scope of this post.
- Dmitry Vostokov @ DumpAnalysis.org -
