Software Diagnostics Library

Five volumes of cross-disciplinary Anthology (dubbed by the author “The Summa Memorianica”) lay the foundation of the scientific discipline of Memoretics (study of computer memory snapshots and their evolution in time) that is also called Memory Dump and Software Trace Analysis.ca

The 5th volume contains revised, edited, cross-referenced, and thematically organized selected DumpAnalysis.org blog posts about crash dump, software trace analysis and debugging written in February 2010 - October 2010 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms, technical support and escalation engineers dealing with complex software issues, and security researchers, malware analysts and reverse engineers. The fifth volume features:

- 25 new crash dump analysis patterns
- 11 new pattern interaction case studies (including software tracing)
- 16 new trace analysis patterns
- 7 structural memory patterns
- 4 modeling case studies for memory dump analysis patterns
- Discussion of 3 common analysis mistakes
- Malware analysis case study
- Computer independent architecture of crash analysis report service
- Expanded coverage of software narratology
- Metaphysical and theological implications of memory dump worldview
- More pictures of memory space and physicalist art
- Classification of memory visualization tools
- Memory visualization case studies
- Close reading of the stories of Sherlock Holmes: Dr. Watson’s observational patterns
- Fully cross-referenced with Volume 1, Volume 2, Volume 3, and Volume 4

Product information:

• Title: Memory Dump Analysis Anthology, Volume 5
• Author: Dmitry Vostokov
• Language: English
• Product Dimensions: 22.86 x 15.24

• Paperback: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-96-4

• Hardcover: 400 pages
• Publisher: Opentask (10 December 2010)
• ISBN-13: 978-1-906717-97-1

Back cover features memory space art image Hot Computation: Memory on Fire.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This new pattern is called Blocked Queue and we provide an example of an ALPC port here. If we see an LPC/ALPC wait chain endpoint or just have a message address (and optionally a port address) we can check the port queue length, for example, for a frozen system we have this (WinDbg output was trimmed to save space and paper):

THREAD fffffa8009db7160  Cid 03b0.2ec0  Teb: 000007fffffd5000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
    fffffa8009db7520  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8a00dbc6650 : queued at port fffffa800577ee60 : owned by process fffffa80056ddb30
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8005691b30       Image:         ServiceA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      39742808       Ticks: 3469954 (0:15:02:11.629)
Context Switch Count      9            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×0000000076cd8e70
Stack Init fffff8800bf60db0 Current fffff8800bf60620
Base fffff8800bf61000 Limit fffff8800bf5b000 Call 0
Priority 10 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`0bf60660 fffff800`016de992 nt!KiSwapContext+0×7a
fffff880`0bf607a0 fffff800`016e0cff nt!KiCommitThreadWait+0×1d2
fffff880`0bf60830 fffff800`016f5d1f nt!KeWaitForSingleObject+0×19f
fffff880`0bf608d0 fffff800`019ddac6 nt!AlpcpSignalAndWait+0×8f
fffff880`0bf60980 fffff800`019dba50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`0bf609e0 fffff800`019d8fcb nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`0bf60b00 fffff800`016d6993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`0bf60bb0 00000000`76d105aa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0bf60c20)
00000000`01efe638 000007fe`fec0aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01efe640 000007fe`fecacb64 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`01efe700 000007fe`fecacd55 RPCRT4!NdrpClientCall3+0×244
00000000`01efe9c0 000007fe`fcbf18a1 RPCRT4!NdrClientCall3+0xf2
[…]

0: kd> !alpc /m fffff8a00dbc6650
Message @ fffff8a00dbc6650
  MessageID             : 0x0720 (1824)
  CallbackID            : 0x257C575 (39306613)
  SequenceNumber        : 0x00000002 (2)
  Type                  : LPC_REQUEST
  DataLength            : 0x0044 (68)
  TotalLength           : 0x006C (108)
  Canceled              : No
  Release               : No
  ReplyWaitReply        : No
  Continuation          : Yes
  OwnerPort             : fffffa8006a4bb10 [ALPC_CLIENT_COMMUNICATION_PORT]
  WaitingThread         : fffffa8009db7160
  QueueType             : ALPC_MSGQUEUE_PENDING
  QueuePort             : fffffa800577ee60 [ALPC_CONNECTION_PORT]
  QueuePortOwnerProcess : fffffa80056ddb30 (ServiceB.exe)
  ServerThread          : fffffa8007ead4d0
  QuotaCharged          : No
  CancelQueuePort       : 0000000000000000
  CancelSequencePort    : 0000000000000000
  CancelSequenceNumber  : 0×00000000 (0)
  ClientContext         : 0000000002a60f40
  ServerContext         : 0000000000000000
  PortContext           : 000000000227a370
  CancelPortContext     : 0000000000000000
  SecurityData          : 0000000000000000
  View                  : 0000000000000000

0: kd> !alpc /p fffffa800577ee60
Port @ fffffa800577ee60
  Type                      : ALPC_CONNECTION_PORT
  CommunicationInfo         : fffff8a0022435d0
    ConnectionPort          : fffffa800577ee60
    ClientCommunicationPort : 0000000000000000
    ServerCommunicationPort : 0000000000000000
  OwnerProcess              : fffffa80056ddb30 (ServiceB.exe)
  SequenceNo                : 0×0000481A (18458)
  CompletionPort            : fffffa8005728e80
  CompletionList            : 0000000000000000
  MessageZone               : 0000000000000000
  ConnectionPending         : No
  ConnectionRefused         : No
  Disconnected              : No
  Closed                    : No
  FlushOnClose              : Yes
  ReturnExtendedInfo        : No
  Waitable                  : No
  Security                  : Static
  Wow64CompletionList       : No

  Main queue is empty.

  Large message queue is empty.

  Pending queue has 698 message(s)

    fffff8a002355aa0 00000404 0000000000001344:0000000000001358 0000000000000000 fffffa8004c0cb60 LPC_REQUEST
    fffff8a00a52f030 00000644 0000000000001078:00000000000024c0 0000000000000000 fffffa80072f1b60 LPC_REQUEST
    fffff8a00abb5030 000007a8 000000000000103c:000000000000050c 0000000000000000 fffffa800725b580 LPC_REQUEST
    fffff8a00239cab0 000000b8 0000000000000480:00000000000015f8 0000000000000000 fffffa80077f0b60 LPC_REQUEST
    fffff8a00ac81a90 00000a18 00000000000028ac:0000000000001e54 0000000000000000 fffffa8007fba060 LPC_CANCELED
    fffff8a005879140 00000f80 0000000000001260:0000000000000730 fffffa8006432060 fffffa8006b18060 LPC_REQUEST
    fffff8a013720d00 00000c6c 0000000000003764:00000000000032a8 0000000000000000 fffffa8006b00a60 LPC_CANCELED
    fffff8a00ac82660 00000810 0000000000003af4:0000000000002a98 0000000000000000 fffffa80068c0b60 LPC_CANCELED
    fffff8a00bdeca50 00000ec8 000000000000233c:00000000000013f8 0000000000000000 fffffa80079455b0 LPC_CANCELED
    fffff8a00b662830 000005cc 00000000000005e4:0000000000000e0c fffffa800791a7a0 fffffa8007376580 LPC_REQUEST
    fffff8a003d57150 00000f08 0000000000002678:0000000000003e0c 0000000000000000 fffffa8007e4a870 LPC_CANCELED
    fffff8a00cd08830 00000750 0000000000003408:0000000000003adc 0000000000000000 fffffa8008631b60 LPC_CANCELED
    fffff8a01855b2f0 000004f4 0000000000002c74:0000000000002d00 0000000000000000 fffffa800746b890 LPC_CANCELED
    fffff8a00da0d0b0 00000db0 0000000000001a34:0000000000002d80 0000000000000000 fffffa800aff4b60 LPC_CANCELED
    fffff8a00eddb030 0000059c 0000000000003f34:0000000000003c8c 0000000000000000 fffffa8008f96060 LPC_CANCELED
    fffff8a017a14d00 00000920 0000000000003850:0000000000002588 0000000000000000 fffffa8009f66060 LPC_CANCELED
    fffff8a01792d030 000007f8 0000000000003844:00000000000028d0 0000000000000000 fffffa800ad56260 LPC_CANCELED
    fffff8a00f8d6ae0 00000f30 000000000000239c:0000000000001694 0000000000000000 fffffa8008b86060 LPC_CANCELED
    fffff8a01395ab80 00000cdc 0000000000003630:00000000000018f8 0000000000000000 fffffa8005bc0770 LPC_CANCELED
    fffff8a0166ff800 00000984 00000000000005e4:00000000000025f4 fffffa8009718910 fffffa8008cbfb60 LPC_REQUEST
    fffff8a012b9f5a0 00000ac8 0000000000002d34:0000000000001b24 0000000000000000 fffffa8009cd8410 LPC_CANCELED
    fffff8a014313830 00000afc 00000000000005e4:00000000000023bc fffffa80073f0230 fffffa80054d7060 LPC_REQUEST
    fffff8a00a34a6b0 00000ca8 0000000000002534:0000000000002dd0 0000000000000000 fffffa80064c3980 LPC_CANCELED
[...]
    fffff8a00ad8f610 00000e64 0000000000003714:00000000000030b8 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a015720710 00001594 0000000000003638:00000000000029b8 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a009bac560 00001508 0000000000003994:0000000000001aac 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a00b6e78f0 00001574 0000000000002938:0000000000001998 0000000000000000 fffffa800aeea9f0 LPC_REQUEST
    fffff8a00b5716b0 00001570 0000000000002938:0000000000001698 0000000000000000 fffffa800a3b8620 LPC_REQUEST
    fffff8a018531d00 00000db8 00000000000016d8:00000000000031c4 0000000000000000 fffffa800b5359a0 LPC_REQUEST
    fffff8a01112f410 000014b0 0000000000001b6c:0000000000001618 0000000000000000 fffffa800a3b8620 LPC_CANCELED

  Canceled queue is empty.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’m pleased to announce that MDAA, Volume 4 is available in PDF format:

www.dumpanalysis.org/Memory+Dump+Analysis+Anthology+Volume+4

It features:

- 15 new crash dump analysis patterns
- 13 new pattern interaction case studies
- 10 new trace analysis patterns
- 6 new Debugware patterns and case study
- Workaround patterns
- Updated checklist
- Fully cross-referenced with Volume 1, Volume 2 and Volume 3
- Memory visualization tutorials
- Memory space art

Its table of contents is available here:

http://www.dumpanalysis.org/MDAA/MDA-Anthology-V4-TOC.pdf

Paperback and hardcover versions should be available in a week or two. I also started working on Volume 5 that should be available in December.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The previously published script to dump raw stack of all threads dumps only 64-bit raw stack from 64-bit WOW64 process memory dumps (a 32-bit process saved in a 64-bit dump). In order to dump WOW64 32-bit raw stack from such 64-bit dumps we need another script. After I found a location of 32-bit TEB pointers (WOW64 TEB32) inside a 64-bit TEB structure I was able to create such a script:

0:000> .load wow64exts

0:000> !teb
Wow64 TEB32 at 000000007efdd000

Wow64 TEB at 000000007efdb000
    ExceptionList:        000000007efdd000
    StackBase:            000000000008fd20
    StackLimit:           0000000000082000
    SubSystemTib:         0000000000000000
    FiberData:            0000000000001e00
    ArbitraryUserPointer: 0000000000000000
    Self:                 000000007efdb000
    EnvironmentPointer:   0000000000000000
    ClientId:             0000000000000f34 . 0000000000000290
    RpcHandle:            0000000000000000
    Tls Storage:          0000000000000000
    PEB Address:          000000007efdf000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

0:000:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0x7efdd000
TEB64: 0x7efdb000

32 bit, StackBase   : 0×1a0000
        StackLimit  : 0×190000
        Deallocation: 0xa0000

64 bit, StackBase   : 0x8fd20
        StackLimit  : 0x82000
        Deallocation: 0x50000

[...]

0:000:x86> dd 000000007efdd000 L4
7efdd000  0019fa84 001a0000 00190000 00000000

So the script obviously should be this:

~*e r? $t1 = ((ntdll!_NT_TIB *)@$teb)->ExceptionList; !wow64exts.info; dds poi(@$t1+8) poi(@$t1+4)

Before running it against a freshly opened user dump we need to execute the following commands first after setting symbols right:

.load wow64exts;
.effmach x86

I’ve also created a page to put all such scripts together:

Raw Stack Analysis Scripts 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The choice of webinars below mostly reflects my personal preferences and long time desire to speak on topics like systems thinking, troubleshooting tool design and development. The other topic about BSOD minidump analysis was requested by participants in an ongoing survey. There will be more topics in 2011. All forthcoming webinars will be hosted by Memory Dump Analysis Services. The planning list includes:

Systems Thinking in Memory Dump and Software Trace Analysis

Software Troubleshooting and Debugging Tools: Objects, Components, Patterns and Frameworks with UML

• UML basics
• DebugWare patterns
• Unified Troubleshooting Framework
• RADII software development process
• Hands-on exercise: designing and building a tool

Blue Screen Of Death Analysis Done Right: Minidump Investigation for System Administrators

• Making sense of !analyze –v output
• Get extra troubleshooting information with additional WinDbg commands
• Guessing culprits with raw stack analysis
• Who’s responsible: hardware or software?
• Checklist and patterns
• Including hands-on exercises: send your own minidumps

More detailed information will be available soon. 

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This forthcoming full color book is the complete transcript of a Webinar organized by Memory Dump Analysis Services (www.DumpAnalysis.com).

It discusses user vs. kernel vs. physical (complete) memory space, challenges of complete memory dump analysis, common WinDbg commands, patterns and pattern-driven analysis methodology, common mistakes, fiber bundles, DumpAnalysis.org case studies and illustrates step by step a hands-on exercise in a complete memory dump analysis.

• Title: Fundamentals of Complete Crash and Hang Memory Dump Analysis
• Author: Dmitry Vostokov
• Publisher: OpenTask (October 2010)
• Language: English
• Product Dimensions: 28.0 x 21.6
• Paperback: 48 pages
• ISBN-13: 978-1906717155

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Memory Dump Analysis Services (DumpAnalysis.com) organizes a free webinar

Date: 18th of August 2010
Time: 21:00 (BST) 16:00 (Eastern) 13:00 (Pacific)
Duration: 90 minutes

Topics include:

- User vs. kernel vs. physical (complete) memory space
- Challenges of complete memory dump analysis
- Common WinDbg commands
- Patterns
- Common mistakes
- Fiber bundles
- Hands-on exercise: a complete memory dump analysis
- A guide to DumpAnalysis.org case studies

Prerequisites: working knowledge of basic user process and kernel memory dump analysis or live debugging using WinDbg 

The webinar link will be posted before 18th of August on DumpAnalysis.com

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Our future sponsor has been registered in Ireland and has its own independent website and logo: DumpAnalysis.com

More information will be available later this month.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

I’d always seen the empty field Attached Process in !thread command output:

1: kd> !thread fffffa802c2cfbb0 ff
THREAD fffffa802c2cfbb0  Cid 43b8.470c  Teb: 000007fffffda000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
    fffffa802acfc970  QueueObject
    fffffa802c2cfc68  NotificationTimer
Not impersonating
DeviceMap                 fffff88000008e00
Owning Process            fffffa802af8ac10       Image:         winlogon.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      428658         Ticks: 3 (0:00:00:00.046)
Context Switch Count      4            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address RPCRT4!ThreadStartRoutine (0×000007fefea07780)
Stack Init fffffa6029203db0 Current fffffa60292037e0
Base fffffa6029204000 Limit fffffa60291fe000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`29203820 fffff800`01a6b9fa nt!KiSwapContext+0×7f
fffffa60`29203960 fffff800`01a6ee94 nt!KiSwapThread+0×13a
fffffa60`292039d0 fffff800`01cd1cd7 nt!KeRemoveQueueEx+0×4b4
fffffa60`29203a80 fffff800`01ca8b2d nt!IoRemoveIoCompletion+0×47
fffffa60`29203b00 fffff800`01a69233 nt!NtRemoveIoCompletion+0×13d
fffffa60`29203bb0 00000000`778c6daa nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`29203c20)
00000000`017ff9f8 00000000`7769f65c ntdll!NtRemoveIoCompletion+0xa
00000000`017ffa00 000007fe`fea25d0d kernel32!GetQueuedCompletionStatus+0×48
00000000`017ffa60 000007fe`fea25b93 RPCRT4!COMMON_ProcessCalls+0×7d
00000000`017ffaf0 000007fe`fea07769 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0×133
00000000`017ffba0 000007fe`fea07714 RPCRT4!ProcessIOEventsWrapper+0×9
00000000`017ffbd0 000007fe`fea077a4 RPCRT4!BaseCachedThreadRoutine+0×94
00000000`017ffc10 00000000`7769be3d RPCRT4!ThreadStartRoutine+0×24
00000000`017ffc40 00000000`778a6a51 kernel32!BaseThreadInitThunk+0xd
00000000`017ffc70 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Until recently I got this stack trace from winlogon.exe deep in win32k.sys. Because csrss.exe is a session-specific user-space counterpart to win32k.sys it make sense to see it attached:

1: kd> !thread fffffa802b2e6bb0 ff
THREAD fffffa802b2e6bb0  Cid 43b8.74d0  Teb: 000007fffffdc000 Win32Thread: fffff900c0016690 RUNNING on processor 1
Not impersonating
DeviceMap                 fffff88000008e00
Owning Process            fffffa802af8ac10       Image:         winlogon.exe
Attached Process          fffffa80174d7040       Image:         csrss.exe
Wait Start TickCount      428661         Ticks: 0
Context Switch Count      212                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.031
Win32 Start Address 0×00000000ff860260
Stack Init fffffa60294c15f0 Current fffffa60294c0ec0
Base fffffa60294c3000 Limit fffffa60294b9000 Call fffffa60294c1840
Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`294c0340 fffff800`01a77197 nt!MiAllocatePagedPoolPages+0×69d
fffffa60`294c0410 fffff800`01b49f07 nt!ExpAllocateBigPool+0xa7
fffffa60`294c04f0 fffff960`00082f28 nt!ExAllocatePoolWithTag+0×767
fffffa60`294c05c0 fffff960`00094863 win32k!EngAllocMem+0×3c
fffffa60`294c05f0 fffff960`00094749 win32k!ttfdOpenFontContextInternal+0xbf
fffffa60`294c0630 fffff960`000976d9 win32k!ttfdOpenFontContext+0×1d
fffffa60`294c0670 fffff960`0009762c win32k!ttfdQueryFontData+0×49
fffffa60`294c06c0 fffff960`0008c335 win32k!ttfdSemQueryFontData+0×7c
fffffa60`294c0720 fffff960`0008989a win32k!PDEVOBJ::QueryFontData+0×79
fffffa60`294c0780 fffff960`0008bacb win32k!RFONTOBJ::bGetDEVICEMETRICS+0×6a
fffffa60`294c07d0 fffff960`0004d0e1 win32k!RFONTOBJ::bRealizeFont+0×2df
fffffa60`294c08f0 fffff960`0004caa5 win32k!RFONTOBJ::vInit+0×379
fffffa60`294c0bb0 fffff960`00048fdd win32k!RFONTOBJ::vInitEUDC+0×5e5
fffffa60`294c0d80 fffff960`0008c516 win32k!RFONTOBJ::wpgdGetLinkMetricsPlus+0×33d
fffffa60`294c0e00 fffff960`0009b1b2 win32k!RFONTOBJ::bGetGlyphMetrics+0×1b6
fffffa60`294c0e80 fffff960`00082699 win32k!RFONTOBJ::bGetWidthTable+0×262
fffffa60`294c1080 fffff960`00082395 win32k!iGetPublicWidthTable+0×28d
fffffa60`294c1430 fffff800`01a69233 win32k!NtGdiSetupPublicCFONT+0×25
fffffa60`294c1460 000007fe`fe23c55a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`294c1460)

1: kd> !thread fffffa802bd9f060 ff
THREAD fffffa802bd9f060  Cid 7624.28b8  Teb: 000007fffffdd000 Win32Thread: fffff900c0016690 RUNNING on processor 0
Not impersonating
DeviceMap                 fffff88000008e00
Owning Process            fffffa802b18d040       Image:         winlogon.exe
Attached Process          fffffa802ad2fc10       Image:         csrss.exe
Wait Start TickCount      428661         Ticks: 0
Context Switch Count      196                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.046
Win32 Start Address 0×00000000ff860260
Stack Init fffffa60296b3db0 Current fffffa60296b1980
Base fffffa60296b4000 Limit fffffa60296aa000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffffa60`296b22b8 fffff960`0009022a win32k!itrp_CINDEX+0×4f
fffffa60`296b22c0 fffff960`00092817 win32k!itrp_InnerExecute+0×36
fffffa60`296b22f0 fffff960`0009022a win32k!itrp_CALL+0×26f
fffffa60`296b2360 fffff960`00092817 win32k!itrp_InnerExecute+0×36
fffffa60`296b2390 fffff960`0009022a win32k!itrp_CALL+0×26f
fffffa60`296b2400 fffff960`0009a6e3 win32k!itrp_InnerExecute+0×36
fffffa60`296b2430 fffff960`00099720 win32k!itrp_Execute+0×384
fffffa60`296b2540 fffff960`0009968e win32k!itrp_ExecutePrePgm+0×78
fffffa60`296b2590 fffff960`00096da6 win32k!fsg_RunPreProgram+0×222
fffffa60`296b25e0 fffff960`00096af4 win32k!fs__Contour+0×256
fffffa60`296b26a0 fffff960`0009796c win32k!bGetGlyphOutline+0×125
fffffa60`296b26d0 fffff960`000978d8 win32k!lGetGlyphBitmap+0×4c
fffffa60`296b2890 fffff960`0009762c win32k!ttfdQueryFontData+0×248
fffffa60`296b28e0 fffff960`0008c335 win32k!ttfdSemQueryFontData+0×7c
fffffa60`296b2940 fffff960`0008c213 win32k!PDEVOBJ::QueryFontData+0×79
fffffa60`296b29a0 fffff960`0008bf4f win32k!RFONTOBJ::bInitCache+0×15f
fffffa60`296b2a40 fffff960`00086337 win32k!RFONTOBJ::bRealizeFont+0×763
fffffa60`296b2b60 fffff960`0008aac8 win32k!RFONTOBJ::bInit+0×523
fffffa60`296b2c70 fffff960`00037597 win32k!GreGetTextMetricsW+0×48
fffffa60`296b2cb0 fffff960`00038a42 win32k!GetTextMetricsW+0×17
fffffa60`296b2d30 fffff960`00047951 win32k!GetCharDimensions+0×26
fffffa60`296b2db0 fffff960`00049aae win32k!xxxSetNCFonts+0×181
fffffa60`296b2e60 fffff960`0005be45 win32k!xxxSetWindowNCMetrics+0×3e
fffffa60`296b30e0 fffff960`000768bf win32k!xxxInitWindowStation+0xa1
fffffa60`296b3140 fffff960`00077daf win32k!xxxCreateWindowStation+0×1cf
fffffa60`296b3500 fffff800`01a69233 win32k!NtUserCreateWindowStation+0×4b3
fffffa60`296b3bb0 00000000`777b1a6a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`296b3c20)
00000000`000ff338 00000000`00000000 USER32!NtUserCreateWindowStation+0xa

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Although the first 2 parameters are passed via registers RCX and RDX they are saved on a stack as the part of a function prolog (as can be seen in many examples from my book x64 Windows Debugging: Practical Foundations):

0:000> uf arithmetic
FunctionParameters!arithmetic [c:\dumps\wdpf-x64\functionparameters\arithmetic.cpp @ 2]:
    2 00000001`40001020 mov     dword ptr [rsp+10h],edx
    2 00000001`40001024 mov     dword ptr [rsp+8],ecx
    2 00000001`40001028 push    rdi
    3 00000001`40001029 mov     eax,dword ptr [rsp+10h]
    3 00000001`4000102d mov     ecx,dword ptr [rsp+18h]
    3 00000001`40001031 add     ecx,eax
    3 00000001`40001033 mov     eax,ecx
    3 00000001`40001035 mov     dword ptr [rsp+18h],eax
    4 00000001`40001039 mov     eax,dword ptr [rsp+10h]
    4 00000001`4000103d add     eax,1
    4 00000001`40001040 mov     dword ptr [rsp+10h],eax
    5 00000001`40001044 mov     eax,dword ptr [rsp+10h]
    5 00000001`40001048 imul    eax,dword ptr [rsp+18h]
    5 00000001`4000104d mov     dword ptr [rsp+18h],eax
    7 00000001`40001051 mov     eax,dword ptr [rsp+18h]
    8 00000001`40001055 pop     rdi
    8 00000001`40001056 ret

Notice that RDI is saved too. This helps us later in a more complex case. If we put a breakpoint at arithmetic entry we see that WinDbg is not able to get parameters from RCX and RDX:

0:000> bp 00000001`40001020

0:000> g
ModLoad: 000007fe`ff4d0000 000007fe`ff5d8000   C:\Windows\system32\ADVAPI32.DLL
ModLoad: 000007fe`fef80000 000007fe`ff0c3000   C:\Windows\system32\RPCRT4.dll
Breakpoint 0 hit
FunctionParameters!arithmetic:
00000001`40001020 89542410        mov     dword ptr [rsp+10h],edx ss:00000000`0012fe88=cccccccc

0:000> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d

This seems correct approach in general because at the time of any other breakpoint in the middle of the code parameter passing registers could be already overwritten, for example, RCX at 0000000140001031. However, as soon as we execute the first two MOV instruction one by one, parameters appear on kv output one by one too:

0:000> t
ModLoad: 000007fe`fd810000 000007fe`fd845000   C:\Windows\system32\apphelp.dll
FunctionParameters!arithmetic+0x4:
00000001`40001024 894c2408        mov     dword ptr [rsp+8],ecx ss:00000000`0012fe80=cccccccc

0:000> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`cccccccc cccccccc`00000001 cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic+0×4
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d

0:000> t
FunctionParameters!arithmetic+0x8:
00000001`40001028 57              push    rdi

0:000> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0012fe78 00000001`400010a5 : cccccccc`00000001 cccccccc`00000001 cccccccc`cccccccc cccccccc`cccccccc : FunctionParameters!arithmetic+0×8
00000000`0012fe80 00000001`4000137c : 00000000`00000001 00000000`00282960 00000000`00000000 00000000`00000000 : FunctionParameters!main+0×35
00000000`0012fec0 00000001`4000114e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!__tmainCRTStartup+0×21c
00000000`0012ff30 00000000`7776be3d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : FunctionParameters!mainCRTStartup+0xe
00000000`0012ff60 00000000`778a6a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0×1d

Now we come to the more complex example:

1: kd> kv
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffffa60`166a7020 fffffa60`07e9dbf2 : fffffa80`1dbd8820 00000000`00000000 fffffa80`1ec3b7a8 fffffa60`166a7278 : Driver!DeviceWrite+0xae
fffffa60`166a7050 fffffa60`062ae7cb : 00000000`00000000 fffffa80`1dbd8820 fffffa60`166a7340 fffffa80`1df4f520 : Driver!RawWrite+0×8a
[…]

1: kd> r
Last set context:
rax=000000000083a03b rbx=fffffa801ec3b800 rcx=fffffa8018cdc000
rdx=0000000000000004 rsi=fffffa801ec3b9f0 rdi=0000000005040000
rip=fffffa6007ea006e rsp=fffffa60166a7020 rbp=fffffa801ec3b7a8
 r8=fffff6fd400f61e0  r9=000000000083a03b r10=fffffa801ec3b9f8
r11=fffffa801ec3b9f8 r12=fffffa801e7c9000 r13=0000000000000000
r14=000000000038011b r15=fffffa8019891670
iopl=0         nv up ei ng nz na po cy
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010287
Driver!DeviceWrite+0xae:
fffffa60`07ea006e mov     rax,qword ptr [rdi+10h] ds:002b:00000000`05040010=????????????????

We know that the first parameter to Write function is a pointer to some structure we want to explore because we see from the disassembly that some member from that structure was used ([rcx+320h]) and it was used as a pointer (assigned to RDI) that was trapping ([rdi+10h]):

1: kd> .asm no_code_bytes
Assembly options: no_code_bytes

1: kd> u Driver!DeviceWrite Driver!DeviceWrite+0xae+10
Driver!DeviceWrite:
fffffa60`07e9ffc0 mov     qword ptr [rsp+8],rbx
fffffa60`07e9ffc5 mov     qword ptr [rsp+10h],rbp
fffffa60`07e9ffca mov     qword ptr [rsp+18h],rsi
fffffa60`07e9ffcf push    rdi
fffffa60`07e9ffd0 sub     rsp,20h
fffffa60`07e9ffd4 mov     rdi,qword ptr [rcx+320h]
[…]
fffffa60`07ea006e mov     rax,qword ptr [rdi+10h] ; TRAP
[…]

Unfortunately RCX was not saved on the stack and fffffa80`1dbd8820 from kv was just the value of the saved RBX. This can be double-checked by verifying that parameter+320 doesn’t point to RDI value (05040000) at the time of the trap:

1: kd> dq fffffa80`1dbd8820+320 l1
fffffa80`1dbd8b40  00000000`00020000

Looking at DeviceWrite caller we see that RCX was initialized from RDI:

1: kd> ub fffffa60`07e9dbf2
Driver!RawWrite+0x66:
fffffa60`07e9dbce mov     rax,qword ptr [rdi+258h]
fffffa60`07e9dbd5 mov     qword ptr [rcx-18h],rax
fffffa60`07e9dbd9 mov     rax,qword ptr [rdi+260h]
fffffa60`07e9dbe0 mov     qword ptr [rcx-20h],rax
fffffa60`07e9dbe4 mov     dword ptr [rdx+10h],ebp
fffffa60`07e9dbe7 mov     rdx,rsi
fffffa60`07e9dbea mov     rcx,rdi
fffffa60`07e9dbed call    Driver!DeviceWrite (fffffa60`07e9ffc0)

We also see that RDI was saved at the function prolog so we can get our real first parameter from the raw stack bearing in mind that 0×20 was subtracted from RSP too:

1: kd> dq esp
fffffa60`166a7020  fffffa80`1ec3b800 00000000`05040000 ; SUB RSP, 20H
fffffa60`166a7030  fffffa60`07edc5dd fffffa60`07ee6f8f ;
fffffa60`166a7040  fffffa80`1ec3b520 fffffa60`07e9dbf2 ; Saved RDI - Return Address
fffffa60`166a7050  fffffa80`1dbd8820 00000000`00000000 ; Saved RBX and RBP
fffffa60`166a7060  fffffa80`1ec3b7a8 fffffa60`166a7278 ; Saved RSI 
fffffa60`166a7070  fffffa60`166a7250 fffffa60`01ab5180
fffffa60`166a7080  fffffa80`1e2937c8 fffff800`018a928a
fffffa60`166a7090  00000003`0004000d 00000026`0024000d

We see that saved RDI value +320 points to the right expected address:

1: kd> dq fffffa80`1ec3b520+320 l1
fffffa80`1ec3b840  00000000`05040000

Now we can investigate the structure but this is beyond the scope of this post. 

- Dmitry Vostokov @ DumpAnalysis.org -