Archive for the ‘Kernel Development’ Category

Windows® Debugging Notebook

Friday, April 25th, 2008

This is the next scheduled book from Crash Dump Analysis Publishing Roadmap:

  • Title: Windows® Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
  • Authors: Roberto Alexis Farah, Dmitry Vostokov
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • ISBN-13: 978-1-906717-00-1
  • Publisher: Opentask (1 December 2009)
  • Paperback: 256 pages
  • ISBN-13: 978-0-9558328-5-7
  • Publisher: Opentask (1 February 2010)
  • Hardcover (Cloth): 256 pages

Draft Table of Contents will be published next month together with a sample chapter.

- Dmitry Vostokov @ DumpAnalysis.org -

MDAA Volume One Goes Digital

Friday, April 25th, 2008

Due to demand from people that prefer ebooks I published Memory Dump Analysis Anthology, Volume 1 in a digital format that can be purchased in Crash Dump Analysis Store. This format has color pictures inside.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugcheck Callbacks

Wednesday, April 23rd, 2008

There are some improvements in Vista and Windows Server 2008 regarding various WER callbacks to write user-defined data in the case of application crashes and hangs. See MSDN documentation:

What’s New in WER

However I have found that many engineers are not aware that the similar mechanism exists in kernel for many years:

Writing a Bug Check Callback Routine

You can check this data using !bugdump and .enumtag WinDbg commands:

0: kd> !bugdump
**** Dump of Bug Check Data ****
8526ba7c: Bug check callback record could not be read

We get “could not be read” message probably because for systems newer than Windows XP SP1 !bugdump command shows callback data written to memory after the crash dump was saved. So it is useful for live debugging only. However we can see that bugcheck callbacks form a linked list:

0: kd> dps 8526ba7c
8526ba7c  849eca7c
8526ba80  81b36ce0 nt!KeBugCheckCallbackListHead
8526ba84  858a7dea ndis!ndisBugcheckHandler
8526ba88  8526b438
8526ba8c  00000b28
8526ba90  8594dd76 ndis! ?? ::LNCPHCLB::`string’
8526ba94  90461ac0
8526ba98  00000001
8526ba9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa4  85969274 ndis!ethFilterDprIndicateReceivePacket
8526baa8  8de66c5c bthpan!MpReturnPacket
8526baac  8526ea80
8526bab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
8526bab4  8526b438
8526bab8  00000000

0: kd> !list -x "dps @$extret l10" 81b36ce0
81b36ce0  8526ba7c
81b36ce4  81ddbe40 hal!HalpCallbackRecord
81b36ce8  00000000
81b36cec  00000001
81b36cf0  00000000
81b36cf4  00000000
81b36cf8  00000101
81b36cfc  00000001
81b36d00  00000000
81b36d04  00000000
81b36d08  00000000
81b36d0c  00000000
81b36d10  00000000
81b36d14  00000000
81b36d18  00000000
81b36d1c  00000000

8526ba7c  849eca7c
8526ba80  81b36ce0 nt!KeBugCheckCallbackListHead
8526ba84  858a7dea ndis!ndisBugcheckHandler
8526ba88  8526b438
8526ba8c  00000b28
8526ba90  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8526ba94  90461ac0
8526ba98  00000001
8526ba9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8526baa4  85969274 ndis!ethFilterDprIndicateReceivePacket
8526baa8  8de66c5c bthpan!MpReturnPacket
8526baac  8526ea80
8526bab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
8526bab4  8526b438
8526bab8  00000000

849eca7c  849ea72c
849eca80  8526ba7c
849eca84  858a7dea ndis!ndisBugcheckHandler
849eca88  849ec438
849eca8c  00000b28
849eca90  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849eca94  8fbe2ac0
849eca98  00000001
849eca9c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ecaa0  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ecaa4  859432ca ndis!ndisMIndicatePacket
849ecaa8  00000000
849ecaac  00000000
849ecab0  859495ef ndis!ndisSynchReturnPacketsForTranslation
849ecab4  849ec438
849ecab8  00000000

849ea72c  849c272c
849ea730  849eca7c
849ea734  858a7dea ndis!ndisBugcheckHandler
849ea738  849ea0e8
849ea73c  00000b28
849ea740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849ea744  8fbe0770
849ea748  00000001
849ea74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ea750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849ea754  85969274 ndis!ethFilterDprIndicateReceivePacket
849ea758  00000000
849ea75c  00000000
849ea760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849ea764  849ea0e8
849ea768  00000000

849c272c  849c172c
849c2730  849ea72c
849c2734  858a7dea ndis!ndisBugcheckHandler
849c2738  849c20e8
849c273c  00000b28
849c2740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849c2744  8fbb8770
849c2748  00000001
849c274c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c2750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c2754  85969274 ndis!ethFilterDprIndicateReceivePacket
849c2758  85df579a tunmp!TunMpReturnPacket
849c275c  84a45538
849c2760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849c2764  849c20e8
849c2768  00000000

849c172c  849a072c
849c1730  849c272c
849c1734  858a7dea ndis!ndisBugcheckHandler
849c1738  849c10e8
849c173c  00000b28
849c1740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849c1744  8fbb7770
849c1748  00000001
849c174c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c1750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849c1754  859432ca ndis!ndisMIndicatePacket
849c1758  00000000
849c175c  00000000
849c1760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849c1764  849c10e8
849c1768  00000000

849a072c  8499d72c
849a0730  849c172c
849a0734  858a7dea ndis!ndisBugcheckHandler
849a0738  849a00e8
849a073c  00000b28
849a0740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
849a0744  8fb96770
849a0748  00000001
849a074c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849a0750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
849a0754  859432ca ndis!ndisMIndicatePacket
849a0758  00000000
849a075c  00000000
849a0760  859495ef ndis!ndisSynchReturnPacketsForTranslation
849a0764  849a00e8
849a0768  00000000

8499d72c  8499f72c
8499d730  849a072c
8499d734  858a7dea ndis!ndisBugcheckHandler
8499d738  8499d0e8
8499d73c  00000b28
8499d740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8499d744  8fb93770
8499d748  00000001
8499d74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499d750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499d754  859432ca ndis!ndisMIndicatePacket
8499d758  00000000
8499d75c  00000000
8499d760  859495ef ndis!ndisSynchReturnPacketsForTranslation
8499d764  8499d0e8
8499d768  00000000

8499f72c  81ddbe40 hal!HalpCallbackRecord
8499f730  8499d72c
8499f734  858a7dea ndis!ndisBugcheckHandler
8499f738  8499f0e8
8499f73c  00000b28
8499f740  8594dd76 ndis! ?? ::LNCPHCLB::`string'
8499f744  8fb95770
8499f748  00000001
8499f74c  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499f750  85936767 ndis!ndisMDispatchReceiveNetBufferLists
8499f754  859432ca ndis!ndisMIndicatePacket
8499f758  00000000
8499f75c  00000000
8499f760  859495ef ndis!ndisSynchReturnPacketsForTranslation
8499f764  8499f0e8
8499f768  00000000

81ddbe40  81b36ce0 nt!KeBugCheckCallbackListHead
81ddbe44  8499f72c
81ddbe48  81dcebdc hal!HalpBugCheckCallback
81ddbe4c  00000000
81ddbe50  00000000
81ddbe54  81dc2550 hal!HalName
81ddbe58  03b9112c
81ddbe5c  00000001
81ddbe60  00000000
81ddbe64  00000000
81ddbe68  00000000
81ddbe6c  00000000
81ddbe70  6d46da80
81ddbe74  00000000
81ddbe78  00000000
81ddbe7c  00000000

Another WinDbg command .enumtag shows data written before saving a crash dump and therefore useful for postmortem crash dump analysis (binary output is removed for visual clarity):

0: kd> .enumtag
{BC5C008F-1E3A-44D7-988D86F6884C6758} - 0x5cd bytes
  ...$............
  ................
  Apple Inc..    M
  M21.88Z.009A.B00
  .0706281359.06/2
  8/07............
  ................
  .Apple Inc..Macm
  ini2,1.1.0.    
        .System SK
  UNumber.Napa Mac
  ................
  ..Apple Inc..Mac
  -F4208EAA.PVT. .
  .Part Compon
  ent.............
  ..........Apple
  Inc..Mac-F4208EA
  A.           . 
  ............J6H1
  :1-X CMOS CLEAR(
  default); J8H1:1
  -X BIOS RECOVERY
  ...........None.
  Ethernet........
  ...None.DVI.....
  ......None.USB0.
  ..........None.U
  SB1...........No
  ne.USB2.........
  ..None.USB3.....
  ....!.None.FireW
  ire0...........N
  one.Audio Line I
  n...........None
  .Audio Line Out.
  ..............Ai
  rPort........Int
  egrated Graphics
  Controller ....
  ....Yukon Ethern
  et Controller...
  .....Azalia Audi
  o Codec........S
  ATA........PATA.
  ..........#.....
  .............&.&
  .A..........Inte
  l(R) Core(TM)2 C
  PU         T.Int
  el(R) Corporatio
  n.U2E1.       ..
[...]
  .......Intel(R)
  Core(TM)2 CPU  
       T.Intel(R)
  Corporation.U2E
  1.       .......
[...]
  ...........DIMM0
  .BANK 0.0x2C0000
  0000000000.    
      .       .0x
  3848544636343634
  4844592D36363744
  3320....!.......
  .. .$........"..
  ...@.@..........
  ......DIMM1.BANK
  1.0x2C000000000
  00000.         
  .       .0x38485
  4463634363448445
  92D363637443320.
[...]
{6C7AC389-4313-47DC-9F34A8800A0FB56C} - 0x266 bytes
  ....~.M.H.z.....
  ......)...,...C.
  o.m.p.o.n.e.n.t.
  .I.n.f.o.r.m.a.
  t.i.o.n.........
  ..&...C.o.n.f.i.
  g.u.r.a.t.i.o.n.
  .D.a.t.a.......
  ........I.d.e.n.
  t.i.f.i.e.r.....
  ..B...x.8.6. .F.
  a.m.i.l.y. .6. .
  M.o.d.e.l. .1.5.
  .S.t.e.p.p.i.n.
  g. .2...(...P.r.
  o.c.e.s.s.o.r.N.
  a.m.e.S.t.r.i.n.
  g.......`...I.n.
  t.e.l.(.R.). .C.
  o.r.e.(.T.M.).2.
  .C.P.U. . . . .
  . . . . .T.5.6.
  0.0. . .@. .1...
  8.3.G.H.z..."...
  U.p.d.a.t.e. .S.
  i.g.n.a.t.u.r.e.
  ..............W.
  ......U.p.d.a.t.
  e. .S.t.a.t.u.s.
  ..............".
  ..V.e.n.d.o.r.I.
  d.e.n.t.i.f.i.e.
  r...........G.e.
  n.u.i.n.e.I.n.t.
  e.l.......M.S.R.
[...]
{D03DC06F-D88E-44C5-BA2AFAE035172D19} - 0x438 bytes
  ............Genu
  ntelineI....Genu
  ntelineI........
[...]
  ........Intel(R)
  Core(TMIntel(R)
  Core(TM........
  )2 CPU         T
  )2 CPU         T
  ........5600  @
  1.83GHz.5600  @
  1.83GHz.........
[...]
{E83B40D2-B0A0-4842-ABEA71C9E3463DD1} - 0x184 bytes
  APICh.....APPLE
  Apple00.....Loki
  _.......FACP....
  .aAPPLE Apple00.
  ....Loki_......>
  HPET8.....APPLE
  Apple00.....Loki
  _.......MCFG<...
  ..APPLE Apple00.
  ....Loki_.......
  ASF!.... .APPLE
  Apple00.....Loki
  _.......SBST0...
  ..APPLE Apple00.
  ....Loki_.......
  ECDTS....9APPLE
  Apple00.....Loki
  _.......SSDTO...
  .>APPLE SataPri.
  ....INTL... SSDT
  O....>APPLE Sata
  Pri.....INTL...
  SSDTO....>APPLE
  SataPri.....INTL
{270A33FD-3DA6-460D-BA893C1BAE21E39B} - 0xfc8 bytes
  ........H.......
  H.......H.......
[...]

Of course, this is much more useful if your drivers save additional data for troubleshooting and you have written a WinDbg extension to interpret it.

- Dmitry Vostokov @ DumpAnalysis.org -

The First Windows® Memory Dump Analysis Book!

Tuesday, April 15th, 2008

I’m very proud to announce that it is finally available in both paperback and hardback. Why have I made available both editions? Because I personally prefer hardcover books. You can order the book today and it will be printed in 3-5 days (paperback) or 5-10 days (hardcover) and sent to you:

Memory Dump Analysis Anthology, Volume 1

Note: although listed on Amazon and other online bookstores it is not immediately available at these stores at the moment due to the late submission. I apologize for this. However, I expect that in a few weeks pre-orders taken there will be eventually fulfilled. In the mean time, if you want the book now, you can use the link above.

- Dmitry Vostokov @ DumpAnalysis.org -

AW Reprints Device Drivers Book

Saturday, March 29th, 2008

Just noticed that this month Addison-Wesley Professional reprints in paperback its out of stock hardcover book originally published in 1999:

Developing Windows NT Device Drivers: A Programmer’s Handbook (paperback)

Buy from Amazon

Highly recommended. Almost all book material is still relevant today even in the light of new WDF model. Please also see my post Moving to kernel space (updated references).

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump Analysis Anthology, Volume 2

Tuesday, March 25th, 2008

Although the first volume has not been published yet (scheduled for 15th of April, 2008) the planning for the second volume has already begun. Preliminary information is:

  • Title: Memory Dump Analysis Anthology, Volume 2
  • Paperback: 512 pages (*)
  • ISBN-13: 978-0-9558328-7-1
  • Author: Dmitry Vostokov
  • Publisher: Opentask (01 Oct 2008)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

Hardcover version is also planned. PDF version will be available for download too.

(*) subject to change

- Dmitry Vostokov @ DumpAnalysis.org -

Windows® Device Drivers

Thursday, March 20th, 2008

Why do we need yet another book about device drivers? There are couple of reasons here:

  1. Old books are more about developing the narrow range of legacy drivers than troubleshooting and debugging them.

  2. New books shift towards WDF and ignore legacy drivers.

  3. Windows Internals book is too big and something lightweight is desperately needed.

  4. No published driver books use UML as communication device and discuss driver developement as software factory.

  5. Existing books mostly view device drivers as hardware device drivers.

I started collecting and organizing information about Windows drivers 2 years ago and published a few selected materials so you can get an approximate flavour of what is expected in the forthcoming book scheduled for the next year:

UML and Device Drivers

  • Title:  Windows Device Drivers: An Introduction
  • Author: Dmitry Vostokov
  • Paperback: 128 pages
  • ISBN-13: 978-0-9558328-4-0
  • Publisher: Opentask (15 Apr 2009)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

- Dmitry Vostokov @ DumpAnalysis.org -

WinDbg book to be published after MDAA V1

Thursday, March 20th, 2008

This is a forthcoming reference book for technical support and escalation engineers troubleshooting and debugging complex software issues. The book is also invaluable for software maintenance and development engineers debugging unmanaged, managed and native code.

  • Title: Windows® Debugging Notebook: Essential Concepts, WinDbg Commands and Tools
  • Author: Dmitry Vostokov
  • Hardcover: 256 pages
  • ISBN-13: 978-0-9558328-5-7
  • Publisher: Opentask (1 September 2008)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

- Dmitry Vostokov @ DumpAnalysis.org -

Memory Dump Analysis Anthology, Volume 1

Thursday, February 7th, 2008

It is very easy to become a publisher nowadays. Much easier than I thought. I registered myself as a publisher under the name of OpenTask which is my registered business name in Ireland. I also got the list of ISBN numbers and therefore can announce product details for the first volume of Memory Dump Analysis Anthology series:

Memory Dump Analysis Anthology, Volume 1

  • Paperback: 720 pages (*)
  • ISBN-13: 978-0-9558328-0-2
  • Hardcover: 720 pages (*)
  • ISBN-13: 978-0-9558328-1-9
  • Author: Dmitry Vostokov
  • Publisher: Opentask (15 Apr 2008)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

(*) subject to change 

PDF file will be available for download too.

- Dmitry Vostokov @ DumpAnalysis.org -

Moving to kernel space (updated references)

Sunday, August 26th, 2007

If you are developing and debugging user space applications (and/or doing crash dump analysis in user space) and you want to understand Windows kernel dumps and device drivers better (and probably start writing your own kernel tools) here is the reading list I found the most effective over the last 4 years:

0. Read and re-read Windows Internals book in parallel while reading all other books. I read all editions by the way. It will show you the big picture and some useful WinDbg commands and techniques but you need to read device driver books to fill the gaps and be confident in kernel space:

Buy from Amazon

1. Start with “The Windows 2000 Device Driver Book: A Guide for Programmers (2nd Edition)”. This short book will show you the basics and you can start writing your drivers and kernel tools immediately.

Buy from Amazon

2. Next read “Windows NT Device Driver Development” book to consolidate your knowledge. This book has been reprinted by OSR:

Buy from Amazon

3. Don’t stop here. Read “Developing Windows NT Device Drivers:
 A Programmer’s Handbook”. This is very good book explaining everything in great detail and good pictures. You will finally understand various buffering methods.

Buy from Amazon

4. Continue with WDM drivers and modern presentation: “Programming the Microsoft Windows Driver Model, Second Edition”. Must read even if your drivers are not WDM.

Buy from Amazon

5. Finally read “Developing Drivers with the Windows Driver Foundation” book as this is the future and it also covers ETW (event tracing for Windows), WinDbg extensions, PREfast and static driver verifier.

Buy from Amazon

Additional reading (not including DDK Help which you will use anyway) can be done in parallel after finishing “Windows NT Device Driver Development” book:

1. OSR NT Insider articles. I have their full printed collection 1996 - 2006

http://www.osronline.com/

2. “Windows NT File System Internals” reprinted by OSR:

Buy from Amazon

3. “Rootkits: Subverting the Windows Kernel” book will show you Windows kernel from hacker perspective. In addition you will find overview of kernel areas not covered in other books.

Buy from Amazon

Of course, you must know C language and its idioms really well. Really know it down to assembly language level! I’ll publish another reading list soon. Stay tuned.

- Dmitry Vostokov @ DumpAnalysis.org -

Yet another look at Zw* and Nt* functions

Tuesday, April 10th, 2007

While reading the new book “Professional Rootkits” by Ric Vieler I encountered the following macro definition to get function index in system service table:

#define HOOK_INDEX(function2hook) *(PULONG)((PUCHAR)function2hook+1)

Couldn’t understand the code until looked at disassembly of a typical ntdll!Zw and nt!Zw function (x86 W2K3):

lkd> u ntdll!ZwCreateProcess
ntdll!NtCreateProcess:
7c821298 b831000000      mov     eax,31h
7c82129d ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
7c8212a2 ff12            call    dword ptr [edx]
7c8212a4 c22000          ret     20h
7c8212a7 90              nop
ntdll!ZwCreateProcessEx:
7c8212a8 b832000000      mov     eax,32h
7c8212ad ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
7c8212b2 ff12            call    dword ptr [edx]

lkd> u nt!ZwCreateProcess
nt!ZwCreateProcess:
8083c2a3 b831000000      mov     eax,31h
8083c2a8 8d542404        lea     edx,[esp+4]
8083c2ac 9c              pushfd
8083c2ad 6a08            push    8
8083c2af e8c688ffff      call    nt!KiSystemService (80834b7a)
8083c2b4 c22000          ret     20h
nt!ZwCreateProcessEx:
8083c2b7 b832000000      mov     eax,32h
8083c2bc 8d542404        lea     edx,[esp+4]

You can notice that user space ntdll!Nt and ntdll!Zw variants are the same. This is not the case in kernel space:

lkd> u nt!NtCreateProcess
nt!NtCreateProcess:
808f80ea 8bff            mov     edi,edi
808f80ec 55              push    ebp
808f80ed 8bec            mov     ebp,esp
808f80ef 33c0            xor     eax,eax
808f80f1 f6451c01        test    byte ptr [ebp+1Ch],1
808f80f5 0f8549d10600    jne     nt!NtCreateProcess+0xd (80965244)
808f80fb f6452001        test    byte ptr [ebp+20h],1
808f80ff 0f8545d10600    jne     nt!NtCreateProcess+0×14 (8096524a)

nt!Zw functions are dispatched through service table. nt!Nt functions are actual code. 

For completeness let’s look at AMD x64 W2K3. User space x64 call:

0:001> u ntdll!ZwCreateProcess
ntdll!NtCreateProcess:
00000000`78ef1ab0 4c8bd1          mov     r10,rcx
00000000`78ef1ab3 b882000000      mov     eax,82h
00000000`78ef1ab8 0f05            syscall
00000000`78ef1aba c3              ret
00000000`78ef1abb 666690          xchg    ax,ax
00000000`78ef1abe 6690            xchg    ax,ax
ntdll!NtCreateProfile:
00000000`78ef1ac0 4c8bd1          mov     r10,rcx
00000000`78ef1ac3 b883000000      mov     eax,83h

User space x86 call in x64 W2K3:

0:001> u ntdll!ZwCreateProcess
ntdll!ZwCreateProcess:
7d61d428 b882000000      mov     eax,82h
7d61d42d 33c9            xor     ecx,ecx
7d61d42f 8d542404        lea     edx,[esp+4]
7d61d433 64ff15c0000000  call    dword ptr fs:[0C0h]
7d61d43a c22000          ret     20h
7d61d43d 8d4900          lea     ecx,[ecx]
ntdll!ZwCreateProfile:
7d61d440 b883000000      mov     eax,83h
7d61d445 33c9            xor     ecx,ecx

Kernel space call in x64 W2K3:

kd> u nt!ZwCreateProcess nt!ZwCreateProcess+20
nt!ZwCreateProcess:
fffff800`0103dd70 488bc4          mov     rax,rsp
fffff800`0103dd73 fa              cli
fffff800`0103dd74 4883ec10        sub     rsp,10h
fffff800`0103dd78 50              push    rax
fffff800`0103dd79 9c              pushfq
fffff800`0103dd7a 6a10            push    10h
fffff800`0103dd7c 488d057d380000  lea     rax,[nt!KiServiceLinkage (fffff800`01041600)]
fffff800`0103dd83 50              push    rax
fffff800`0103dd84 b882000000      mov     eax,82h
fffff800`0103dd89 e972310000      jmp     nt!KiServiceInternal (fffff800`01040f00)
fffff800`0103dd8e 6690            xchg    ax,ax

kd> u nt!NtCreateProcess
nt!NtCreateProcess:
fffff800`01245832 53               push    rbx
fffff800`01245833 4883ec50         sub     rsp,50h
fffff800`01245837 4c8b9c2488000000 mov     r11,qword ptr [rsp+88h]
fffff800`0124583f b801000000       mov     eax,1
fffff800`01245844 488bd9           mov     rbx,rcx
fffff800`01245847 488b8c2490000000 mov     rcx,qword ptr [rsp+90h]
fffff800`0124584f 41f6c301         test    r11b,1
fffff800`01245853 41ba00000000     mov     r10d,0

Here is the same as in kernel x86: Zw functions are dispatched but Nt functions are actual code. If you want to remember which function variant is dispatched and which is actual code I propose the mnemonic “Z-dispatch”.

- Dmitry Vostokov -

Crash Dump Analysis Patterns (Part 2)

Tuesday, October 31st, 2006

Another pattern I would like to discuss is Dynamic Memory Corruption (and its user and kernel variants called Heap Corruption and Pool Corruption). You might have already guessed it :-) It is so ubiquitous. And its manifestations are random and usually crashes happen far away from the original corruption point. In your user mode and space part of exception threads (don’t forget about Multiple Exceptions pattern) you would see something like this:

ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlFreeHeap+0x142
MSVCRT!free+0xda
componentA!xxx

or this

ntdll!RtlpCoalesceFreeBlocks+0x10c
ntdll!RtlpExtendHeap+0x1c1
ntdll!RtlAllocateHeap+0x3b6
componentA!xxx

or any similar variants and you need to know exact component that corrupted application heap (which usually is not the same as componentA.dll you see in crashed thread stack).

For this common recurrent problem we have a general solution: enable heap checking. This general solution has many variants applied in a specific context:

  • parameter value checking for heap functions

  • user space software heap checks before or after certain checkpoints (like “malloc”/”new” and/or “free”/”delete” calls): usually implemented by checking various fill patterns, etc.

  • hardware/OS supported heap checks (like using guard and nonaccessible pages to trap buffer overruns)

The latter variant is the mostly used according to my experience and mainly due to the fact that most heap corruptions originate from buffer overflows. And it is easier to rely on instant MMU support than on checking fill patterns. Here is the article from Citrix support web site describing how you can enable full page heap. It uses specific process as an example: Citrix Independent Management Architecture (IMA) service but you can substitute any application name you are interested in debugging:

How to enable full page heap

and another article:

How to check in a user dump that full page heap was enabled

The following Microsoft article discusses various heap related checks:

How to use Pageheap.exe in Windows XP and Windows 2000

The Windows kernel analog to user mode and space heap corruption is called page and nonpaged pool corruption. If we consider Windows kernel pools as variants of heap then exactly the same techniques are applicable there, for example, the so called special pool enabled by Driver Verifier is implemented by nonaccessible pages. Refer to the following Microsoft article for further details:

How to use the special pool feature to isolate pool damage

- Dmitry Vostokov @ DumpAnalysis.org -

Reverse Engineering Citrix ThinWire

Tuesday, October 24th, 2006

Crash dumps (and live debugging) can be very useful for reverse engineering component dependencies. Let’s look at MS Video Driver Architecture UML component diagram (synthesized after reading various articles from OSR and DDK):

Coupled with this understanding and armed with Citrix symbol files (which are freely downloadable from Citrix support and you don’t really need them to see component dependencies) I was able to transform this thread stack below and other similar stacks into the following UML component diagram (some functions are shown as module!xxx and offsets are removed for clarity):

nt!KiSwapContext
nt!KiSwapThread
nt!KeWaitForSingleObject
tcpip!xxx
tcpip!TCPDispatch
nt!IofCallDriver
nt!xxx
nt!xxx
TDTCP!xxx
TDTCP!xxx
TDTCP!TdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver

pdrframe!xxx
pdrframe!PdIoctl

termdd!_IcaCallSd
termdd!IcaCallNextDriver

pdcrypt1!xxx
pdcrypt1!PdIoctl

termdd!_IcaCallSd
termdd!IcaCallNextDriver

WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!WdIoctl

termdd!IcaCallStack
termdd!IcaCallDriver
termdd!IcaDeviceControlChannel
termdd!IcaDeviceControl
termdd!IcaDispatch

win32k!GreDeviceIoControl
win32k!EngDeviceIoControl

vdtw30!xxx
vdtw30!xxx

win32k!vMovePointer
win32k!GreMovePointer
win32k!xxxMoveEventAbsolute
win32k!ProcessMouseInput
win32k!InputApc

nt!KiDeliverApc
nt!KiSwapThread
nt!KeWaitForMultipleObjects
win32k!xxxMsgWaitForMultipleObjects
win32k!xxxDesktopThread
win32k!xxxCreateSystemThreads
win32k!NtUserCallOneParam

nt!KiSystemServiceCopyEnd
nt!KiSwapThread
nt!KeWaitForSingleObject
win32k!EngWaitForSingleObject
vdtw30!xxx
vdtw30!xxx
vdtw30!xxx
vdtw30!DrvTw2SaveScreenBits

win32k!GreSaveScreenBits
win32k!CreateSpb
win32k!zzzChangeStates
win32k!zzzBltValidBits
win32k!xxxEndDeferWindowPosEx
win32k!xxxSetWindowPos
win32k!xxxShowWindow
win32k!NtUserShowWindow

nt!KiSystemService
USER32!NtUserShowWindow
USER32!InternalDialogBox
USER32!DialogBoxIndirectParamAorW
USER32!DialogBoxParamW

We replace MS components with Citrix ones:

  • Video Display with vdtw30.dll
  • Video Miniport with icacdd.sys
  • Hardware and HAL with Terminal Services stack components (MS termdd.sys, Citrix wdica.sys, etc)

twarchitecture.JPG

- Dmitry Vostokov -

UML and Device drivers

Sunday, October 8th, 2006

I got the impression after reading numerous books and articles about device drivers that UML is almost never used in describing kernel and device driver design and architecture. Everything is described either by words or using proprietary notations. If you don’t know about UML (Unified Modeling Language) it is time to learn because it is an industry standard general purpose modeling language with graphical notation. You can find many good tutorials on the Web and I can recommend the book to start:

UML Distilled: A Brief Guide to the Standard Object Modeling Language, Third Edition

Buy from Amazon

Recently I created some diagrams based on my past experience in using UML to describe and communicate architecture and design:

0. Component diagram depicting major driver interfaces 

driverinterfaces2.JPG

1. Class and object diagram depicting relationship between drivers and devices

 Drivers and Devices

2. Component diagram showing dependencies and interfaces when calling Win32 API function ReadFile

iomanager.JPG

3. Component diagram showing IRP flow in a driver stack (driver-to-driver communication)

Actually I found that some driver books incorrectly depict the ordering of I/O stack locations in IRP stack corresponding to driver or device stack. The correct layout is depicted above. IRP I/O stack locations grow down (to lower addresses) in memory like any other Wintel stack. You can see it from kernel dumps or the following macro from DDK header file wdm.h which shows that next IRP I/O stack location is down in memory (1 is subtracted from current stack location pointer):

#define IoGetNextIrpStackLocation( Irp ) (\
    (Irp)->Tail.Overlay.CurrentStackLocation - 1 )

Dumps (and live debugging) are good in studying component relationships, reconstructing sequence diagrams, etc. For example, this edited fragment below is from crash dump and it shows who calls whom and component dependencies be reconstructed from call stack of Win32 API function GetDriveType: SHELL32 (calls it) -> kernel32 -> ntdll -> nt (ntoskrnl.exe). You can also see various Citrix hooks and filters here (CtxSbxXXX):

kd> kL
CtxSbx!xxx
nt!IovCallDriver
nt!IofCallDriver
CtxAltStr!xxx
nt!IovCallDriver
nt!IofCallDriver
nt!IopParseDevice
nt!ObpLookupObjectName
nt!ObOpenObjectByName
nt!IopCreateFile
nt!IoCreateFile
nt!NtOpenFile
nt!KiSystemService
SharedUserData!SystemCallStub
ntdll!ZwOpenFile
CtxSbxHook!xxx
kernel32!GetDriveTypeW
SHELL32!CMountPoint::_EnumMountPoints

- Dmitry Vostokov -

Studying Linux kernel

Thursday, October 5th, 2006

I believe studying Linux kernel and playing with it will broaden your conceptual understanding of kernel development and issues and you can apply it to Wintel stuff too. I’m not a complete Windows guy as you might think after reading my previous posts. I spent 1.5 years (before joining Citrix) under RedHat Linux writing C++ software quality tools in C++ using Emacs editor (working for Programming Research Ltd www.programmingresearch.com). And I did multi platform (Windows - Linux - Solaris) architecture, design and programming for Boeing Commercial Airplanes Group 6 years ago (when working for the biggest Russian outsourcing company Luxoft www.luxoft.com). Coupled with all this prior knowledge about Linux I’m on my journey to study the latest Linux kernel (2.6) and I would recommend 2 wonderful books I’m reading now:

Linux Kernel Development, 2nd Edition

Buy from Amazon

Understanding Linux Kernel, 3rd Edition

Buy from Amazon

and another fantastic book about Unix internals in general:

UNIX Internals

Buy from Amazon

- Dmitry Vostokov -

Moving to kernel space (annotated references)

Thursday, September 28th, 2006

The post was updated and can be found here:

http://www.dumpanalysis.org/blog/index.php/2007/08/26/moving-to-kernel-space-updated-references/

- Dmitry Vostokov -