Archive for March 10th, 2011

Reading Notebook: 04-March-11

Thursday, March 10th, 2011

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

HKLM\S\MountedDevices and basic disk volume partition offset (pp. 667 - 668)

General reparse points; symbolic links and mount points as their applications (p. 669)

Device object -> VPB, !vpb WinDbg command (p. 670) - here’s on my x64 W2K8 system:

0: kd> dt _DEVICE_OBJECT
ntdll!_DEVICE_OBJECT
+0x000 Type             : Int2B
+0x002 Size             : Uint2B
+0x004 ReferenceCount   : Int4B
+0x008 DriverObject     : Ptr64 _DRIVER_OBJECT
+0x010 NextDevice       : Ptr64 _DEVICE_OBJECT
+0x018 AttachedDevice   : Ptr64 _DEVICE_OBJECT
+0x020 CurrentIrp       : Ptr64 _IRP
+0x028 Timer            : Ptr64 _IO_TIMER
+0x030 Flags            : Uint4B
+0x034 Characteristics  : Uint4B
   +0×038 Vpb              : Ptr64 _VPB
+0×040 DeviceExtension  : Ptr64 Void
+0×048 DeviceType       : Uint4B
+0×04c StackSize        : Char
+0×050 Queue            : <unnamed-tag>
+0×098 AlignmentRequirement : Uint4B
+0×0a0 DeviceQueue      : _KDEVICE_QUEUE
+0×0c8 Dpc              : _KDPC
+0×108 ActiveThreadCount : Uint4B
+0×110 SecurityDescriptor : Ptr64 Void
+0×118 DeviceLock       : _KEVENT
+0×130 SectorSize       : Uint2B
+0×132 Spare1           : Uint2B
+0×138 DeviceObjectExtension : Ptr64 _DEVOBJ_EXTENSION
+0×140 Reserved         : Ptr64 Void

0: kd> dt _VPB
ntdll!_VPB
+0x000 Type             : Int2B
+0x002 Size             : Int2B
+0x004 Flags            : Uint2B
+0x006 VolumeLabelLength : Uint2B
+0x008 DeviceObject     : Ptr64 _DEVICE_OBJECT
+0x010 RealDevice       : Ptr64 _DEVICE_OBJECT
+0x018 SerialNumber     : Uint4B
+0x01c ReferenceCount   : Uint4B
+0x020 VolumeLabel      : [32] Wchar

FS -> Volume I/O (pp. 674 - 675) - we can also see driver stack from IRP I/O stack locations:

2: kd> !irp fffffa8017492b80
[...]
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000
>[  4,34]  1c e0 fffffa800dfe2060 00000000 fffff88001186f30-00000000 Success Error Cancel
\Driver\Disk  partmgr!PmReadWriteCompletion
Args: 00001000 00000000 b99a9000 00000000
[  4, 0]  1c e0 fffffa800dfe2b90 00000000 fffff88001197180-fffffa800da89e20 Success Error Cancel
\Driver\partmgr     volmgr!VmpReadWriteCompletionRoutine
Args: 148ce8c5bed 00000000 b99a9000 00000000
[  4, 0]   c e0 fffffa800da89cd0 00000000 fffff88001968150-fffffa800dfe7190 Success Error Cancel
\Driver\volmgr      volsnap!VspRefCountCompletionRoutine
Args: 00001000 00000000 148ce8c5be9 00000000
[  4, 0]   c e1 fffffa800dfe7040 00000000 fffff88001a464f4-fffff88002777a10 Success Error Cancel pending
\Driver\volsnap     Ntfs!NtfsMasterIrpSyncCompletionRoutine
Args: 00001000 00000000 b996a000 00000000
[  4, 0]   0  0 fffffa800dfed030 fffffa800da958e0 00000000-00000000
\FileSystem\Ntfs
Args: 00001000 00000000 01afc000 00000000
[…]

BitLocker architecture diagram (p.678) - parts can be seen from IRP I/O stack locations:

 kd> !irp 85e7ee00
[...]
cmd  flg cl Device   File     Completion-Context
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
[  0, 0]   0  0 00000000 00000000 00000000-00000000   

                  Args: 00000000 00000000 00000000 00000000
>[  3,34]  10 e0 857b9030 00000000 8353724e-00000000 Success Error Cancel
\Driver\Disk     partmgr!PmReadWriteCompletion
Args: 00001000 00000000 400d6000 00000000
[  3, 0]  10  0 857b9d18 00000000 00000000-00000000
\Driver\partmgr
Args: 6bad71d7 00000000 400d6000 00000000
[  3, 0]  10 e0 8478b5f0 00000000 835487a4-857bc2f0 Success Error Cancel
\Driver\DriverA   volmgr!VmpReadWriteCompletionRoutine
Args: 00001000 00000000 400d6000 00000000
[  3, 0]   0 e0 857bc238 00000000 872c83e2-857bfb70 Success Error Cancel
\Driver\volmgr   fvevol!FvePassThroughCompletion
Args: 00001000 00000000 6bad70ba 00000000
[  3, 0]   0 e0 857bfab8 00000000 8709807a-859a2118 Success Error Cancel
\Driver\fvevol   Ntfs!NtfsMasterIrpAsyncCompletionRoutine
Args: 00001000 00000000 40097000 00000000
[  3, 0]   0  1 857e2020 8584ca40 00000000-00000000    pending
\FileSystem\Ntfs
Args: 00001000 00000000 0329e000 00000000
[…]

VMK -> FVEK: possibility for rekeying (p. 679)

Maximum protection: TPM+USB+PIN (p. 679)

Diffuser to protect from manipulations with AES-encrypted ciphertext (p. 681)

Software Trace Analysis Checklist

Thursday, March 10th, 2011

Because the number of software trace patterns is growing I’m starting another checklist in addition to memory dump analysis checklist. The goal is to help experienced engineers not to miss any important information. The checklist doesn’t prescribe any specific steps, just lists all possible points to double check when looking at a software trace. Of course, it is not complete at the moment and any suggestions are welcome. This post will be modified on the ongoing basis.

General:

• Check overall trace time delta
• Check no trace metafile message density
• Check whether a trace is a multi-part or a circular
• Check for basic facts and the story (software narrative)
• Check for any exceptions, non-false positive errors and periodic errors
• Check for significant events
• Check for discontinuities in the time domain
• Check for message current and acceleration in the frequency domain

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

10 Common Mistakes in Memory Analysis (Part 10)

Thursday, March 10th, 2011

The last common mistake in this set is when an engineer doesn’t compare stack traces or other debugger output to normal reference stack traces. For example, a 3rd party service was hanging on a W2K8 server and a user memory dump was saved for offline analysis. The following thread was identified as blocked waiting for a response:

STACK_TEXT: 
0041f1bc 75a30816 ntdll_77e50000!ZwWaitForSingleObject+0x15
0041f228 76671184 KERNELBASE!WaitForSingleObjectEx+0x98
0041f240 76671138 kernel32!WaitForSingleObjectExImplementation+0x75
0041f254 75b17be6 kernel32!WaitForSingleObject+0x12
0041f2f8 75b18040 sechost!ScSendResponseReceiveControls+0xea
0041f3ac 75b18662 sechost!ScDispatcherLoop+0xc2
0041f3ec 01271bb4 sechost!StartServiceCtrlDispatcherW+0xb0
0041fa7c 01271dd6 ServiceA!WinMain+0×254
0041fb0c 76673677 ServiceA!__tmainCRTStartup+0×160
0041fb18 77e89d42 kernel32!BaseThreadInitThunk+0xe
0041fb58 77e89d15 ntdll_77e50000!__RtlUserThreadStart+0×70
0041fb70 00000000 ntdll_77e50000!_RtlUserThreadStart+0×1b

Unfortunately, this thread wasn’t recognized as a normal main service thread. Typical Internet search for ScSendResponseReceiveControls function points to a sample analysis log where we can find such thread stacks in the variety of other standard services:

THREAD fffffa8005362060  Cid 0a1c.0b68  Teb: 000007fffffde000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8004dc0a60  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a000008c10
Owning Process            fffffa800540e060       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      37051          Ticks: 3239 (0:00:00:50.528)
Context Switch Count      13            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address svchost!wmainCRTStartup (0x00000000ffa9246c)
Stack Init fffff88005abbdb0 Current fffff88005abb900
Base fffff88005abc000 Limit fffff88005ab6000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`05abb940 fffff800`01a93992 nt!KiSwapContext+0x7a
fffff880`05abba80 fffff800`01a95cff nt!KiCommitThreadWait+0x1d2
fffff880`05abbb10 fffff800`01d871d2 nt!KeWaitForSingleObject+0x19f
fffff880`05abbbb0 fffff800`01a8b993 nt!NtWaitForSingleObject+0xb2
fffff880`05abbc20 00000000`7781fefa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05abbc20)
00000000`001ffab8 000007fe`fda910ac ntdll!NtWaitForSingleObject+0xa
00000000`001ffac0 000007fe`ff4eaffb KERNELBASE!WaitForSingleObjectEx+0x79
00000000`001ffb60 000007fe`ff4e9d61 sechost!ScSendResponseReceiveControls+0×13b
00000000`001ffc50 000007fe`ff4e9c16 sechost!ScDispatcherLoop+0×121
00000000`001ffd60 00000000`ffa91d3a sechost!StartServiceCtrlDispatcherW+0×14e
00000000`001ffdb0 00000000`ffa9257a svchost!wmain+0×110
00000000`001ffde0 00000000`776cf56d svchost!ScCreateWellKnownSids+0×2fd
00000000`001ffe20 00000000`77803281 kernel32!BaseThreadInitThunk+0xd
00000000`001ffe50 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Also, studying OS architecture and deliberate practice in memory dump analysis helps in recognition of problem and normal structural and behavioral patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -