Here is the presentation PDF file:
Citrix Tools - everything you need for troubleshooting, optimization and analysis
It is based on my previous presentation with a few added slides which can be downloaded from Citrix support web site:
Selected Citrix Troubleshooting Tools
- Dmitry Vostokov @ DumpAnalysis.org -
There are few tools available. Please drop a comment if you know any other. As the focus here is mainly troubleshooting Citrix terminal service environments I put links to Citrix articles where possible:
- ASC/AMC: Using Citrix Diagnostic Facility and the Access Suite Console for Tracing
- tracelog: How to Collect System Startup Traces Using the Microsoft Utility Tracelog.exe
- Dmitry Vostokov @ DumpAnalysis.org -
Component Age Diagram (CAD) helps to visualize and pinpoint anomalies in component timestamps. Excel helps here. We can import the output of lmt WinDbg command and get these graphs where peaks can be used to identify old modules. For example, here is a CAD from my Windows Vista SP1 running on MacMini:
Here is another CAD from Windows 2000 server where the oldest driver is easily identified:
The following CAD diagram is created from lmt output in Module Variety pattern example:
- Dmitry Vostokov @ DumpAnalysis.org -
Thanks to Volker who noticed WinDbg online help I was able to quickly update my HTML version of CDA Poster to point to online links instead of the local help CHM file:
http://www.dumpanalysis.org/CDAPoster.html
It is also featured on http://windbg.org
I’m also working on the better version that will be released simultaneuosly with WDN book.
- Dmitry Vostokov @ DumpAnalysis.org -
As always you can quickly get it through WinDbg Quick Links page:
- Dmitry Vostokov @ DumpAnalysis.org -
Some applications are written using Standard Template Library and it is good that there is !stl WinDbg extension which works with a few types from Plauger’s STL implementation used in Visual C++ CRT library:
0:000> !stl
!stl [options] <varname>
stl [options] <varname> - dumps an STL variable
stl [options] -n <type-name> <address>
currently works with string, wstring
vector<string>, vector<wstring>
list<string>, vector<wstring>
(and pointer varieties therein)
[options]
-n <type-name> The name of the type. If the
type has spaces, surround with
parentheses ().
-v verbose output
-V extremely verbose output
If we have public symbols and know variable names we can simply dump their values, for example:
0:000> dv /i /V
prv local @ecx @ecx this = 0x0012fbdc
prv local 0012fbf8 @ebp-0x2c MyName = class std::basic_string<char,std::char_traits<char>,std::allocator<char> >
0:000> !stl MyName
[da 0x12fbfc]
0012fbfc "COMPANY__NAME"
We can also supply full STL type name:
0:000> !stl -n (std::basic_string<char,std::char_traits<char>,std::allocator<char> >) 0012fbf8
[da 0x12fbfc]
0012fbfc "COMPANY__NAME"
Let’s dump this string type internal structure to be able to recognize it later in raw data:
0:000> dt -r -n std::basic_string<char,std::char_traits<char>,std::allocator<char> > 0012fbf8
application!std::basic_string<char,std::char_traits<char>,std::allocator<char> >
+0x000 _Alval : std::allocator<char>
=00400000 npos : 0x905a4d
+0×004 _Bx : std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Bxty
+0×000 _Buf : [16] “COMPANY__NAME”
+0×000 _Ptr : 0×43415250 “”
+0×014 _Mysize : 0xd
+0×018 _Myres : 0xf
We see that for short strings less than 16 bytes std::basic_string<char> data starts from offset +4 and followed by the actual string size and its reserved size:
0:000> dd 0012fbf8
0012fbf8 00000000 43415250 45434954 53504d5f
0012fc08 41bf0033 0000000d 0000000f 41bf3b72
0012fc18 0012fc6c 0046107b 00000000 0012fc78
0012fc28 0041a441 00000000 41bf3b2e 00ed6380
0012fc38 00000003 00ed6128 00ed6128 00f41b00
0012fc48 00ed6128 41bf3b3e 0012fc3c 00000000
0012fc58 0000000f 00f41b98 00f469a0 00000000
0012fc68 014487c8 0012fcfc 00463fdd 00000002
For bigger strings implementation starts with a pointer from offset +4 to the actual string data and then followed by 12 bytes of garbage and then by the actual string size and its reserved size:
0:000> dt -r -n std::basic_string<char,std::char_traits<char>,std::allocator<char> >
application!std::basic_string<char,std::char_traits<char>,std::allocator<char> >
+0x000 _Alval : std::allocator<char>
=00400000 npos : Uint4B
+0×004 _Bx : std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Bxty
+0×000 _Buf : [16] Char
+0×000 _Ptr : Ptr32 Char
+0×014 _Mysize : Uint4B
+0×018 _Myres : Uint4B
0:000> dt -r -n std::basic_string<char,std::char_traits<char>,std::allocator<char> > 0012ff08
application!std::basic_string<char,std::char_traits<char>,std::allocator<char> >
+0x000 _Alval : std::allocator<char>
=00400000 npos : 0x905a4d
+0×004 _Bx : std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Bxty
+0×000 _Buf : [16] “???”
+0×000 _Ptr : 0×00ed4ba0 “/h /c:100 /enum”
+0×014 _Mysize : 0×10
+0×018 _Myres : 0×1f
In such cases dpa or dpu commands help to show this additional dereference:
0:000> dpa 0012ff08
0012ff08 00ed2f90 "."
0012ff0c 00ed4ba0 “/h /c:100 /enum”
0012ff10 41eafd01
0012ff14 0012ffc0 “…”
0012ff18 0045890a “……U..SVWUj”
0012ff1c 00000010
0012ff20 0000001f
0012ff24 41bf3996
0012ff28 0012ffc0 “…”
0012ff2c 0044b528 “.E..}.”
0012ff30 00400000 “MZ.”
SDbgExt has commands to interrogate additional STL types.
- Dmitry Vostokov @ DumpAnalysis.org -
I’m presenting Citrix Tools at the famous PubForum event. It is a non-commercial Microsoft Terminal Services, Citrix, Virtualization and Server-Based Computing event. This year PubForum is held in Dublin, Republic of Ireland. See the final event details and agenda:
My presentation date, time and topic are:
Friday, May 9, 19:00 “Citrix Tools - everything you need for troubleshooting, optimization and analysis”
See you there 
- Dmitry Vostokov @ DumpAnalysis.org -
Optimized VM Layout is a specialization of the general Changed Environment pattern where the whole modules are moved in virtual memory by changing their load order and load addresses. This can result in dormant bugs being exposed and one of workarounds usually is to disable such external optimization programs or services or adding applications that behave improperly to exclusion lists. Some optimized virtual memory cases can easily be detected by looking at module list where system DLLs are remapped to lower addresses instead of 0×7X000000 range:
0:000> lm
start end module name
00400000 00416000 Application
00470000 0050b000 advapi32
00520000 00572000 shlwapi
02340000 023cb000 oleaut32
04b80000 0523e000 System_Data_ni
1a400000 1a524000 urlmon
4dd60000 4df07000 GdiPlus
5f120000 5f12e000 ntlanman
5f860000 5f891000 netui1
5f8a0000 5f8b6000 netui0
637a0000 63d28000 System_Xml_ni
64890000 6498c000 System_Configuration_ni
64e70000 6515c000 System_Data
65ce0000 65ecc000 System_Web_Services_ni
71bd0000 71be1000 mpr
71bf0000 71bf8000 ws2help
71c00000 71c17000 ws2_32
71c20000 71c32000 tsappcmp
71c40000 71c97000 netapi32
73070000 73097000 winspool
75e90000 75e97000 drprov
75ea0000 75eaa000 davclnt
76190000 761a2000 msasn1
761b0000 76243000 crypt32
76a80000 76a92000 atl
76b80000 76bae000 credui
76dc0000 76de8000 adsldpc
76df0000 76e24000 activeds
76f00000 76f08000 wtsapi32
76f10000 76f3e000 wldap32
771f0000 77201000 winsta
77670000 777a9000 ole32
77ba0000 77bfa000 msvcrt
78130000 781cb000 msvcr80
79000000 79046000 mscoree
79060000 790b6000 mscorjit
790c0000 79bf6000 mscorlib_ni
79e70000 7a3ff000 mscorwks
7a440000 7ac2a000 System_ni
7ade0000 7af7c000 System_Drawing_ni
7afd0000 7bc6c000 System_Windows_Forms_ni
7c340000 7c396000 msvcr71
7c8d0000 7d0ce000 shell32
7d4c0000 7d5f0000 kernel32
7d600000 7d6f0000 ntdll
7d800000 7d890000 gdi32
7d8d0000 7d920000 secur32
7d930000 7da00000 user32
7da20000 7db00000 rpcrt4
7dbd0000 7dcd3000 comctl32
7df50000 7dfc0000 uxtheme
7e020000 7e02f000 samlib
The similar address space reshuffling happens with ASLR-enabled applications with the difference that system modules are never remapped below 0×70000000.
- Dmitry Vostokov @ DumpAnalysis.org -
This is a must have book for system administrators of complex Windows server platforms and client workstations to understand and choose the best course of action to address system and application crashes, hangs, CPU spikes and memory leaks. It is also invaluable to general Windows users and technical support engineers.
- Dmitry Vostokov @ DumpAnalysis.org -
OpenTask, the publisher of Crash Dump Analysis books, joins PMA, the Independent Book Publishers Association.
- Dmitry Vostokov @ DumpAnalysis.org -