Software Diagnostics Library

Welcome to Inside Citrix. This monthly column gives a glimpse of different aspects of Citrix through our people. Our guests have different areas of responsibility and expertise to give you an idea of what is happening behind the scenes. We discuss items of interest with people from Product Readiness, Escalation, Technical Support, and Engineering just to name a few.

In this installment of Inside Citrix, we discuss the meaning of life with Dmitry Vostokov, EMEA Development Analysis Team Lead.

Q: Hello Dmitry, how are you? I am very happy to conduct this interview as you are a creative and prolific worker. I wonder…has fame caught up to you yet, due to your creativity?

A: I’m fine, thank you! I believe there is a synergistic effect going on here. I make the company famous and the company makes me famous.

Q: So, before I get too far ahead of myself, please tell everyone a bit of your history. Where are you from? What did you do before Citrix? How long have you been with us? What kinds of things have you been doing at Citrix during your tenure?

A: I’m from Russia. I was born near Moscow and I spent 14 years there after enrolling at Moscow State University to study chemistry. In that university, I saw a computer and immediately started programming. My first program was written in FORTRAN and had almost 200 lines. My second program had commercial success: I ported 800 FORTRAN lines to about 2000 PDP-11 assembler lines and achieved a 25 percent increase in speed (the program calculated rocket fuel properties for weeks). Since then I’d been working from home for some U.S. and Russian ISV companies (mostly in speech and image processing domains) until 1999, when I went to work in an office to see a large software factory from the inside out.

In 2001 I went to Ireland to learn English. My first job in Ireland was with Ericsson in a small town as a Senior Software Designer. The title sounded great to me, but I heard rumors that the only engineers in Ericsson were hardware engineers. So that job didn’t last long because I was headhunted by a company called Programming Research and I relocated to Dublin. I spent 1.5 years there and after working briefly for a security company (that company is extinct now) I was hired by Citrix. I’ve already spent 3.16 years here. For Citrix I analyze crash dumps and provide recommendations. It’s like being a computer psychologist assessing brain damage. I also do a bit of escalation work when I have time. I like to provide full escalation and software maintenance cycles whenever I have sufficient resources to analyze the problem, contact the customer, and provide the resolution. I also have an opportunity here to apply my software design and programming skills by writing various troubleshooting tools.

Q: Most people probably didn’t know all of that. I guarantee you that Escalation knows you well. How is the blogging going? How can readers get to your blog?

A: I love blogging. I didn’t even think about blogging until I suddenly realized its potential in information sharing. When I joined the company there was no sufficient information available about crash dump analysis, so I had to learn on my own. Now I’m happy to share what I have learnt to everyone.

One topic I like to write about in my blog at the moment is crash dump analysis patterns and anti-patterns, where I summarize general solutions you can apply or should not apply in specific contexts to common recurrent dump analysis problems.

More will come…

Q: And the tools that you create, very useful! Can you take a moment to talk about each of the ones you have created? Which ones have you gotten the best feedback about? Which ones have been the most useful?

A: Thanks! I use them too. The tool I got the most complaints about is RepairCBDChain; the tool with the fewest complaints is SystemDump. I got the best feedback about PDBFinder.

All of them are useful in certain troubleshooting scenarios. I’m preparing a presentation about all these tools and I will present it to the EMEA TRM team in December. I’ll definitely publish it as soon as I get feedback about that training.

Here are brief descriptions of these tools (most of them have different versions for various platforms, and some were even ported to Windows Mobile):

• RepairCBDChain: Repairs clipboard functionality and magically you are able to copy/paste again (not always actually – I promise to write a blog post explaining why).

• ADSCleaner: Cleans Windows NT File System (NTFS) file streams created by Citrix memory optimization code if you no longer need this feature (it also frees disk space, by the way).

• ProcessHistory: Tracks processes, threads, and modules on 32-bit and 64-bit platforms. I’m going to release a Windows Mobile version soon.

• MessageHistory: Tracks window messages. It’s similar to Spy++ but much easier to use for troubleshooting and it works on 64-bit platforms too.

• WindowHistory: Tracks windows as they change their appearance, are created, and are destroyed and saves a log file. This is what Spy++ lacks and it was the primary motivation to write this tool.

• SystemDump: Forces a dump immediately or after a specified period of time. This can be done remotely too. It works on both 32-bit and 64-bit Windows! My primary motivation was that the OSR “bang” tool doesn’t work on 64-bit Windows.

• PDBFinder: Helps to find symbol files if you have zillions of them.

• DumpCheck: Verifies that you have a valid dump and even provides recommendations to avoid common mistakes before sending dumps to support.

• CtxHidEx32: Can hide any annoying windows or message boxes and reduce unnecessary support calls. It also has a peculiar feature: you can specify an action to do before hiding the window. When the Media Player window appears it can send a message to your boss.

• Dump2Wave: My most controversial tool that allows you to hear the sound of memory corruption. Some people say it’s useless but I would say it is entertaining.

Some other upcoming tools I’m working days and nights on (when I have free time) are:

• DumpDepends: Helps to automate repetitive dumping.

• DumpAlerts: Provides notification whenever new dump is saved.

• SessionHistory: Tracks session information.

• HistoryToolbar: Organizes “History” tools into one coherent super tool.

• DumpPlayer: Plays musical dumps in real-time and provides visual images based on crash dump memory contents. I coined a term—Dump Tomography—for this.

Q: They must take some upkeep, as we see a lot of improvements, updates, and so on. I also see you provide a lot of training information on escalation techniques, debugging, analysis, and more. What do you believe is the most important characteristic of a successful escalation engineer?

A: As Winston Churchill said: “Never, never, never give up!”?

Q: Any advice for Citrix administrators who might be reading this on how to avoid trouble or have their environment best situated to speed resolution, should an issue occur?

A: If you are asked to generate and/or collect crash dumps, please tell support personnel how you got that dump. And ensure that you are sending the right dump for the right issue.

I started writing Dumps for Dummies blog posts to explain dumps and I promise to continue and expand them.

Q: What do you find most challenging about your job?

A: To work with enormous amounts of information and make quick decisions at the same time.

Q: Is there anything you can share with us about new Citrix products or technologies (not giving away confidential information) that you are excited about?

A: I would tell you that with whatever new technology comes along, crash dumps will be the same! And this gives me some optimism. Whether there will be more or less crash dumps in the future is pretty confidential though…

Q: Any plans to visit Citrix headquarters in Fort Lauderdale, Florida?

A: I’m actually visiting Citrix headquarters at the end of this month! See you there.

Q: Not so much a question, make us laugh!

A: One day we got a fax from a customer where all of the blue screen information was written down by hand—hundreds of digits… How long it took to copy all that from the screen and whether or not he made any mistakes, we will never know. The copy from that fax is still hanging on my desk wall.

Q: What do you do in your free time besides analyzing dumps, debugging and programming?

A: Read books. I read lots of them and about quite diverse subjects. However, my favorite subject for the last four years has been math—the more abstract the better.

It really helps in improving the critical thinking skills required for my job.

Thanks, Dmitry. People will know to look you up online…

Crash dumps (and live debugging) can be very useful for reverse engineering component dependencies. Let’s look at MS Video Driver Architecture UML component diagram (synthesized after reading various articles from OSR and DDK):

Coupled with this understanding and armed with Citrix symbol files (which are freely downloadable from Citrix support and you don’t really need them to see component dependencies) I was able to transform this thread stack below and other similar stacks into the following UML component diagram (some functions are shown as module!xxx and offsets are removed for clarity):

nt!KiSwapContext
nt!KiSwapThread
nt!KeWaitForSingleObject
tcpip!xxx
tcpip!TCPDispatch
nt!IofCallDriver
nt!xxx
nt!xxx
TDTCP!xxx
TDTCP!xxx
TDTCP!TdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
pdrframe!xxx
pdrframe!PdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
pdcrypt1!xxx
pdcrypt1!PdIoctl
termdd!_IcaCallSd
termdd!IcaCallNextDriver
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!xxx
WDICA!WdIoctl
termdd!IcaCallStack
termdd!IcaCallDriver
termdd!IcaDeviceControlChannel
termdd!IcaDeviceControl
termdd!IcaDispatch
win32k!GreDeviceIoControl
win32k!EngDeviceIoControl
vdtw30!xxx
vdtw30!xxx
win32k!vMovePointer
win32k!GreMovePointer
win32k!xxxMoveEventAbsolute
win32k!ProcessMouseInput
win32k!InputApc
nt!KiDeliverApc
nt!KiSwapThread
nt!KeWaitForMultipleObjects
win32k!xxxMsgWaitForMultipleObjects
win32k!xxxDesktopThread
win32k!xxxCreateSystemThreads
win32k!NtUserCallOneParam
nt!KiSystemServiceCopyEnd
nt!KiSwapThread
nt!KeWaitForSingleObject
win32k!EngWaitForSingleObject
vdtw30!xxx
vdtw30!xxx
vdtw30!xxx
vdtw30!DrvTw2SaveScreenBits
win32k!GreSaveScreenBits
win32k!CreateSpb
win32k!zzzChangeStates
win32k!zzzBltValidBits
win32k!xxxEndDeferWindowPosEx
win32k!xxxSetWindowPos
win32k!xxxShowWindow
win32k!NtUserShowWindow
nt!KiSystemService
USER32!NtUserShowWindow
USER32!InternalDialogBox
USER32!DialogBoxIndirectParamAorW
USER32!DialogBoxParamW

We replace MS components with Citrix ones:

• Video Display with vdtw30.dll
• Video Miniport with icacdd.sys
• Hardware and HAL with Terminal Services stack components (MS termdd.sys, Citrix wdica.sys, etc)

- Dmitry Vostokov -

I’ve just found this very good article on Citrix support web site and I see it was recently updated reflecting new tools written at Citrix or discovered since its previous publication a year ago:

Troubleshooting Tools for Citrix Environments

- Dmitry Vostokov -

I’m writing a presentation about the history of Voice Recognition on Windows platforms and found the following wikipedia article Covox Speech Thing about pioneering work of Covox Inc. where I was a remote employee in Moscow, Russia in 1992-1993 designing and implementing components for their VoiceBlaster speech recognition application (as far as I remember subject to the past war with Creative Labs and their SoundBlaster trademark)

Looking back almost 15 years ago I envisage that 15 years in the future someone might write an article called “Citrix Access Thing” featuring schematics of Citrix hardware and software appliances. Hope this will not happen.

- Dmitry Vostokov -

Q: What proportion of Citrix customers uses 64-bit server environment?

Hint: search Citrix support web site but don’t run complex search queries - think Citrix tools

A: If we look at download counts for two Citrix tools ADSCleaner and WindowHistory which were published separately for 32-bit and 64-bit platforms but at the same time (links might require free registration):

http://support.citrix.com/kb/category.jspa?categoryID=682 

ADSCleaner 2.0  - 299 downloads

ADSCleaner64 2.0 (for x64 platforms)  - 33 downloads

WindowHistory 2.1  - 140 downloads

WindowHistory64 2.1 (for x64 platforms)  - 31 downloads

So we have approximately 10 - 20% of Citrix customers using 64-bit environments. 20% is the upper limit. What about lower limit? Let’s look at another published tool:

Repair CBD Chain  - 4282 downloads

RepairCBDChain64  - 60 downloads

Therefore the lower limit is less than 2%.

Conclusion: 2 - 20% of Citrix customers use 64-bit environments

- Dmitry Vostokov -

Don’t expect me to talk about Netscaler stuff. I’m a Windows guy. It’s started in 1989 when I got PS/2 with 2Mb of memory on board and Windows 2.x as a GUI appliance to an IBM thermal printer. And then suddenly Windows 3.0 appeared and I didn’t have a clue about programming on it (I was an MS DOS guy). Thanks to BBS (some of you probably have never heard about it - it was mini Internet at that time) I got a text file - that wonderful book “Programming Windows” 1st edition written by Charles Petzold and read it twice and being facinated by Windows GUI independence from hardware went straight programming Norton Commander variant. Enough nostalgia. Let’s come back to Citrix and hardware.

I’m a big fan of OSR. Read their articles and bought some hardware from them to learn about USB driver programming, like this one:

This is a real USB device! You connect it via cable to your USB port and you have a button, switches and indicators. Inspired by this device I’m writing a driver which will monitor the health of a Citrix server by lighting appropiate indicators when your IMA service is gone, showing the number of sessions active, etc. And there is a button which could force a server to show a blue screen (in another words force a system dump to do an analysis later - that’s my job in Citrix) if things go beyond control of a Citrix administrator. Something like a magic to me. Stay tuned. 

- Dmitry Vostokov -