Archive for the ‘Crash Dump Analysis’ Category

Malware Analysis Patterns (Part 4)

Saturday, January 19th, 2013

The next pattern is closely linked to packed and/or obfuscated code. We call it Pre-Obfuscation Residue. Depending on a level of obfuscation and/or packing some initial code and data structures and patterns including fragments of strings may leak in post-obfuscation data giving a clue to intended software behavior:

0:000> s-sa 00000000`00fd4000 L6000
[...]
00000000`00fd943d  "o__"
00000000`00fd9449  "91!We"
00000000`00fd945d  "H5!"
00000000`00fd94d2  "zQ@"
00000000`00fd94dd  "ommandS"
00000000`00fd94f4  “IsDeb”
00000000`00fd94fd  “uggerP”
00000000`00fd9507  “Enc”
00000000`00fd950c  “v)3Po4t”
00000000`00fd9515  “DeXU”
00000000`00fd9520  “xFe”
00000000`00fd952a  “5Eb”
00000000`00fd9533  “SI=l8kev”
00000000`00fd953e  “Z_1m”
00000000`00fd9547  “@IF”
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Malware Analysis Patterns (Part 3)

Saturday, January 19th, 2013

Packed Code is frequent ingredient of armored malware. Here we demonstrate a few WinDbg commands to detect UPX packed modules with little or no expected strings:

0:000> !dh 00000000`00fd40b0

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
3 number of sections
time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL

OPTIONAL HEADER VALUES
10B magic #
11.00 linker version
6000 size of code
1000 size of initialized data
F000 size of uninitialized data
15600 address of entry point
10000 base of code
----- new -----
0000000010000000 image base
1000 section alignment
200 file alignment
2 subsystem (Windows GUI)
6.00 operating system version
0.00 image version
6.00 subsystem version
17000 size of image
1000 size of headers
0 checksum
0000000000100000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
140  DLL characteristics
Dynamic base
NX compatible
16274 [      AC] address [size] of Export Directory
161DC [      98] address [size] of Import Directory
16000 [     1DC] address [size] of Resource Directory
0 [       0] address [size] of Exception Directory
0 [       0] address [size] of Security Directory
16320 [      10] address [size] of Base Relocation Directory
0 [       0] address [size] of Debug Directory
0 [       0] address [size] of Description Directory
0 [       0] address [size] of Special Directory
0 [       0] address [size] of Thread Storage Directory
157CC [      48] address [size] of Load Configuration Directory
0 [       0] address [size] of Bound Import Directory
0 [       0] address [size] of Import Address Table Directory
0 [       0] address [size] of Delay Import Directory
0 [       0] address [size] of COR20 Header Directory
0 [       0] address [size] of Reserved Directory
SECTION HEADER #1
UPX0 name

F000 virtual size
1000 virtual address
0 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E0000080 flags
Uninitialized Data
(no align specified)
Execute Read Write

SECTION HEADER #2
UPX1 name

6000 virtual size
10000 virtual address
5A00 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
E0000040 flags
Initialized Data
(no align specified)
Execute Read Write

SECTION HEADER #3
.rsrc name
1000 virtual size
16000 virtual address
400 size of raw data
5E00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
(no align specified)
Read Write

0:000> s-sa 00000000`00fd40b0 L6600
00000000`00fd40fd  "!This program cannot be run in D"
00000000`00fd411d  "OS mode."
00000000`00fd4188  "Rich"
00000000`00fd4290  “UPX0″
00000000`00fd42b8  “UPX1″

00000000`00fd42e0  “.rsrc”
00000000`00fd448b  “3.08″
00000000`00fd4490  “UPX!”
00000000`00fd449b  “YhHM4″
00000000`00fd44d1  “vqx”
[…]

Such in-memory modules (not yet initialized by a loader) can be saved to disk using .writemem command and unpacked. Once loaded and relocated into some address they still have UPX sections but also they now have more strings:

0:000> s-sa 00000000`691c0000 L300
00000000`691c004d  "!This program cannot be run in D"
00000000`691c006d  "OS mode."
00000000`691c00d8  "Rich"
00000000`691c01e0  “UPX0″
00000000`691c0207  “`UPX1″

00000000`691c022f  “`.rsrc”
[…]
00000000`691d620b  “uGC”
00000000`691d621c  “KERNEL32.DLL”
00000000`691d622a  “LoadLibraryA”
00000000`691d6238  “GetProcAddress”
00000000`691d6248  “VirtualProtect”
00000000`691d6258  “VirtualAlloc”
00000000`691d6266  “VirtualFree”

[…]

0:000> s-su 00000000`691c0000 L(00000000`691d7000-00000000`691c0000)
[...]
00000000`691c8178  “http://www.dumpanalysis.com”
00000000`691c8260  “mscoree.dll”
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 194)

Friday, January 18th, 2013

Whereas some false positives can be considered soft debugger bugs false negatives can have more severe impact on software behavior analysis especially in malware analysis. We name this pattern Debugger Omission. Typical example here is current .imgscan command which according to documentation should by default scan virtual process space for MZ/PE signatures. Unfortunately it doesn’t detect such signatures in resource pages (we haven’t checked stack regions yet):

0000000000fd0000 image base

SECTION HEADER #4
.rsrc name
6430 virtual size
4000 virtual address
6600 size of raw data
1600 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

0:000> .imgscan /r 00000000`00fd4000 L200

0:000> s -[l2]sa 00000000`00fd4000 l200
00000000`00fd40b0  "MZ"
00000000`00fd40fd  "!This program cannot be run in D"
00000000`00fd411d  "OS mode."
00000000`00fd4188  "Rich"
00000000`00fd4198  "PE"

0:000> !dh 00000000`00fd40b0

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
3 number of sections
time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL
[...]

Another other analysis scenarios found will be added to this pattern. Milder version of it includes !analyze -v that shows us a breakpoint instead of an exception violation from a parallel thread.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Static Code Analysis Patterns (Part 1)

Friday, January 11th, 2013

Static program analysis is used to eliminate certain coding errors that may lead to abnormal software behaviour. So it is naturally a part of software diagnostics but at source code level. Our goal here is to identify certain patterns directly linkable to patterns we see in memory dumps and software logs and collect them into a catalog. One such pattern candidate is called Loop Construct. It covers conditional and unconditional loops, for example, in one of modern languages:

extern bool soonToBeTrue; 
int mediumValue = ...;
while (true)

{

  TRACE("Waiting");

  sleep(mediumValue);

  if (soonToBeTrue)

  {

    break;

  }

  doHeavyWork();

}
while (--pControl->aFewPasses)

{

  TRACE("Waiting");

  sleep(mediumValue);

  doHeavyWork();

}

Such loops may potentially lead to Spiking Thread memory dump analysis and High Message Current and Density trace analysis patterns. Of course, we shouldn’t suspect every loop but only some that have potential to be altered by Local Buffer Overflow (for mediumValue) or Shared Buffer Overwrite (for Control.aFewPasses) or by a race condition (soonToBeTrue).

We expect things to get more interesting when we start associating source code that uses certain API with patterns of abnormal behavior.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 193)

Wednesday, January 9th, 2013

Sometimes we have a Broken Link for some reason, either from memory corruption, Lateral Damage or Truncated Dump. For example, an active process list enumeration stops after showing some processes (!for_each_thread and !vm also don’t work):

0: kd> !process 0 ff

[...]

TYPE mismatch for process object at fffffa80041da5c0

0: kd> !validatelist nt!PsActiveProcessHead
Blink at address fffffa80041da748 does not point back to previous at fffffa8005bc8cb8

Here we can either try to repair or navigate links manually or use other means such as dumping pool allocations for process structures with Proc pool tag:

0: kd> !poolfind Proc

Searching NonPaged pool (fffffa80032fc000 : ffffffe000000000) for Tag: Proc

*fffffa80033879a0 size:  510 previous size:   a0  (Allocated) Proc (Protected)
*fffffa80033ffad0 size:  530 previous size:  280  (Allocated) Proc (Protected)
*fffffa80041a2af0 size:  510 previous size:   90  (Allocated) Proc (Protected)
*fffffa800439c5c0 size:  530 previous size:   80  (Allocated) Proc (Protected)
[...]
*fffffa8007475ad0 size:  530 previous size:   30  (Allocated) Proc (Protected)
*fffffa80074e8490 size:  530 previous size:  100  (Allocated) Proc (Protected)
*fffffa80075ee0b0 size:  530 previous size:   b0  (Free)      Pro.
*fffffa800761d000 size:  530 previous size:    0  (Free)      Pro.
*fffffa8007645ad0 size:  530 previous size:   b0  (Allocated) Proc (Protected)

0: kd> dc fffffa8007645ad0
fffffa80`07645ad0  0253000b e36f7250 07644030 fffffa80  ..S.Pro.0.d.....
fffffa80`07645ae0  00001000 00000528 00000068 fffff800  ....(...h.......
fffffa80`07645af0  01a1a940 fffff800 00080090 00490024  @...........$.I.
fffffa80`07645b00  000000c4 00000000 00000008 00000000  ................
fffffa80`07645b10  00000000 00000000 00080007 00300033  ............3.0.
fffffa80`07645b20  01a1a940 fffff800 013cfeae fffff8a0  @.........<.....
fffffa80`07645b30  00580003 00000000 05ba19a0 fffffa80  ..X………….
fffffa80`07645b40  05ba19a0 fffffa80 07645b48 fffffa80  ……..H[d…..

0: kd> !process fffffa80`07645b30 ff
PROCESS fffffa8007645b30
SessionId: 0  Cid: 14c4    Peb: 7fffffd4000  ParentCid: 02c4
DirBase: 7233e000  ObjectTable: fffff8a0014d4220  HandleCount: 399.
Image: AppA.exe
VadRoot fffffa80072bc5b0 Vads 239 Clone 0 Private 24675. Modified 23838. Locked 0.
DeviceMap fffff8a0000088f0
Token                             fffff8a000f28060
ElapsedTime                       00:00:53.066
UserTime                          00:00:00.000
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         0
QuotaPoolUsage[NonPagedPool]      0
Working Set Sizes (now,min,max)  (11960, 50, 345) (47840KB, 200KB, 1380KB)
PeakWorkingSetSize                74346
VirtualSize                       331 Mb
PeakVirtualSize                   478 Mb
PageFaultCount                    92214
MemoryPriority                    BACKGROUND
BasePriority                      8
CommitCharge                      25905

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 42k)

Tuesday, January 8th, 2013

Here we provide examples of threads waiting for pushlocks as they are not normally seen in crash dumps:

THREAD fffffa80033b5b50  Cid 0004.0030  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880021d9750  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          fffffa800439c620       Image:         AppA.exe
Wait Start TickCount      30819          Ticks: 14746574 (2:15:54:08.028)
Context Switch Count      2800
UserTime                  00:00:00.000
KernelTime                00:00:00.374
Win32 Start Address nt!ExpWorkerThread (0xfffff8000189e530)
Stack Init fffff880021d9db0 Current fffff880021d9470
Base fffff880021da000 Limit fffff880021d4000 Call 0
Priority 12 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`021d94b0 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`021d95f0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`021d9680 fffff800`018c4bf8 nt!KeWaitForSingleObject+0×19f
fffff880`021d9720 fffff800`01c2915d nt!ExfAcquirePushLockShared+0×138
fffff880`021d97a0 fffff800`01c6da31 nt!MmEnumerateAndReferenceImages+0×6d
[…]
fffff880`021d9cb0 fffff800`01b2be5a nt!ExpWorkerThread+0×111
fffff880`021d9d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`021d9d80 00000000`00000000 nt!KxStartSystemThread+0×16

THREAD fffffa8003c9d600  Cid 0004.00ac  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880023d1b30  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      177686         Ticks: 14599707 (2:15:15:56.888)
Context Switch Count      1590
UserTime                  00:00:00.000
KernelTime                00:00:00.124
Win32 Start Address 0xfffff80001bac754
Stack Init fffff880023d1db0 Current fffff880023d1850
Base fffff880023d2000 Limit fffff880023cc000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`023d1890 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`023d19d0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`023d1a60 fffff800`01886183 nt!KeWaitForSingleObject+0×19f
fffff880`023d1b00 fffff800`01cd9982 nt!ExfAcquirePushLockExclusive+0×188
[…]
fffff880`023d1d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`023d1d80 00000000`00000000 nt!KxStartSystemThread+0×16

Instead of explaining what a pushlock is we provide a link to ntdebugging blog article.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Process Patterns

Saturday, January 5th, 2013

A page to reference all different kinds of process related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Thread Patterns

Saturday, January 5th, 2013

A page to reference all different kinds of thread related patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 13i)

Saturday, January 5th, 2013

Although we briefly mentioned session pool in Insufficient Memory (kernel pool) pattern we decided to factor it into a separate (sub)pattern and provide WinDbg commands to analyze possible leaks. The following output shows the sequence of commands that gives you an idea although the example itself was taken from a healthy dump so no red coloring (from my memory leaks in session pool happened mostly in 32-bit past):

1: kd> !vm 4

Terminal Server Memory Usage By Session:

Session ID 0 @ fffff8800324d000:
Paged Pool Usage:        4128K
Commit Usage:            7488K

Session ID 1 @ fffff88002f65000:
Paged Pool Usage:       32852K
Commit Usage:           36488K

1: kd> !session
Sessions on machine: 2
Valid Sessions: 0 1
Error in reading current session

1: kd> !session -s 1
Sessions on machine: 2
Implicit process is now fffffa80`07d79730
Using session 1

1: kd> !poolused 8
Sorting by Session Tag

Pool Used:
NonPaged            Paged
Tag    Allocs     Used    Allocs     Used
TOTAL           4     4208      9500 33475120
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

2008-2012 in Retrospection

Friday, January 4th, 2013

Before deciding on whether to retrospect on 2012 we found that since March 14, 2008 this site has had more than 1 million visitors with more than 33% returning. So instead of just 2012 we decided to retrospect on that interval up to December 31, 2012. Google Analytics has improved since last January, 2012 and now made our task easier. So we start with the first one hundred sites referring to us:


Source / Medium

Visits

google

698156

(direct)

164142

bing

27923

google.com

17868

windbg.org

12994

yahoo

8682

stackoverflow.com

7194

yandex

5985

windbg.dumpanalysis.org

5375

dumpanalysis.com

5369

live

5310

google.co.in

4598

blogs.msdn.com

4385

baike.baidu.com

3475

twitter.com

2972

facebook.com

2733

dumpanalysis.org

2708

images.google.com

2314

t.co

2095

baidu

1916

winvistaclub.com

1862

google.co.uk

1449

advancedwindowsdebugging.com

1427

jasonhaley.com

1370

search

1328

rsdn.ru

1294

en.wikipedia.org

1276

msn

1256

nynaeve.net

1256

blog.codeimproved.net

1213

google.de

1074

google.ca

979

reddit.com

951

bytetalk.net

908

citrixblogger.org

819

stumbleupon.com

819

linkedin.com

780

social.technet.microsoft.com

774

analyze-v.com

757

naver

750

forum.sysinternals.com

735

google.ru

710

blogs.microsoft.co.il

693

kumo.com

678

google.co.kr

658

google.com.au

654

blog.naver.com

646

reconstructer.org

645

community.citrix.com

632

blog.not-a-kernel-guy.com

604

itdatabase.com

601

advanceddotnetdebugging.com

581

serverfault.com

564

voneinem-windbg.blogspot.com

561

support.citrix.com

555

debuggingexperts.com

549

blog.miniasp.com

527

google.fr

495

caloni.com.br

488

google.com.br

479

ask

459

msuiche.net

439

insidewindows.kr

432

google.es

430

gynvael.coldwind.pl

430

blog.flexilis.com

429

aol

418

netfxharmonics.com

416

advdbg.org

413

images.google.co.uk

401

google.it

391

images.google.co.in

391

google.nl

354

serious-code.net

340

admin.itdatabase.com

337

blogs.technet.com

334

brianmadden.com

327

google.pl

319

google.com.ua

318

experts-exchange.com

316

delicious.com

312

images.google.de

305

opentask.com

301

codemachine.com

296

driveronline.org

287

google.com.tw

282

wasm.ru

275

debuglab.com

265

isisaka.com

262

literatescientist.com

261

blog.zoller.lu

258

shellexecute.wordpress.com

257

google.com.hk

256

managementbits.com

253

d.hatena.ne.jp

251

bloglines.com

249

google.com.tr

248

clausbrod.de

246

bing.com

243

Next table is distribution of visits among countries:


Country / Territory

Visits

United States

342291

India

89303

United Kingdom

76131

Russia

46472

Germany

44472

China

40155

Canada

34781

Japan

24985

France

24084

South Korea

21056

Australia

20606

Taiwan

17949

Netherlands

15607

Ireland

15579

Israel

13514

Ukraine

13449

Italy

12542

Brazil

11834

Spain

11786

Singapore

11703

Sweden

11201

Poland

10340

Romania

9423

(not set)

8909

Czech Republic

8355

Belgium

6731

Switzerland

6624

Finland

6596

Norway

5585

Malaysia

5289

Philippines

5052

Austria

5046

Denmark

4980

Hong Kong

4914

Turkey

4728

Slovakia

4599

New Zealand

4369

Portugal

4228

Argentina

3712

Belarus

3518

Hungary

3465

Bulgaria

3301

Mexico

2960

South Africa

2945

Vietnam

2721

Greece

2712

Indonesia

2527

Croatia

1881

Serbia

1843

Iran

1842

Thailand

1726

Pakistan

1660

Egypt

1519

Malta

1422

Estonia

1385

Slovenia

1334

Lithuania

1304

United Arab Emirates

1167

Chile

1104

Saudi Arabia

1096

Colombia

1067

Latvia

922

Kazakhstan

725

Peru

649

Morocco

585

Sri Lanka

516

Luxembourg

516

Moldova

439

Uruguay

435

Venezuela

431

Jordan

425

Tunisia

425

Bolivia

418

Armenia

371

Algeria

362

Costa Rica

355

Iceland

353

Panama

352

Macedonia [FYROM]

347

Bosnia and Herzegovina

327

Cyprus

317

Bangladesh

314

Nigeria

298

Puerto Rico

296

Jamaica

251

Ecuador

248

Kuwait

239

Lebanon

218

Qatar

217

Kenya

195

Georgia

194

Mongolia

189

Dominican Republic

163

Macau

156

Trinidad and Tobago

147

Bahrain

143

Uzbekistan

142

Guatemala

141

Azerbaijan

134

Mauritius

128

Oman

117

Nepal

110

El Salvador

106

Syria

103

Iraq

102

Ghana

96

Kyrgyzstan

86

Cambodia

72

Albania

71

Serbia and Montenegro

63

Ethiopia

63

Uganda

61

Brunei

57

Honduras

55

Isle of Man

55

Yemen

55

Cuba

54

Sudan

54

Palestinian Territories

52

Barbados

49

Myanmar [Burma]

48

Paraguay

45

Liechtenstein

43

Montenegro

43

Rwanda

42

Libya

41

Namibia

41

Jersey

40

Maldives

40

Turks and Caicos Islands

39

Bermuda

38

Zimbabwe

34

Fiji

32

Nicaragua

32

Tanzania

29

Réunion

27

Gibraltar

26

New Caledonia

26

Bahamas

25

Monaco

25

Netherlands Antilles

24

Aruba

24

Botswana

24

Cayman Islands

23

Angola

22

Madagascar

20

Guam

19

Afghanistan

17

Côte d’Ivoire

17

Papua New Guinea

17

Dominica

16

Guernsey

16

Guyana

16

Suriname

16

Andorra

14

Belize

14

Congo [DRC]

14

Lesotho

14

Mozambique

13

Antigua and Barbuda

12

Laos

12

French Polynesia

11

Zambia

11

Saint Lucia

10

San Marino

10

Senegal

10

Saint Vincent and the Grenadines

10

Benin

9

Guinea

9

Guadeloupe

9

Malawi

9

Turkmenistan

9

U.S. Virgin Islands

8

Faroe Islands

7

Grenada

7

Haiti

7

British Virgin Islands

7

Cameroon

6

French Guiana

6

Greenland

6

Martinique

6

Seychelles

6

Timor-Leste

6

Mali

5

Tajikistan

5

Gabon

4

Anguilla

3

Å land Islands

3

Swaziland

3

Burundi

2

Congo [Republic]

2

Cape Verde

2

Djibouti

2

Saint Kitts and Nevis

2

Liberia

2

Somalia

2

Togo

2

Vanuatu

2

Burkina Faso

1

Bhutan

1

Falkland Islands [Islas Malvinas]

1

Gambia

1

Equatorial Guinea

1

Guinea-Bissau

1

Comoros

1

Mauritania

1

Palau

1

Sierra Leone

1

Vatican City

1

Samoa

1


Then the first 100 network locations:

Service Provider

Visits

microsoft corp

33646

comcast cable communications inc.

18544

road runner holdco llc

16529

internet service provider

12815

comite gestor da internet no brasil

10995

hewlett-packard company

10961

deutsche telekom ag

9889

japan network information center

9746

verizon internet services inc.

7851

network of citrix systems inc

6945

intel corporation

6873

symantec corporation

6812

chunghwa telecom data communication business group

6381

ip pools

6314

insignium llc

6206

reliance communications ltd

5870

charter communications

5583

uunet non-portable customer assignment

4931

verizon online llc

4900

comcast cable communications holdings inc

4700

at&t internet services

4617

eircom

4567

cox communications

4540

proxad / free sas

4451

korea telecom

4397

abts (karnataka)

4251

nib (national internet backbone)

4243

chinanet guangdong province network

4189

comcast cable communications

3896

unknown

3279

xo communications

3274

chinanet shanghai province network

3248

shaw communications inc.

3179

qwest communications company llc

3156

telstra internet

3130

tw telecom holdings inc.

3091

citrix systems inc.

3029

data general corporation

2998

cox communications inc.

2946

bellsouth.net inc.

2925

optimum online (cablevision systems)

2853

china unicom beijing province network

2850

chtd chunghwa telecom co. ltd.

2791

krnic

2786

ntt communications corporation

2779

psinet inc.

2599

emc corporation

2499

comcast cable communications ip services

2435

arcor ag

2371

cisco systems inc.

2364

(not set)

2335

broadband multiplay project o/o dgm bb noc bsnl bangalore

2285

research in motion limited

2283

samtel

2257

rcs & rds s.a.

2246

computer associates international

2166

honeywell international inc.

2106

telus communications inc.

2103

customers ie

1954

sympatico hse

1929

comcast business communications llc

1853

telefonica de espana sau

1843

iinet limited

1840

ziggo consumers

1810

easynet ltd

1758

comcast business communications inc.

1738

microsoft

1717

kaspersky lab internet

1698

appense

1687

chinanet jiangsu province network

1665

dell computer corporation

1656

eircom ltd

1644

taipei taiwan

1612

abts tamilnadu

1594

network of ign arch. and design gb

1578

starhub cable vision ltd

1555

wipro technologies

1537

level 3 communications inc.

1522

tpg internet pty ltd.

1510

siemens ag

1483

microsoft corporation

1478

global crossing

1433

singnet pte ltd

1429

dynamic pools

1426

this space is statically assigned.

1425

videsh sanchar nigam ltd - india.

1414

provider local registry

1403

abts delhi

1385

qwest communications corporation

1356

kla instruments corp.

1316

telia network services

1311

cncgroup beijing province network

1278

frontier communications of america inc.

1264

telecom italia s.p.a. tin easy lite

1257

videotron ltee

1255

oracle datenbanksysteme gmbh

1234

neostrada plus

1228

suddenlink communications

1214

dynamic ip pool for broadband customers

1202

eset s.r.o.

1200

Then the first 100 search keywords and phrases that led to us:


Keyword

Visits

(not provided)

53903

kifastsystemcallret

10644

crash dump analysis

10348

crash dump

9863

ntdll!kifastsystemcallret

4305

dump analysis

4143

adplus

3332

win32 error 0n2

2553

windbg commands

2198

memory dump analysis

2183

windbg

2131

crash dumps

1825

dumpanalysis.org

1818

nt!_gshandlercheck_seh

1734

dmitry vostokov

1718

crashdump

1683

symbol file could not be found

1669

bugcheck 3b

1458

memory dump analysis anthology

1393

crash dump analyzer

1360

warning: frame ip not in any known module. following frames may be wrong.

1347

windbg cheat sheet

1318

windbg crash dump analysis

1271

minidump analysis

1259

adplus download

1214

core dump analysis

1167

fnodobfm

1159

dumpanalysis

1142

windows 7 crash dump

1142

windbg analyze

1118

kisystemservicecopyend

1066

frame ip not in any known module

1010

getcontextstate failed, 0×80070026

949

crash dump windows 7

930

the stored exception information can be accessed via .ecxr.

925

windbg script

922

error: symbol file could not be found

912

vista crash dump

895

windows crash dump analysis

888

system_thread_exception_not_handled

857

анализ дампа памяти

857

dump analyzer

847

дамп памяти

821

pool corruption

820

time travel debugging

776

system_service_exception

772

kernel_mode_exception_not_handled

741

ntdll kifastsystemcallret

741

the stored exception information can be accessed via .ecxr

734

kmode_exception_not_handled

726

trap frame

719

idna trace

695

windbg crash dump

694

kiuserexceptiondispatcher

691

minidump analyzer

672

bugcheck 7e

670

kernel32!pnlsuserinfo

643

windbg scripts

641

rtlpwaitoncriticalsection

635

minidump

628

bugcheck system_service_exception

621

exception_double_fault

597

warning: stack unwind information not available. following frames may be wrong.

584

application_fault_status_breakpoint

583

crash dump vista

582

memory dump analysis tool

576

getcontextstate failed, 0xd0000147

575

memoretics

544

dumpanalysis.org/asmpedia

537

failure_bucket_id

524

“dec 15″ module windbg

511

error: symbol file could not be found.

511

download adplus

507

basethreadinitthunk

505

dr watson vista

505

ntkrnlmp.exe crash dump

499

ntdll.dll!kifastsystemcallret

492

rtlplowfragheapfree

488

analyze minidump

477

adplus tutorial

473

application_hang_blockedon_fileio

468

bios disassembly ninjutsu uncovered

460

ntdll.kifastsystemcallret

460

analyze crash dump

459

windows dump analysis

459

debug_flr_image_timestamp

456

system_thread_exception_not_handled (7e)

456

windbg dump analysis

446

windbg hang

438

windows debugging: practical foundations

434

crash dump analysis windbg

432

dynamicbase aslr

422

crash dump analysis tool

419

nt!kebugcheckex

414

rtluserthreadstart

414

type referenced: kernel32!pnlsuserinfo

407

error: symbol file could not be found. defaulted to export symbols for ntkrnlmp.exe

405

memory dump

403

warning: frame ip not in any known module. following frames may be wrong

399

application_hang_busyhang

398

Then browser stats (we have never thought that there are so many of them):


Browser

Visits

Internet Explorer

446051

Firefox

356686

Chrome

184535

Opera

45787

Safari

24123

Mozilla

3780

Mozilla Compatible Agent

2401

Android Browser

1337

Konqueror

1057

IE with Chrome Frame

982

Opera Mini

705

SeaMonkey

503

Safari (in-app)

197

Lunascape

144

BlackBerry8900

128

Camino

126

RockMelt

124

(not set)

96

Netscape

72

Playstation 3

36

IUC

34

Googlebot

29

Lynx

24

Unsupported Browser Version

22

BlackBerry9630

21

NetFront

17

BlackBerry9700

15

Microsoft-Symbol-Server

14

BlackBerry9000

12

Galeon

11

Midori

9

NokiaE63

9

Yahoo! Slurp

9

BlackBerry8530

8

BlackBerry8520

7

PagePeeker.com

7

SAMSUNG-SGH-I617

7

BlackBerry9530

6

JUC

6

MSR-ISRCCrawler

6

OpenWave

6

anonimo

5

BlackBerry9300

5

HTC_HD2_T8585 Opera

5

Nokia5233

5

Space Bison

5

-Vasya

4

Blazer

4

Uzbl

4

-^_^- Hello :)

3

<?echo ‘<pre>’; system

3

12345

3

BlackBerry9330

3

BlackBerry9650

3

HTC_P3700 Opera

3

HTC_TyTN_II Mozilla

3

NOKIAN78

3

Playstation Portable

3

PPC; 240×320; HTC_P3450

3

undefined GoogleToolbarBB

3

anonymous

2

Empty

2

GreatBrowse

2

Helyi user agent

2

HTC_Touch_Pro2_T7373 Opera

2

HTC_Touch2_T3333 Opera

2

J2ME

2

Mozilla 5.0

2

NokiaC1-01

2

NokiaC3-00

2

NokiaC7-00

2

NokiaX2-01

2

nwzfq

2

test

2

1

?M5

1

“PagePeeker.com”

1

<?include

1

<script>alert

1

<SCRIPT>window.location=’http:

1

2.0.0.11

1

31337′

1

8900b

1

AltaVista Intranet V2.0 evreka.com crawler@evreka.com

1

annoying

1

AppEngine-Google;

1

BlackBerry9500

1

BlackBerry9550

1

bwh3_user_agent

1

Citrix

1

EBABrowser

1

EY

1

fake_user_agent Mozilla

1

FAST Enterprise Crawler 6 used by Reed Exhibitions

1

foo

1

General Browser

1

GOOGLEBOT

1

HD_mini_T5555 Opera

1

Hellbrowser 6.66

1

holy_teacher FirePHP

1

HTC_P3490 Opera

1

HTC_P4550 Mozilla

1

HTC_Polaris Mozilla

1

HTC_Touch_3G_T3232 Opera

1

HTC_Touch_HD_T8282 Opera

1

HTC_Touch_Pro_T7272 Opera

1

HTC_Touch2_T3320 Opera

1

HTC-8900

1

IE 8

1

IE6

1

iTunes

1

Keep Out

1

KraftwayBrowser2

1

Links

1

Maemo Browser

1

Medusa

1

MERONG

1

Motorola_ES405B

1

mozilla

1

Mozilla Firefox

1

MS-OC 4.0

1

msie

1

NCSA Mosaic

1

NightDynamo AdminPanel v0.2.1

1

Nokia2700c-2

1

Nokia2730c-1

1

Nokia305

1

Nokia5230

1

Nokia5310XpressMusic

1

Nokia5800 XpressMusic

1

Nokia6300

1

Nokia6700c-1

1

NokiaC2-01

1

NokiaC2-02

1

NokiaC2-03

1

NokiaC5-03

1

nokiac6-00

1

NokiaC6-00

1

NOKIAE65

1

NokiaE66

1

NokiaE71

1

NokiaE71-2;Mozilla

1

NokiaE72-1

1

NokiaN-GageQD

1

NokiaN70-1

1

NokiaNokia 6210s

1

NoneOfYourBusiness

1

nothisname_wangxiaoyang3

1

OmniWeb

1

Palm750

1

Peeplo Screenshot Bot

1

PerTrUsTsQuiD

1

pippos.7

1

PPC; 480×800; HTC_Touch_HD_T8282; OpVer 34.159.1.612

1

PriceGoblin User Agent

1

Private

1

Privoxy

1

Read Later

1

SAMSUNG-GT-E2222

1

samsung-gt-s3653

1

samsung-gt-s3653 UNTRUSTED

1

SAMSUNG-S8000

1

SAMSUNG-SGH-I637

1

Samsung-SPHM540 Polaris

1

SmallProxy 3.5.4

1

SonyEricssonK750

1

Surf

1

tdhbrowser

1

TiFiC Client Z

1

union update table sd_users set userid=9 where username=’coco

1

unknown

1

Unknown

1

UNTRUSTED

1

Updownerbot

1

WIN

1

WinXP SP2

1

Wlwap

1

WM5 PIE

1

Xda_orbit_2; 240×320

1

Xyi znat kakoi browser MRA 5.7

1

ZooShot 0.1a

1

ZooShot 0.42

1

and finally mobile devices stats (you may find your own device there):


Mobile Device Info

Visits

Apple iPhone

2292

Apple iPad

1940

(not set)

1099

Samsung GT-I9100 Galaxy S II

167

Apple iPod Touch

112

Asus Eee Pad Transformer TF101

112

SonyEricsson LT15i Xperia Arc

94

Motorola Xoom

47

Samsung Galaxy Nexus

47

Samsung GT-I9000 Galaxy S

34

Samsung GT-P7510 Galaxy Tab 10.1

30

Google Nexus S Samsung Nexus S

26

HTC EVO 4G

26

Google Nexus 7

21

RIM BlackBerry Bold Touch 9900 Dakota

21

Samsung GT-N7000 Galaxy Note

21

Acer A500 Picasso

17

Asus Eee Pad TF201 Transformer Prime

17

HTC Desire HD

17

Motorola DroidX

17

Motorola XT862 Droid 3

17

Samsung GT-S5830 Galaxy Ace

17

Samsung SGH-I747 Galaxy SIII

17

Samsung SGH-i917 Omnia 7

17

Verizon Droid2

17

Google Nexus One

13

Google Nexus One HTC Nexus One

13

HTC ADR6300 Incredible

13

Motorola Droid 2

13

Samsung GT-P7500 P4

13

Samsung SHW-M250K GALAXY S II (KT)

13

Apple iPod

9

BlackBerry 9780

9

BlackBerry 9800 Torch

9

Dell Venue Pro

9

HTC Desire

9

HTC G2 HTC Sappire

9

HTC HD7

9

HTC T9292 HD7

9

Motorola MB860 Atrix

9

Nokia E63

9

RIM BlackBerry 8530 Curve

9

Samsung GT-I9001

9

Samsung GT-I9300 Galaxy S3

9

Samsung GT-N8000 Galaxy Note 10.1

9

Samsung GT-P1000 Galaxy Tab

9

Sharp IS03 IS03 for KDDI

9

T-Mobile myTouch4G

9

Toshiba AT100

9

ZTE N860

9

Acer A101 Vangogh

4

Acer A200 Picasso_E

4

Acer Acer E310 Liquid Mini

4

Asus TF300T Transformer Pad TF300T

4

BlackBerry 8520 Curve

4

BlackBerry 9900 Dakota

4

DoCoMo L-05D Optimus it

4

DoCoMo P502i

4

Fujitsu F-12C F-12C for DoCoMo

4

Google Nexus S

4

Google Wireless Transcoder

4

HTC A8181 Desire

4

HTC ADR6350 Droid Incredible 2

4

HTC ADR6400L Thunderbolt

4

HTC ADR6400L Thunderbolt 4G

4

HTC APC715CKT EVO Design 4G

4

HTC Bravo

4

HTC Desire X0H6T

4

HTC Glacier

4

HTC Incredible S Incredible S

4

HTC Inspire 4G

4

HTC ISW12HT EVO 3D ISW12HT for KDDI

4

HTC Mozart 7 Mozart

4

HTC PC36100 EVO 4G

4

HTC PJ83100 One X

4

HTC Radar 4G

4

HTC S510e Desire S

4

HTC T7380 TouchFLO

4

HTC X515 EVO 3D

4

Huawei M860 Ascend

4

Huawei u8800 Ideos X5

4

kddi ISW11HT HTC EVO WiMAX ISW11HT for KDDI

4

LG C900 Quantum

4

LG E900 Optimus 7

4

LG LS670 Optimus S

4

LG MS690 Optimus M

4

LG VM670 Optimus V

4

LG VS910 4G Revolution

4

Motorola A953 MILESTONE 2

4

Motorola ISW11M PHOTON ISW11M for KDDI

4

Motorola MB501

4

Motorola MB525 DEFY

4

Motorola MB611

4

Motorola MOTXT912B Droid Razr 4G

4

Motorola MZ601 Xoom

4

Motorola MZ604 Xoom

4

Motorola MZ605 Xoom

4

Motorola xt875 Droid Bionic

4

Nokia 5800d XpressMusic

4

Nokia C3-00

4

Nokia C5-03 C5

4

Nokia C6-00

4

Nokia Lumia 710

4

Nokia Lumia 800

4

RIM BlackBerry 9300 Curve 3G

4

RIM BlackBerry 9700 Bold

4

RIM BlackBerry 9800 Torch

4

RIM Blackberry Bold Touch 9930

4

Samsung GT i5700 Galaxy Spica

4

Samsung GT I9000T Galaxy S

4

Samsung GT-I9100G Galaxy S II

4

Samsung GT-I9100P Galaxy S II NFC

4

Samsung GT-I9103

4

Samsung GT-I9300 Galaxy SIII

4

Samsung GT-N8010 Galaxy Note 10.1

4

Samsung GT-P7500 Galaxy Tab 10.1

4

Samsung SCH-I500 Fascinate

4

Samsung SCH-I535 4G Galaxy SIII

4

Samsung SGH-i717 Galaxy Note

4

Samsung SGH-I747 Galaxy S3

4

Samsung SGH-I777

4

Samsung SGH-I777 Galaxy S II

4

Samsung SGH-I897 Galaxy S Captivate

4

Samsung SHW-M250S GALAXY S II (SKT)

4

Samsung SPH-D700 Epic 4G

4

Samsung SWH-M110S

4

Sharp 003SH Sharp Galapagos 003SH for SoftBank

4

Softbank 001DL DELL Streak

4

SonyEricsson LT26i Xperia Arc HD

4

Xiaomi MI-ONE Plus M1 Plus

4


- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 192)

Friday, January 4th, 2013

It is common to get dozens of process memory dumps saved sequentially, for example, after each second. Then we can first analyze memory dumps corresponding to changes in their file sizes ignoring plateaus to save analysis time. We call such pattern Step Dumps by an analogy with step functions. For example, we have this dump set (it was reported that an application was freezing for some time until its disappearance from a user screen) with comments from WinDbg analysis sessions:

C:\MemoryDumps>dir
[...]
12/30/2012  8:33 PM  218,252,862 AppA-1.dmp // normal
12/30/2012  8:34 PM  218,541,762 AppA-2.dmp // slightly increased CPU consumption for thread #11
12/30/2012  8:37 PM  218,735,848 AppA-3.dmp // spiking thread #11
12/30/2012  8:38 PM  218,735,848 AppA-4.dmp
12/30/2012  8:38 PM  218,735,848 AppA-5.dmp
12/30/2012  8:39 PM  218,735,848 AppA-6.dmp
12/30/2012  8:39 PM  218,735,848 AppA-7.dmp
12/30/2012  8:39 PM  218,735,848 AppA-8.dmp
12/30/2012  8:40 PM  218,735,848 AppA-9.dmp
12/30/2012  8:40 PM  218,735,848 AppA-10.dmp
12/30/2012  8:41 PM  218,735,848 AppA-11.dmp 

12/30/2012  8:41 PM  218,735,848 AppA-12.dmp // spiking thread #11
12/30/2012  8:42 PM  219,749,040 AppA-13.dmp // spiking thread #11, another thread blocked in ALPC
12/30/2012  8:42 PM  219,048,842 AppA-14.dmp // only one thread left
[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 191)

Monday, December 31st, 2012

Sometimes we need to check under what security principal or group we run a process or what privileges it has or whether it has impersonating threads. We call this pattern Deviant Token when we find, for example, an unexpected token with a different security identifier, for example, Network Service instead of Local System (SID: S-1-5-18):

PROCESS 8f218d88  SessionId: 0  Cid: 09c4    Peb: 7ffdf000  ParentCid: 0240
DirBase: bffd4260  ObjectTable: e10eae90  HandleCount:  93.
Image: ServiceA.exe
VadRoot 8f1f70e8 Vads 141 Clone 0 Private 477. Modified 2. Locked 0.
DeviceMap e10038d8
Token                             e10ff5d8
[…]

0: kd> !token e10ff5d8
_TOKEN e10ff5d8
TS Session ID: 0
User: S-1-5-20
[…]

Well-known SIDs can be found in this MS article: http://support.microsoft.com/kb/243330

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Malware Analysis Patterns (Part 2)

Saturday, December 29th, 2012

As was announced earlier we start cataloguing elemental malware detection and analysis patterns. We skip Part 1 because we assign Deviant Module to it. Part 2 deals with Fake Module pattern where one of loaded modules masquerades as a legitimate system DLL or a widely known value adding DLL from some popular 3rd party product. To illustrate this pattern we modeled it as Victimware: a process crashed after loading a malware module:

0:000> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
00000000`0026f978 00000001`3f89103a 0x0
00000000`0026f980 00000001`3f8911c4 FakeModule!wmain+0x3a
00000000`0026f9c0 00000000`76e3652d FakeModule!__tmainCRTStartup+0x144
00000000`0026fa00 00000000`7752c521 kernel32!BaseThreadInitThunk+0xd
00000000`0026fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

When we inspect loaded modules we don’t find anything suspicious:

0:000> lmp
start             end                 module name
00000000`76e20000 00000000`76f3f000   kernel32 <none>
00000000`77500000 00000000`776a9000   ntdll    <none>
00000001`3f890000 00000001`3f8a6000   FakeModule <none>
000007fe`f8cb0000 000007fe`f8cc7000   winspool <none>
000007fe`fdb30000 000007fe`fdb9c000   KERNELBASE <none>

However, when checking modules images for any modifications we find that winspool was not compared with existing binary from Microsoft symbol server:

0:000> !for_each_module "!chkimg -v -d @#ModuleName"
Searching for module with expression: kernel32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\kernel32.dll\503285C111f000\kernel32.dll
No range specified

Scanning section:    .text
Size: 633485
Range to scan: 76e21000-76ebba8d
Total bytes compared: 633485(100%)
Number of errors: 0
0 errors : kernel32
Searching for module with expression: ntdll
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\ntdll.dll\4EC4AA8E1a9000\ntdll.dll
No range specified

Scanning section:    .text
Size: 1049210
Range to scan: 77501000-7760127a
Total bytes compared: 1049210(100%)
Number of errors: 0

Scanning section:       RT
Size: 474
Range to scan: 77602000-776021da
Total bytes compared: 474(100%)
Number of errors: 0
0 errors : ntdll
Searching for module with expression: FakeModule
Error for FakeModule: Could not find image file for the module. Make sure binaries are included in the symbol path.
Searching for module with expression: winspool
Error for winspool: Could not find image file for the module. Make sure binaries are included in the symbol path.

Searching for module with expression: KERNELBASE
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: C:\WSDK8\Debuggers\x64\sym\KERNELBASE.dll\503285C26c000\KERNELBASE.dll
No range specified

Scanning section:    .text
Size: 302047
Range to scan: 7fefdb31000-7fefdb7abdf
Total bytes compared: 302047(100%)
Number of errors: 0
0 errors : KERNELBASE

Checking module data reveals that it was loaded not from System32 folder and doesn’t have any version information:

0:000> lmv m winspool
start             end                 module name
000007fe`f8cb0000 000007fe`f8cc7000   winspool   (deferred)
Image path: C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
Image name: winspool.drv
Timestamp:        Fri Dec 28 22:22:42 2012 (50DE1BB2)
CheckSum:         00000000
ImageSize:        00017000
File version:     0.0.0.0
Product version:  0.0.0.0
File flags:       0 (Mask 0)
File OS:          0 Unknown Base
File type:        0.0 Unknown
File date:        00000000.00000000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

We could see that path from running this command as well :

0:000> !for_each_module
00: 0000000076e20000  0000000076f3f000         kernel32 C:\Windows\System32\kernel32.dll                      kernel32.dll
01: 0000000077500000  00000000776a9000            ntdll C:\Windows\System32\ntdll.dll                            ntdll.dll
02: 000000013f890000  000000013f8a6000       FakeModule C:\Work\AWMA\FakeModule\x64\Release\FakeModule.exe  FakeModule.exe
03: 000007fef8cb0000  000007fef8cc7000         winspool C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
04: 000007fefdb30000  000007fefdb9c000       KERNELBASE C:\Windows\System32\KERNELBASE.dll                  KERNELBASE.dll

or from PEB:

0:000> !peb
PEB at 000007fffffdf000
[...]
7fef8cb0000 50de1bb2 Dec 28 22:22:42 2012 C:\Work\AWMA\FakeModule\x64\Release\winspool.drv
[…]

Another sign is module size in memory which is much smaller than real winspool.drv:

0:000> ? 000007fe`f8cc7000 - 000007fe`f8cb0000
Evaluate expression: 94208 = 00000000`0001700

Module size can help if legitimate module from well-known folder was replaced. Module debug directory and the size of export and import directories are also different with the former revealing the development folder:

0:000> !dh 000007fe`f8cb0000
[...]
   0 [       0] address [size] of Export Directory
[…]
9000 [     208] address [size] of Import Address Table Directory
[…]
Debug Directories(2)
Type       Size     Address  Pointer
cv           49        e2c0     cac0 Format: RSDS, guid, 1, C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb

This can also be seen from the output of !lmi command:

0:000> !lmi 7fef8cb0000
Loaded Module Info: [7fef8cb0000]
Module: winspool
Base Address: 000007fef8cb0000
Image Name: winspool.drv
Machine Type: 34404 (X64)
Time Stamp: 50de1bb2 Fri Dec 28 22:22:42 2012
Size: 17000
CheckSum: 0
Characteristics: 2022
Debug Data Dirs: Type  Size     VA  Pointer
CODEVIEW    49,  e2c0,    cac0 RSDS - GUID: {29D85193-1C9D-4997-95BA-DD190FA3C1BF}
Age: 1, Pdb: C:\Work\AWMA\FakeModule\x64\Release\winspool.pdb
??    10,  e30c,    cb0c [Data not mapped]
Symbol Type: DEFERRED - No error - symbol load deferred
Load Report: no symbols loaded

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 190)

Monday, December 24th, 2012

In addition to stack trace collection we often are interested in Module Collection (we called this pattern initially Vendor Collection), especially if we would like to check if some vendor DLL is present in some process address space in a complete memory dump (kernel module list or module list from a process memory dump is trivial). Or we need to check for some vendor information from problem description (lmv command). If we have a complete memory dump from x64 system then listing modules for each process is not enough. For example, we might have this:

0: kd> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)

AppA is a 32-bit process and has an additional 32-bit module list that is more useful. We can set x86 context for a thread from that process and get the list of 32-bit modules:

0: kd> .load wow64exts

0: kd> .thread /w fffffa800e372060
Implicit thread is now fffffa80`0e372060
x86 context set

0: kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

0: kd:x86> lmu
start             end                 module name
00000000`00ab0000 00000000`00ae8000   AppA    (deferred)
00000000`73490000 00000000`73515000   COMCTL32   (deferred)
00000000`73520000 00000000`735c3000   MSVCR90    (deferred)
00000000`735d0000 00000000`7365e000   MSVCP90    (deferred)
00000000`74920000 00000000`7493e000   USERENV    (deferred)
00000000`74940000 00000000`74ade000   comctl32_74940000   (deferred)
00000000`74af0000 00000000`74b02000   MSASN1     (deferred)
00000000`74b10000 00000000`74c03000   CRYPT32    (deferred)
00000000`74dc0000 00000000`74e5b000   MSVCR80    (deferred)
00000000`74f60000 00000000`74fd6000   NETAPI32   (deferred)
00000000`74fe0000 00000000`7502e000   wow64win   (deferred)
00000000`75030000 00000000`75075000   wow64      (deferred)
00000000`750b0000 00000000`750ba000   WTSAPI32   (deferred)
00000000`750c0000 00000000`750c9000   wow64cpu   (deferred)
00000000`75cf0000 00000000`75d50000   Secur32    (deferred)
00000000`75d50000 00000000`76861000   SHELL32    (deferred)
00000000`76a10000 00000000`76aa0000   GDI32      (deferred)
00000000`76b30000 00000000`76b90000   IMM32      (deferred)
00000000`76be0000 00000000`76cf0000   kernel32   (deferred)
00000000`76e30000 00000000`76f75000   ole32      (deferred)
00000000`76f80000 00000000`7702a000   msvcrt     (deferred)
00000000`77030000 00000000`77037000   PSAPI      (deferred)
00000000`77040000 00000000`77110000   USER32     (deferred)
00000000`77110000 00000000`77169000   SHLWAPI    (deferred)
00000000`77170000 00000000`771ed000   USP10      (deferred)
00000000`77380000 00000000`7740d000   OLEAUT32   (deferred)
00000000`77640000 00000000`77649000   LPK        (deferred)
00000000`776e0000 00000000`777d0000   RPCRT4     (deferred)
00000000`777d0000 00000000`77898000   MSCTF      (deferred)
00000000`778a0000 00000000`77966000   ADVAPI32   (deferred)
00000000`77b70000 00000000`77cf7000   ntdll      (pdb symbols)
00000000`77d30000 00000000`77e90000   ntdll_77d30000 # (pdb symbols)

So it looks like we need to dump modules for each thread. However, the output would be enormous unless we skip threads having the same PID. After some tinkering I wrote this WinDbg script with moderate output volume:

.load wow64exts
!for_each_thread ".thread @#Thread; .if (@$t0 != @@c++(@$thread->Cid.UniqueProcess)) {.reload /user;lmvu;.thread /w @#Thread;.reload /user;lmvu;r $t0 = @@c++(@$thread->Cid.UniqueProcess);.effmach AMD64; }"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 189)

Sunday, December 23rd, 2012

Although a handle leak may lead to Insufficient Memory it is not always the case especially if pool structures are small such as events. So we describe another pattern called Handle Leak that covers high memory usage (including fat structures), high handle counts and also abnormal differences in allocations and deallocations. As an example for the latter here is a nonpaged pool leak of Event objects and correlated pooltag ABCD. Although memory usage footprint is small compared with other nonleaking pooltags we see the difference between Allocs and Frees is surely abnormal correlating with high handle counts:

0: kd> !poolused 3
Sorting by  NonPaged Pool Consumed

Pool Used:
NonPaged                    Paged
Tag    Allocs    Frees     Diff     Used   Allocs    Frees     Diff     Used
[…]
ABCD  1778517  1704538    73979  4734656        0        0        0        0 UNKNOWN pooltag ‘ABCD’, please update pooltag.txt
Even  6129633  6063728    65905  4224528        0        0        0        0 Event objects
[…]

0: kd> !process 0 0

[...]

PROCESS d2b85360  SessionId: 2  Cid: 1bf4    Peb: 7ffdf000  ParentCid: 1688
DirBase: 7d778dc0  ObjectTable: e53dda08  HandleCount: 18539.
Image: AppA.exe

PROCESS b2fcd670  SessionId: 2  Cid: 0818    Peb: 7ffd4000  ParentCid: 1688
DirBase: 7d778400  ObjectTable: b3ffd8c0  HandleCount: 36252.
Image: AppB.exe

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 188)

Sunday, December 23rd, 2012

Critical Stack Trace pattern addresses abnormal behaviour such as page fault processing or any other critical system activity that is waiting too long. Such activity is either finishes quickly or lead to normal bugcheck processing code. For example, this thread is stuck in page fault processing for 32 minutes while loading a resource:

THREAD fffffa80f0603c00  Cid 376.3d6  Teb: 000007fffffd6000 Win32Thread: fffff900c09e0640 WAIT: (Executive) KernelMode Non-Alertable
[...]
Wait Start TickCount      6281298        Ticks: 123391 (0:00:32:04.102)
[…]
Child-SP          RetAddr           Call Site
fffff880`3fc99030 fffff800`01882bd2 nt!KiSwapContext+0×7a
fffff880`3fc99170 fffff800`01893f8f nt!KiCommitThreadWait+0×1d2
fffff880`3fc99200 fffff880`016283ff nt!KeWaitForSingleObject+0×19f
fffff880`3fc992a0 fffff880`01620fc6 Ntfs!NtfsNonCachedIo+0×23f
fffff880`3fc99470 fffff880`01622a68 Ntfs!NtfsCommonRead+0×7a6
fffff880`3fc99610 fffff880`00fb4bcf Ntfs!NtfsFsdRead+0×1b8
fffff880`3fc99820 fffff880`00fb36df fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×24f
fffff880`3fc998b0 fffff800`018b44f5 fltmgr!FltpDispatch+0xcf
fffff880`3fc999a0 fffff800`018b3fc9 nt!IoPageRead+0×255

fffff880`3fc99a30 fffff800`0189a85a nt!MiIssueHardFault+0×255
fffff880`3fc99ac0 fffff800`0188b2ee nt!MmAccessFault+0×146a
fffff880`3fc99c20 00000000`779da643 nt!KiPageFault+0×16e (TrapFrame @ fffff880`3fd99c20)

00000000`039ff4f0 00000000`779d8b1e ntdll!LdrpGetRcConfig+0xcd
00000000`039ff580 00000000`779da222 ntdll!LdrIsResItemExist+0×1e
00000000`039ff5c0 00000000`779f82c4 ntdll!LdrpSearchResourceSection_U+0xa4
00000000`039ff6e0 000007fe`fe0075c1 ntdll!LdrFindResource_U+0×44
00000000`039ff720 000007fe`fb217777 KERNELBASE!FindResourceExW+0×85
[…]

The Top Blocking Module is NTFS so we might want then to look for other similar stack traces from stack trace collection.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 187, Mac OS X)

Saturday, November 17th, 2012

Here we publish Active Thread pattern that was previously introduced in Accelerated Mac OS X Core Dump Analysis training. Basically it is a thread that is not waiting or suspended (most threads are). However, from a memory dump it is not possible to find out whether it was spiking at the dump generation time (unless we have a set of memory snapshots and in each one we have the same or similar back trace) and we don’t have any paratext with CPU consumption stats for threads. For example, in one core dump we have these threads:

(gdb) info threads
12 0x98c450ee in __workq_kernreturn ()
11 0x98c4280e in semaphore_wait_trap ()
10 0x98c448e2 in __psynch_cvwait ()
9 0×00110171 in std::_Rb_tree<int, std::pair<int const, _iCapture*>, std::_Select1st<std::pair<int const, _iCapture*> >, std::less<int>, std::allocator<std::pair<int const, _iCapture*> > >::find ()
8 0×98c428e6 in mach_wait_until ()
7 0×98c448e2 in __psynch_cvwait ()
6 0×98c427d2 in mach_msg_trap ()
5 0×98c427d2 in mach_msg_trap ()
4 0×98c428e6 in mach_wait_until ()
3 0×98c427d2 in mach_msg_trap ()
2 0×98c459ae in kevent ()
*  1 0×014bcee0 in cgGLGetLatestProfile ()

Threads #9 and #1 are not waiting so we inspect their back traces:

(gdb) bt
#0  0x014bcee0 in cgGLGetLatestProfile ()
#1  0x99060dd5 in exit ()
#2  0x001ef859 in os_exit ()
#3  0x001dc873 in luaD_precall ()
#4  0x001e7d9e in luaV_execute ()
#5  0x001dc18b in luaD_rawrunprotected ()
#6  0x001dced4 in lua_resume ()
#7  0x0058a526 in ticLuaManager::executeProgram ()
#8  0x005a09af in ticLuaScript::_execute ()
#9  0x003a6480 in darcScript::execute ()
#10 0x003af4d8 in darcTimeline::execute ()
#11 0x0034a2ba in darcSequenceur::executeAll ()
#12 0x0036904b in darcEventManager::ExecuteEventHandler ()
#13 0x003a37d2 in darcScene::process ()
#14 0x0034a2ba in darcSequenceur::executeAll ()
#15 0x0036904b in darcEventManager::ExecuteEventHandler ()
#16 0x00343ec0 in darcContext::process ()
#17 0x00347339 in darcContext::main ()
#18 0x003cf73d in darcPlayerImpl::renderOneFrame ()
#19 0x003cf078 in darcPlayerImpl::render ()
#20 0x000b1f6f in Run ()
#21 0x000b1fe9 in tiMain ()
#22 0x000c73ee in main ()

(gdb) thread 9
[Switching to thread 9 (core thread 8)]
0x00110171 in std::_Rb_tree<int, std::pair<int const, _iCapture*>, std::_Select1st<std::pair<int const, _iCapture*> >, std::less<int>, std::allocator<std::pair<int const, _iCapture*> > >::find ()

(gdb) bt
#0  0x00110171 in std::_Rb_tree<int, std::pair<int const, _iCapture*>, std::_Select1st<std::pair<int const, _iCapture*> >, std::less<int>, std::allocator<std::pair<int const, _iCapture*> > >::find ()
#1  0x0010f936 in ticVideoManager::isPaused ()
#2  0x00201801 in ticMLT_VideoCapture::Execute ()
#3  0x0020aa0b in ticModuleGraph::runOnce ()
#4  0x002632be in TrackingApp::ProcessTracking ()
#5  0x005b2f5d in ticMLTTracking::processInternal ()
#6  0x005b322d in ticMLTTracking::processThread ()
#7  0x005b36f3 in trackingThread ()
#8  0x004eaf1e in ticThread::threadFunc ()
#9  0x99023557 in _pthread_start ()
#10 0x9900dcee in thread_start ()

Windows equivalent would be a process memory dump which doesn’t have any information saved for !runaway WinDbg command.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Optimization Patterns

Friday, November 16th, 2012

A page to reference all different kinds of optimization patterns is necessary, so I created this post:

I’ll update it as soon as I add more similar patterns.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

WinDbg Reference Cards Version 2 (Page 1)

Thursday, November 15th, 2012

Finally, the new version of WinDbg: A Reference Poster and Learning Cards is under development. This time every page is published online for comments, suggestions and corrections which are very welcome. The format of every page follows colored memory space diagram where red cards are for native kernel space commands, blue cards are for unmanaged user space, and green cards are for managed .NET space (click on a picture to open a PDF file):

Download page 1 PDF file

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Crash Dump Analysis Patterns (Part 186)

Tuesday, November 13th, 2012

Not all processes are linked into a list that some commands traverse such as !process 0 0. A processes may unlink itself or be in an initialization stage. However, a process structure is allocated from nonpaged pool and such pool can be searched for ”Proc” pool tag (unless a process changes that in memory). We call such pattern Hidden Process. For example:

0: kd> !poolfind Proc

Searching NonPaged pool (83c3c000 : 8bc00000) for Tag: Proc

*87b15000 size:  298 previous size:    0  (Free)      Pro.
*87b18370 size:  298 previous size:   98  (Allocated) Proc (Protected)
[...]
*8a35e900 size:  298 previous size:   30  (Allocated) Proc (Protected)
*8a484000 size:  298 previous size:    0  (Allocated) Proc (Protected)
*8a4a2d68 size:  298 previous size:   28  (Allocated) Proc (Protected)
[…]

One such structure is missing from active process linked list (note that it has a parent PID):

0: kd> !process 8a484000+20
PROCESS 8a484020  SessionId: 0  Cid: 05a0    Peb: 00000000  ParentCid: 0244
DirBase: bffc2200  ObjectTable: e17e6a78  HandleCount:   0.
Image: AppChild.exe
VadRoot 8a574f80 Vads 4 Clone 0 Private 3. Modified 0. Locked 0.
DeviceMap e1002898
Token                             e1a36030
ElapsedTime                       00:00:00.000
UserTime                          00:00:00.000
KernelTime                        419 Days 13:24:16.625
QuotaPoolUsage[PagedPool]         7580
QuotaPoolUsage[NonPagedPool]      160
Working Set Sizes (now,min,max)  (12, 50, 345) (48KB, 200KB, 1380KB)
PeakWorkingSetSize                12
VirtualSize                       1 Mb
PeakVirtualSize                   1 Mb
PageFaultCount                    5
MemoryPriority                    BACKGROUND
BasePriority                      8
CommitCharge                      156

No active threads

We may think that this process is zombie (note that unlike terminated processes it has a non-zero data such as VAD and object table and zero PEB and elapsed time) but inspection of its parent process thread stacks reveals that it was in the process of creation (note an attached process field):

THREAD 8a35dad8  Cid 0244.0248  Teb: 7ffdd000 Win32Thread: bc3aa688 WAIT: (Unknown) KernelMode Non-Alertable
ba971608  NotificationEvent
Impersonation token:  e2285030 (Level Impersonation)
DeviceMap                 e1a31a58
Owning Process            8a35e920       Image:         AppParent.exe
Attached Process          8a484020       Image:         AppChild.exe
Wait Start TickCount      2099           Ticks: 1 (0:00:00:00.015)
Context Switch Count      279                 LargeStack
UserTime                  00:00:00.046
KernelTime                00:00:00.046
Win32 Start Address AppParent!mainCRTStartup (0×0100d303)
Start Address kernel32!BaseProcessStartThunk (0×77e617f8)
Stack Init ba972000 Current ba971364 Base ba972000 Limit ba96e000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
ba97137c 80833f2d nt!KiSwapContext+0×26
ba9713a8 80829c72 nt!KiSwapThread+0×2e5
ba9713f0 bad3c9db nt!KeWaitForSingleObject+0×346
[…]
ba971b94 8094cfc3 nt!MmCreatePeb+0×2cc
ba971ce4 8094d42d nt!PspCreateProcess+0×5a9
ba971d38 8088b4ac nt!NtCreateProcessEx+0×77

ba971d38 7c82845c nt!KiFastCallEntry+0xfc (TrapFrame @ ba971d64)
0006f498 7c826d09 ntdll!KiFastSystemCallRet
0006f49c 77e6cf95 ntdll!ZwCreateProcessEx+0xc
0006fcc0 7d1ec670 kernel32!CreateProcessInternalW+0×15e5
0006fd0c 01008bcf ADVAPI32!CreateProcessAsUserW+0×108

[…]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -