Crash Dump Analysis Patterns (Part 294)
Monday, March 18th, 2024Generative AI LLM models such as GPT-4 can be used as Analysis Summary devices (although such summaries should be treated with caution and double-checked). For example, let’s look at the default !analyze -v WinDbg command summary:
Summarize:
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: ffffbc06b64895a0, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff80403c61981, address which referenced memory
Debugging Details:
------------------
Unable to load image \??\C:\WINDOWS\system32\drivers\myfault.sys, Win32 error 0n2
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 1717
Key : Analysis.Elapsed.mSec
Value: 7052
Key : Analysis.IO.Other.Mb
Value: 0
Key : Analysis.IO.Read.Mb
Value: 3
Key : Analysis.IO.Write.Mb
Value: 10
Key : Analysis.Init.CPU.mSec
Value: 82327
Key : Analysis.Init.Elapsed.mSec
Value: 75855690
Key : Analysis.Memory.CommitPeak.Mb
Value: 577
Key : Bugcheck.Code.KiBugCheckData
Value: 0xd1
Key : Bugcheck.Code.LegacyAPI
Value: 0xd1
Key : Bugcheck.Code.TargetModel
Value: 0xd1
Key : Failure.Bucket
Value: AV_myfault!unknown_function
Key : Failure.Hash
Value: {9745090a-9bce-ccba-c096-ca6e9ca04c64}
Key : Hypervisor.Enlightenments.Value
Value: 8480
Key : Hypervisor.Enlightenments.ValueHex
Value: 2120
Key : Hypervisor.Flags.AnyHypervisorPresent
Value: 1
Key : Hypervisor.Flags.ApicEnlightened
Value: 0
Key : Hypervisor.Flags.ApicVirtualizationAvailable
Value: 0
Key : Hypervisor.Flags.AsyncMemoryHint
Value: 0
Key : Hypervisor.Flags.CoreSchedulerRequested
Value: 0
Key : Hypervisor.Flags.CpuManager
Value: 0
Key : Hypervisor.Flags.DeprecateAutoEoi
Value: 0
Key : Hypervisor.Flags.DynamicCpuDisabled
Value: 0
Key : Hypervisor.Flags.Epf
Value: 0
Key : Hypervisor.Flags.ExtendedProcessorMasks
Value: 0
Key : Hypervisor.Flags.HardwareMbecAvailable
Value: 0
Key : Hypervisor.Flags.MaxBankNumber
Value: 0
Key : Hypervisor.Flags.MemoryZeroingControl
Value: 0
Key : Hypervisor.Flags.NoExtendedRangeFlush
Value: 1
Key : Hypervisor.Flags.NoNonArchCoreSharing
Value: 0
Key : Hypervisor.Flags.Phase0InitDone
Value: 1
Key : Hypervisor.Flags.PowerSchedulerQos
Value: 0
Key : Hypervisor.Flags.RootScheduler
Value: 0
Key : Hypervisor.Flags.SynicAvailable
Value: 0
Key : Hypervisor.Flags.UseQpcBias
Value: 0
Key : Hypervisor.Flags.Value
Value: 536584
Key : Hypervisor.Flags.ValueHex
Value: 83008
Key : Hypervisor.Flags.VpAssistPage
Value: 1
Key : Hypervisor.Flags.VsmAvailable
Value: 0
Key : Hypervisor.RootFlags.AccessStats
Value: 0
Key : Hypervisor.RootFlags.CrashdumpEnlightened
Value: 0
Key : Hypervisor.RootFlags.CreateVirtualProcessor
Value: 0
Key : Hypervisor.RootFlags.DisableHyperthreading
Value: 0
Key : Hypervisor.RootFlags.HostTimelineSync
Value: 0
Key : Hypervisor.RootFlags.HypervisorDebuggingEnabled
Value: 0
Key : Hypervisor.RootFlags.IsHyperV
Value: 0
Key : Hypervisor.RootFlags.LivedumpEnlightened
Value: 0
Key : Hypervisor.RootFlags.MapDeviceInterrupt
Value: 0
Key : Hypervisor.RootFlags.MceEnlightened
Value: 0
Key : Hypervisor.RootFlags.Nested
Value: 0
Key : Hypervisor.RootFlags.StartLogicalProcessor
Value: 0
Key : Hypervisor.RootFlags.Value
Value: 0
Key : Hypervisor.RootFlags.ValueHex
Value: 0
Key : SecureKernel.HalpHvciEnabled
Value: 0
Key : WER.OS.Branch
Value: co_release
Key : WER.OS.Version
Value: 10.0.22000.1
BUGCHECK_CODE: d1
BUGCHECK_P1: ffffbc06b64895a0
BUGCHECK_P2: 2
BUGCHECK_P3: 0
BUGCHECK_P4: fffff80403c61981
FILE_IN_CAB: MEMORY-WM.DMP
VIRTUAL_MACHINE: VMware
READ_ADDRESS: ffffbc06b64895a0 Paged pool
BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXWINLOGON: 1
PROCESS_NAME: notmyfault64.exe
TRAP_FRAME: ffffa7848ee7d7c0 -- (.trap 0xffffa7848ee7d7c0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=000000000210f204 rbx=0000000000000000 rcx=ffffbc06aa200380
rdx=0000000000000880 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80403c61981 rsp=ffffa7848ee7d950 rbp=ffffa7848ee7dbc1
r8=ffffbc06b9320aa0 r9=0000000000000000 r10=ffffbc06aa200300
r11=ffffbc06b647e590 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz na po nc
myfault+0x1981:
fffff804`03c61981 mov eax,dword ptr [rbx] ds:00000000`00000000=????????
Resetting default scope
STACK_TEXT:
ffffa784`8ee7d678 fffff804`04e28da9 : 00000000`0000000a ffffbc06`b64895a0 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
ffffa784`8ee7d680 fffff804`04e24f00 : ffffa784`8ee7d878 ffffa784`8ee7d7fc ffffbc06`b647e580 fffff804`04cfcba7 : nt!KiBugCheckDispatch+0x69
ffffa784`8ee7d7c0 fffff804`03c61981 : 00000000`00000000 ffff8008`aaed9ef0 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x440
ffffa784`8ee7d950 fffff804`03c61d3d : ffff8008`0210f204 fffff906`021a0010 00000000`00000000 00000000`00000000 : myfault+0x1981
ffffa784`8ee7d980 fffff804`03c61ea1 : ffff8008`b0f05120 fffff804`050a4524 ffff8008`aa135b00 00000000`00001dc9 : myfault+0x1d3d
ffffa784`8ee7dac0 fffff804`04d03115 : ffff8008`b0f05120 00000000`00000002 00000000`000000f0 00000000`00000000 : myfault+0x1ea1
ffffa784`8ee7db20 fffff804`0516bbf2 : 00000000`00000001 ffff8008`b0f05120 ffffa784`8ee7dbc1 fffff804`04d02ed3 : nt!IofCallDriver+0x55
ffffa784`8ee7db60 fffff804`0516b9d2 : ffff8008`00000000 ffffa784`8ee7dea0 00000000`83360018 ffff8008`b0f05120 : nt!IopSynchronousServiceTail+0x1d2
ffffa784`8ee7dc10 fffff804`0516ad36 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xc82
ffffa784`8ee7dd40 fffff804`04e28775 : 00000000`00000000 00000000`00000000 ffff0cbf`113fcb36 00000000`00070000 : nt!NtDeviceIoControlFile+0x56
ffffa784`8ee7ddb0 00007ffe`4d263834 : 00007ffe`4a893ffb 00000000`0008040a 00000014`00000048 00007ffe`4b031434 : nt!KiSystemServiceCopyEnd+0x25
0000009b`cc0fedd8 00007ffe`4a893ffb : 00000000`0008040a 00000014`00000048 00007ffe`4b031434 00000000`00000000 : ntdll!NtDeviceIoControlFile+0x14
0000009b`cc0fede0 00007ffe`4bb95f91 : 00000000`83360018 00000000`00000000 00000000`00000000 00007ffe`4cda99f9 : KERNELBASE!DeviceIoControl+0x6b
0000009b`cc0fee50 00007ff7`6544342f : 00000000`0008050e 0000009b`cc0fef39 0000009b`cc0fef39 00000000`00000000 : KERNEL32!DeviceIoControlImplementation+0x81
0000009b`cc0feea0 00000000`0008050e : 0000009b`cc0fef39 0000009b`cc0fef39 00000000`00000000 00000000`00000000 : 0x00007ff7`6544342f
0000009b`cc0feea8 0000009b`cc0fef39 : 0000009b`cc0fef39 00000000`00000000 00000000`00000000 00000000`00000000 : 0x8050e
0000009b`cc0feeb0 0000009b`cc0fef39 : 00000000`00000000 00000000`00000000 00000000`00000000 0000009b`cc0feee0 : 0x0000009b`cc0fef39
0000009b`cc0feeb8 00000000`00000000 : 00000000`00000000 00000000`00000000 0000009b`cc0feee0 00000000`00000000 : 0x0000009b`cc0fef39
SYMBOL_NAME: myfault+1981
MODULE_NAME: myfault
IMAGE_NAME: myfault.sys
STACK_COMMAND: .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET: 1981
FAILURE_BUCKET_ID: AV_myfault!unknown_function
OS_VERSION: 10.0.22000.1
BUILDLAB_STR: co_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {9745090a-9bce-ccba-c096-ca6e9ca04c64}
Followup: MachineOwner
---------
This summary is based on a Windows bugcheck (blue screen of death) analysis report. The specific error encountered is DRIVER_IRQL_NOT_LESS_OR_EQUAL with a stop code of 0xD1. This error indicates that a driver attempted to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that was too high, which is typically a result of drivers using improper addresses.
Key details from the analysis include:
- Memory Referenced (Arg1): ffffbc06b64895a0. This is the memory address that was being accessed when the error occurred.
- IRQL (Arg2): 2, indicating the interrupt request level at the time of the error.
- Operation Type (Arg3): 0, meaning a read operation was being performed.
- Address that Referenced Memory (Arg4): fffff80403c61981, the address within the driver or system component that attempted the illegal memory access.
Additional debugging details highlight that the error was caused by the myfault.sys driver, which is notably used for generating system crashes for testing purposes. The error occurred while running the process notmyfault64.exe, further suggesting that this was a deliberate test of system stability or error handling.
The analysis also included various technical details about the system state, such as CPU and memory usage metrics, the virtual machine environment (VMware), and other system flags. The failure bucket identifies the issue as an access violation (AV) within the myfault driver, without specifying a known function, indicating a potentially unknown or unexpected path within the driver causing the crash.
Given the context, it appears this crash was induced as part of a debugging or system testing exercise, likely to analyze how the system and its error handling mechanisms respond to driver-induced errors.
The following are Analysis Summaries from the two latest pattern cooperation case studies:
Spiking Thread, Top Module, Module Hint, and Memory Fluctuation
The analysis revolves around diagnosing a sluggish performance issue on a Windows 7 system, attributed to high CPU and physical memory usage by one of the svchost.exe processes. A process memory dump revealed a significantly large file size (almost 1.5 GB), hinting at abnormal behavior warranting a deeper investigation using crash and hang dump analysis techniques.
Key Observations:
- Spiking Thread Pattern: Initial analysis identified a single thread consuming a disproportionate amount of CPU time, indicative of a Spiking Thread pattern. This thread, identified through the !runaway command in WinDbg, significantly outpaced other threads in both user and kernel mode time.
- Windows Update Agent Activity: The stack trace of the spiking thread pointed to activity related to the Windows Update Agent (wuaueng.dll), suggesting that the Windows Update process was contributing to the system’s sluggish performance.
- Heap Usage and Potential Memory Leak: Examination of the process address space showed nearly 1 GB occupied by the process heap, with many large virtual blocks allocated. This was suspected to be indicative of a memory leak.
- Temporary Condition: Interestingly, the system’s performance normalized after about 20 minutes, and a subsequent memory dump was significantly smaller (less than 150 KB). Analysis of this dump showed a different thread (#32) as having consumed CPU time in the past, but with the system no longer exhibiting high CPU or memory usage.
- Memory Fluctuation: The investigation concluded that the observed behavior was a case of Memory Fluctuation, associated with the temporary demands of the Windows Update process. The initial large memory usage and CPU consumption were temporary and resolved without intervention, as evidenced by the normalization of resource usage and the significant reduction in the size of the process memory dump.
Conclusion:
The investigation showcases the importance of crash and hang dump analysis in diagnosing and understanding transient performance issues in Windows systems. It highlights how system processes, in this case, the Windows Update Agent, can temporarily consume significant system resources, leading to performance degradation. The analysis also demonstrates the system’s ability to recover from such states, emphasizing the temporary nature of certain performance issues.
The analysis provided deals with an investigation into a series of crashes of the backgroundTaskHost.exe on a system, focusing on the crash dumps found in a specified folder set up through the LocalDumps Windows Error Reporting (WER) registry. These crashes are characterized by a consistent Exception Stack Trace and a Stored Exception that suggests a security vulnerability (stack buffer overrun) potentially exploitable by malicious actors. The investigation involves detailed WinDbg command outputs, providing insight into the nature of the crashes, the involved modules, and the potential causes.
Key Findings:
- Consistent Crash Cause: All crash dumps share the same Exception Stack Trace and Stored Exception, indicating a consistent failure mode across multiple instances. The failure is identified as a stack buffer overrun (c0000409), a type of vulnerability that could allow unauthorized control over the application.
- Exception Stack Trace Analysis: The stack trace points to several modules, including ucrtbase, vccorlib140_app, and Microsoft_Applications_Telemetry_Windows, among others. The presence of multiple calls to DllGetActivationFactory and the involvement of telemetry-related functions suggest the crashes may be related to the handling of telemetry data.
- Translated and Hidden Exceptions: The analysis reveals a discrepancy between the exception addresses and contexts reported by different WinDbg commands, suggesting the presence of translated exceptions. Further examination of the execution residue uncovers hidden exceptions, indicating that the initial crash may be concealing additional underlying issues.
- Module Involvement and Symbolic Information: The Microsoft_Applications_Telemetry_Windows module is highlighted as a significant factor in the crashes, particularly due to its association with telemetry data processing functions. However, the analysis also suggests that the symbolic information for this module may be coincidental, due to the absence of component symbols and an unusually large offset for the getJsonFormattedEvent function.
- Function Size Discrepancy: Discrepancies in the expected size of the getJsonFormattedEvent function, compared to its offset, raise questions about the accuracy of the symbolic information provided. This discrepancy also complicates the search for pointers to JSON strings in the memory dump, as typical commands do not reveal expected patterns.
Overall, the analysis suggests that the crashes may be related to the processing of telemetry data, potentially due to a stack buffer overrun vulnerability in the backgroundTaskHost.exe. The involvement of the Microsoft_Applications_Telemetry_Windows module and discrepancies in symbolic information point to the need for further investigation, including the possibility of missing or inaccurate debugging symbols that could clarify the root cause of these crashes.
- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -