Skip to content

[BUG] Native test run crashes with 0xC0000005 after TestStartAtOldestRecord passes #238

Open

Description

@bobsira

Description

The broader native test run terminates with access violation 0xC0000005 (signed process exit code -1073741819) after TestStartAtOldestRecord reports success. This prevents the remaining tests from completing and prevents a complete TRX result from being produced.

This was observed while validating #237 and reproduced independently using an isolated, unmodified main checkout at d07608d5d39ae08af75430b33c2d1a03b90b4956. It is not introduced by the custom single-line formatting change in #237.

Environment

  • Windows development host with existing Application event-log history.
  • Visual Studio 2022 Enterprise; VSTest 17.14.0 (x64).
  • CMake build, x64 Release, native LogMonitorTests.dll.
  • Baseline source: d07608d5d39ae08af75430b33c2d1a03b90b4956.
  • Also observed on candidate cefcd10 for Fix single-line formatting for custom Event Log output #237.

Reproduction

Using an isolated checkout of the baseline commit, configure the existing CMake build with the repository's required Visual Studio/native-test and nlohmann-json dependencies, then run:

cmake --build .\LogMonitor\build --config Release --target LogMonitorTests --parallel 4

& 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe' `
    .\LogMonitor\build\Release\LogMonitorTests.dll `
    '/TestCaseFilter:FullyQualifiedName~EventMonitorTests|FullyQualifiedName~JsonProcessorTests|FullyQualifiedName~UtilityTests' `
    '/Logger:trx;LogFileName=event-monitor-repro.trx'

$LASTEXITCODE

TestStartAtOldestRecord subscribes to existing Application records. Reproduction may depend on the host's historical event backlog and timing; the minimum required backlog has not been established.

Observed results

On the unmodified baseline, the last reported test outcomes were:

Passed TestErrorLevelMonitor [2 s]
Passed TestInformationLevelMonitor [2 s]
Passed TestStartAtOldestRecord [5 s]
Process exit code: -1073741819 (0xC0000005)

The candidate also crashed after the historical-replay test reported success, with replay output continuing. Excluding that test allowed all 12 selected EventMonitor, configuration, and utility tests on the candidate to pass:

(FullyQualifiedName~EventMonitorTests&FullyQualifiedName!~TestStartAtOldestRecord)|FullyQualifiedName~JsonProcessorTests|FullyQualifiedName~UtilityTests

Exclusion is a diagnostic workaround, not a fix. A reported pass for the historical-replay test does not mean the test process shut down safely.

Expected behavior

Historical event replay should stop safely when the monitor is destroyed, and the remaining selected tests should complete with a valid final test report.

Investigation context

At the baseline commit, EventMonitor::~EventMonitor waits up to five seconds for the worker but does not handle a timeout/failure before returning. EnumerateResults drains events without checking the stop event between batches/events. These lifecycle paths are consistent with a worker outliving its monitor during backlog replay, but no crash dump or stack trace was captured for these particular failing runs, so their exact faulting instruction is not established.

Related PR #235 already includes EventMonitor shutdown/lifetime fixes and replay-shutdown regression coverage; its description reports full native-suite passes. Link this report to that work and verify the reproduction against its fix rather than adding another implementation in #237.

Issue #199 also reports 0xC0000005, but for a deployed LogMonitor.exe startup failure; no common cause has been established.

Activity

  1. sunny-kavixo commented on Oct 24, 2026

    @sunny-kavixo

    Hi bob sira (@bobsira),

    Thanks for the detailed report and for confirming that the crash also occurs on an unmodified main checkout.

    I'm interested in helping investigate and independently validate this issue.

    Since PR #235 already addresses EventMonitor shutdown and lifetime behavior, I'd prefer to verify the existing fix rather than introduce a duplicate implementation.

    I'd like to focus on:

    • Comparing the reported crash scenario against the changes in PR Fix Log Monitor lint and EventMonitor shutdown #235.
    • Checking worker-thread shutdown and object lifetime during historical event replay.
    • Reviewing regression coverage for TestStartAtOldestRecord.
    • Investigating whether additional edge-case tests are needed for shutdown timeouts and large event backlogs.
    • Sharing reproducible findings before proposing any further changes.

    I understand that the exact faulting instruction hasn't been established without a crash dump or stack trace.

    Would independent validation or additional regression coverage be useful here?

    Thanks!
    Putluru Sandeep (Putluru Sandeep (@sunny-kavixo))

  2. bobsira commented on Oct 24, 2026

    @bobsira
    ContributorAuthor

    Thanks Putluru Sandeep (@sunny-kavixo) — yes, independent validation would be very helpful!

    Comparing the baseline at d07608d with #235 is the right approach. Please use the reproduction command in this issue, ideally on the same host with the same Application event-log backlog. The important check is that the entire test run completes and produces a final report, not just that TestStartAtOldestRecord reports success.

    Please share the source commits, Windows/build environment, commands, exit codes, and whether the results hold across repeated runs. If you reproduce the baseline crash, a crash dump or relevant stack trace would help establish the exact failure behind this report.

    #235 already includes immediate-shutdown, replay-shutdown, and handle-leak regression tests, along with reported timeout/failure-path validation. Reviewing those first and identifying any remaining gaps—particularly with large backlogs—would be valuable.

    Please post your findings here before proposing further changes so we can coordinate with #235 and avoid duplicating work. Thanks for offering to help!

  3. sunny-kavixo commented on Oct 25, 2026

    @sunny-kavixo

    Hi bob sira (@bobsira),

    Thank you for welcoming independent validation.

    I've completed a read-only review of the baseline and PR #235. I identified one potential regression-coverage gap worth discussing before making changes.

    The existing TestShutdownDuringReplay does not deterministically establish that shutdown occurs while historical replay is actively processing a batch with outstanding event handles. Increasing event counts or adding delays would not reliably establish that condition.

    One possible approach is a narrowly scoped event-source abstraction that allows a test-controlled source to pause replay with an outstanding batch, trigger shutdown, and verify cleanup and worker termination.

    However, that would touch production code and overlap with PR #235, so I don't want to introduce an uncoordinated architectural change.

    Would you prefer this deterministic test seam to be considered within PR #235, or as a separate follow-up contribution?

    For transparency, this is a source-level assessment only. I have not reproduced the Windows crash or executed native VSTest. The existing Windows verification remains pending.

    Thanks!
    Putluru Sandeep (Putluru Sandeep (@sunny-kavixo))

  4. sunny-kavixo commented on Oct 25, 2026

    @sunny-kavixo

    I completed an independent, same-runner Windows comparison of the issue baseline and PR #235.

    Environment:

    • Windows Server 2022 Datacenter, build 20348, x64
    • Git 2.56.0.windows.3
    • CMake 3.31.6
    • Visual Studio 2022 Enterprise / VSTest 17.0.36015.10
    • Application event log contained 131 records on this fresh runner

    Results:

    • Baseline d07608d5d39ae08af75430b33c2d1a03b90b4956: 3/3 complete process runs, exit code 0x00000000, complete TRX each time, 11/11 tests passed, 0 failed/error/aborted. Durations: 9.893 s, 3.598 s, 3.576 s.
    • PR Fix Log Monitor lint and EventMonitor shutdown #235 head c41a5384fb26b3d4fe73ab867fb30787c3a8bd1b: 3/3 complete process runs, exit code 0x00000000, complete TRX each time, 14/14 tests passed, 0 failed/error/aborted. Durations: 4.175 s, 4.202 s, 4.155 s.

    The successful run, including uploaded environment and summary artifacts, is here:

    https://github.com/sunny-kavixo/windows-container-tools/actions/runs/40174725987

    The verification workflow and script are preserved at:

    sunny-kavixo@fd75965

    Important limitation: this fresh runner had only 131 Application log records. The run therefore validates clean native execution and full-report completion on one Windows environment, but it does not reproduce or disprove the backlog-sensitive crash reported on the original host. Establishing shutdown during active historical replay with a large backlog still requires either the affected host or a deterministic test seam. I have not proposed a production abstraction or competing fix while the ownership of that seam remains unresolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions