Skip to content

fix: Docker container probe uses Podman-only container exists, and diagnostics flags transient mirror-node startup errors #5914

Description

@JeffreyDallas

Describe the bug

Two separate problems observed after a local one-shot deploy with a block node.

1. docker container exists is a Podman-only subcommand

DockerClient.containerExists() probes for the kind node container with:

docker container exists <node-name>

Docker has no container exists subcommand (it is Podman-only), so this probe fails 100% of the time on every Docker host. Two consequences:

  • An alarming ERROR is written to ~/.solo/logs/solo.log on every invocation:
[15:35:38.621] ERROR: Error executing: 'docker' {
  commandExitCode: 1,
  commandExitSignal: null,
  commandOutput: [],
  errOutput: [
    'docker: unknown command: docker container exists',
    'Usage:  docker container',
    "Run 'docker container --help' for more information"
  ],
  ...
  • Because the Docker probe never succeeds, resolveKindContainerCommand() always silently falls through to the Podman detection path, even on Docker-only machines. This is a functional bug, not merely log noise.

Contributing factor: ShellRunner.run() logs at ERROR unconditionally on a non-zero exit, before rejecting the promise. Callers that legitimately expect a probe to fail (and already catch and handle it) have no way to keep the failure out of solo.log.

2. deployment diagnostics logs reports transient mirror-node startup errors as findings

solo deployment diagnostics logs surfaced three findings, of which two were false alarms:

1. Application ERROR detected in pod log: mirror-1-importer-...
- line 478: ERROR ... o.h.m.i.d.b.CompositeBlockSource Failed to get block from BLOCK_NODE source: No block node can provide block 0
2. Application ERROR detected in pod log: mirror-1-restjava-...
- line 34:  WARN ... o.h.orm.jdbc.error ERROR: relation "file_data" does not exist
- line 39: ERROR ... TaskUtils$LoggingErrorHandler Unexpected error occurred in scheduled task ... [ERROR: relation "file_data" does not exist
3. ERROR detected in solo.log
- line 310: [15:35:38.621] ERROR: Error executing: 'docker' {

Root causes:

  • No block node can provide block 0 — a suppression rule already exists in POD_LOG_ERROR_SUPPRESSIONS, but its regex requires a literal space after source:
    /...BLOCK_NODE source .*No block node can provide block \d+/i
    The real Mirror Node message is source: No block node ... (colon), so the rule never matched and was dead code.
  • relation "file_data" does not exist — genuine coverage gap. mirror-restjava starts accepting scheduled work before the importer's Flyway migration has created the tables it queries. The only existing rule for this message is scoped to the shared Postgres log file, not the rest/restjava pod log.
  • Incorrect first block item case ROUND_HEADER — also present in the same importer log, a block-stream alignment race during bring-up, with no suppression rule at all.
  • Finding 3 is a direct consequence of bug 1.

3. Unit tests write ERROR entries into the developer's real ~/.solo/logs/solo.log

SoloPinoLogger derives its log destination from constants.SOLO_LOGS_DIR rather than from the injected InjectTokens.HomeDirectory, so it ignores the home directory the container was configured with. The test container is correctly configured with test/data/tmp, but the logger writes to the real ~/.solo/logs anyway.

Consequently any unit test that triggers an ERROR-level log pollutes the real log, and deployment diagnostics logs then reports it as a finding for a binary Solo never runs:

[16:58:33.933] ERROR: Error executing: 'INVALID_PROGRAM' {
  error: {
    message: 'spawn INVALID_PROGRAM ENOENT',
    stack: 'Error: Executing command: INVALID_PROGRAM \n' +
      '    at BaseCommand.run (/Users/jeffrey/solo/src/core/shell-runner.ts:49:31)\n' +
      '    at Context.<anonymous> (/Users/jeffrey/solo/test/unit/commands/base.test.ts:90:28)\n' +
      '    at callFn (/Users/jeffrey/solo/node_modules/mocha/lib/runnable.js:366:21)\n' +

INVALID_PROGRAM appears only in test/unit/commands/base.test.ts:90 and test/unit/core/dependency-managers/dependency-manager.test.ts:26; it is never spawned by production code. Measured growth of the real solo.log across one full unit-suite run: +1762 bytes.

Two test suites compound the problem independently of the logger:

  • test/unit/core/config-manager.test.ts calls container.clearInstances(), which drops value registrations entirely (verified: resolving HomeDirectory afterwards throws Attempted to resolve unregistered dependency token), then constructs a logger — so it falls back to the real home.
  • test/unit/core/dependency-injection/container-init.test.ts restores constants.SOLO_HOME_DIR in afterEach "so other suites are unaffected", which actually points every subsequent suite at the real home.

Describe the expected behavior

  1. The Docker container-existence probe should use a command Docker actually supports, succeed on Docker hosts, and not fall back to Podman when Docker owns the kind node container. No ERROR should be written to solo.log for a probe that is expected to fail sometimes.
  2. deployment diagnostics logs should suppress known-benign transient errors that occur during component bring-up, while still surfacing the same messages if they appear after the startup window (or without evidence that the condition self-healed). A clean deploy should not produce false-alarm findings.
  3. Running the unit test suite should not write anything to the developer's real ~/.solo/logs/solo.log. The logger should honour the home directory its container was configured with.

To Reproduce

  1. Deploy Solo locally with a block node:

    ONE_SHOT_WITH_BLOCK_NODE=true BLOCK_STREAM_STREAM_MODE=BLOCKS npm run solo-test -- one-shot single deploy
    
  2. Run the diagnostics command:

    npm run solo-test -- deployment diagnostics logs
    
  3. Observe the false-alarm findings for mirror-*-importer and mirror-*-restjava in the terminal output and in diagnostics-analysis.txt.

  4. Inspect ~/.solo/logs/solo.log and observe repeated ERROR: Error executing: 'docker' entries with docker: unknown command: docker container exists.

  5. Confirm the command is invalid for Docker directly:

    docker container exists <kind-node-name>   # -> docker: unknown command
    docker container inspect --format '{{.Id}}' <kind-node-name>   # -> exit 0, prints container id
    

Additional Context

Environment: macOS (darwin 25.4.0), Docker 29.7.2, kind cluster solo-cluster, Solo main @ 0.88.0.

Affected code:

  • src/integration/container-engine/docker-client.tscontainerExists(), resolveKindContainerCommand(), readContainerState()
  • src/core/shell-runner.ts / src/core/shell-run-options.ts — unconditional ERROR logging on non-zero exit
  • src/integration/container-engine/container-engine-resource-inspector.ts — best-effort docker info probe
  • src/commands/util/diagnostics-analyzer.tsPOD_LOG_ERROR_SUPPRESSIONS

Note that the existing unit test suppresses mirror importer begin-phase block-node source errors only after repeated parse success passed despite the broken regex, because its fixture used an invented message shape (source org.hiero...Exception: No block node ...) rather than the real one (source: No block node ...).

Remaining related noise, likely worth a separate issue: the importer also logs AccountBalancesDownloader Error downloading signature files ... Unable to load credentials from any of the providers in the chain every ~30s indefinitely (S3 polling with no AWS credentials in a local deploy). This is a persistent configuration condition rather than a startup race, so it is intentionally left visible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugA error that causes the feature to behave differently than what was expected based on design docsP1-💎Current Milestone & Goals

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions