Describe the bug
Two separate problems observed after a local one-shot deploy with a block node.
1. docker container exists is a Podman-only subcommand
DockerClient.containerExists() probes for the kind node container with:
docker container exists <node-name>
Docker has no container exists subcommand (it is Podman-only), so this probe fails 100% of the time on every Docker host. Two consequences:
- An alarming
ERROR is written to ~/.solo/logs/solo.log on every invocation:
[15:35:38.621] ERROR: Error executing: 'docker' {
commandExitCode: 1,
commandExitSignal: null,
commandOutput: [],
errOutput: [
'docker: unknown command: docker container exists',
'Usage: docker container',
"Run 'docker container --help' for more information"
],
...
- Because the Docker probe never succeeds,
resolveKindContainerCommand() always silently falls through to the Podman detection path, even on Docker-only machines. This is a functional bug, not merely log noise.
Contributing factor: ShellRunner.run() logs at ERROR unconditionally on a non-zero exit, before rejecting the promise. Callers that legitimately expect a probe to fail (and already catch and handle it) have no way to keep the failure out of solo.log.
2. deployment diagnostics logs reports transient mirror-node startup errors as findings
solo deployment diagnostics logs surfaced three findings, of which two were false alarms:
1. Application ERROR detected in pod log: mirror-1-importer-...
- line 478: ERROR ... o.h.m.i.d.b.CompositeBlockSource Failed to get block from BLOCK_NODE source: No block node can provide block 0
2. Application ERROR detected in pod log: mirror-1-restjava-...
- line 34: WARN ... o.h.orm.jdbc.error ERROR: relation "file_data" does not exist
- line 39: ERROR ... TaskUtils$LoggingErrorHandler Unexpected error occurred in scheduled task ... [ERROR: relation "file_data" does not exist
3. ERROR detected in solo.log
- line 310: [15:35:38.621] ERROR: Error executing: 'docker' {
Root causes:
No block node can provide block 0 — a suppression rule already exists in POD_LOG_ERROR_SUPPRESSIONS, but its regex requires a literal space after source:
/...BLOCK_NODE source .*No block node can provide block \d+/i
The real Mirror Node message is source: No block node ... (colon), so the rule never matched and was dead code.
relation "file_data" does not exist — genuine coverage gap. mirror-restjava starts accepting scheduled work before the importer's Flyway migration has created the tables it queries. The only existing rule for this message is scoped to the shared Postgres log file, not the rest/restjava pod log.
Incorrect first block item case ROUND_HEADER — also present in the same importer log, a block-stream alignment race during bring-up, with no suppression rule at all.
- Finding 3 is a direct consequence of bug 1.
3. Unit tests write ERROR entries into the developer's real ~/.solo/logs/solo.log
SoloPinoLogger derives its log destination from constants.SOLO_LOGS_DIR rather than from the injected InjectTokens.HomeDirectory, so it ignores the home directory the container was configured with. The test container is correctly configured with test/data/tmp, but the logger writes to the real ~/.solo/logs anyway.
Consequently any unit test that triggers an ERROR-level log pollutes the real log, and deployment diagnostics logs then reports it as a finding for a binary Solo never runs:
[16:58:33.933] ERROR: Error executing: 'INVALID_PROGRAM' {
error: {
message: 'spawn INVALID_PROGRAM ENOENT',
stack: 'Error: Executing command: INVALID_PROGRAM \n' +
' at BaseCommand.run (/Users/jeffrey/solo/src/core/shell-runner.ts:49:31)\n' +
' at Context.<anonymous> (/Users/jeffrey/solo/test/unit/commands/base.test.ts:90:28)\n' +
' at callFn (/Users/jeffrey/solo/node_modules/mocha/lib/runnable.js:366:21)\n' +
INVALID_PROGRAM appears only in test/unit/commands/base.test.ts:90 and test/unit/core/dependency-managers/dependency-manager.test.ts:26; it is never spawned by production code. Measured growth of the real solo.log across one full unit-suite run: +1762 bytes.
Two test suites compound the problem independently of the logger:
test/unit/core/config-manager.test.ts calls container.clearInstances(), which drops value registrations entirely (verified: resolving HomeDirectory afterwards throws Attempted to resolve unregistered dependency token), then constructs a logger — so it falls back to the real home.
test/unit/core/dependency-injection/container-init.test.ts restores constants.SOLO_HOME_DIR in afterEach "so other suites are unaffected", which actually points every subsequent suite at the real home.
Describe the expected behavior
- The Docker container-existence probe should use a command Docker actually supports, succeed on Docker hosts, and not fall back to Podman when Docker owns the kind node container. No
ERROR should be written to solo.log for a probe that is expected to fail sometimes.
deployment diagnostics logs should suppress known-benign transient errors that occur during component bring-up, while still surfacing the same messages if they appear after the startup window (or without evidence that the condition self-healed). A clean deploy should not produce false-alarm findings.
- Running the unit test suite should not write anything to the developer's real
~/.solo/logs/solo.log. The logger should honour the home directory its container was configured with.
To Reproduce
-
Deploy Solo locally with a block node:
ONE_SHOT_WITH_BLOCK_NODE=true BLOCK_STREAM_STREAM_MODE=BLOCKS npm run solo-test -- one-shot single deploy
-
Run the diagnostics command:
npm run solo-test -- deployment diagnostics logs
-
Observe the false-alarm findings for mirror-*-importer and mirror-*-restjava in the terminal output and in diagnostics-analysis.txt.
-
Inspect ~/.solo/logs/solo.log and observe repeated ERROR: Error executing: 'docker' entries with docker: unknown command: docker container exists.
-
Confirm the command is invalid for Docker directly:
docker container exists <kind-node-name> # -> docker: unknown command
docker container inspect --format '{{.Id}}' <kind-node-name> # -> exit 0, prints container id
Additional Context
Environment: macOS (darwin 25.4.0), Docker 29.7.2, kind cluster solo-cluster, Solo main @ 0.88.0.
Affected code:
src/integration/container-engine/docker-client.ts — containerExists(), resolveKindContainerCommand(), readContainerState()
src/core/shell-runner.ts / src/core/shell-run-options.ts — unconditional ERROR logging on non-zero exit
src/integration/container-engine/container-engine-resource-inspector.ts — best-effort docker info probe
src/commands/util/diagnostics-analyzer.ts — POD_LOG_ERROR_SUPPRESSIONS
Note that the existing unit test suppresses mirror importer begin-phase block-node source errors only after repeated parse success passed despite the broken regex, because its fixture used an invented message shape (source org.hiero...Exception: No block node ...) rather than the real one (source: No block node ...).
Remaining related noise, likely worth a separate issue: the importer also logs AccountBalancesDownloader Error downloading signature files ... Unable to load credentials from any of the providers in the chain every ~30s indefinitely (S3 polling with no AWS credentials in a local deploy). This is a persistent configuration condition rather than a startup race, so it is intentionally left visible.
Describe the bug
Two separate problems observed after a local one-shot deploy with a block node.
1.
docker container existsis a Podman-only subcommandDockerClient.containerExists()probes for the kind node container with:Docker has no
container existssubcommand (it is Podman-only), so this probe fails 100% of the time on every Docker host. Two consequences:ERRORis written to~/.solo/logs/solo.logon every invocation:resolveKindContainerCommand()always silently falls through to the Podman detection path, even on Docker-only machines. This is a functional bug, not merely log noise.Contributing factor:
ShellRunner.run()logs atERRORunconditionally on a non-zero exit, before rejecting the promise. Callers that legitimately expect a probe to fail (and already catch and handle it) have no way to keep the failure out ofsolo.log.2.
deployment diagnostics logsreports transient mirror-node startup errors as findingssolo deployment diagnostics logssurfaced three findings, of which two were false alarms:Root causes:
No block node can provide block 0— a suppression rule already exists inPOD_LOG_ERROR_SUPPRESSIONS, but its regex requires a literal space aftersource:/...BLOCK_NODE source .*No block node can provide block \d+/iThe real Mirror Node message is
source: No block node ...(colon), so the rule never matched and was dead code.relation "file_data" does not exist— genuine coverage gap.mirror-restjavastarts accepting scheduled work before the importer's Flyway migration has created the tables it queries. The only existing rule for this message is scoped to the shared Postgres log file, not the rest/restjava pod log.Incorrect first block item case ROUND_HEADER— also present in the same importer log, a block-stream alignment race during bring-up, with no suppression rule at all.3. Unit tests write ERROR entries into the developer's real
~/.solo/logs/solo.logSoloPinoLoggerderives its log destination fromconstants.SOLO_LOGS_DIRrather than from the injectedInjectTokens.HomeDirectory, so it ignores the home directory the container was configured with. The test container is correctly configured withtest/data/tmp, but the logger writes to the real~/.solo/logsanyway.Consequently any unit test that triggers an ERROR-level log pollutes the real log, and
deployment diagnostics logsthen reports it as a finding for a binary Solo never runs:INVALID_PROGRAMappears only intest/unit/commands/base.test.ts:90andtest/unit/core/dependency-managers/dependency-manager.test.ts:26; it is never spawned by production code. Measured growth of the realsolo.logacross one full unit-suite run: +1762 bytes.Two test suites compound the problem independently of the logger:
test/unit/core/config-manager.test.tscallscontainer.clearInstances(), which drops value registrations entirely (verified: resolvingHomeDirectoryafterwards throwsAttempted to resolve unregistered dependency token), then constructs a logger — so it falls back to the real home.test/unit/core/dependency-injection/container-init.test.tsrestoresconstants.SOLO_HOME_DIRinafterEach"so other suites are unaffected", which actually points every subsequent suite at the real home.Describe the expected behavior
ERRORshould be written tosolo.logfor a probe that is expected to fail sometimes.deployment diagnostics logsshould suppress known-benign transient errors that occur during component bring-up, while still surfacing the same messages if they appear after the startup window (or without evidence that the condition self-healed). A clean deploy should not produce false-alarm findings.~/.solo/logs/solo.log. The logger should honour the home directory its container was configured with.To Reproduce
Deploy Solo locally with a block node:
Run the diagnostics command:
Observe the false-alarm findings for
mirror-*-importerandmirror-*-restjavain the terminal output and indiagnostics-analysis.txt.Inspect
~/.solo/logs/solo.logand observe repeatedERROR: Error executing: 'docker'entries withdocker: unknown command: docker container exists.Confirm the command is invalid for Docker directly:
Additional Context
Environment: macOS (darwin 25.4.0), Docker 29.7.2, kind cluster
solo-cluster, Solomain@ 0.88.0.Affected code:
src/integration/container-engine/docker-client.ts—containerExists(),resolveKindContainerCommand(),readContainerState()src/core/shell-runner.ts/src/core/shell-run-options.ts— unconditionalERRORlogging on non-zero exitsrc/integration/container-engine/container-engine-resource-inspector.ts— best-effortdocker infoprobesrc/commands/util/diagnostics-analyzer.ts—POD_LOG_ERROR_SUPPRESSIONSNote that the existing unit test
suppresses mirror importer begin-phase block-node source errors only after repeated parse successpassed despite the broken regex, because its fixture used an invented message shape (source org.hiero...Exception: No block node ...) rather than the real one (source: No block node ...).Remaining related noise, likely worth a separate issue: the importer also logs
AccountBalancesDownloader Error downloading signature files ... Unable to load credentials from any of the providers in the chainevery ~30s indefinitely (S3 polling with no AWS credentials in a local deploy). This is a persistent configuration condition rather than a startup race, so it is intentionally left visible.