Add structured debug logging to config merge pipeline (#318)

[leynos]

Summary

Closes #318

Makes the four-layer configuration merge auditable, as proposed:

• tracing::debug! at each layer push (defaults, file, environment, CLI), including the failure paths that previously accumulated errors silently.
• Config file discovery logs the explicit path vs discovery mode, each discovered layer's path, and the no-layers case.
• Validation rejections log structured key/reason fields.
• CLI override logging records override keys only — values may echo user-supplied paths/host lists that do not belong in logs.
• Suppression in diag_json mode: the subscriber is never initialised when JSON diagnostics are active, so stderr stays machine-readable. For operators, tracing now initialises before the merge when --verbose is passed on the command line (init_tracing switched to try_init, making later init calls no-ops). Config-file-driven verbosity still applies post-merge as before.

Testing

• New test_support::tracing_capture module (reusable thread-local event capture, mirroring the pattern used by the manifest-expansion tracing tests).
• tests/cli_tests/merge_logging.rs: asserts one event per layer, override-key logging for --jobs, and key/reason fields when a file-sourced output_format = "json" is rejected.

Validation

• make check-fmt / make lint / make test — pass (37 suites; complexity ceiling respected via extracted per-layer helpers)

🤖 Generated with Claude Code

Summary by Sourcery

Add structured debug logging and observability around the configuration merge pipeline and its validation.

Enhancements:

• Emit structured debug events for each configuration layer (defaults, files, environment, CLI) and for validation rejections, including key and reason metadata.
• Ensure CLI override logging records only override keys to avoid leaking user-supplied values in logs.
• Initialize tracing earlier for --verbose runs while keeping diagnostic JSON mode stderr output machine-readable via try_init-based tracing setup.
• Refactor merge logic into per-layer helper functions and improve file layer discovery logging, including explicit-path and no-layer cases.

Build:

• Add tracing and tracing-subscriber dependencies to the test_support crate for use in observability tests.

Tests:

• Introduce a reusable tracing_capture utility to capture tracing events in tests without modifying the global subscriber.
• Add CLI merge_logging tests that assert per-layer debug events, CLI override-key logging, and structured key/reason validation logging for rejected configurations.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8f0d9479-face-44b7-a683-85cc5e84986c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-318-merge-pipeline-debug-logging

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Reviewer's Guide

Adds structured debug logging and observability for each configuration merge layer (defaults, file, environment, CLI), wires early tracing initialisation for --verbose while preserving JSON diagnostic mode, and introduces reusable tracing capture utilities plus CLI tests to assert merge logging behaviour and validation logs.

Sequence diagram for config merge logging and early tracing init

sequenceDiagram
    actor User
    participant run_with_args
    participant DiagMode
    participant init_tracing
    participant merge_with_config
    participant MergeComposer
    participant tracing_debug

    User->>run_with_args: run_with_args
    run_with_args->>DiagMode: from_json_enabled
    alt mode is not json and parsed_cli.verbose
        run_with_args->>init_tracing: init_tracing Level_DEBUG
        init_tracing-->>run_with_args: fmt try_init
    end

    run_with_args->>merge_with_config: merge_with_config
    merge_with_config->>MergeComposer: MergeComposer::with_capacity

    merge_with_config->>merge_with_config: push_defaults_layer
    merge_with_config->>tracing_debug: tracing::debug! layer defaults

    merge_with_config->>merge_with_config: push_file_layers
    merge_with_config->>MergeComposer: push_discovered_layers
    MergeComposer->>tracing_debug: tracing::debug! layer file

    merge_with_config->>merge_with_config: push_environment_layer
    merge_with_config->>tracing_debug: tracing::debug! layer environment

    merge_with_config->>merge_with_config: push_cli_layer
    merge_with_config->>tracing_debug: tracing::debug! layer cli override_keys

    merge_with_config->>merge_with_config: validate_output_format_source
    alt output_format json from non_cli
        merge_with_config->>tracing_debug: tracing::debug! key output_format reason
    end

    merge_with_config-->>run_with_args: merged Cli
    run_with_args-->>User: exit_code_or_ok

Loading

File-Level Changes

Change	Details	Files
Refactored the configuration merge pipeline to emit structured debug logs at each layer and during validation, while keeping error accumulation behaviour intact.	Introduced a MergeError type alias for shared OrthoError references and kept the existing error-accumulation pattern using a Vec. Extracted per-layer helper functions push_defaults_layer, push_environment_layer, and push_cli_layer from merge_with_config, each responsible for pushing its layer into the MergeComposer and emitting tracing::debug! events on success or failure. Extended merge_with_config to build a LayerComposition from composer layers and accumulated errors, then added a call to validate_output_format_source that logs key and reason when validation rejects JSON output requested outside the CLI. Ensured CLI override logging only records override_keys derived from the Value object to avoid leaking user paths or host lists into logs, while logging when no CLI overrides are present.	src/cli/merge.rs
Made configuration file discovery and layer pushing observable via structured debug logs.	Logged whether configuration loading uses an explicit path or discovery mode, including the explicit file path when present. Added push_discovered_layers helper to iterate discovered configuration file layers, logging each layer path and the empty-discovery case before pushing onto the MergeComposer. Logged file discovery failures via tracing::debug! and preserved error propagation into the existing errors vector.	src/cli/discovery.rs
Adjusted tracing initialisation to support pre-merge debug logging under --verbose without breaking JSON diagnostic mode, and made repeated initialisation safe.	Before running the merge pipeline in run_with_args, initialise tracing with Level::DEBUG when --verbose is passed and diagnostics mode is not JSON, ensuring merge-time debug events are visible. Left JSON diagnostics mode untouched by skipping early tracing initialisation to keep stderr machine-readable, while maintaining post-merge config-driven verbosity via configure_runtime. Changed init_tracing to use tracing_subscriber::fmt::Subscriber::try_init wrapped in drop, making multiple init_tracing calls harmless by letting the first caller win.	src/main.rs
Added reusable tracing capture utilities for tests and new CLI tests to assert merge logging behaviour.	Added tracing and tracing-subscriber as dependencies of the test_support crate to support local subscribers in tests. Introduced test_support::tracing_capture module providing CapturedEvents, a CapturedEventsLayer, and with_test_subscriber, which installs a thread-local subscriber capturing DEBUG+ events as joined name=value field strings. Registered the tracing_capture module in test_support::lib, and wired a new merge_logging test module into the CLI tests suite. Implemented tests/cli_tests/merge_logging.rs to run the CLI merge with a test subscriber, asserting that debug events are emitted for each merge layer, that CLI overrides log override_keys and job key names, and that validation rejection of file-sourced output_format="json" logs structured key and reason fields.	test_support/Cargo.toml test_support/src/lib.rs test_support/src/tracing_capture.rs tests/cli_tests/mod.rs tests/cli_tests/merge_logging.rs Cargo.lock

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#318	Add structured debug logging to the configuration merge pipeline, including debug events at each layer boundary (defaults, file discovery, environment, CLI), logging discovery attempts and file layers (paths tried, found/not found), and logging failure paths that previously accumulated errors silently.	✅
#318	Log validation rejections from the merge pipeline with structured fields, including at least the rejected configuration key and the reason for rejection.	✅
#318	Ensure debug log output from the merge pipeline is suppressed in diag_json mode so as not to pollute machine-readable JSON diagnostics, while still allowing tracing to be initialised when verbose logging is requested.	✅

Possibly linked issues

• #(not specified): PR adds the requested structured debug logging, file discovery logs, validation logs, and diag_json suppression in merge.rs.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.