Fix local card storage root resolution

[arjunxplorer]

PR Type

• Bug fix
• New feature
• Core Runtime change (higher bar -- see CONTRIBUTING.md)
• Docs / tooling
• Refactoring

Summary

Keep local and spin cards under the resolved artifact datastore root when no explicit card root is configured. Also ensure an explicit METAFLOW_CARD_LOCALROOT is returned and used correctly.

Issue

Fixes #2139

Reproduction

Runtime: Local datastore with a separate datastore sysroot, reproducing the storage configuration used with local Kubernetes.

Commands to run:

python3 -m pytest \
  test/unit/test_card_datastore.py \
  test/core/test_card_local_storage.py \
  -v

Where evidence shows up: The filesystem card location and the card list output exercised by the integration test.

Before (error / log snippet)

AssertionError: Expected cards below configured datastore root
<sysroot>/.metaflow/mf.cards.
assert []

The card HTML was written under <cwd>/.metaflow/mf.cards instead of the configured local datastore root.

The explicit-root unit test also failed because get_storage_root() returned None when CARD_LOCALROOT was configured.

After (evidence that fix works)

test/unit/test_card_datastore.py ... PASSED
test/core/test_card_local_storage.py::test_card_round_trip_uses_configured_local_datastore_root PASSED

10 passed

The card is written under <sysroot>/.metaflow/mf.cards, no stray <cwd>/.metaflow directory is created, and card list finds the card.

Root Cause

Artifact storage resolves METAFLOW_DATASTORE_SYSROOT_LOCAL, but card storage independently searched upward from the current working directory. Consequently, artifacts and cards could use different roots.

Additionally, when CARD_LOCALROOT was configured, the local/spin branch assigned it to a variable but did not return it, causing the method to return None.

Why This Fix Is Correct

Local and spin card roots now follow a single resolution order:

1. Return the explicitly configured CARD_LOCALROOT.
2. Otherwise, append mf.cards to the already-resolved artifact datastore root.
3. If neither root is available, retain the legacy nearest-parent datastore-directory lookup.

The card writer and reader both use this resolution logic, restoring the invariant that cards are read from the same location where they were written.

Failure Modes Considered

1. An explicit CARD_LOCALROOT continues to take precedence over the artifact datastore root.
2. The legacy nearest-parent and current-working-directory fallbacks remain available when no resolved datastore root is supplied.
3. Spin storage follows the corrected local-storage behavior.
4. S3, Azure, and Google Storage root resolution remains unchanged.

Tests

• Unit tests added/updated
• Reproduction script provided (required for Core Runtime)
• CI passes
• If tests are impractical: explain why below and provide manual evidence above

Local verification:

10 passed in 1.32s

CI remains unchecked until the pull request checks complete.

Non-Goals

This change does not alter cloud card storage behavior, change card file layout, or remove the existing local fallback behavior.

AI Tool Usage

• No AI tools were used in this contribution
• AI tools were used (describe below)

Cursor was used to assist with issue investigation, implementation, comments, and regression tests. I reviewed the changes and ran the relevant tests locally.

[greptile-apps]

Greptile Summary

This PR fixes local card storage root resolution. The main changes are:

• Local and spin card roots now prefer an explicit card root.
• Card roots now fall under the resolved artifact datastore root when available.
• Empty datastore roots now fall back to the legacy local lookup.
• Reader and CLI paths now use the shared card root resolver.
• Tests cover configured local roots, empty roots, spin roots, and cloud roots.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
metaflow/plugins/cards/card_datastore.py	Updates local and spin card root resolution to use explicit card roots, resolved datastore roots, and legacy fallback in that order.
metaflow/plugins/cards/card_client.py	Routes local and spin reader roots through the shared card root resolver.
metaflow/plugins/cards/card_cli.py	Passes the resolved artifact datastore root into card root resolution for CLI card operations.
test/core/test_card_local_storage.py	Adds an end-to-end local card round-trip test for configured datastore roots.
test/unit/test_card_datastore.py	Adds unit tests for local, spin, empty-root, explicit-root, fallback, and cloud root behavior.

_{Reviews (2): Last reviewed commit: "Handle empty local card datastore roots" | Re-trigger Greptile}

[iamsgarg-ob]

cc @valayDave

[Shriprasad-P]

The fresh-run storage change looks correct, but I found a backward-compatibility regression for cards created before this change.

With a configured local datastore root, the previous behavior could store artifacts under <sysroot>/.metaflow while cards were written under the cwd-based .metaflow/mf.cards location.

I reproduced a run using the base revision with that layout. After switching to this PR, the reader uses the persisted ds-root and resolves the card location as <sysroot>/.metaflow/mf.cards. Both card list and the client API then fail to find the existing card in its legacy location.

Please keep the new resolved datastore location for new writes, but preserve a reader fallback to the legacy local/spin card root before reporting a card as missing.

There is also an empty-root edge case: CARD_LOCALROOT="" is currently treated as an explicit root because the check uses is not None. That leaves the card datastore with an empty root and can produce incorrect absolute paths or inconsistent read behavior. Please treat an empty value as unset, or reject it during configuration validation, and add coverage for it.

The fresh local/spin writer-reader symmetry, explicit non-empty card roots, and cloud backends otherwise look correct.