Compare differences in local metadata vs. metadata service

[saikonen]

When running Metaflow with local metadata, a bunch of files are persisted on disk. There might exist discrepancies in data that is recorded when relying on local metadata compared to when using a remote metadata service.

For example, can both implementations reliably

• determine when a run has finished
• by polling or other means, track when a run, step, or task has started. What are the data points required for this?
• by polling or other means, track when a run, step or task has failed. What are the data points for this?

The findings in this might also lend towards solving issue 2293

Goals

• Explore the differences between data recorded with local metadata, and the metadata service.
• Report any severe differences between the two, if any exist.
• Propose changes if needed.

Instructions

This issue is meant to be a venue for asking further questions on the topic. Implementation details and proposals are to be kept separate.

When you have met the goals of the issue, it is time to show and tell.
You can do this for example with your own fork with an issue or pull request and messaging a mentor on Slack with a link.

[rawadhossain]

Hey @saikonen, I've investigated this and made a report on it. Here's what I found:

TL;DR: No semantic mismatches were found across tested scenarios. Both backends expose consistent lifecycle signals, no runtime code changes appear necessary based on these results.

What I did:
I ran the same flow under both --metadata=local and --metadata=service across three scenarios: baseline success with a retry probe, controlled failure at post_join, and mid run termination. I built three reproducible tools to make the investigation repeatable: a deterministic probe flow, a snapshot collector, and a comparator script.

Key finding:
run.finished, run.successful, end_task_exists, terminal task state, and retry metadata all matched semantically across both backends in every tested scenario.

Two things noted for follow up:

• Service polling in this dev setup was ~6x slower than local (~12s vs ~2s average), which could affect time sensitive monitoring in production
• Mid run termination behavior is time sensitive and needs more systematic testing

Full report, probe flow, and scripts are in my fork: PR Link

Happy to explore more if there are scenarios I missed.

[Kio-1]

Hey @saikonen

Following up on the GSoC task to compare local vs. service metadata. I've completed a detailed investigation into the friction points of running the current UI in a pure local environment.

Key Findings:

Identified a 500 Internal Server Error in the DAG view when pointing to a local datastore.

Observed 'Metadata Fragmentation' where system tags (user, version) are present on disk but not consistently discovered by the UI backend.

Verified 'Artifact Invisibility' issues for simple variables.

You can find my full report and reproduction notes on my fork here: Report link

Looking forward to discussing how these findings can shape the Standalone Mode implementation plan!