Metadata Client fetches unbounded child collections (no pagination/limit/server-side filter) — O(N) for wide foreach scans

[prabhaharanv]

Problem

ServiceMetadataProvider._get_object_internal
(metaflow/plugins/metadata_providers/service.py) fetches child collections with
a bare, unbounded GET and then filters client-side:

url += "/%ss" % sub_type          # e.g. /tasks, /artifacts
v, _ = cls._request(None, url, "GET")
return MetadataProvider._apply_filter(v, filters)

There are no limit, pagination, or time-range parameters, so the response size
grows with the number of objects, and any filtering happens after the full
list is transferred. For Client-side workflows over wide runs this scales poorly.

Motivation / where it hurts

While profiling the Client API (patching ServiceMetadataProvider._request to
count HTTP calls), I saw two patterns worth improving:

• Simple status checks fan out into several metadata requests plus artifact
  downloads (e.g. reading run.successful on a small flow triggers multiple
  metadata GETs and a _success artifact fetch).
• For a foreach with N parallel tasks, finding which tasks failed is
  effectively O(N): there is no way to filter or batch server-side, so the
  client pulls every task/artifact and filters locally.

This is precisely the kind of overhead that matters for programmatic and
agent-style consumers of the Client that poll or scan runs repeatedly.

Prior art in the same file

filter_tasks_by_metadata already introduced a server-side filtered_tasks
endpoint with metadata_field_name / pattern query params — so the pattern of
pushing filtering to the service exists. _get_object_internal predates that and
still does the naive full-list fetch.

Possible directions (seeking maintainer input)

I'd like guidance on the preferred approach before writing code, since this
touches Core Runtime (Metadata) and likely the metadata service contract:

1. Pagination / limit params on the child-collection GETs, with the client
   iterating pages lazily.
2. Server-side filtering for common Client queries (status, attempt,
   time-range), extending the filtered_* pattern rather than downloading then
   _apply_filter-ing.
3. Batch/targeted fetches for the foreach failure-scan case specifically
   (e.g. request only failed tasks / only the _success markers).

Some of these are client-only; others need metadata-service changes. Happy to
prototype whichever direction aligns with the roadmap — this also overlaps with
making the Client more efficient for automated/agent use.

Environment

• Metaflow version: 2.19.35 (also present on current master)
• File: metaflow/plugins/metadata_providers/service.py (_get_object_internal)

(Filing as a design discussion first, per the Core Runtime contribution
guidelines — not a PR yet.)

[yamantaka-singh]

This is a great catch. I can't dictate the metadata service contract changes, but once the maintainers provide guidance on whether to go with pagination or server-side filtering, I'd be happy to help review the PR or test it against my own large runs.