Rework olf e2e readiness and Dagster launch retries

[malon64]

Context

make local-e2e and make aws-e2e can pass Kubernetes pod health then fail while launching Dagster jobs with a GraphQL HTTP 500. During the observed local failure, the Dagster webserver was reachable through the runner-managed port-forward, but its product gRPC code locations were temporarily unavailable (UNAVAILABLE / DEADLINE_EXCEEDED).

The current pod-health check queries every pod in the namespace and treats every non-succeeded pod as a platform health failure. This gives two incorrect outcomes:

• historical or completed Dagster-run Jobs can make preflight wait five minutes before the actual suite begins;
• a failed one-shot operational Job, such as an OpenMetadata crawler/ingestion pipeline, blocks readiness even when the suite does not depend on that Job. In AWS this occurred for om-job-aws-glue-metadata-ingestion; dbt-trino materialization and native OpenLineage remained healthy.

This is readiness-policy behavior, not a missing port-forward: olf e2e run already opens a scoped kubectl port-forward to the Dagster webserver.

Scope

• Define health classes for olf e2e: long-running required services, transient startup workloads, suite-owned Jobs, and independent operational/ingestion Jobs.
• Preflight only on the required long-running services and make readiness wait for the required Dagster repositories/code locations to be loadable, not only for Kubernetes containers to report Ready.
• Retry bounded, transient GraphQL 5xx failures while launching a run, with diagnostics when the retry budget is exhausted.
• Treat failed one-shot Jobs outside the suite's explicit dependencies as warnings: print their identity, owner/pipeline labels, exit status, and bounded logs, but do not block preflight.
• Preserve failure detection for Jobs launched by the current suite and for any pipeline explicitly required by a suite assertion.
• Retain actionable diagnostics: unhealthy service pods, failed/warned Job identity, recent Dagster webserver and user-code logs, and GraphQL response/status.
• Add unit coverage for health classification, readiness, warnings, and retry behavior.

Acceptance criteria

• A temporary gRPC location reload during startup waits/retries and does not fail the suite immediately.
• Historical failed Dagster runs and unrelated failed OpenMetadata ingestion Jobs are warnings, not preflight blockers.
• An unavailable required service fails within a bounded time with useful diagnostics.
• A failed run launched by the suite, or a failed ingestion pipeline the suite explicitly asserts, fails the suite with useful evidence.
• A persistent Dagster GraphQL 5xx fails within a bounded time and reports the relevant diagnostics.

Related

• Complements Add a fresh local full-e2e main or nightly gate #60 (fresh local full-E2E gate); this issue makes the existing runner reliable enough for that gate.
• Observed while validating the dbt-trino Gold migration in ADR: evaluate dbt-trino as the Gold transformation engine #17.

[malon64]

Completed and merged in #63.