Skip to content

Make OpenLineage endpoint configurable for native OpenMetadata ingestion #450

Description

@malon64

Context

OpenMetadata's native OpenLineage endpoint is /api/v1/openlineage/lineage, while Floe currently appends /api/v1/lineage unconditionally to lineage.url. With Floe 0.6.8, setting lineage.url to an OpenMetadata base URL therefore cannot reach the native OpenMetadata endpoint without a proxy or URL workaround.

Current behavior

The lineage client builds the request URL as:

{lineage.url}/api/v1/lineage

This is documented in docs/lineage.md and prevents direct ingestion at OpenMetadata's native route.

Requested change

Please add a backward-compatible optional lineage.endpoint (default api/v1/lineage) and join it safely to lineage.url. This should allow configurations such as:

lineage:
  url: http://openmetadata:8585
  endpoint: api/v1/openlineage/lineage

A separate optional lineage.dataset_namespace would also be useful so the OpenLineage job namespace can remain a stable orchestration identity (for example dagster) while Iceberg dataset namespaces map to the canonical catalog service in OpenMetadata. It should default to the existing lineage.namespace for compatibility.

Acceptance criteria

  • Existing configurations continue posting to /api/v1/lineage.
  • Custom endpoints do not produce doubled paths or leading-slash bugs.
  • Tests cover URL joining and distinct job/dataset namespaces.
  • Documentation includes the OpenMetadata native endpoint example.

OpenLakeForge issue: malon64/openlakeforge#17

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions