Skip to content

write_modes.md

Latest commit

 

History

History
81 lines (70 loc) · 4.74 KB

write_modes.md

File metadata and controls

81 lines (70 loc) · 4.74 KB
Raw

Write Modes

Floe resolves accepted/rejected write behavior from sink.write_mode.

Supported values:

  • overwrite (default): existing output parts are replaced before writing new output.
  • append: new output parts are added without deleting existing ones.
  • merge_scd1: SCD1 upsert keyed by schema.primary_key on Delta or DuckDB accepted sinks.
  • merge_scd2: SCD2 merge keyed by schema.primary_key on Delta or DuckDB accepted sinks.

Standard Delta writes (append, overwrite)

For accepted Delta sinks, standard write modes stay strict by default.

  • schema.schema_evolution.mode: strict preserves the previous behavior: source and target Delta schemas must already be compatible.
  • schema.schema_evolution.mode: add_columns enables additive-only Delta schema evolution for append and overwrite.
    • Only new columns present in the source schema may be added to the target Delta table.
    • Existing columns must remain compatible; drop/rename/type-change flows are rejected.
    • When enabled and no new columns are present, Floe records a no-op schema-evolution report block.
    • When columns are added, Floe emits a structured schema_evolution_applied event and records the added column names in the entity run report.
    • Adding columns to already partitioned Delta tables is rejected in this phase.

merge_scd1 semantics

  • Accepted sink format must be delta or duckdb.
  • schema.primary_key is required and is used as merge key.
  • Source rows are validated for merge-key uniqueness during row checks. In policy.severity=warn, duplicate merge-key rows are rejected before merge so only unambiguous rows are merged.
  • On key match: update non-key columns from source.
  • sink.accepted.merge.ignore_columns can exclude additional business columns from SCD1 update sets.
  • On missing key: insert source row.
  • Strict schema handling remains the default behavior.
  • schema.schema_evolution.mode: add_columns enables additive-only Delta schema evolution for merge writes.
    • New non-key business columns may be added before the merge commit.
    • Existing columns must remain compatible; drop/rename/type-change flows are rejected.
    • Merge-key columns cannot be introduced by schema evolution.
  • DuckDB merge writes use DuckDB's native MERGE INTO; DuckDB schema evolution is not supported.
  • Delta merge writes use Delta native merge (MERGE INTO) through delta-rs/DataFusion.
  • Single-writer assumption: sink-level commit or lock conflicts are returned as clear write errors.

merge_scd2 semantics

  • Accepted sink format must be delta or duckdb.
  • schema.primary_key is required and is used as merge key.
  • Source rows are validated for merge-key uniqueness during row checks. In policy.severity=warn, duplicate merge-key rows are rejected before merge so only unambiguous rows are merged.
  • Floe manages SCD2 system columns in Delta and DuckDB target tables:
    • __floe_is_current
    • __floe_valid_from
    • __floe_valid_to
  • sink.accepted.merge.scd2 can override these system column names.
  • Changed current rows are closed (is_current=false, valid_to=current_timestamp()), then new current versions are inserted.
  • Missing keys are inserted as new current rows.
  • Unchanged current rows are left as-is.
  • Reported merge metrics for merge_scd2:
    • closed_count = rows closed in the current target set.
    • inserted_count = rows inserted as new current versions (changed + brand new keys).
    • unchanged_count = source rows that matched active target rows with no change.
    • updated_count is preserved for backward compatibility and matches closed_count.
  • Change detection columns are resolved as:
    • sink.accepted.merge.compare_columns when configured
    • otherwise all non-key business columns minus sink.accepted.merge.ignore_columns
  • Strict schema handling remains the default behavior.
  • schema.schema_evolution.mode: add_columns enables additive-only Delta schema evolution for merge writes.
    • New business columns may be added while Floe preserves SCD2 system columns.
    • Existing columns must remain compatible; drop/rename/type-change flows are rejected.
    • Merge-key columns cannot be introduced by schema evolution.
  • DuckDB merge writes use DuckDB's native MERGE INTO; DuckDB schema evolution is not supported.
  • Single-writer assumption: sink-level commit or lock conflicts are returned as clear write errors.

Rejected output behavior

sink.write_mode is shared with rejected output:

  • overwrite: first rejected write in a run starts fresh, subsequent rejected files append.
  • append: rejected files append.
  • merge_scd1: rejected files append (merge logic applies only to the accepted Delta/DuckDB sink).
  • merge_scd2: rejected files append (merge logic applies only to the accepted Delta/DuckDB sink).