Skip to content

Write human-readable partition paths for temporal transforms#1111

Closed
raghav-reglobe wants to merge 1 commit into
duckdb:v1.5-variegatafrom
raghav-reglobe:human-readable-partition-paths
Closed

Write human-readable partition paths for temporal transforms#1111
raghav-reglobe wants to merge 1 commit into
duckdb:v1.5-variegatafrom
raghav-reglobe:human-readable-partition-paths

Conversation

@raghav-reglobe

@raghav-reglobe raghav-reglobe commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Partitioned writes for the year/month/day/hour transforms name the
on-disk hive directory after the raw integer ordinal, e.g. .../<field>=634/...
for month(ts). The partition routing column is the date_diff(...) ordinal and
DuckDB core stringifies that value directly into the directory name.

The Apache Iceberg reference implementation (Java) instead formats the path segment
with Transform.toHumanString (PartitionSpec.partitionToPath):

transform DuckDB today Apache Iceberg (Java)
month =634 =2022-11
year =52 =2022
day =N =2022-11-15
hour =N =2022-11-15-13

This is not a correctness problem for Iceberg readers — files are tracked by full
path in the manifest and the partition tuple is stored in the manifest entry, so the
directory string is never authoritative. But it diverges from the Apache Iceberg
reference implementation's convention, the human-readable form is easier to
understand when browsing storage, and a table written by both DuckDB and Spark/Java
otherwise ends up with mixed =634/ and =2022-11/ directories for the same logical
partition.

Fix

The directory string is built by DuckDB core from the partition routing column's
value, so the extension's only lever is the value/type of that column. This PR
changes only the string representation of the temporal routing value and
leaves partition assignment untouched:

  • New iceberg_partition_to_human(transform, ordinal) scalar formats a temporal
    ordinal into the human-readable segment, reusing the existing
    IcebergTransform::PartitionValueToString. GetPartitionExpression wraps the
    date_diff ordinal with it, so the routing value (hence the directory) is
    human-readable. Rows still group by the same ordinal.
  • The manifest partition value is sourced back from that routing value in
    IcebergInsertGlobalState::AddFiles, so a new inverse
    IcebergTransform::PartitionStringToValue converts the human string back to the
    raw ordinal there. The manifest partition value is byte-identical to before.
  • PartitionValueToString's hour case is fixed to Iceberg's yyyy-MM-dd-HH (it
    previously emitted a full timestamp string), and year is 4-digit padded, so
    the read-side output matches the written segment.

bucket, truncate and identity are unchanged — they already write the correct
human value.

Backward compatibility

Because the manifest partition value is unchanged, files written by older DuckDB
(=634/) and by this change (=2022-11/) land in the same logical partition
Iceberg matches on the manifest value, not the directory. No re-partitioning, and a
subsequent rewrite_data_files normalizes the physical layout over time. (If the
ordinal were ever lost, the INSERT would fail the manifest's BIGINT cast, so the
round-trip is verified by construction.)

Test

test/.../insert/partitions/temporal/test_human_readable_partition_paths.test
asserts the human-readable directory for all four temporal transforms (and that the
old =634/ form is gone) while the data round-trips and partition pruning still
resolves.

@Tishj

Tishj commented Jun 25, 2026

Copy link
Copy Markdown
Member

"Diverges from the spec convention"
Can you please link that section? Thanks

@Tishj

Tishj commented Jun 25, 2026

Copy link
Copy Markdown
Member

And in general, can you please refrain from adding more comments than code?

@raghav-reglobe
raghav-reglobe force-pushed the human-readable-partition-paths branch from dba1204 to 537f34c Compare June 25, 2026 05:43
Partitioned writes for the year/month/day/hour transforms named the on-disk
hive directory after the raw integer ordinal (e.g. `<field>=634/`): the
partition routing column is the `date_diff(...)` ordinal and DuckDB core
stringifies that value directly into the directory name. Apache Iceberg's
reference writers use `Transform.toHumanString` for the path segment, producing
`<field>=2022-11/` (month), `=2022/` (year), `=2022-11-15/` (day) and
`=2022-11-15-13/` (hour).

Match that convention without changing partition assignment:

- Add an `iceberg_partition_to_human(transform, ordinal)` scalar that formats a
  temporal ordinal into the human-readable segment, reusing the existing
  `IcebergTransform::PartitionValueToString`. The temporal partition routing
  expression is wrapped with it, so the routing value (and therefore the
  directory) is human-readable. Rows still group by the same `date_diff`
  ordinal -- only the string representation differs.
- Restore the raw ordinal for the manifest on the write-back path via a new
  inverse `IcebergTransform::PartitionStringToValue` (in
  `IcebergInsertGlobalState::AddFiles`), so the manifest partition value is
  byte-identical to before. Files written by older code (`=634/`) and by this
  change (`=2022-11/`) resolve to the same logical partition; readers match on
  the manifest value, not the path, and a later `rewrite_data_files` normalizes
  the physical layout.
- Fix `PartitionValueToString`'s `hour` case to Iceberg's `yyyy-MM-dd-HH` (it
  emitted a full timestamp string) and 4-digit-pad `year`, so the read-side log
  path agrees with the written segment.

bucket, truncate and identity are unchanged -- they already write the correct
human value.

Adds a regression test asserting the human-readable directory for month, year,
day and hour transforms while the data round-trips and partition pruning still
resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Raghvendra Singh <raghav@cashify.in>
@raghav-reglobe
raghav-reglobe force-pushed the human-readable-partition-paths branch from 537f34c to 285e2b3 Compare June 25, 2026 07:47
@raghav-reglobe

Copy link
Copy Markdown
Contributor Author

You're right, "spec convention" was imprecise — the table spec doesn't mandate a path format. Files are tracked by full path in the manifest and the partition tuple is stored in the manifest entry, so the directory string is never authoritative.

What I meant is the convention of the Apache Iceberg reference implementation (Java): PartitionSpec.partitionToPath formats each segment as name = escape(transform.toHumanString(type, value)). Beyond matching that, the human-readable form is genuinely useful in storage — …month=2022-11/ is immediately understandable when browsing the data vs the opaque …month=634/ — and a table written by both DuckDB and Spark/Java currently ends up with a mix of =634/ and =2022-11/ for the same logical partition. I've reworded the PR description to drop the "spec" framing. Thanks for the catch!

@raghav-reglobe

Copy link
Copy Markdown
Contributor Author

@Tishj reworded the description per your note — anything else you'd like changed before this is good to go?

One branching question: this targets v1.5-variegata, and since main has diverged onto the 1.6 line the change won't reach main from here. Happy to open a matching PR against main if you'd like it on 1.6 too — just let me know which you'd prefer (or both).

@Tishj

Tishj commented Jun 25, 2026

Copy link
Copy Markdown
Member

This is not a correctness problem for Iceberg readers

I don't understand why we would want to make this change.
It's adding complexity where there was none, for no apparent reason

@Tishj Tishj closed this Jun 25, 2026
@Tmonster

Copy link
Copy Markdown
Member

I think changing the partition function so it outputs the human readable string, then reinterpreting the string back to the ordinal for the manifest files introduces complexity we would rather not have. I would suggest instead to first add functionality to core that allows a user to determine the file naming pattern per partition value, then you can have duckdb-iceberg use that to write the human readable dates

@raghav-reglobe
raghav-reglobe deleted the human-readable-partition-paths branch June 25, 2026 13:32
@raghav-reglobe

Copy link
Copy Markdown
Contributor Author

@Tmonster that makes sense — threading the format through a core hook is cleaner than reinterpreting the string back to the ordinal, and it lets the extension partition on the raw value directly. Before I put a PR together I want to make sure the shape matches what you had in mind.

From a quick look, the only place core turns a partition value into the name=value directory segment is GetOrCreateDirectory in physical_copy_to_file.cpp (the value.ToString() call) — and it's already independent of both the routing key (HivePartitionKey) and the returned WRITTEN_FILE_STATISTICS partition map, which is built separately. So I was picturing something minimal: an optional std::function<string(idx_t, const Value &)> partition_value_formatter on PhysicalCopyToFile, threaded into GetOrCreateDirectory and defaulting to value.ToString() when unset. An extension that builds the copy op directly (like iceberg does in PlanCopyForInsert) sets it; the SQL COPY path is unaffected.

On the iceberg side that then lets us partition on the raw ordinal and drop the iceberg_partition_to_human + PartitionStringToValue round-trip entirely — the manifest reads the ordinal, only the directory gets formatted.

Does that match what you were picturing, and would core be open to a hook like that on the copy operator? Happy to adjust the API (or put it somewhere other than PhysicalCopyToFile) before I open anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants