Malformed schema in manifest header

[jvansanten]

I see the following error in iceberg-rest-fixture 1.10.1 when trying to drop a table created by DuckDB:

[qtp999522307-16] INFO org.apache.iceberg.BaseMetastoreCatalog - Table loaded by catalog: rest_backend.lsst.stream_jW1rl7BPK582gJ5bBFus5aHzaMBZ6D0QLfucUhJd1UQ
[qtp999522307-17] INFO org.apache.iceberg.BaseMetastoreTableOperations - Refreshing table metadata from new version: /private/var/folders/1_/hs0775c97ms0m4p0yvl38vm40000gn/T/pytest-of-jakob/pytest-1115/warehouse0/lsst/stream_jW1rl7BPK582gJ5bBFus5aHzaMBZ6D0QLfucUhJd1UQ/metadata/00000-36ad4fad-cf16-487a-bdd8-3cc6f614b91d.metadata.json
[qtp999522307-17] INFO org.apache.iceberg.CatalogUtil - Manifests to delete: GenericManifestFile{content=DATA, path=/private/var/folders/1_/hs0775c97ms0m4p0yvl38vm40000gn/T/pytest-of-jakob/pytest-1115/warehouse0/lsst/stream_jW1rl7BPK582gJ5bBFus5aHzaMBZ6D0QLfucUhJd1UQ/metadata/09668e1d-74c1-4fd3-90fa-8302209736b6-m0.avro, length=3399, partition_spec_id=0, added_snapshot_id=6767045734509594085, added_data_files_count=1, added_rows_count=1, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=null, key_metadata=null, sequence_number=1, min_sequence_number=1, first_row_id=null}
[iceberg-worker-pool-0] WARN org.apache.iceberg.CatalogUtil - Failed to get deleted files: this may cause orphaned data files
java.lang.IllegalArgumentException: Cannot parse type from json: {"type":"array","items":{"type":"record","name":"k126_k127","fields":[{"name":"key","type":"int","id":126},{"name":"value","type":"binary","id":127}]}}
        at org.apache.iceberg.SchemaParser.typeFromJson(SchemaParser.java:195)
        at org.apache.iceberg.SchemaParser.structFromJson(SchemaParser.java:235)
        at org.apache.iceberg.SchemaParser.typeFromJson(SchemaParser.java:186)
        at org.apache.iceberg.SchemaParser.structFromJson(SchemaParser.java:235)
        at org.apache.iceberg.SchemaParser.typeFromJson(SchemaParser.java:186)
        at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:284)
        at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:104)
        at org.apache.iceberg.SchemaParser.lambda$fromJson$1(SchemaParser.java:303)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
        at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:303)
        at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:144)
        at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:129)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:138)
        at org.apache.iceberg.ManifestFiles.open(ManifestFiles.java:311)
        at org.apache.iceberg.ManifestFiles.open(ManifestFiles.java:304)
        at org.apache.iceberg.CatalogUtil.lambda$deleteFiles$1(CatalogUtil.java:179)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
        at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
[qtp999522307-17] INFO org.apache.iceberg.jdbc.JdbcCatalog - Dropped table: lsst.stream_jW1rl7BPK582gJ5bBFus5aHzaMBZ6D0QLfucUhJd1UQ

This appears to happen because the JSON string written into the Avro header at "schema" is:

{
  "type": "struct",
  "schema-id": 0,
  "fields": [
    {
      "id": 0,
      "name": "status",
      "required": true,
      "type": "int"
    },
    {
      "id": 1,
      "name": "snapshot_id",
      "required": false,
      "type": "long"
    },
    {
      "id": 3,
      "name": "sequence_number",
      "required": false,
      "type": "long"
    },
    {
      "id": 4,
      "name": "file_sequence_number",
      "required": false,
      "type": "long"
    },
    {
      "id": 2,
      "name": "data_file",
      "required": true,
      "type": {
        "type": "struct",
        "fields": [
          {
            "id": 134,
            "name": "content",
            "required": true,
            "type": "int"
          },
          {
            "id": 100,
            "name": "file_path",
            "required": true,
            "type": "string"
          },
          {
            "id": 101,
            "name": "file_format",
            "required": true,
            "type": "string"
          },
          {
            "id": 102,
            "name": "partition",
            "required": true,
            "type": {
              "type": "struct",
              "fields": []
            }
          },
          {
            "id": 103,
            "name": "record_count",
            "required": true,
            "type": "long"
          },
          {
            "id": 104,
            "name": "file_size_in_bytes",
            "required": true,
            "type": "long"
          },
          {
            "id": 125,
            "name": "lower_bounds",
            "required": false,
            "type": {
              "type": "array",
              "items": {
                "type": "record",
                "name": "k126_k127",
                "fields": [
                  {
                    "name": "key",
                    "type": "int",
                    "id": 126
                  },
                  {
                    "name": "value",
                    "type": "binary",
                    "id": 127
                  }
                ]
              }
            }
          },
          {
            "id": 128,
            "name": "upper_bounds",
            "required": false,
            "type": {
              "type": "array",
              "items": {
                "type": "record",
                "name": "k129_k130",
                "fields": [
                  {
                    "name": "key",
                    "type": "int",
                    "id": 129
                  },
                  {
                    "name": "value",
                    "type": "binary",
                    "id": 130
                  }
                ]
              }
            }
          },
          {
            "id": 110,
            "name": "null_value_counts",
            "required": false,
            "type": {
              "type": "array",
              "items": {
                "type": "record",
                "name": "k121_k122",
                "fields": [
                  {
                    "name": "key",
                    "type": "int",
                    "id": 121
                  },
                  {
                    "name": "value",
                    "type": "binary",
                    "id": 122
                  }
                ]
              }
            }
          }
        ]
      }
    }
  ]
}

This is not the Iceberg schema of the referenced table required by the spec, but nearly the Iceberg schema of the manifest itself, with some Avro schema syntax mixed in ("record" instead of "struct", "array" instead of "list", etc.). Looking at some examples of manifests written by Spark, it looks like the Iceberg schema of the manifest should be in "iceberg.schema", though I can't actually find this mentioned in the spec.

[Tmonster]

Thanks for the issue. We've been having a lot of difficulties with spark where it requires a very specific details in metadata avro schemas that are technically not required by the spec. Even in this case, looking at the schema, it's unclear what duckdb is doing incorrectly. I have also looked at many spark created avro files, and they use "record" and "array" as well, so I can't be certain that is the issue.

Do you know with what version the DuckDB table was written?
And can you still delete the table with DuckDB?

[jvansanten]

Thanks for the issue. We've been having a lot of difficulties with spark where it requires a very specific details in metadata avro schemas that are technically not required by the spec. Even in this case, looking at the schema, it's unclear what duckdb is doing incorrectly.

As far as I can tell, there are 3 schemas in the headers of Spark-created manifests:

1. schema: the Iceberg schema of the table at the time the manifest was written (required by the spec)
2. iceberg.schema: the Iceberg schema of the manifest itself (not sure what uses it)
3. avro.schema: the Avro schema of the manifest itself
   I think the issue is that duckdb was trying to write (2) at the key where org.apache.iceberg.ManifestReader expects to find (1), and also using mixing Avro and Iceberg syntax.

I have also looked at many spark created avro files, and they use "record" and "array" as well, so I can't be certain that is the issue.

"record" and "array" will certainly appear in the headers of Spark-created avro files, but, as far as I understand, only on avro.schema, not in the Iceberg-related header fields.

Do you know with what version the DuckDB table was written?

Sorry, should have mentioned that I first ran across this with a local build of duckdb-iceberg based on the head of PR #766, but it also appears with 1.4.4.

And can you still delete the table with DuckDB?

Yes, the delete operation succeeds, and the catalog cleans up the metadata files (I've set PURGE_REQUESTED true when attaching the catalog). Because of the way org.apache.iceberg.ManifestReader greedily parses the schema in the manifest header, though, it silently fails before it gets around to deleting the data files, so they sit around in storage until cleaned up manually.

#801 contains a suggested fix.

[djouallah]

Hitting this same issue from a different downstream consumer. Tables written by the DuckDB Iceberg writer fail conversion in Microsoft Fabric / OneLake's XTable-based Iceberg→Delta sync with:

IllegalArgumentException: Cannot parse type from json: 
{"type":"array","items":{"type":"record","name":"k126_k127",
"fields":[{"name":"key","type":"int","id":126},
{"name":"value","type":"binary","id":127}]}}
    at org.apache.iceberg.SchemaParser.typeFromJson(SchemaParser.java:184)

Confirmed via read_avro on the manifest entry files: lower_bounds and upper_bounds come through as struct(key integer, value blob)[] rather than MAP(INTEGER, BLOB), matching the malformed embedded schema in this issue. The table is unpartitioned with only primitive columns, so the bug is entirely in the manifest header schema, not user data.
Adds a second concrete downstream that breaks on this beyond the iceberg-rest-fixture drop case — XTable sync is a hard blocker for any pipeline writing Iceberg with DuckDB and consuming via OneLake.

[dataders]

I took a Codex-assisted pass at this and opened #1016 with a focused regression test: #1016

[Tishj]

Thanks for making this issue and apologies for the delay in fixing.
It seems that our testing set up doesn't lend itself nicely to verifying situations like this, which made verifying the fix very difficult.

[djouallah]

i can confirm, it fixed a writing issue with UC, now it works !!!

[tikhdm]

Bumped in the same issue with 1.5.3. The malformed schema in the manifest header also breaks Athena OPTIMIZE (any partition-spec-aware rewrite).
Seems that rewrite re-opens each manifest via ManifestReader.readPartitionSpec → SchemaParser.fromJson(<embedded schema>) and hits the same IllegalArgumentException: Cannot parse type from json: {"type":"array",...}.

But Athena surfaces it as:

  ```
  OPTIMIZE <table> REWRITE DATA USING BIN_PACK
  GENERIC_INTERNAL_ERROR: Missing required files to delete: <s3 paths that actually exist>
  ```

The listed files exist on S3 and are referenced by the current snapshot, so the error is misleading - the real failure is parsing the partition spec.

Update:
I mis-attributed this. The Athena OPTIMIZE "Missing required files to delete" failure I hit is not caused by this malformed schema — patching the manifest schema to the table schema did not fix OPTIMIZE. The actual cause is un-rounded partition field-summary bounds (separate bug, filing separately).