feat: Support compilation of properties on Read Operator

[zhenzhongxu]

What happened?

Per Substrait spec, the Read operator supports Properties field with a list of name/value pairs associated. I can't seem to be able to figure out a way to compile a table definition with WITH Statement. One concrete example below. Am I missing something or this can be a new feature request?

import ibis
from ibis_substrait.compiler.core import SubstraitCompiler

bid_schema = ibis.schema(
    {
        "auction": "int64",
        "bidder": "int64",
        "price": "float64",
        "datetime": "timestamp(3)"
    }
)

bid_table = ibis.table(name="Bid", schema=bid_schema)
with_statements = [("watermark", "10"), ("url", "bla")]
compiler = SubstraitCompiler()
compiler.compile(bid_table) 

What is an Ibis equivalent implementation for this DDL that can generate to the expected Substrait plan:

CREATE TABLE bids (
    auction BIGINT,
    bidder BIGINT,
    price DOUBLE,
    datetime TIMESTAMP(3)
) WITH (
    'watermark' = 10,
    'url' = 'bla'
);

Generated Substrait Plan

relations {
  root {
    input {
      read {
        common {
          direct {
          }
        }
        base_schema {
          names: "auction"
          names: "bidder"
          names: "price"
          names: "datetime"
          struct {
            types {
              i64 {
                nullability: NULLABILITY_NULLABLE
              }
            }
            types {
              i64 {
                nullability: NULLABILITY_NULLABLE
              }
            }
            types {
              fp64 {
                nullability: NULLABILITY_NULLABLE
              }
            }
            types {
              timestamp {
                nullability: NULLABILITY_NULLABLE
              }
            }
            nullability: NULLABILITY_REQUIRED
          }
        }
        named_table {
          names: "Bid"
        }
        properties {
          "watermark": "10",
          "url": "bla"
        }
      }
    }
    names: "auction"
    names: "bidder"
    names: "price"
    names: "datetime"
  }
}
version {
  minor_number: 54
  producer: "ibis-substrait"
}

What version of ibis-substrait are you using?

v4.0.1

What substrait consumer(s) are you using, if any?

N/A

Relevant log output

No response

[gforsyth]

What is an Ibis equivalent implementation for this DDL that can generate to the expected Substrait plan:

You aren't missing anything, that just doesn't exist. There is Ibis DDL for creating a table with properties, but those calls are eager and are not part of the IR, so there's nothing to "translate" to substrait.

Adding deferred table creation statements to Ibis would be a long-term project.

I think the better move short-term is to edit the generated substrait plan with the watermark properties

[zhenzhongxu]

thanks, will take the short-term workaround for now. Do let me know when the deferred table creation will be included!