Skip to content

[Feature Request]: ClickHouseIO: Add Decimal(P, S) support for fixed-point columns #39840

Closed

Description

@tomnewton

What would you like to happen?

Problem

ClickHouseIO has no notion of ClickHouse's fixed-point Decimal family — Decimal(P, S),
Decimal(P), bare Decimal, and the width aliases Decimal32(S), Decimal64(S),
Decimal128(S), Decimal256(S). The column-type parser rejects these type strings, so
ClickHouseIO.getTableSchema() throws on the DESCRIBE TABLE output and pipelines cannot
write to any table containing a Decimal column — a blocker for monetary and other
fixed-point data.

Proposed solution

  1. Schema model

    • Add TypeName.DECIMAL.
    • Extend ColumnType with a scale field alongside the existing precision field
      (introduced for DateTime64), plus a validated factory
      ColumnType.decimal(int precision, int scale) (P in [1, 76], S in [0, P]).
  2. Parser

    • Grammar rules for Decimal[(P[, S])] (bare Decimal = Decimal(10, 0),
      Decimal(P) = Decimal(P, 0), matching ClickHouse) and the width aliases
      Decimal32/64/128/256(S) (precisions 9/18/38/76), reachable through
      Nullable(...) and Array(...).
  3. Beam field type mapping

    • Decimal(P, S) maps to the FixedPrecisionNumeric.of(P, S) logical type
      (base type DECIMAL, values are BigDecimal), preserving the declared
      precision/scale through getEquivalentSchema — the same logical type JdbcIO
      uses for NUMERIC columns.
  4. Writer

    • Serialize as ClickHouse's RowBinary representation: the unscaled value as a
      little-endian signed integer whose width (32/64/128/256 bits) is chosen from the
      declared precision, exactly per ClickHouse's Decimal(P, S) → Decimal32/64/128/256
      synonym rule.
  5. Defaults

    • Support DEFAULT <literal> on Decimal columns in
      ColumnType.parseDefaultExpression — without it, schema fetch fails for any table
      with a defaulted Decimal column.
  6. Tests

    • Parser/mapping unit tests, byte-level writer tests, and round-trip tests against the
      ClickHouse test container for all four width buckets, Nullable(Decimal),
      Array(Decimal), DEFAULT handling, and getTableSchema on the server's canonicalized
      DESCRIBE output.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions