Feature: Add first-class support for Avro logical types
Description
Avro lets you tag a basic type with a "logical type" that adds extra meaning. KSML currently ignores the tag, so bad values get through silently. The issue asks KSML to start enforcing it.
What is an Avro logical type?
Avro has basic types: int, long, string, bytes. A logical type is a tag on top of one of those that says what the bytes are supposed to mean. Examples:
uuid is a string that must look like a UUID.date is an int that means "days since 1970-01-01".time-millis is an int that means "milliseconds since midnight" (valid: 0 to 86,399,999).time-micros is a long that means "microseconds since midnight" (valid: 0 to 86,399,999,999).timestamp-millis is a long that means "milliseconds since the Unix epoch".timestamp-micros is a long that means "microseconds since the Unix epoch".local-timestamp-millis and local-timestamp-micros are the same but without a timezone.decimal is a bytes array that must encode a number with a declared precision and scale.
Avro just records the tag. It does not check that the value matches. That part is the application's job.
What KSML does today
Nothing. KSML looks at the basic type (INT, STRING, BYTES) and passes the value through. The logical-type tag is ignored.
So:
- A
uuid field accepts "not-a-uuid-at-all". No error.
- A
time-millis field accepts -1. No error. Downstream shows "yesterday at 23:59:59".
- A
decimal field accepts random bytes. No error. The consumer reads the wrong number.
Why it matters
Real systems use these tags for things that have to be exact: prices, customer IDs, timestamps in audit logs. When KSML lets a bad value through, the bug surfaces far from the producer — in a customer dashboard, a database join, a financial report — and is hard to track back.
The most damaging case is decimal. A decimal(10, 2) field is supposed to encode $1234.56 as the integer 123456 in two's-complement bytes. If a producer accidentally writes 1234 (without the cents), the consumer decodes 12.34 — wrong by a factor of 100, with no warning anywhere.
Feature: Add first-class support for Avro logical types
Description
Avro lets you tag a basic type with a "logical type" that adds extra meaning. KSML currently ignores the tag, so bad values get through silently. The issue asks KSML to start enforcing it.
What is an Avro logical type?
Avro has basic types:
int,long,string,bytes. A logical type is a tag on top of one of those that says what the bytes are supposed to mean. Examples:uuidis astringthat must look like a UUID.dateis anintthat means "days since 1970-01-01".time-millisis anintthat means "milliseconds since midnight" (valid: 0 to 86,399,999).time-microsis alongthat means "microseconds since midnight" (valid: 0 to 86,399,999,999).timestamp-millisis alongthat means "milliseconds since the Unix epoch".timestamp-microsis alongthat means "microseconds since the Unix epoch".local-timestamp-millisandlocal-timestamp-microsare the same but without a timezone.decimalis abytesarray that must encode a number with a declared precision and scale.Avro just records the tag. It does not check that the value matches. That part is the application's job.
What KSML does today
Nothing. KSML looks at the basic type (
INT,STRING,BYTES) and passes the value through. The logical-type tag is ignored.So:
uuidfield accepts"not-a-uuid-at-all". No error.time-millisfield accepts-1. No error. Downstream shows "yesterday at 23:59:59".decimalfield accepts random bytes. No error. The consumer reads the wrong number.Why it matters
Real systems use these tags for things that have to be exact: prices, customer IDs, timestamps in audit logs. When KSML lets a bad value through, the bug surfaces far from the producer — in a customer dashboard, a database join, a financial report — and is hard to track back.
The most damaging case is
decimal. Adecimal(10, 2)field is supposed to encode$1234.56as the integer123456in two's-complement bytes. If a producer accidentally writes1234(without the cents), the consumer decodes12.34— wrong by a factor of 100, with no warning anywhere.