Summary
The Kafka Streams runner creates its own internal topics (a bootstrap topic per Impulse and Read, a repartition topic per GroupByKey). Today only the partition count (--internalParallelism) and replication factor are configurable.
A user should be able to set the rest of the topic-level configuration — retention, compaction, segment settings, min.insync.replicas and so on. Two options discussed on #39546: pass a JSON-serialized dictionary of topic configs, and/or expose the most common ones as separate pipeline options.
Part of #18479.
Summary
The Kafka Streams runner creates its own internal topics (a bootstrap topic per Impulse and Read, a repartition topic per GroupByKey). Today only the partition count (
--internalParallelism) and replication factor are configurable.A user should be able to set the rest of the topic-level configuration — retention, compaction, segment settings, min.insync.replicas and so on. Two options discussed on #39546: pass a JSON-serialized dictionary of topic configs, and/or expose the most common ones as separate pipeline options.
Part of #18479.