diff --git a/src/blog/polymorphic-messages-in-kafka-streams.md b/src/blog/polymorphic-messages-in-kafka-streams.md index eaa7a9a8..40155379 100644 --- a/src/blog/polymorphic-messages-in-kafka-streams.md +++ b/src/blog/polymorphic-messages-in-kafka-streams.md @@ -2,20 +2,18 @@ title: Polymorphic Messages in Kafka Streams image: "/img/polymorphic-messages-in-kafka-streams.png" imageMeta: - attribution: '' - attributionLink: '' + attribution: "" + attributionLink: "" author: MC publish: 2020-08-26 08:29:00 +0000 layout: Post category: TECHNOLOGY tags: [] - --- ## Things usually start simple... You are designing a Kafka Streams application which must read commands and produce the corresponding business event. - The Avro models you’re expecting to read look like this: ![](/img/1-3.png) @@ -27,7 +25,6 @@ You know you can leverage the **sbt-avrohugger** plugin to generate the correspo ![](/img/3-1.png) Since the messages themselves are pretty straightforward, you decide to create a **monomorphic function** to map properties between each command and the corresponding event. - The resulting topology ends up looking like this: ![](/img/4-1.png) @@ -35,21 +32,17 @@ The resulting topology ends up looking like this: ## ...But then the domain widens Today new functional requirements have emerged: your application must now handle **multiple types** of assets, each with its own unique properties. - You are pondering how to implement this requirement and make your application more resilient to further changes in behavior. ### Multiple streams You could split both commands and events into **multiple topics**, one per asset type, so that the corresponding Avro schema stays consistent and its compatibility is ensured. - This solution, however, would have you replicate pretty much the same topology multiple times, so it’s not recommended unless the business logic has to be customized for each asset type. ### “All-and-none” messages Avro doesn’t support inheritance between records, so any OOP strategy to have assets inherit properties from a common ancestor is unfortunately not viable. - You could however create a “Frankenstein” object with **all the properties** of each and every asset and fill in only those required for each type of asset. - This is definitely the worst solution from an evolutionary and maintainability point of view. ### Union types @@ -65,7 +58,6 @@ Luckily for you, Avro offers an interesting feature named **union types**: you c ### Objects with no shape To cope with this advanced polymorphism, you leverage the [**shapeless**](https://github.com/milessabin/shapeless) library, which introduces the Coproduct type, the perfect companion for the Avro union type. - First of all, you update the custom types mapping of sbt-avrohugger, so that it generates an additional **sealed trait** for each Avro protocol containing multiple records: ![](/img/7.png) @@ -87,7 +79,6 @@ Changes to the topology are minimal, as you’d expect: ### A special kind of Serde Now for the final piece of the puzzle, Serdes. Introducing the [**avro4s**](https://github.com/sksamuel/avro4s) library, which takes Avro GenericRecords above and beyond. - You create a **type class** to extend a plain old Serde providing a brand new method: ![](/img/11.png) @@ -102,4 +93,4 @@ Finally, the main program where you combine all ingredients: ## Conclusions -When multiple use cases share (almost) the same business logic, you can create a stream processing application with **ad-hoc polymorphism** and reduce the duplication of code to the minimum, while making your application even more future-proof. \ No newline at end of file +When multiple use cases share (almost) the same business logic, you can create a stream processing application with **ad-hoc polymorphism** and reduce the duplication of code to the minimum, while making your application even more future-proof.