feat(transform): Add checkpoint file

[tomverelst]

Allow appending to an existing checkpoint snapshot file. The existing snapshot will be considered as work that is already done. This will help the user in case of errors that occur during the transformation.

Warning

This only works properly when sticking with the batch model. This does not work with a streaming model, thus if the transform command does NOT stream any records and only outputs the fully transformed snapshot when allll work is done. Streaming output could cause the same records to be applied multiple times when retrying. A journal/checkpoint needs to be considered as well for the apply step.

Tip

tee can write stdin to stdout and files. Look at this as alternative.

Example

$ cat input.csv
topic1,group1,0,100
topic2,group2,0,200

$ echo input.csv | kbridge calculate --checkpoint done.csv ...  # Fails halfway

$ cat done.csv
topic1,group1,0,50

$ echo input.csv | kbridge calculate --checkpoint done.csv ...  #  Runs to completion
topic1,group1,0,50
topic2,group2,0,100

What if we cannot read from or write to the checkpoint file?

The command will print an error message and exit with a non-zero code