Repository navigation
[YAML] Add SplunkIO write SchemaTransform and WriteToSplunk - #40481
schizophrenicmaniac wants to merge 4 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #40481 +/- ##
============================================
+ Coverage 56.16% 58.68% +2.51%
- Complexity 2288 13953 +11665
============================================
Files 1124 2594 +1470
Lines 178336 272368 +94032
Branches 1489 11286 +9797
============================================
+ Hits 100170 159850 +59680
- Misses 75645 106505 +30860
- Partials 2521 6013 +3492
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
waiting on author |
|
Merged master and resolved the CHANGES.md conflict, keeping both entries. |
derrickaw
left a comment
There was a problem hiding this comment.
A few comments on yaml related code.
|
LGTM for yaml/python @kennknowles, can you review the java part? Thanks |
Adds a SchemaTransform for SplunkIO writes and exposes it in Beam YAML as
WriteToSplunk. It follows the same approach as the Datadog one (#38362). SplunkIO in Java only supports writes, so there is noReadFromSplunkhere.Each input row becomes a
SplunkEvent.event(string) is required.time(int64),host,source,sourcetype,indexandfieldsare optional.fieldscan be a nested row or a JSON object string, and other columns are ignored. The config covers everythingSplunkIO.Writesupports (url, token, batch_count, parallelism, disable_certificate_validation, root_ca_certificate_path, enable_batch_logs, enable_gzip_http_compression) plus error_handling. The error output has the same shape as Datadog's: rows that can't be converted come back withfailed_rowand a 400, and HEC write failures come back with the HTTP status and the payload. Without error_handling, a failed write fails the pipeline.While testing this I found that
SplunkWriteErrorloses all of its values once it goes through a coder. It has@DefaultSchema(AutoValueSchema.class), but its accessors aren'tget-prefixed, so the inferred schema has no fields and every error decodes as null/null/null.SplunkEventWriterTestdoesn't catch this becausecontainsInAnyOrderdecodes both sides with the same coder. I added aSplunkWriteErrorCoder(same idea asDatadogWriteErrorCoder) and set it on the output ofSplunkIO.Write. This changes the output coder ofSplunkIO.Write, so I listed it under bugfixes in CHANGES.md.Changes:
SplunkWriteSchemaTransformProvider/SplunkWriteSchemaTransformConfiguration, registered in the IO expansion serviceWriteToSplunkin standard_io.yaml and the matching entry in standard_external_transforms.yamlSplunkWriteErrorCoder, used bySplunkIO.WriteFixes #40470