Skip to content

[YAML] Add SplunkIO write SchemaTransform and WriteToSplunk - #40481

Open
schizophrenicmaniac wants to merge 4 commits into
apache:masterfrom
schizophrenicmaniac:fix/40470-feature-request-normalize-splunkio-for
Open

schizophrenicmaniac wants to merge 4 commits into
apache:masterfrom
schizophrenicmaniac:fix/40470-feature-request-normalize-splunkio-for

Conversation

@schizophrenicmaniac

Copy link
Copy Markdown
Contributor

Adds a SchemaTransform for SplunkIO writes and exposes it in Beam YAML as WriteToSplunk. It follows the same approach as the Datadog one (#38362). SplunkIO in Java only supports writes, so there is no ReadFromSplunk here.

Each input row becomes a SplunkEvent. event (string) is required. time (int64), host, source, sourcetype, index and fields are optional. fields can be a nested row or a JSON object string, and other columns are ignored. The config covers everything SplunkIO.Write supports (url, token, batch_count, parallelism, disable_certificate_validation, root_ca_certificate_path, enable_batch_logs, enable_gzip_http_compression) plus error_handling. The error output has the same shape as Datadog's: rows that can't be converted come back with failed_row and a 400, and HEC write failures come back with the HTTP status and the payload. Without error_handling, a failed write fails the pipeline.

While testing this I found that SplunkWriteError loses all of its values once it goes through a coder. It has @DefaultSchema(AutoValueSchema.class), but its accessors aren't get-prefixed, so the inferred schema has no fields and every error decodes as null/null/null. SplunkEventWriterTest doesn't catch this because containsInAnyOrder decodes both sides with the same coder. I added a SplunkWriteErrorCoder (same idea as DatadogWriteErrorCoder) and set it on the output of SplunkIO.Write. This changes the output coder of SplunkIO.Write, so I listed it under bugfixes in CHANGES.md.

Changes:

  • SplunkWriteSchemaTransformProvider / SplunkWriteSchemaTransformConfiguration, registered in the IO expansion service
  • WriteToSplunk in standard_io.yaml and the matching entry in standard_external_transforms.yaml
  • SplunkWriteErrorCoder, used by SplunkIO.Write
  • Java tests against a mock HEC server, and a YAML integration test with a fake HEC fixture (tests/splunk.yaml)

Fixes #40470

@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.68%. Comparing base (76f6e6d) to head (22694d6).
⚠️ Report is 143 commits behind head on master.

Files with missing lines Patch % Lines
...n/apache_beam/yaml/test_utils/splunk_test_utils.py 0.00% 99 Missing ⚠️
sdks/python/apache_beam/yaml/integration_tests.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40481      +/-   ##
============================================
+ Coverage     56.16%   58.68%   +2.51%     
- Complexity     2288    13953   +11665     
============================================
  Files          1124     2594    +1470     
  Lines        178336   272368   +94032     
  Branches       1489    11286    +9797     
============================================
+ Hits         100170   159850   +59680     
- Misses        75645   106505   +30860     
- Partials       2521     6013    +3492     
Flag Coverage Δ
java 65.04% <ø> (-10.75%) ⬇️
python 79.61% <0.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
R: @kennknowles for label java.
R: @derrickaw for label yaml.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@derrickaw

Copy link
Copy Markdown
Collaborator

waiting on author

@schizophrenicmaniac

Copy link
Copy Markdown
Contributor Author

Merged master and resolved the CHANGES.md conflict, keeping both entries.

@derrickaw derrickaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments on yaml related code.

Comment thread sdks/python/apache_beam/yaml/test_utils/splunk_test_utils.py Outdated
Comment thread sdks/python/apache_beam/yaml/test_utils/splunk_test_utils.py
Comment thread sdks/python/apache_beam/yaml/test_utils/splunk_test_utils.py Outdated
@derrickaw

Copy link
Copy Markdown
Collaborator

LGTM for yaml/python

@kennknowles, can you review the java part? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Normalize splunkIO for yaml

2 participants