Feature: Concatenate Partial/Split Container Logs

[dtbayles]

Summary

• Add parse-multiline-json.conf which allows users to concatenate split container logs without having to publish their own image (see tutorial here)
• This PR does not change or remove any existing behavior

Testing

• integration tests passing

make debug succeeded: yes
Integ tests succeeded: yes
New tests cover the changes: yes

Description for the changelog

Add .conf for concatenating split container logs

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dtbayles]

Context: this reduces the need for consumers of this service to publish their own image to ECR with just a few added lines in a .conf file and allows them to choose the new .conf file in their CDK infra. Ultimately saves time having to build one-off ECR publishing steps into existing CI/CD. Very lightweight change.

[ShelbyZ]

@dtbayles it may make more sense to get this into fluent-bit upstream as we are potentially looking to pull parser files from that repository instead of managing local parser conf files here or from the archived fluent-bit-docker-image.

[jric-at-taxbit]

Since getting a public contribution accepted is a higher bar, and since this is a very lightweight change, I think it could make the most sense to merge this now and follow up getting this pushed upstream? My CI/CD pipelines are paying the price while this remains open.

[dtbayles]

@ShelbyZ This config primarily provides a reusable pipeline preset rather than a new parser definition. Since upstream Fluent Bit appears to ship parser primitives rather than opinionated pipeline configurations, I believe this belongs in aws-for-fluent-bit rather than upstream.

[ShelbyZ]

Two easier options come to mind via s3:

1. Provide a configuration via config-file-value in the firelens configuration - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html#firelens-taskdef2
2. Use init tag to provide a configuration - https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md

Both do not require modifying the image when supplying configuration via S3. Only the second option works with Fargate tasks

[jric-at-taxbit]

Thank you for the solution @ShelbyZ .

We are using Fargate, so we are trying out the init tag. I didn't understand your point about it being simpler though. Whereas if this PR were merged, we would use the AWS image out of the box. With the init approach we have deployed the config file to S3, enabled permissions so that it can be accessed from our logging processes (b/c we don't have a public bucket AFAIK), and configured the init mechanism to use the new config. It seems objectively more complicated, but LMK if I'm missing something here. If there were a publicly hosted version of this multi-line parsing config file on S3 somewhere, I could see that closing some of the gap.