From 6f2e5e86e093f0e78d21af5ddf2eff32de92d329 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 31 Jul 2026 17:28:18 +0200 Subject: [PATCH] docs: pipeline: outputs: gcs: add Google Cloud Storage output plugin page Document the out_gcs plugin introduced in Fluent Bit 5.1, including all configuration parameters, service account authentication, object key format, and local buffering behavior. Add the page to SUMMARY.md and allow "Google Cloud Storage" in the Vale heading capitalization exceptions. Note: covers missing doc PR for a code change. Signed-off-by: Eric D. Schabell --- SUMMARY.md | 1 + pipeline/outputs/gcs.md | 104 +++++++++++++++++++++++++++++ vale-styles/FluentBit/Headings.yml | 1 + 3 files changed, 106 insertions(+) create mode 100644 pipeline/outputs/gcs.md diff --git a/SUMMARY.md b/SUMMARY.md index cafaf1057..64eef78bb 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -192,6 +192,7 @@ * [Graylog Extended Log Format (GELF)](pipeline/outputs/gelf.md) * [Google Chronicle](pipeline/outputs/chronicle.md) * [Google Cloud BigQuery](pipeline/outputs/bigquery.md) + * [Google Cloud Storage](pipeline/outputs/gcs.md) * [HTTP](pipeline/outputs/http.md) * [InfluxDB](pipeline/outputs/influxdb.md) * [Kafka Producer](pipeline/outputs/kafka.md) diff --git a/pipeline/outputs/gcs.md b/pipeline/outputs/gcs.md new file mode 100644 index 000000000..deb53b577 --- /dev/null +++ b/pipeline/outputs/gcs.md @@ -0,0 +1,104 @@ +--- +description: Send logs to Google Cloud Storage +--- + +# Google Cloud Storage + +{% hint style="info" %} +**Supported event types:** `logs` +{% endhint %} + +The _Google Cloud Storage_ (`gcs`) output plugin lets you upload the records received through the input plugin to a [Google Cloud Storage](https://cloud.google.com/storage) bucket. Records are buffered locally and uploaded as objects when the configured upload timeout elapses. + +This plugin is available in Fluent Bit version 5.1 and greater. It's included in official builds. If you compile Fluent Bit from source, the `FLB_OUT_GCS` build option is enabled by default. + +## Google Cloud configuration + +Before using the plugin, you must: + +1. Create a [Google Cloud service account](https://docs.cloud.google.com/iam/docs/service-accounts-create) for Fluent Bit and [create a JSON key](https://docs.cloud.google.com/iam/docs/keys-create-delete) for it. +1. [Create the bucket](https://docs.cloud.google.com/storage/docs/creating-buckets) that receives your data. Fluent Bit doesn't create buckets. +1. Grant the service account permission to write objects to that bucket. The plugin requests the `https://www.googleapis.com/auth/devstorage.read_write` `OAuth 2.0` scope. +1. Make the credentials available to Fluent Bit, either by setting the `google_service_credentials` option to the absolute path of the JSON key file, or by setting the `GOOGLE_SERVICE_CREDENTIALS` environment variable to that path. + +The plugin uploads objects to `storage.googleapis.com` over TLS using the Cloud Storage JSON API. + +## Configuration parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `bucket` | Name of the Cloud Storage bucket that receives the objects. | _none_ | +| `canned_acl` | [Predefined ACL](https://docs.cloud.google.com/storage/docs/access-control/lists#predefined-acl) applied to uploaded objects. Accepted values: `authenticated-read`, `bucket-owner-full-control`, `bucket-owner-read`, `private`, `project-private`, and `public-read`. The camel case forms used by the Cloud Storage API, such as `publicRead`, are also accepted. | _none_ | +| `compression` | Compression applied to uploaded objects. Accepted values: `none`, `gzip`. | `none` | +| `content_type` | Value of the `Content-Type` metadata set on uploaded objects. | `application/json` | +| `gcs_key_format` | Format string for object names in the bucket. See [Object key format](#object-key-format). | `fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | +| `gcs_key_format_tag_delimiters` | Characters used to split the tag into the parts referenced by `$TAG[n]` in `gcs_key_format`. | `.` | +| `google_service_credentials` | Absolute path to a Google Cloud service account credentials JSON file. | Value of the environment variable `$GOOGLE_SERVICE_CREDENTIALS`. | +| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This option prevents shuffling by using queue logic for uploads. | `false` | +| `send_content_md5` | Send the `Content-MD5` header with uploads so that Cloud Storage verifies object integrity. | `false` | +| `static_file_path` | Disables the behavior where a random suffix appends to the object name when `$UUID` isn't provided in `gcs_key_format`. | `false` | +| `store_chunk_limit` | Maximum number of buffered chunks kept in `store_dir`. Set to `0` for unlimited. | `0` | +| `store_dir` | Directory used to locally buffer data before uploading it. | `/tmp/fluent-bit/gcs` | +| `store_dir_limit_size` | Limits the amount of data buffered in `store_dir` to limit disk usage. When the limit is reached, data is discarded. Set to `0` for unlimited. | `0` | +| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads the buffered data and starts a new object. Set to `60m` to upload a new object every hour. | `10m` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | + +## Object key format + +The `gcs_key_format` option supports the same formatters as the [Amazon S3](s3.md#s3-key-format-and-tag-delimiters) output plugin: + +- `$TAG`: The full tag. +- `$TAG[n]`: The nth part of the tag, with the index starting at zero. Tag parts are separated using the characters set in `gcs_key_format_tag_delimiters`. +- `$UUID`: A random string. +- `$INDEX`: An integer that increments with each upload. The value is stored in `store_dir` so it survives a restart. +- [strftime](https://man7.org/linux/man-pages/man3/strftime.3.html) formatters such as `%Y`, `%m`, and `%d`. The time used is the timestamp of the first record in the object. + +When `gcs_key_format` doesn't contain `$UUID` and `static_file_path` is `false`, Fluent Bit appends a random suffix to each object name so that concurrent uploads don't overwrite each other. Set `static_file_path` to `true` to keep the object name exactly as the format string produces it. + +## Buffering + +This plugin buffers records as files in `store_dir` and uploads them when `upload_timeout` elapses, so it requires a writeable filesystem. Because the plugin has its own buffering system, the `storage.total_limit_size` parameter isn't meaningful. Use `store_dir_limit_size` and `store_chunk_limit` to limit disk usage instead. + +## Get started + +The following configuration buffers CPU metrics and uploads a gzip-compressed object to the `my-logs` bucket every five minutes: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: cpu + tag: cpu + + outputs: + - name: gcs + match: '*' + bucket: my-logs + google_service_credentials: /path/to/credentials.json + gcs_key_format: /fluent-bit/$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz + compression: gzip + upload_timeout: 5m +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + Name gcs + Match * + Bucket my-logs + Google_Service_Credentials /path/to/credentials.json + Gcs_Key_Format /fluent-bit/$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz + Compression gzip + Upload_Timeout 5m +``` + +{% endtab %} +{% endtabs %} diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index bcbc230eb..df3393364 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -76,6 +76,7 @@ exceptions: - Google Cloud - Google Cloud BigQuery - Google Cloud Platform + - Google Cloud Storage - GPG - GPU - Grafana