Skip to content

out_azure_kusto: support for otel log schema - #10173

Merged
edsiper merged 5 commits into
fluent:masterfrom
dceravigupta:otellogs
May 30, 2025
Merged

out_azure_kusto: support for otel log schema#10173
edsiper merged 5 commits into
fluent:masterfrom
dceravigupta:otellogs

Conversation

@dceravigupta

@dceravigupta dceravigupta commented Apr 4, 2025

Copy link
Copy Markdown
Contributor

As outlined in the issue below, when the out_azure_kusto plugin was used alongside the in_opentelemetry plugin, it previously generated multiple logs for each incoming OTEL log. For instance:

[62] v1_logs: [[-1.000000000, {"schema"=>"otlp", "resource_id"=>0, "scope_id"=>11}], {"resource"=>{"attributes"=>{"telemetry.sdk.name"=>"opentelemetry", "telemetry.sdk.language"=>"dotnet", "telemetry.sdk.version"=>"1.8.1", "service.name"=>"unknown_service:dotnet"}}, "schema_url"=>"", "scope"=>{"name"=>"Microsoft.Extensions.HttpClient.Logging.Internal.HttpLoggingHandler"}}]
[63] v1_logs: [[1741372257.966227535, {"otlp"=>{"observed_timestamp"=>1741372257825108400, "timestamp"=>1741372257825108400, "severity_number"=>17, "severity_text"=>"Error", "attributes"=>{"resolved_host"=>"<some url>.com", "httpMethod"=>"GET", "httpHost"=>"<removed>", "httpPath"=>"<removed>", "duration"=>274, "{OriginalFormat}"=>"{httpMethod} {httpHost}/{httpPath}", "httpStatusCode"=>401}, "trace_flags"=>0}}], {"message"=>"GET <removed>"}]
[64] v1_logs: [[-2.000000000, {}], {}]

Fixes:
#10065

The fix implemented in this PR is take from similar fix done in out_splunk plugin: #10002

Output log after this change:

{
  "resource": {
    "attributes": {
      "telemetry.sdk.name": "opentelemetry",
      "telemetry.sdk.language": "dotnet",
      "telemetry.sdk.version": "1.8.1",
      "service.name": "unknown_service:dotnet",
    }
  },
  "schema_url": "",
  "scope": {
    "name": "Microsoft.Extensions.HttpClient.Logging.Internal.HttpLoggingHandler"
  },
  "otlp": {
    "observed_timestamp": 1743791176558409200,
    "timestamp": 1743791176558409200,
    "severity_number": 17,
    "severity_text": "Error",
    "attributes": {
      "resolved_host": "<removed>",
      "httpMethod": "GET",
      "httpHost": "<removed>",
      "httpPath": "<removed>",
      "duration": 36,
      "{OriginalFormat}": "<removed>",
      "httpStatusCode": 403
    },
    "trace_flags": 0
  }
}

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A ] Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A ] Run local packaging test showing all targets (including any new ones) build.
  • [ N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A ] Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@dceravigupta

Copy link
Copy Markdown
Contributor Author

@edsiper can you please take a look at this PR? It is based on the pr you did for out_splunk plugin: #10002

Signed-off-by: Ravi Gupta <dceravigupta@gmail.com>
@dceravigupta

Copy link
Copy Markdown
Contributor Author

@leonardo-albertovich could you take a look and, if everything looks good, go ahead and merge it?

Comment thread plugins/out_azure_kusto/azure_kusto.c Outdated
Comment thread plugins/out_azure_kusto/azure_kusto.c Outdated
Comment thread plugins/out_azure_kusto/azure_kusto.c
Signed-off-by: Ravi Gupta <dceravigupta@gmail.com>
@dceravigupta

Copy link
Copy Markdown
Contributor Author

@edsiper seems like CI build is failing for some other reason not related to my PR. Is there any way for me to re-trigger it?

Signed-off-by: Ravi Gupta <dceravigupta@gmail.com>
@dceravigupta

Copy link
Copy Markdown
Contributor Author

@patrick-stephens can you help merging this pr since it has already been approved?

@patrick-stephens

Copy link
Copy Markdown
Contributor

I think @edsiper needs to confirm he's happy with the changes.

@edsiper edsiper added this to the Fluent bit v4.0.3 milestone May 30, 2025
@edsiper
edsiper merged commit cb28d0b into fluent:master May 30, 2025
rghouzra pushed a commit to rghouzra/fluent-bit_oracle that referenced this pull request Jun 14, 2025
---------

Signed-off-by: Ravi Gupta <dceravigupta@gmail.com>
nourdouf pushed a commit to seveas/fluent-bit that referenced this pull request Sep 23, 2025
---------

Signed-off-by: Ravi Gupta <dceravigupta@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants