docs: L7 logs restructure and eBPF collector#2724
Conversation
New L7 observability pages for the eBPF-based collector (tigera/calico-private#11749): - configure-bpf.mdx: enable via L7ObservabilityEnabled on FelixConfiguration; comparison with the Istio Waypoint collector - configure-istio-waypoint.mdx: placeholder - datatypes-wip.mdx: redraft of the L7 schema with a Populated by column and new protocol / collector_name / tls_* fields - aggregation.mdx: aggregation key fields with an interactive playground component - sidebar entries for all four pages Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
❌ Deploy Preview for tigera failed. Why did it fail? →Built without sensitive environment variables
|
There was a problem hiding this comment.
Pull request overview
This PR (marked WIP) introduces draft Calico Enterprise documentation for a new eBPF-based L7 observability collector, including a new L7 log aggregation page with an interactive sandbox component, and wires the new pages into the Enterprise sidebar navigation.
Changes:
- Adds new draft L7 documentation pages for eBPF configuration, Istio Waypoint (placeholder), aggregation, and an expanded (WIP) L7 schema reference.
- Introduces a new React component (
L7AggregationSandbox) used to provide an interactive aggregation-key playground in docs. - Updates the Calico Enterprise sidebar to surface the new L7 pages.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| sidebars-calico-enterprise.js | Adds the new L7 doc pages to the Enterprise “L7 logs” sidebar category. |
| calico-enterprise/observability/elastic/l7/configure-bpf.mdx | Draft guide for enabling/viewing eBPF-based L7 logs, including a comparison table and limitations. |
| calico-enterprise/observability/elastic/l7/configure-istio-waypoint.mdx | Placeholder page scaffold for Istio Ambient/Waypoint-based L7 collection. |
| calico-enterprise/observability/elastic/l7/aggregation.mdx | Draft documentation describing FelixConfiguration aggregation fields and embedding an interactive sandbox. |
| calico-enterprise/observability/elastic/l7/datatypes-wip.mdx | Draft schema reference adding eBPF/TLS fields and a “Populated by” column. |
| calico-enterprise/_includes/components/L7AggregationSandbox/index.js | New interactive React component that demonstrates how aggregation settings merge L7 log rows and update count. |
Comments suppressed due to low confidence (4)
calico-enterprise/observability/elastic/l7/configure-bpf.mdx:70
- Same issue as above:
L7LogsFileDirectoryis a FelixConfiguration (resource) field and should link toreference/resources/felixconfig.mdx#l7LogsFileDirectory(note the lowercaselin the anchor) rather than the Felix config file/env-var page.
The eBPF collector writes L7 events as JSON to a log file on each node. By default the file is at `/var/log/calico/l7logs/l7.log`. The directory is controlled by the [L7LogsFileDirectory](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileDirectory) field on `FelixConfiguration`.
calico-enterprise/observability/elastic/l7/aggregation.mdx:25
- The per-field links here target
.../component-resources/node/felix/configuration.mdx#L7Logs..., but the generated anchors for FelixConfiguration fields use the YAML field names (for example#l7LogsFileAggregationSourceInfo) and live onreference/resources/felixconfig.mdx. Please update the doc path and anchor casing for these aggregation field links so they resolve.
| [L7LogsFileAggregationSourceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationSourceInfo) | Source aggregated name, namespace, and type appear on each row. | Requests are merged across sources. |
| [L7LogsFileAggregationDestinationInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationDestinationInfo) | Destination aggregated name, namespace, and type appear on each row. | Requests are merged across destinations. |
| [L7LogsFileAggregationServiceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationServiceInfo) | Destination service name, namespace, and port appear on each row. | Requests are merged across services. |
| [L7LogsFileAggregationHTTPMethod](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationHTTPMethod) | HTTP method appears on each row. | Requests are merged across methods. |
| [L7LogsFileAggregationTrimURL](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationTrimURL) | URL (or a trimmed prefix) appears on each row. Intermediate values trim the query string, then the path, before excluding the URL entirely. | Requests are merged across URLs. |
calico-enterprise/observability/elastic/l7/aggregation.mdx:28
- These links have the same issue as above (wrong target doc + uppercase anchors). Also,
L7LogsFileAggregationTLSSNIdoes not appear in the currentconfig-params.jsondata set, so the anchor will remain broken until the FelixConfig JSON is regenerated to include it.
| [L7LogsFileAggregationResponseCode](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationResponseCode) | Response code appears on each row. | Requests are merged across response codes. |
| [L7LogsFileAggregationHTTPHeaderInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationHTTPHeaderInfo) | User agent and request type appear on each row. | Requests are merged across user agents and types. |
| [L7LogsFileAggregationTLSSNI](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationTLSSNI) | TLS Server Name Indication appears on each TLS row. | TLS flows to the same destination are merged regardless of SNI. |
calico-enterprise/observability/elastic/l7/aggregation.mdx:35
- The second table here also starts rows with
||, which will render as an extra empty column. Use a single leading|for Markdown tables.
| Field | Description |
| --- | --- |
| [L7LogsFileAggregationNumURLPath](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationNumURLPath) | Maximum number of URL path components retained. Default `5`. A negative value disables this truncation. |
| [L7LogsFileAggregationURLCharLimit](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationURLCharLimit) | Maximum URL length in characters; longer URLs are sliced. Default `250`. |
| | Additional infrastructure | Waypoint Envoy pods per namespace/service | None — runs inside Felix | | ||
| | L7 coverage | Only traffic routed through a waypoint | All TCP traffic on the node | | ||
| | Network path overhead | Two extra proxy hops | None | | ||
| | Plaintext access | Decrypted at the waypoint (mTLS terminated) | Captured at the kernel TCP layer, before TLS encryption | |
|
|
||
| ## Configure L7 logs | ||
|
|
||
| Enable the eBPF L7 collector by setting the [L7ObservabilityEnabled](../../../reference/component-resources/node/felix/configuration.mdx#L7ObservabilityEnabled) field to `true` on the default `FelixConfiguration`. The setting is independent of `bpfEnabled` — the collector works with any dataplane. |
| The following `FelixConfiguration` fields control the L7 log aggregation key. For accepted values and defaults, follow the link to the [Felix Configuration reference](../../../reference/component-resources/node/felix/configuration.mdx#l7-logs). | ||
|
|
||
| | Field | Effect when included | Effect when excluded | | ||
| | --- | --- | --- | | ||
| | [L7LogsFileAggregationSourceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationSourceInfo) | Source aggregated name, namespace, and type appear on each row. | Requests are merged across sources. | | ||
| | [L7LogsFileAggregationDestinationInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationDestinationInfo) | Destination aggregated name, namespace, and type appear on each row. | Requests are merged across destinations. | | ||
| | [L7LogsFileAggregationServiceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationServiceInfo) | Destination service name, namespace, and port appear on each row. | Requests are merged across services. | | ||
| | [L7LogsFileAggregationHTTPMethod](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationHTTPMethod) | HTTP method appears on each row. | Requests are merged across methods. | | ||
| | [L7LogsFileAggregationTrimURL](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationTrimURL) | URL (or a trimmed prefix) appears on each row. Intermediate values trim the query string, then the path, before excluding the URL entirely. | Requests are merged across URLs. | | ||
| | [L7LogsFileAggregationResponseCode](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationResponseCode) | Response code appears on each row. | Requests are merged across response codes. | | ||
| | [L7LogsFileAggregationHTTPHeaderInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationHTTPHeaderInfo) | User agent and request type appear on each row. | Requests are merged across user agents and types. | | ||
| | [L7LogsFileAggregationTLSSNI](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationTLSSNI) | TLS Server Name Indication appears on each TLS row. | TLS flows to the same destination are merged regardless of SNI. | | ||
|
|
||
| Two additional fields control how the URL is truncated before it enters the aggregation key, rather than whether the URL participates at all: | ||
|
|
||
| | Field | Description | | ||
| | --- | --- | | ||
| | [L7LogsFileAggregationNumURLPath](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationNumURLPath) | Maximum number of URL path components retained. Default `5`. A negative value disables this truncation. | | ||
| | [L7LogsFileAggregationURLCharLimit](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationURLCharLimit) | Maximum URL length in characters; longer URLs are sliced. Default `250`. | |
| 'observability/elastic/l7/configure-istio-waypoint', | ||
| 'observability/elastic/l7/aggregation', | ||
| 'observability/elastic/l7/datatypes', | ||
| 'observability/elastic/l7/datatypes-wip', |
| | Aspect | Istio Waypoint collector | eBPF L7 collector | | ||
| | --- | --- | --- | | ||
| | Additional infrastructure | Waypoint Envoy pods per namespace/service | None — runs inside Felix | | ||
| | L7 coverage | Only traffic routed through a waypoint | All TCP traffic on the node | | ||
| | Network path overhead | Two extra proxy hops | None | | ||
| | Plaintext access | Decrypted at the waypoint (mTLS terminated) | Captured at the kernel TCP layer, before TLS encryption | | ||
| | Protocol coverage | HTTP/1, HTTP/2, gRPC | HTTP/1.0, HTTP/1.1, TLS handshake metadata | |
| | Field | Effect when included | Effect when excluded | | ||
| | --- | --- | --- | | ||
| | [L7LogsFileAggregationSourceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationSourceInfo) | Source aggregated name, namespace, and type appear on each row. | Requests are merged across sources. | | ||
| | [L7LogsFileAggregationDestinationInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationDestinationInfo) | Destination aggregated name, namespace, and type appear on each row. | Requests are merged across destinations. | | ||
| | [L7LogsFileAggregationServiceInfo](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationServiceInfo) | Destination service name, namespace, and port appear on each row. | Requests are merged across services. | | ||
| | [L7LogsFileAggregationHTTPMethod](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationHTTPMethod) | HTTP method appears on each row. | Requests are merged across methods. | | ||
| | [L7LogsFileAggregationTrimURL](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileAggregationTrimURL) | URL (or a trimmed prefix) appears on each row. Intermediate values trim the query string, then the path, before excluding the URL entirely. | Requests are merged across URLs. | |
| | Name | Datatype | Populated by | Description | | ||
| | ------------------------ | -------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | `host` | keyword | All | Name of the node that collected the L7 log entry. | | ||
| | `start_time` | date | All | Start time of log collection in Unix timestamp format. | | ||
| | `end_time` | date | All | End time of log collection in Unix timestamp format. | |
Aggregation sandbox redesign: - Group rows by aggregation bucket; expand/collapse to reveal merged events - Two-tone highlight (green for added/count up, red for removed/count down) - Color legend above the table; intro/footer condensed - More diverse sample data spanning cart, payments, and TLS scenarios Pages: - Promote the WIP data types page to datatypes.mdx; move the original to datatypes-legacy.mdx - Add collector_type and protocol_version fields - Sidebar reordered with "+ " prefix on the new pages and shorter labels (Configure with eBPF, Configure with Istio Waypoint, Aggregation, Data types) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- dataplane / dataplanes → data plane / data planes - waypoint → Waypoint (Calico term rule) - Plaintext / plaintext → Plain text / plain text - Rephrase HTTP pipelining bullet to drop dictionary-rejected words Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add Overview page (overview.mdx) with big picture, value, collectors, and a comparison table — moves the shared framing out of per-collector pages - Rename configure-bpf.mdx → enable-ebpf.mdx, refocus on enablement steps - Rename configure-istio-waypoint.mdx → enable-waypoint.mdx (placeholder) - Drop datatypes-legacy.mdx (no inbound references) - Sidebar labels: Overview, Enable eBPF/Waypoint/Envoy collector, Data types, Aggregation - Datatypes page: HTTP-only / TLS-only fields tagged inline in the description - Repoint external "L7 logs" links (dashboards, get-started-cem, kibana) to overview.mdx Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Istio Waypoint collector will land in a separate PR. For now: - enable-waypoint.mdx: add draft: true; Docusaurus excludes it from production builds and auto-filters the sidebar entry - overview.mdx, datatypes.mdx: remove inline Waypoint references; leave TODO notes in single-line JSX comments so the next-PR author can find the spots - overview.mdx: drop the Istio Waypoint column from the comparison table for this PR - datatypes.mdx: change "Populated by" from "eBPF, Waypoint" to "eBPF" Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
||
| # L7 logs | ||
|
|
||
| ## Big picture |
|
|
||
| L7 logs complement L3/4 flow logs, which expose only the source/destination pair without request-level detail. | ||
|
|
||
| ## Collectors |
There was a problem hiding this comment.
Might be worth saying what a collector actually is.
|
|
||
| ## Collectors | ||
|
|
||
| $[prodname] supports multiple L7 collectors. They all feed the same aggregation and reporting pipeline; each entry on the log file is tagged with `collector_type` and `collector_name` so consumers can filter or join sources as needed. |
There was a problem hiding this comment.
It's not clear what the distinction is between type and name. Are you saying we can have an epbf type called A and another ebpf type called B?
|
|
||
| Multiple collectors can coexist in the same cluster, with each L7 log entry tagged by `collector_type` and `collector_name` so consumers can filter or join sources as needed. | ||
|
|
||
| :::note |
There was a problem hiding this comment.
This should not be buried in a note. Probably best as a prerequisite to each 'enable' procedure.
For clarity, consider working through an example. "For example, if you want to maintain a 7-day retention period for a three-node cluster, you need to add at least 21 GB to your existing log storage."
| @@ -0,0 +1,42 @@ | |||
| --- | |||
There was a problem hiding this comment.
The file name is misleading. Perhaps enable-ebpf-collector or simlar.
| To disable the collector, set the field back to `false`. Felix tears down its BPF programs and cleans up pinned maps; no node reboot is required. | ||
|
|
||
| ```bash | ||
| kubectl patch felixconfiguration default --type=merge -p '{"spec":{"l7ObservabilityEnabled":false}}' | ||
| ``` |
There was a problem hiding this comment.
Is there an expectation that it wouldn't be cleaned up? I'm wondering if we can remove all of this.
|
|
||
| ## Enable the collector | ||
|
|
||
| Enable the eBPF L7 collector by setting the [L7ObservabilityEnabled](../../../reference/component-resources/node/felix/configuration.mdx#L7ObservabilityEnabled) field to `true` on the default `FelixConfiguration`. The setting is independent of `bpfEnabled` — the collector works with any data plane. |
There was a problem hiding this comment.
For a single step procedure, use an unordered list:
| Enable the eBPF L7 collector by setting the [L7ObservabilityEnabled](../../../reference/component-resources/node/felix/configuration.mdx#L7ObservabilityEnabled) field to `true` on the default `FelixConfiguration`. The setting is independent of `bpfEnabled` — the collector works with any data plane. | |
| * Enable the eBPF L7 collector by setting the [L7ObservabilityEnabled](../../../reference/component-resources/node/felix/configuration.mdx#L7ObservabilityEnabled) field to `true` on the default `FelixConfiguration`. The setting is independent of `bpfEnabled` — the collector works with any data plane. |
| ```bash | ||
| tail -f /var/log/calico/l7logs/l7.log | ||
| ``` | ||
|
|
There was a problem hiding this comment.
| ## Additional information | |
| * L7 log data types |
| ## View L7 logs | ||
|
|
||
| The eBPF collector writes L7 events as JSON to a log file on each node. By default the file is at `/var/log/calico/l7logs/l7.log`. The directory is controlled by the [L7LogsFileDirectory](../../../reference/component-resources/node/felix/configuration.mdx#L7LogsFileDirectory) field on `FelixConfiguration`. | ||
|
|
||
| To tail the log on a node: | ||
|
|
||
| ```bash | ||
| tail -f /var/log/calico/l7logs/l7.log | ||
| ``` |
There was a problem hiding this comment.
I think this can all move to a single line after the enable step.
This could also act as a verification step, come to think of it.
- To verify that L7 logs are being collected, look for output in the default log file location on each node:
|
|
||
| import L7AggregationKeyDemo from '../../../_includes/components/L7AggregationKeyDemo'; | ||
|
|
||
| # L7 log aggregation |
There was a problem hiding this comment.
I think the new component is handy for illustrating how aggregation works. At this stage, I have two thoughts.
-
To what extent is this L7-specific? I get that the fields may be different, but it seems that the main idea here is to help people understand aggregation generally, not L7 aggregation. If that's the case, perhaps it would be better to use a tool like this in a general doc about aggregation?
-
For me, it would be useful to see all parts of this dynamic in action. 1) a full list of all the communications 2) the configuration (as a code block, rendered dynamically according to the checkboxes, and 3) the actual flow logs (what the log would actually look like, not the isolated terms as cells in a table.
Summary
Restructures the L7 logs docs section to introduce the new eBPF-based collector alongside the existing Envoy collector, with a shared overview and per-collector enable pages.
The Istio Waypoint collector will land in a follow-up PR — its placeholder page is set to
draft: trueso it's visible during local development but excluded from production builds.L7 logs section, after this PR
overview.mdxenable-ebpf.mdxL7ObservabilityEnabledonFelixConfiguration. Limitations: kernel 5.17+, HTTP/1.0–1.1 only, TLS metadata only.enable-waypoint.mdxdraft: trueso it's hidden in production builds. Content lands in the Waypoint PR.configure.mdxdatatypes.mdxprotocol,protocol_version,collector_type,collector_name, andtls_*fields. HTTP-only / TLS-only fields tagged inline (e.g._HTTP only._ URL that the request was made against.).aggregation.mdxL7LogsFileAggregation*fields (including the newL7LogsFileAggregationTLSSNI) and includes an interactive aggregation playground — a React component (_includes/components/L7AggregationSandbox) where readers toggle aggregator checkboxes and watch sample events regroup with two-tone highlight (green = added / count up, red = removed / count down) and per-group expand/collapse.Other changes
dashboards.mdx,get-started-cem.mdx,kibana.mdx: external "L7 logs" links repointed fromconfigure.mdx(Envoy-only) to the newoverview.mdx.alp.mdx: left pointing atconfigure.mdx#kibanaper maintainer note (page will be deprecated).data plane,Waypoint,plain text).Open items / known gaps
L7ObservabilityEnabledandL7LogsFileAggregationTLSSNIwon't render in the Felix Configuration tables until_includes/components/FelixConfig/config-params.jsonis regenerated from upstream calico-private. Field-level anchors (#L7ObservabilityEnabled,#L7LogsFileAggregationTLSSNI) will activate at that time.nexttree (calico-enterprise/) only.Test plan
yarn start-next) compiles and renders all six pages..docusaurusroute data.configure-bpf.mdx,configure-istio-waypoint.mdx,datatypes-wip.mdx,datatypes-legacy.mdx).yarn build-next) excludesenable-waypoint.mdx— to be verified by reviewers.🤖 Generated with Claude Code