new(nok8s): json schema validation for tracingpolicies by FedeDP · Pull Request #5322 · cilium/tetragon

FedeDP · 2026-07-22T14:11:40Z

Description

Since in nok8s builds we can't rely upon kubernetes validation, we were lacking any sort of validation for the tracing policy yaml object.
This patch introduces a validation based upon the json schema of the crds that were previously generated and used only in the kubeconform make target.

New dependency: github.com/santhosh-tekuri/jsonschema/v6; only used in nok8s build.
Size change:

main: 49532
HEAD: 50752

~800K.

Most of the line changes come from:

the new pkg/tracingpolicy/schemas/ json schemas
the new vendored dependency

Changelog

new(nok8s): json schema + CEL validation for tracingpolicies

FedeDP · 2026-07-22T14:38:34Z

Run static checks / build-nok8s (pull_request)Failing after 1m

So, this is failing because in the tetragon-nok8s target i clean up the pkg/tracingpolicy/schemas/ folder, but then tetra-nok8s fails to build because go:embed can't find the file.
2 possibilities:

we keep the install/kubernetes tetragon/schemas/ folder alive (and we re-generate it in generate target); then both tetragon-nok8s and tetra-nok8s will rely upon a pkg/tracingpolicy/schemas/ target that just cp -r install/kubernetes/tetragon/schemas/ pkg/tracingpolicy/schemas/; finally, we gitignore pkg/tracingpolicy/schemas/.
both tetra-nok8s and tetragon-nok8s will depend upon pkg/tracingpolicy/schemas/ target that generates json schemas from crds, and both targets clean pkg/tracingpolicy/schemas/ folder up once they are done

For now, i implement 1, but am open to switch to 2 if requested.

EDIT: in the end, i chose to follow what we already do for crds: make generate will re-generate the files directly in the desired folder (pkg/tracingpolicy/schemas/) and they are then versioned. This allows go test -tags nok8s or just go build -tags nok8s to work like a charm since the required files are correctly there.

netlify · 2026-07-23T07:15:14Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`616ed9e`
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/6a6b2f200be0660008efebf0
😎 Deploy Preview	https://deploy-preview-5322--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

FedeDP · 2026-07-23T10:21:47Z

Example output from feeding a wrong syntax policy:

level=error msg="Failed to execute tetragon" error="failed to validate TracingPolicy: jsonschema validation failed with 'file:///tracingpolicy#'\n- at '/spec': additional properties 'lsmhoooks' not allowed"

In main, same policy:

level=info msg="Added TracingPolicy with success" TracingPolicy=/home/fdipierr/pol-lsm-wrong.yaml metadata.namespace="" metadata.name=protection-policy-cve-2024-21626

Policy file (note lsmhoooks :D )

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "protection-policy-cve-2024-21626"
spec:
  lsmhoooks:
  - hook: "bprm_check_security"
    args:
    - index: 0 # struct linux_binprm *bprm
      type: "path"
      resolve: "mm.owner.fs.pwd"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/sys/fs/cgroup"
      matchActions:
      - action: Post
    message: cve-2024-21626

andrewstrohman · 2026-07-23T20:12:03Z

I confirmed that this came from the vendor's repo, and it's not something that we created by accident. (I guess our CI would catch it, if it came from us).

andrewstrohman · 2026-07-23T21:20:35Z

It might be worth doing an explicit validation for drift between the JSON schema and the CRDs in CI.

FedeDP · 2026-07-24T07:49:01Z

It might be worth doing an explicit validation for drift between the JSON schema and the CRDs in CI.

This should never happen since make generate takes care of rebuilding both.

kkourt

Note: x-kubernetes-validations cel expressions are not supported; we would need to extend the json schema validation and manually support them; it's not super easy to do, but it would allow us to cleanup some code.

Is this something that we plan to do?

Because if we can't do it, then I think the best option would be to do just duplicate the validation logic in go since we are using CEL expressions extensively in our spec.

FedeDP · 2026-07-24T12:28:25Z

Is this something that we plan to do?

I think we might want to commit to do that in the future; unfortunately, it is a bit complex and would take quite a bit of time.

Because if we can't do it, then I think the best option would be to do just duplicate the validation logic in go since we are using CEL expressions extensively in our spec.

I am not sure, it quickly becomes risky to rely upon the double validation (sooner or later we would forget to add it) and also why pay the perf penalty (for validation) in k8s builds when k8s already validates for us?

mtardy · 2026-07-27T10:36:34Z

 	github.com/prometheus/client_model v0.6.2
 	github.com/prometheus/common v0.70.0
 	github.com/prometheus/procfs v0.21.1
+	github.com/santhosh-tekuri/jsonschema/v6 v6.0.2


how much do we trust this new dep?

Seems widely used and updated. But can't really say :/

I'm also concerned about that. It seems to be widely used, but most of the projects that use it in the dependency graph seem to have 0 stars/forks: https://github.com/santhosh-tekuri/jsonschema/network/dependents

Some notable exceptions:

https://github.com/apache/doris

https://github.com/redpanda-data/console

https://github.com/elastic/go-elasticsearch

Would https://github.com/google/jsonschema-go fit our purposes?

One thought that came to mind is to do the validation after dropping privileges/capabilities. This is not easy, and AFAICT the best way to achieve it would be by executing a binary (it can even be tetragon).

Would https://github.com/google/jsonschema-go fit our purposes?

After a quick look, i don't think it supports registering custom vocabulary and validate in a single step like we do with jsonschema (very elegantly).
And i'd really love to avoid the huge "take all x-kubernetes-validations keyword values in the json schema, then for each build the context (ie: which strucutre they are attached), and finally manually execute all cel programs with their own context after the json schema validation".
It becomes a real mess :) And it was the thing i thought we had to do (and that's why i said "it could be done but it's really hard) before discovering about the custom vocabulary feat of https://github.com/santhosh-tekuri/jsonschema.

IMO the 3 "notable exceptions" you provided are pretty big.

This is not easy, and AFAICT the best way to achieve it would be by executing a binary (it can even be tetragon).

😱

mtardy · 2026-07-27T10:41:01Z

-kubeconform: ## Validate Helm chart using kubeconform
-kubeconform:
-	mkdir -p $(JSON_SCHEMAS)/ $(SCHEMAS_CACHE)/
-	$(PYTHON) /bin/bash -c "pip install pyyaml && python /code/install/kubernetes/openapi2jsonschema.py /code/$(CRDS_RELATIVE_DIR)/*"
-	mv $(ROOT_DIR)/install/kubernetes/*-cilium.io.json $(JSON_SCHEMAS)/
+kubeconform: $(TETRAGON_CHART)/schemas ## Validate Helm chart using kubeconform


could you explain a little bit in the git commit log the why of these changes? I'm not super familiar on why we were doing all this before

I'm not super familiar on why we were doing all this before

I'm not either :) i just split the schemas generation code from being part of the kubeconform target to its own.
(I just restored the SCHEMAS_CACHE thing because it's still useful).
I expanded the commit message to be more precise :)

Also, note that i tested make kubeconform and it works fine.

FedeDP · 2026-07-27T15:37:19Z

Moving to draft to fix CEL stuff.

FedeDP · 2026-07-28T07:25:45Z

Fixed CEL stuff.

We already generated schemas as part of the `kubeconform` make target; move the schemas generation to their own target, and let `kubeconform` depend upon the new target. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Also, run `make generate` to generate them. Next commit will use the schemas through `go:embed` to validate policies. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Prior to this change, feeding a wrong-syntax policy spec to nok8s build would've accepted it just fine. After this change, syntax errors are caught and correctly thrown. Note that `x-kubernetes-validations` CEL expressions validation will come in the follow up commit. New dependency: github.com/santhosh-tekuri/jsonschema/v6. Impact of the change: * Binary size: * main: `49532K` * HEAD: `50752K` * Running memory: * main: `70252K` * HEAD: `73168` Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

mtardy

thanks I think overall lgtm!

Never a big fan of adding new deps to the projects, especially ones that belong to one individual author but I if the feature is worth it!

We register a `vocabulary` for the `jsonschema.Compiler` that basically maps each `"x-kubernetes-validations"` entry to a function that loads the CEL `rule` and `message`, compiles the rule and store the resulting CEL program. The resulting `celSchema` will then validate all stored CEL programs during the jsonschema validation. `k8s_xvalidation_test` tests are not `k8s` builds specific anymore. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Before this change, we would've tried to evaluate `!return` even when the rule did not include any `return` statement, leading to a warning in the new nok8s `celSchema.Validate()`. Consequently, bumped `CustomResourceDefinitionSchemaVersion`. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

The case is already covered by `tracingpolicy/k8s_xvalidation_test` now. Also, cleaned up unused `build.K8sEnabled()` helper. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2026-07-31T09:23:04Z

@mtardy fixed your suggestions, PTAL 🙏

FedeDP force-pushed the new/nok8s-jsonschema-validation branch 4 times, most recently from 42edb96 to 964380b Compare July 22, 2026 14:22

FedeDP added release-note/minor This PR introduces a minor user-visible change kind/feature This introduces a new functionality labels Jul 22, 2026

FedeDP force-pushed the new/nok8s-jsonschema-validation branch 3 times, most recently from 1554289 to 24206d3 Compare July 23, 2026 07:14

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 24206d3 to 0d2788f Compare July 23, 2026 07:21

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 0d2788f to 76dae24 Compare July 23, 2026 10:23

FedeDP marked this pull request as ready for review July 23, 2026 12:10

FedeDP requested a review from a team as a code owner July 23, 2026 12:10

FedeDP requested review from andrewstrohman and kkourt July 23, 2026 12:10

andrewstrohman reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/tracingpolicy/nok8s.go Outdated

andrewstrohman reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/tracingpolicy/nok8s.go Outdated

andrewstrohman approved these changes Jul 23, 2026

View reviewed changes

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 76dae24 to 0834ad5 Compare July 24, 2026 07:50

FedeDP requested a review from mtardy July 24, 2026 07:50

kkourt reviewed Jul 24, 2026

View reviewed changes

Comment thread pkg/tracingpolicy/nok8s.go

mtardy reviewed Jul 27, 2026

View reviewed changes

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 0834ad5 to ec01176 Compare July 27, 2026 13:46

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from a3f1776 to f76b9ad Compare July 27, 2026 15:35

FedeDP marked this pull request as draft July 27, 2026 15:37

FedeDP force-pushed the new/nok8s-jsonschema-validation branch 3 times, most recently from 6033959 to a6b9a67 Compare July 27, 2026 15:54

FedeDP marked this pull request as ready for review July 27, 2026 16:56

FedeDP requested a review from a team as a code owner July 28, 2026 06:59

FedeDP requested a review from olsajiri July 28, 2026 06:59

FedeDP requested review from andrewstrohman, kkourt and mtardy July 28, 2026 07:25

FedeDP force-pushed the new/nok8s-jsonschema-validation branch 2 times, most recently from 50a4218 to 9d1c7d1 Compare July 28, 2026 07:58

andrewstrohman reviewed Jul 28, 2026

View reviewed changes

Comment thread install/kubernetes/Makefile Outdated

andrewstrohman approved these changes Jul 28, 2026

View reviewed changes

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 9d1c7d1 to 262eca7 Compare July 29, 2026 07:16

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 262eca7 to 616ed9e Compare July 30, 2026 11:01

FedeDP added 2 commits July 30, 2026 13:26

new(make): generate target to also generate tetragon JSON schemas.

ac54d9f

Also, run `make generate` to generate them. Next commit will use the schemas through `go:embed` to validate policies. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 616ed9e to 6681e99 Compare July 30, 2026 11:28

mtardy approved these changes Jul 31, 2026

View reviewed changes

Comment thread pkg/tracingpolicy/nok8s.go Outdated

Comment thread pkg/tracingpolicy/k8s_xvalidation_test.go

FedeDP added 4 commits July 31, 2026 11:20

cleanup(pkg): drop redundant TestKprobeValidationMissingReturnArg.

cdde986

The case is already covered by `tracingpolicy/k8s_xvalidation_test` now. Also, cleaned up unused `build.K8sEnabled()` helper. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore: TestFooValidationReturnWithoutArg to check for error string.

237b5ac

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP force-pushed the new/nok8s-jsonschema-validation branch from 6681e99 to 237b5ac Compare July 31, 2026 09:22

Uh oh!

Conversation

FedeDP commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog

Uh oh!

FedeDP commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for tetragon ready!

Uh oh!

FedeDP commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrewstrohman commented Jul 23, 2026

Uh oh!

FedeDP commented Jul 24, 2026

Uh oh!

kkourt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FedeDP commented Jul 24, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FedeDP Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FedeDP commented Jul 27, 2026

Uh oh!

FedeDP commented Jul 28, 2026

Uh oh!

Uh oh!

mtardy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FedeDP commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FedeDP commented Jul 22, 2026 •

edited

Loading

FedeDP commented Jul 22, 2026 •

edited

Loading

netlify Bot commented Jul 23, 2026 •

edited

Loading

FedeDP Jul 31, 2026 •

edited

Loading