From aedcd4830d7d78a3153f383267f3ad2ff398aad7 Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Fri, 2 Oct 2026 10:07:05 -0700 Subject: [PATCH 1/6] feat: add per-tool CEL expression rail --- .../configure-rails/guardrail-catalog/cel.mdx | 145 ++++++++++ docs/index.yml | 3 + docs/reference/rail-engine-support.mdx | 10 +- docs/telemetry.mdx | 2 +- nemoguardrails/library/cel/__init__.py | 14 + nemoguardrails/library/cel/actions.py | 129 +++++++++ nemoguardrails/library/cel/rail.py | 91 ++++++ nemoguardrails/library/cel/rail_config.py | 95 ++++++ pyproject.toml | 4 + schemas/rails_config.snapshot.json | 49 ++++ .../test_per_tool_cel_rails_iorails.py | 207 ++++++++++++++ tests/rails/llm/test_config.py | 2 + tests/test_cel_actions.py | 225 +++++++++++++++ tests/test_runtime_flow_gate_equivalence.py | 24 ++ uv.lock | 270 +++++++++++------- 15 files changed, 1159 insertions(+), 111 deletions(-) create mode 100644 docs/configure-rails/guardrail-catalog/cel.mdx create mode 100644 nemoguardrails/library/cel/__init__.py create mode 100644 nemoguardrails/library/cel/actions.py create mode 100644 nemoguardrails/library/cel/rail.py create mode 100644 nemoguardrails/library/cel/rail_config.py create mode 100644 tests/guardrails/test_per_tool_cel_rails_iorails.py create mode 100644 tests/test_cel_actions.py diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx new file mode 100644 index 0000000000..07e1610777 --- /dev/null +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -0,0 +1,145 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: "Common Expression Language (CEL)" +sidebar-title: "Common Expression Language (CEL)" +description: "Reference for the per-tool rail that blocks tool calls or tool results matching Common Expression Language (CEL) expressions you write per tool." +content: + type: "reference" +--- + +The CEL rail checks a tool call's arguments, or a tool result's content, against [Common Expression Language (CEL)](https://cel.dev) expressions that you write per tool. CEL is a small, side-effect-free expression language used for policy rules in systems such as Kubernetes admission policies and Envoy. Each expression evaluates to `true` when the tool is being used in a way you want to block. + +Unlike [Regex Detection](/configure-guardrails/guardrail-catalog/community/regex), which matches text patterns, CEL expressions address the structured fields of a tool call directly, so one rule can combine several arguments. Unlike [Tool Safety Check](/configure-guardrails/guardrail-catalog/tool-safety-check), the check is deterministic and calls no model. It is scoped per tool through the same `rails.tool_output.per_tool` and `rails.tool_input.per_tool` dispatch mechanism as both. + +## Installation + +The CEL rail requires the optional `cel` extra, which installs the [`common-expression-language`](https://pypi.org/project/common-expression-language/) package and is available on Python 3.11 and later: + +```bash +pip install "nemoguardrails[cel]" +``` + +If the extra is not installed, a configuration that uses the CEL rail fails to load with an error naming the extra. + +## Configure the Rail + +To use the CEL rail, you should: + +1. Write the expressions for each tool in the `rails.config.cel` section of the `config.yml` file, under `tool_output` to check tool calls and `tool_input` to check tool results: + + ```yaml + rails: + config: + cel: + tool_output: + run_shell: + expressions: + - 'args.command.matches("(?i)\\brm\\s+-[a-z]*r[a-z]*f")' + - 'has(args.cwd) && args.cwd.startsWith("/etc")' + tool_input: + run_shell: + expressions: + - 'type(result) == string && result.contains("BEGIN RSA PRIVATE KEY")' + ``` + +2. Scope the check to the tool in the `tool_output` and `tool_input` rails sections of the `config.yml` file: + + ```yaml + rails: + tool_output: + per_tool: + run_shell: + - "cel check tool output" + tool_input: + per_tool: + run_shell: + - "cel check tool input" + ``` + + The `cel check tool output` flow evaluates the tool's `tool_output` expressions against each call to the tool, and the `cel check tool input` flow evaluates its `tool_input` expressions against each result the tool returns. A tool with no expressions configured for a direction is allowed. + +3. Declare the tool, either in the main model's `parameters.tools` or per request in `options.llm_params.tools`. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. + +The `cel check tool output` and `cel check tool input` rails execute the [`evaluate_tool_output_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) and [`evaluate_tool_input_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) actions respectively. + +## Expression Variables + +Each expression can read the following variables: + +| Flow | Variable | Value | +|---|---|---| +| `cel check tool output` | `args` | The tool call's arguments, as a map | +| | `tool` | The tool's name | +| `cel check tool input` | `result` | The tool result's content, exactly as the tool returned it: a string, a list of content blocks, or `null` | +| | `args` | The arguments of the tool call that the result answers | +| | `tool` | The tool's name | + +Nested arguments are available at any depth, for example `args.options.recursive` or `args.files.exists(f, f.path.startsWith("/etc"))`. + +A tool result that contains JSON text is still a string, so check it with string functions such as `contains()` and `matches()`. + +## How Expressions Are Evaluated + +The rail evaluates every expression configured for the tool: + +- If any expression evaluates to `true`, the rail blocks the tool call or result. The block lists the expressions that matched in its `matched_expressions` metadata. The metadata never includes the checked arguments or content. +- If no expression evaluates to `true`, the rail allows the tool call or result. +- If an expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail logs a warning and continues with the remaining expressions. A match from another expression still blocks. If no expression matched, the rail fails closed: the tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed and why. + +## Write Expressions + +Expressions use [CEL syntax](https://github.com/google/cel-spec/blob/master/doc/langdef.md), including comparisons, `&&` and `||`, the `in` operator, the `exists()` and `all()` macros, and string functions such as `contains()`, `startsWith()`, `endsWith()`, and `matches()`, which uses RE2 regular expression syntax. The rail also provides the cel-go extension functions, such as `lowerAscii()`, `split()`, and `lists.range()`. + +When you write expressions, keep the following in mind: + +- Guard optional arguments with `has()`, for example `has(args.cwd) && args.cwd.startsWith("/etc")`. Without the guard, a call that omits the argument fails to evaluate. +- If a tool can return either a string or a list of content blocks, check the type first, for example `type(result) == string && result.contains("password")`. +- Each expression must evaluate to a boolean. An expression such as `args.command` that returns a string fails to evaluate. + +The rail checks each expression's syntax when the configuration loads, so a syntax error fails the load. A misspelled variable, field, or function, or a type mismatch, is detected only when the expression is evaluated, so test your expressions against representative tool calls and results before deploying them. + +## Complete Example + +The following configuration blocks recursive forced deletes and writes under `/etc` in a `run_shell` tool, and blocks any result that contains a private key: + +```yaml +models: + - type: main + engine: nim + model: meta/llama-3.3-70b-instruct + +rails: + config: + cel: + tool_output: + run_shell: + expressions: + - 'args.command.matches("(?i)\\brm\\s+-[a-z]*r[a-z]*f")' + - 'has(args.cwd) && args.cwd.startsWith("/etc")' + tool_input: + run_shell: + expressions: + - 'type(result) == string && result.contains("BEGIN RSA PRIVATE KEY")' + tool_output: + per_tool: + run_shell: + - "cel check tool output" + tool_input: + per_tool: + run_shell: + - "cel check tool input" +``` + +## Schema Validation + +`cel check tool output` runs behind `@tool_output_validation`: a call to a tool the model has not declared, or whose arguments violate the tool's declared JSON schema, is blocked before any expression is evaluated. `cel check tool input` has no such gate, matching regex detection's asymmetry between the two directions. + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `cel check tool output` | ✗ | ✓ | +| `cel check tool input` | ✗ | ✓ | + +Per-tool checks are an `IORails`-only capability by design. For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/index.yml b/docs/index.yml index c6615d2970..78b48d180c 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -127,6 +127,9 @@ navigation: - page: Tool Safety Check path: configure-rails/guardrail-catalog/tool-safety-check.mdx slug: tool-safety-check + - page: Common Expression Language (CEL) + path: configure-rails/guardrail-catalog/cel.mdx + slug: cel - page: Hallucinations & Fact-Checking path: configure-rails/guardrail-catalog/fact-checking.mdx slug: fact-checking diff --git a/docs/reference/rail-engine-support.mdx b/docs/reference/rail-engine-support.mdx index db96b2e6a0..33871f2f6a 100644 --- a/docs/reference/rail-engine-support.mdx +++ b/docs/reference/rail-engine-support.mdx @@ -40,9 +40,9 @@ The following table summarizes surface support by direction: | Input | 32 | 32 | 31 | | Output | 35 | 35 | 28 | | Retrieval | 11 | 11 | 0 | -| Tool_output | 2 | 0 | 2 | -| Tool_input | 2 | 0 | 2 | -| **Total** | **82** | **78** | **63** | +| Tool_output | 3 | 0 | 3 | +| Tool_input | 3 | 0 | 3 | +| **Total** | **84** | **78** | **65** | `IORails` does not accept a `rails.retrieval` section at all, so every retrieval surface is an `LLMRails` capability. A configuration that declares one routes to `LLMRails` by default. @@ -160,6 +160,7 @@ Per-tool checks are an `IORails`-only capability by design: | Flow | Rail | LLMRails | IORails | Notes | |---|---|:---:|:---:|---| +| `cel check tool output` | Common Expression Language (CEL) | ✗ | ✓ | IORails-only capability by design | | `regex check tool output` | Regex Detection | ✗ | ✓ | IORails-only capability by design | | `tool safety check output` | Tool Safety Check | ✗ | ✓ | IORails-only capability by design | @@ -170,10 +171,11 @@ Per-tool checks are an `IORails`-only capability by design: | Flow | Rail | LLMRails | IORails | Notes | |---|---|:---:|:---:|---| +| `cel check tool input` | Common Expression Language (CEL) | ✗ | ✓ | IORails-only capability by design | | `regex check tool input` | Regex Detection | ✗ | ✓ | IORails-only capability by design | | `tool safety check input` | Tool Safety Check | ✗ | ✓ | IORails-only capability by design | -For worked per-tool configuration examples, refer to [Regular Expression Detection Integration](/configure-guardrails/guardrail-catalog/community/regex) (includes the `$argument=` scoping option) and [Tool Safety Check](/configure-guardrails/guardrail-catalog/tool-safety-check) (an LLM-judged check with `$model=`/`$variant=`). +For worked per-tool configuration examples, refer to [Regular Expression Detection Integration](/configure-guardrails/guardrail-catalog/community/regex) (includes the `$argument=` scoping option), [Tool Safety Check](/configure-guardrails/guardrail-catalog/tool-safety-check) (an LLM-judged check with `$model=`/`$variant=`), and [Common Expression Language (CEL)](/configure-guardrails/guardrail-catalog/cel) (deterministic expressions over a tool call's arguments or a tool result's content). ### Tool Surfaces diff --git a/docs/telemetry.mdx b/docs/telemetry.mdx index 94d96b2446..24b9fb105b 100644 --- a/docs/telemetry.mdx +++ b/docs/telemetry.mdx @@ -43,7 +43,7 @@ In this context, a *session* is the lifetime of a single Python process running Each value identifies a built-in library feature. The built-in rail manifests are the authoritative source. Related manifests can share a stable feature identifier so telemetry remains comparable across releases. -`activefence`, `ai_defense`, `autoalign`, `clavata`, `cleanlab`, `content_safety`, `context_bloat_detection`, `crowdstrike_aidr`, `f5`, `factchecking`, `fiddler`, `gcp_moderate_text`, `gliner`, `guardrails_ai`, `hallucination`, `hf_classifier`, `injection_detection`, `jailbreak_detection`, `llama_guard`, `pangea`, `patronusai`, `policyai`, `polygraf`, `prompt_security`, `regex`, `self_check`, `sensitive_data_detection`, `tool_safety_check`, `topic_safety`, `trend_micro`. +`activefence`, `ai_defense`, `autoalign`, `cel`, `clavata`, `cleanlab`, `content_safety`, `context_bloat_detection`, `crowdstrike_aidr`, `f5`, `factchecking`, `fiddler`, `gcp_moderate_text`, `gliner`, `guardrails_ai`, `hallucination`, `hf_classifier`, `injection_detection`, `jailbreak_detection`, `llama_guard`, `pangea`, `patronusai`, `policyai`, `polygraf`, `prompt_security`, `regex`, `self_check`, `sensitive_data_detection`, `tool_safety_check`, `topic_safety`, `trend_micro`. ## Data Not Collected by Telemetry diff --git a/nemoguardrails/library/cel/__init__.py b/nemoguardrails/library/cel/__init__.py new file mode 100644 index 0000000000..467079831e --- /dev/null +++ b/nemoguardrails/library/cel/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nemoguardrails/library/cel/actions.py b/nemoguardrails/library/cel/actions.py new file mode 100644 index 0000000000..4c47621598 --- /dev/null +++ b/nemoguardrails/library/cel/actions.py @@ -0,0 +1,129 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from typing import Any, Dict, List, Optional, Tuple + +from nemoguardrails import RailsConfig +from nemoguardrails.actions import action +from nemoguardrails.actions.rail_outcome import RailOutcome +from nemoguardrails.guardrails.tool_schema import Tool, ToolResult, tool_output_validation +from nemoguardrails.library.cel.rail_config import CelOptions +from nemoguardrails.types import ToolCall + +log = logging.getLogger(__name__) + + +def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: Optional[CelOptions]) -> RailOutcome: + """Evaluate all of a tool's CEL expressions, blocking when any is true. + + A match blocks even if another expression fails. With no match, a failing expression + (an error or a non-boolean result) raises, so the rail fails closed. + """ + metadata: Dict[str, Any] = {"source": source} + if options is None or not options.compiled_expressions: + log.debug("No CEL expressions configured for tool %r under source: %s", tool_name, source) + return RailOutcome.allow(metadata=metadata) + + # Already imported when the expressions compiled at config load. + import cel + from cel.stdlib import add_stdlib_to_context + + context = cel.Context() + add_stdlib_to_context(context) + context.update(variables) + + matched: List[str] = [] + failures: List[Tuple[str, Exception]] = [] + for program, expression in zip(options.compiled_expressions, options.expressions): + try: + result = program.execute(context) + if not isinstance(result, bool): + raise TypeError(f"returned {type(result).__name__}, not bool") + except Exception as e: + log.warning("CEL expression %r for tool %r failed: %s: %s", expression, tool_name, type(e).__name__, e) + failures.append((expression, e)) + continue + if result: + log.info("CEL expression matched: %s", expression) + matched.append(expression) + + if matched: + return RailOutcome.block(metadata={**metadata, "matched_expressions": matched}) + if failures: + details = "; ".join(f"{expression!r}: {type(error).__name__}: {error}" for expression, error in failures) + raise RuntimeError(f"CEL expressions for tool {tool_name!r} failed: {details}") + return RailOutcome.allow(metadata=metadata) + + +@action(is_system_action=True) +@tool_output_validation +async def evaluate_tool_output_cel( + source: str, + tool_call: ToolCall, + tool_definition: Optional[Tool], + config: RailsConfig, + **kwargs, +) -> RailOutcome: + """Checks a tool call's arguments against that tool's CEL expressions. + + Args: + source: Fixed per surface, always "tool_output". + tool_call: The tool call to check. Its arguments are `args` and its name is `tool`. + tool_definition: The declared tool, used only by @tool_output_validation to check + the call's arguments against its schema before this function runs. + config: The rails configuration object. + + Returns: + RailOutcome that blocks when any expression is true, with the matched expressions in + metadata. + """ + if source != "tool_output": + raise ValueError("source must be 'tool_output'") + + tool_name = tool_call.function.name or tool_call.type + options = getattr(config.rails.config.cel, source).get(tool_name) + variables = {"args": tool_call.function.arguments, "tool": tool_name} + return _evaluate(source, tool_name, variables, options) + + +@action(is_system_action=True) +async def evaluate_tool_input_cel( + source: str, + tool_call: ToolCall, + tool_result: ToolResult, + config: RailsConfig, + **kwargs, +) -> RailOutcome: + """Checks a tool result's content against that tool's CEL expressions. + + Args: + source: Fixed per surface, always "tool_input". + tool_call: The prior call this result answers. Its arguments are `args` and its name + is `tool`. + tool_result: The tool result to check. Its content is `result`, as the tool returned it. + config: The rails configuration object. + + Returns: + RailOutcome that blocks when any expression is true, with the matched expressions in + metadata. + """ + if source != "tool_input": + raise ValueError("source must be 'tool_input'") + + tool_name = tool_call.function.name or tool_call.type + options = getattr(config.rails.config.cel, source).get(tool_name) + variables = {"result": tool_result.content, "args": tool_call.function.arguments, "tool": tool_name} + return _evaluate(source, tool_name, variables, options) diff --git a/nemoguardrails/library/cel/rail.py b/nemoguardrails/library/cel/rail.py new file mode 100644 index 0000000000..ab26545b1e --- /dev/null +++ b/nemoguardrails/library/cel/rail.py @@ -0,0 +1,91 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nemoguardrails.manifests import ( + ActionRef, + Binding, + ConfigSpecRef, + RailActions, + RailConfigSchema, + RailDirection, + RailFlows, + RailManifest, + RailMetadata, + RailPrivacy, + RailRequirements, + RailSpec, + RailSurface, +) + +EVALUATE_TOOL_OUTPUT_CEL = ActionRef( + name="evaluate_tool_output_cel", + target="nemoguardrails.library.cel.actions:evaluate_tool_output_cel", +) +EVALUATE_TOOL_INPUT_CEL = ActionRef( + name="evaluate_tool_input_cel", + target="nemoguardrails.library.cel.actions:evaluate_tool_input_cel", +) +RAIL = RailManifest( + name="cel", + metadata=RailMetadata( + display_name="Common Expression Language (CEL)", + description="Detects and blocks content matching configured Common Expression Language (CEL) expressions.", + categories=("tool_output", "tool_input"), + capabilities=("allow", "block", "moderate"), + tags=("built-in", "cel", "tool-calling"), + docs_url="docs/configure-rails/guardrail-catalog/cel.mdx", + ), + spec=RailSpec( + config_schema=RailConfigSchema( + key="cel", + spec=ConfigSpecRef(target="nemoguardrails.library.cel.rail_config:build_config_spec"), + ), + flows=RailFlows( + # Every surface here is TOOL_OUTPUT/TOOL_INPUT (IORails-only), so this rail + # ships no Colang flow definitions at all. + files=(), + v1_files=(), + flow_names=( + "cel check tool output", + "cel check tool input", + ), + ), + actions=RailActions(refs=(EVALUATE_TOOL_OUTPUT_CEL, EVALUATE_TOOL_INPUT_CEL)), + surfaces=( + RailSurface( + name="cel check tool output", + direction=RailDirection.TOOL_OUTPUT, + action=EVALUATE_TOOL_OUTPUT_CEL, + bindings=( + Binding.literal("source", "tool_output"), + Binding.context("tool_call", "tool_call"), + Binding.context("tool_definition", "tool_definition"), + ), + ), + RailSurface( + name="cel check tool input", + direction=RailDirection.TOOL_INPUT, + action=EVALUATE_TOOL_INPUT_CEL, + bindings=( + Binding.literal("source", "tool_input"), + Binding.context("tool_call", "tool_call"), + Binding.context("tool_result", "tool_result"), + ), + ), + ), + requirements=RailRequirements(extras=("cel",), optional_dependencies=("common-expression-language",)), + privacy=RailPrivacy(), + ), +) diff --git a/nemoguardrails/library/cel/rail_config.py b/nemoguardrails/library/cel/rail_config.py new file mode 100644 index 0000000000..1eaf3b8f0d --- /dev/null +++ b/nemoguardrails/library/cel/rail_config.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import TYPE_CHECKING, Dict, List, Optional + +from nemoguardrails.manifests.config_schema import ( + Field, + PrivateAttr, + RailConfigBaseModel, + RailConfigSpec, + model_validator, + rail_field, +) + +if TYPE_CHECKING: + from cel import Program + + +def _compile(expression: str) -> "Program": + """Compile *expression*, importing the optional CEL library only when a config uses it.""" + try: + import cel + except ImportError as e: + raise ImportError( + "The CEL rail requires the 'common-expression-language' package (Python 3.11+). " + "Install it with `pip install nemoguardrails[cel]`." + ) from e + return cel.compile(expression) + + +class CelOptions(RailConfigBaseModel): + """CEL expressions to check for one tool.""" + + expressions: List[str] = Field( + default_factory=list, + description="List of CEL expressions to evaluate, where any that evaluates to true is a match.", + ) + + _compiled_expressions: List["Program"] = PrivateAttr(default_factory=list) + + @model_validator(mode="after") + def compile_expressions(self) -> "CelOptions": + """Compile the expressions at config load time, so a syntax error fails the load.""" + compiled = [] + for i, expression in enumerate(self.expressions): + try: + compiled.append(_compile(expression)) + except ValueError as e: + raise ValueError(f"Invalid CEL expression at index {i} ({expression!r}): {e}") from e + object.__setattr__(self, "_compiled_expressions", compiled) + return self + + @property + def compiled_expressions(self) -> List["Program"]: + """Return the compiled CEL programs.""" + return self._compiled_expressions + + +class CelConfig(RailConfigBaseModel): + """Configuration for CEL expression checks.""" + + tool_output: Dict[str, CelOptions] = Field( + default_factory=dict, + description="Per-tool CEL expressions to check against a tool call, keyed by tool name.", + ) + tool_input: Dict[str, CelOptions] = Field( + default_factory=dict, + description="Per-tool CEL expressions to check against a tool result, keyed by tool name.", + ) + + +def build_config_spec() -> RailConfigSpec: + return RailConfigSpec( + annotation=Optional[CelConfig], + field_info=rail_field( + default_factory=CelConfig, + description="Configuration for CEL expression checks.", + ), + exports={ + "CelConfig": CelConfig, + "CelOptions": CelOptions, + }, + ) diff --git a/pyproject.toml b/pyproject.toml index 301a7625d0..507fbccfa2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -69,6 +69,8 @@ gcp = ["google-cloud-language (>=2.14.0)"] jailbreak = ["yara-python (>=4.5.1,<5.0.0)"] +cel = ["common-expression-language (>=0.10.0,<0.11.0) ; python_version>='3.11'"] + multilingual = ["fast-langdetect (>=1.0.0)"] chat-ui = ["chainlit (>=2.11.0,<3.0.0)"] @@ -108,6 +110,7 @@ all = [ "opentelemetry-exporter-prometheus (>=0.63b0,<1.0.0)", "prometheus-client (>=0.20.0,<2.0.0)", "chainlit (>=2.11.0,<3.0.0)", + "common-expression-language (>=0.10.0,<0.11.0) ; python_version>='3.11'", ] [dependency-groups] @@ -127,6 +130,7 @@ test_integration = [ "aioresponses>=0.7.6", "streamlit>=1.37.0", "yara-python>=4.5.1,<5.0.0", + "common-expression-language>=0.10.0,<0.11.0 ; python_version>='3.11'", "opentelemetry-api>=1.34.1,<2.0.0", "opentelemetry-sdk>=1.34.1,<2.0.0", "opentelemetry-exporter-prometheus>=0.63b0,<1.0.0", diff --git a/schemas/rails_config.snapshot.json b/schemas/rails_config.snapshot.json index 0a5f0563c2..70584836f9 100644 --- a/schemas/rails_config.snapshot.json +++ b/schemas/rails_config.snapshot.json @@ -53,6 +53,44 @@ "title": "AutoAlignRailConfig", "type": "object" }, + "CelConfig": { + "description": "Configuration for CEL expression checks.", + "properties": { + "tool_input": { + "additionalProperties": { + "$ref": "#/$defs/CelOptions" + }, + "description": "Per-tool CEL expressions to check against a tool result, keyed by tool name.", + "title": "Tool Input", + "type": "object" + }, + "tool_output": { + "additionalProperties": { + "$ref": "#/$defs/CelOptions" + }, + "description": "Per-tool CEL expressions to check against a tool call, keyed by tool name.", + "title": "Tool Output", + "type": "object" + } + }, + "title": "CelConfig", + "type": "object" + }, + "CelOptions": { + "description": "CEL expressions to check for one tool.", + "properties": { + "expressions": { + "description": "List of CEL expressions to evaluate, where any that evaluates to true is a match.", + "items": { + "type": "string" + }, + "title": "Expressions", + "type": "array" + } + }, + "title": "CelOptions", + "type": "object" + }, "ClavataRailConfig": { "description": "Configuration data for the Clavata API", "properties": { @@ -1144,6 +1182,17 @@ "$ref": "#/$defs/AutoAlignRailConfig", "description": "Configuration data for the AutoAlign guardrails API." }, + "cel": { + "anyOf": [ + { + "$ref": "#/$defs/CelConfig" + }, + { + "type": "null" + } + ], + "description": "Configuration for CEL expression checks." + }, "clavata": { "anyOf": [ { diff --git a/tests/guardrails/test_per_tool_cel_rails_iorails.py b/tests/guardrails/test_per_tool_cel_rails_iorails.py new file mode 100644 index 0000000000..7ae5a5eb0d --- /dev/null +++ b/tests/guardrails/test_per_tool_cel_rails_iorails.py @@ -0,0 +1,207 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Integration tests for the per-tool CEL rails wired into IORails. + +The IORails-level companion to tests/test_cel_actions.py, mirroring +test_per_tool_regex_rails_iorails.py and reusing test_tool_rails_iorails.py's transport-mocking +helpers. +""" + +import pytest +import pytest_asyncio + +from nemoguardrails.guardrails.iorails import INTERNAL_ERROR_MESSAGE, REFUSAL_MESSAGE +from nemoguardrails.imports import check_optional_dependency +from tests.guardrails.async_helpers import started_iorails +from tests.guardrails.test_tool_rails_iorails import ( + _inject_forbidden_transport, + _inject_json_response, + _inject_sse_stream, + _sse, + _stream_violation_chunks, + _text_payload, + _tool_call_payload, + _tool_call_sse_lines, +) + +pytestmark = pytest.mark.skipif(not check_optional_dependency("cel"), reason="requires the 'cel' extra (Python 3.11+)") + +RUN_SHELL_TOOL = { + "type": "function", + "function": { + "name": "run_shell", + "parameters": { + "type": "object", + "properties": {"command": {"type": "string"}}, + "required": ["command"], + }, + }, +} + +MAIN_MODEL = { + "type": "main", + "engine": "nim", + "model": "meta/llama-3.3-70b-instruct", + "parameters": {"tools": [RUN_SHELL_TOOL]}, +} + + +def _tool_output_config(*expressions: str) -> dict: + return { + "models": [MAIN_MODEL], + "rails": { + "config": {"cel": {"tool_output": {"run_shell": {"expressions": list(expressions)}}}}, + "tool_output": {"per_tool": {"run_shell": ["cel check tool output"]}}, + }, + } + + +TOOL_OUTPUT_CONFIG = _tool_output_config(r'args.command.matches("\\brm\\s+-rf")') +# Fails on every call: `flags` is never sent, and the expression has no has() guard. +TOOL_OUTPUT_FAILING_CONFIG = _tool_output_config('args.flags.exists(f, f == "--force")') + +TOOL_INPUT_CONFIG = { + "models": [MAIN_MODEL], + "rails": { + "config": {"cel": {"tool_input": {"run_shell": {"expressions": ['result.contains("BEGIN RSA PRIVATE KEY")']}}}}, + "tool_input": {"per_tool": {"run_shell": ["cel check tool input"]}}, + }, +} + +MESSAGES = [{"role": "user", "content": "clean up the temp directory"}] + + +def _tool_conversation(result_content: str) -> list: + return [ + {"role": "user", "content": "show me the key"}, + { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": {"name": "run_shell", "arguments": '{"command": "cat key.pem"}'}, + } + ], + }, + {"role": "tool", "tool_call_id": "call_1", "name": "run_shell", "content": result_content}, + ] + + +def _text_sse_lines(text: str) -> list: + chunks = [ + {"id": "chatcmpl-1", "choices": [{"index": 0, "delta": {"role": "assistant", "content": text}}]}, + {"id": "chatcmpl-1", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}, + ] + return [_sse(chunk) for chunk in chunks] + [b"data: [DONE]\n\n"] + + +async def _collect(stream) -> list: + return [chunk async for chunk in stream] + + +@pytest_asyncio.fixture +async def call_iorails(): + async with started_iorails(TOOL_OUTPUT_CONFIG) as engine: + yield engine + + +@pytest_asyncio.fixture +async def failing_call_iorails(): + async with started_iorails(TOOL_OUTPUT_FAILING_CONFIG) as engine: + yield engine + + +@pytest_asyncio.fixture +async def result_iorails(): + async with started_iorails(TOOL_INPUT_CONFIG) as engine: + yield engine + + +class TestNonStreamingPerToolCallCel: + @pytest.mark.asyncio + async def test_matching_expression_blocked(self, call_iorails): + _inject_json_response(call_iorails, _tool_call_payload("run_shell", '{"command": "rm -rf /tmp"}')) + result = await call_iorails.generate_async(messages=MESSAGES) + assert result == {"role": "assistant", "content": REFUSAL_MESSAGE} + + @pytest.mark.asyncio + async def test_non_matching_expression_passes(self, call_iorails): + _inject_json_response(call_iorails, _tool_call_payload("run_shell", '{"command": "ls /tmp"}')) + result = await call_iorails.generate_async(messages=MESSAGES) + assert result["tool_calls"][0]["function"]["name"] == "run_shell" + + @pytest.mark.asyncio + async def test_schema_violation_blocks_before_evaluation(self, call_iorails): + _inject_json_response(call_iorails, _tool_call_payload("run_shell", '{"command": 123}')) + result = await call_iorails.generate_async(messages=MESSAGES) + assert result == {"role": "assistant", "content": REFUSAL_MESSAGE} + + @pytest.mark.asyncio + async def test_failing_expression_fails_closed_as_internal_error(self, failing_call_iorails): + _inject_json_response(failing_call_iorails, _tool_call_payload("run_shell", '{"command": "ls"}')) + result = await failing_call_iorails.generate_async(messages=MESSAGES) + assert result == {"role": "assistant", "content": INTERNAL_ERROR_MESSAGE} + + +class TestStreamingPerToolCallCel: + @pytest.mark.asyncio + async def test_matching_expression_blocks_stream(self, call_iorails): + _inject_sse_stream(call_iorails, _tool_call_sse_lines("run_shell", ['{"command": "rm -rf /tmp"}'])) + chunks = await _collect(call_iorails.stream_async(MESSAGES)) + violations = _stream_violation_chunks(chunks) + assert len(violations) == 1 + assert violations[0]["error"]["param"] == "tool_output_rails" + + @pytest.mark.asyncio + async def test_non_matching_expression_streams_through(self, call_iorails): + _inject_sse_stream(call_iorails, _tool_call_sse_lines("run_shell", ['{"command": "ls /tmp"}'])) + chunks = await _collect(call_iorails.stream_async(MESSAGES)) + assert _stream_violation_chunks(chunks) == [] + + +class TestNonStreamingPerToolResultCel: + @pytest.mark.asyncio + async def test_matching_expression_blocked_before_generation(self, result_iorails): + forbidden_post = _inject_forbidden_transport(result_iorails) + result = await result_iorails.generate_async(messages=_tool_conversation("-----BEGIN RSA PRIVATE KEY-----")) + assert result == {"role": "assistant", "content": REFUSAL_MESSAGE} + forbidden_post.assert_not_called() + + @pytest.mark.asyncio + async def test_non_matching_expression_passes(self, result_iorails): + _inject_json_response(result_iorails, _text_payload("The file has no key.")) + result = await result_iorails.generate_async(messages=_tool_conversation("no keys here")) + assert result == {"role": "assistant", "content": "The file has no key."} + + +class TestStreamingPerToolResultCel: + @pytest.mark.asyncio + async def test_matching_expression_blocks_stream_before_generation(self, result_iorails): + forbidden_post = _inject_forbidden_transport(result_iorails) + chunks = await _collect(result_iorails.stream_async(_tool_conversation("-----BEGIN RSA PRIVATE KEY-----"))) + violations = _stream_violation_chunks(chunks) + assert len(violations) == 1 + assert violations[0]["error"]["param"] == "tool_input_rails" + forbidden_post.assert_not_called() + + @pytest.mark.asyncio + async def test_non_matching_expression_streams_through(self, result_iorails): + _inject_sse_stream(result_iorails, _text_sse_lines("The file has no key.")) + chunks = await _collect(result_iorails.stream_async(_tool_conversation("no keys here"))) + assert _stream_violation_chunks(chunks) == [] + assert "The file has no key." in "".join(chunk for chunk in chunks if isinstance(chunk, str)) diff --git a/tests/rails/llm/test_config.py b/tests/rails/llm/test_config.py index c85db41090..c94b7c65b6 100644 --- a/tests/rails/llm/test_config.py +++ b/tests/rails/llm/test_config.py @@ -133,6 +133,7 @@ def test_builtin_rails_config_fields_canonical_set_and_legacy_exports(): expected_exports = { "ai_defense": ("AIDefenseRailConfig",), "autoalign": ("AutoAlignOptions", "AutoAlignRailConfig"), + "cel": ("CelConfig", "CelOptions"), "clavata": ("ClavataRailConfig", "ClavataRailOptions"), "content_safety": ("ContentSafetyConfig", "MultilingualConfig", "ReasoningConfig"), "context_bloat_detection": ("ContextBloatDetectionConfig",), @@ -183,6 +184,7 @@ def test_builtin_rails_config_fields_canonical_set_and_legacy_exports(): expected_config_keys = { "ai_defense", "autoalign", + "cel", "clavata", "content_safety", "context_bloat_detection", diff --git a/tests/test_cel_actions.py b/tests/test_cel_actions.py new file mode 100644 index 0000000000..f800da9ad7 --- /dev/null +++ b/tests/test_cel_actions.py @@ -0,0 +1,225 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the CEL rail's config and its tool output / tool input actions. + +The action-level companion to tests/guardrails/test_per_tool_cel_rails_iorails.py, which drives +the same rail through IORails. +""" + +import sys + +import pytest + +from nemoguardrails import RailsConfig +from nemoguardrails.actions.rail_outcome import RailDecision +from nemoguardrails.guardrails.tool_schema import Tool, ToolResult +from nemoguardrails.imports import check_optional_dependency +from nemoguardrails.library.cel.actions import evaluate_tool_input_cel, evaluate_tool_output_cel +from nemoguardrails.library.cel.rail_config import CelOptions +from nemoguardrails.types import ToolCall, ToolCallFunction + +pytestmark = pytest.mark.skipif(not check_optional_dependency("cel"), reason="requires the 'cel' extra (Python 3.11+)") + +RUN_SHELL_TOOL = Tool( + name="run_shell", + arguments_schema={ + "type": "object", + "properties": {"command": {"type": "string"}, "flags": {"type": "array", "items": {"type": "string"}}}, + "required": ["command"], + }, +) + + +def _config(direction: str, *expressions: str, tool: str = "run_shell") -> RailsConfig: + return RailsConfig.from_content( + config={"rails": {"config": {"cel": {direction: {tool: {"expressions": list(expressions)}}}}}} + ) + + +def _shell_call(**arguments) -> ToolCall: + return ToolCall(id="call_1", function=ToolCallFunction(name="run_shell", arguments=arguments)) + + +async def _check_call(config: RailsConfig, call: ToolCall, tool_definition=RUN_SHELL_TOOL): + return await evaluate_tool_output_cel( + source="tool_output", tool_call=call, tool_definition=tool_definition, config=config + ) + + +async def _check_result(config: RailsConfig, content, call: ToolCall = None): + return await evaluate_tool_input_cel( + source="tool_input", + tool_call=call or _shell_call(command="cat key.pem"), + tool_result=ToolResult(call_id="call_1", content=content), + config=config, + ) + + +class TestConfig: + def test_expressions_compile_at_load(self): + config = _config("tool_output", 'args.command == "ls"', "true") + assert len(config.rails.config.cel.tool_output["run_shell"].compiled_expressions) == 2 + + def test_syntax_error_fails_the_load(self): + with pytest.raises(ValueError, match=r"Invalid CEL expression at index 1 \('args.command.matches\('\)"): + _config("tool_output", "true", "args.command.matches(") + + def test_missing_extra_names_it(self, monkeypatch): + monkeypatch.setitem(sys.modules, "cel", None) + with pytest.raises(ImportError, match=r"nemoguardrails\[cel\]"): + CelOptions(expressions=["true"]) + + def test_no_expressions_needs_no_cel(self, monkeypatch): + monkeypatch.setitem(sys.modules, "cel", None) + assert CelOptions().compiled_expressions == [] + + +class TestToolOutput: + @pytest.mark.asyncio + async def test_match_blocks_with_every_matched_expression(self): + config = _config( + "tool_output", + 'args.command.startsWith("rm")', + 'args.command == "ls"', + 'args.command.contains("-rf")', + ) + outcome = await _check_call(config, _shell_call(command="rm -rf /")) + assert outcome.decision is RailDecision.BLOCK + assert outcome.metadata == { + "source": "tool_output", + "matched_expressions": ['args.command.startsWith("rm")', 'args.command.contains("-rf")'], + } + + @pytest.mark.asyncio + async def test_no_match_allows(self): + outcome = await _check_call(_config("tool_output", 'args.command == "rm"'), _shell_call(command="ls")) + assert outcome.decision is RailDecision.ALLOW + assert outcome.metadata == {"source": "tool_output"} + + @pytest.mark.asyncio + async def test_tool_without_expressions_allows(self): + outcome = await _check_call(_config("tool_output", "true", tool="other_tool"), _shell_call(command="rm")) + assert outcome.decision is RailDecision.ALLOW + + @pytest.mark.asyncio + async def test_tool_name_is_available(self): + outcome = await _check_call(_config("tool_output", 'tool == "run_shell"'), _shell_call(command="ls")) + assert outcome.decision is RailDecision.BLOCK + + @pytest.mark.asyncio + async def test_stdlib_extensions_are_available(self): + config = _config("tool_output", 'args.command.lowerAscii().startsWith("rm")') + outcome = await _check_call(config, _shell_call(command="RM -rf /")) + assert outcome.decision is RailDecision.BLOCK + + @pytest.mark.asyncio + async def test_undeclared_tool_blocks_before_evaluation(self): + # The expression raises if evaluated, so a block (not a raise) shows it never ran. + config = _config("tool_output", "args.missing == 1") + outcome = await _check_call(config, _shell_call(command="ls"), tool_definition=None) + assert outcome.decision is RailDecision.BLOCK + assert "not an allowed tool" in outcome.reason + + @pytest.mark.asyncio + async def test_schema_violation_blocks_before_evaluation(self): + config = _config("tool_output", "args.missing == 1") + outcome = await _check_call(config, _shell_call(command=123)) + assert outcome.decision is RailDecision.BLOCK + assert "matched_expressions" not in outcome.metadata + + @pytest.mark.asyncio + async def test_wrong_source_raises(self): + with pytest.raises(ValueError, match="source must be 'tool_output'"): + await evaluate_tool_output_cel( + source="tool_input", + tool_call=_shell_call(command="ls"), + tool_definition=RUN_SHELL_TOOL, + config=_config("tool_output", "true"), + ) + + +class TestFailures: + @pytest.mark.asyncio + async def test_match_blocks_despite_a_failing_expression(self): + config = _config("tool_output", 'args.flags.exists(f, f == "--force")', 'args.command == "rm"') + outcome = await _check_call(config, _shell_call(command="rm")) + assert outcome.decision is RailDecision.BLOCK + assert outcome.metadata["matched_expressions"] == ['args.command == "rm"'] + + @pytest.mark.asyncio + async def test_failures_without_a_match_raise_listing_each(self): + config = _config("tool_output", 'args.flags.exists(f, f == "--force")', "args.command", 'args.command == "rm"') + with pytest.raises(RuntimeError) as excinfo: + await _check_call(config, _shell_call(command="ls")) + message = str(excinfo.value) + assert message.startswith("CEL expressions for tool 'run_shell' failed: ") + assert "'args.flags.exists(f, f == \"--force\")': KeyError: 'flags'" in message + assert "'args.command': TypeError: returned str, not bool" in message + + @pytest.mark.asyncio + async def test_failure_message_carries_no_argument_values(self): + config = _config("tool_output", "args.command > 5") + with pytest.raises(RuntimeError) as excinfo: + await _check_call(config, _shell_call(command="SECRET_TOKEN")) + assert "SECRET_TOKEN" not in str(excinfo.value) + + +class TestToolInput: + @pytest.mark.asyncio + async def test_string_result_match_blocks(self): + config = _config("tool_input", 'result.contains("BEGIN RSA PRIVATE KEY")') + outcome = await _check_result(config, "-----BEGIN RSA PRIVATE KEY-----") + assert outcome.decision is RailDecision.BLOCK + assert outcome.metadata == { + "source": "tool_input", + "matched_expressions": ['result.contains("BEGIN RSA PRIVATE KEY")'], + } + + @pytest.mark.asyncio + async def test_string_result_no_match_allows(self): + config = _config("tool_input", 'result.contains("BEGIN RSA PRIVATE KEY")') + outcome = await _check_result(config, "no keys here") + assert outcome.decision is RailDecision.ALLOW + assert outcome.metadata == {"source": "tool_input"} + + @pytest.mark.asyncio + async def test_content_blocks_are_queryable(self): + config = _config("tool_input", 'result.exists(b, b.type == "text" && b.text.contains("password"))') + outcome = await _check_result(config, [{"type": "text", "text": "password=hunter2"}]) + assert outcome.decision is RailDecision.BLOCK + + @pytest.mark.asyncio + async def test_type_guard_skips_other_shapes(self): + config = _config("tool_input", 'type(result) == string && result.contains("password")') + for content in ([{"type": "text", "text": "password"}], None): + outcome = await _check_result(config, content) + assert outcome.decision is RailDecision.ALLOW + + @pytest.mark.asyncio + async def test_args_and_tool_come_from_the_answered_call(self): + config = _config("tool_input", 'tool == "run_shell" && args.command.endsWith(".pem")') + outcome = await _check_result(config, "ok", call=_shell_call(command="cat key.pem")) + assert outcome.decision is RailDecision.BLOCK + + @pytest.mark.asyncio + async def test_wrong_source_raises(self): + with pytest.raises(ValueError, match="source must be 'tool_input'"): + await evaluate_tool_input_cel( + source="tool_output", + tool_call=_shell_call(command="ls"), + tool_result=ToolResult(call_id="call_1", content="ok"), + config=_config("tool_input", "true"), + ) diff --git a/tests/test_runtime_flow_gate_equivalence.py b/tests/test_runtime_flow_gate_equivalence.py index 38718ac529..3924ae1309 100644 --- a/tests/test_runtime_flow_gate_equivalence.py +++ b/tests/test_runtime_flow_gate_equivalence.py @@ -363,6 +363,20 @@ def _injection_detection_transform_decision(raw_return: Any) -> FlowDecision: output_parser="parse_tool_safety_verdict", ) +CEL_TOOL_OUTPUT = RailSpec( + name="cel_tool_output", + flow="cel check tool output", + direction="tool_output", + action="evaluate_tool_output_cel", +) + +CEL_TOOL_INPUT = RailSpec( + name="cel_tool_input", + flow="cel check tool input", + direction="tool_input", + action="evaluate_tool_input_cel", +) + PRIVATEAI_DETECT_INPUT = RailSpec( name="privateai_detect_input", flow="detect pii on input", @@ -2955,6 +2969,16 @@ def _iorails_case_param(case: FlowEquivalenceCase): ), *_rail_outcome_cases(TOOL_SAFETY_CHECK_OUTPUT), *_rail_outcome_cases(TOOL_SAFETY_CHECK_INPUT), + *_rail_outcome_cases( + CEL_TOOL_OUTPUT, + allow_return=RailOutcome.allow(metadata={"source": "tool_output"}), + block_return=RailOutcome.block(metadata={"source": "tool_output", "matched_expressions": ["true"]}), + ), + *_rail_outcome_cases( + CEL_TOOL_INPUT, + allow_return=RailOutcome.allow(metadata={"source": "tool_input"}), + block_return=RailOutcome.block(metadata={"source": "tool_input", "matched_expressions": ["true"]}), + ), ] IORAILS_FIXTURES = [case for case in FIXTURES if _is_iorails_enabled(case.spec)] + IORAILS_ONLY_FIXTURES diff --git a/uv.lock b/uv.lock index a054e6cdb5..4668aedd6d 100644 --- a/uv.lock +++ b/uv.lock @@ -285,7 +285,7 @@ name = "blis" version = "1.3.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" }, { name = "numpy", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.12.*'" }, ] @@ -550,7 +550,7 @@ name = "cloudpathlib" version = "0.24.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/06/19/58bc6b5d7d0f81c7209b05445af477e147c486552f96665a5912211839b9/cloudpathlib-0.24.0.tar.gz", hash = "sha256:c521a984e77b47e656fe78e20a7e3e260e0ab45fc69e33ac01094227c979e34a", size = 53600, upload-time = "2026-04-30T00:54:43.265Z" } wheels = [ @@ -571,7 +571,7 @@ name = "coloredlogs" version = "15.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "humanfriendly" }, + { name = "humanfriendly", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cc/c7/eed8f27100517e8c0e6b923d5f0845d0cb99763da6fdee00478f91db7325/coloredlogs-15.0.1.tar.gz", hash = "sha256:7c991aa71a4577af2f82600d8f8f3a89f936baeaf9b50a9c197da014e5bf16b0", size = 278520, upload-time = "2021-06-11T10:22:45.202Z" } wheels = [ @@ -590,12 +590,60 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6d/c1/e419ef3723a074172b68aaa89c9f3de486ed4c2399e2dbd8113a4fdcaf9e/colorlog-6.10.1-py3-none-any.whl", hash = "sha256:2d7e8348291948af66122cff006c9f8da6255d224e7cf8e37d8de2df3bad8c9c", size = 11743, upload-time = "2025-10-16T16:14:10.512Z" }, ] +[[package]] +name = "common-expression-language" +version = "0.10.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "prompt-toolkit", marker = "python_full_version >= '3.11'" }, + { name = "pygments", marker = "python_full_version >= '3.11'" }, + { name = "rich", marker = "python_full_version >= '3.11'" }, + { name = "typer", marker = "python_full_version >= '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f0/92/cb479d91a71eed6cfdfea673c7dc341bcf627856acbe0f978900e207fa31/common_expression_language-0.10.0.tar.gz", hash = "sha256:bb2b6a2e50094219e4366cf36d342d74140c97414545912b34a7e8b8b95c92e6", size = 231354, upload-time = "2026-09-15T08:29:23.132Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/53/ef86249f04fcc82f1f30ebce49ef1a16e7d711126863e4f6583eaa84b02b/common_expression_language-0.10.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:29417ed47959aab5bad3ec12b21b3d332be0008043b8a1ea6c97e262dc41b7dd", size = 1594313, upload-time = "2026-09-15T08:28:04.808Z" }, + { url = "https://files.pythonhosted.org/packages/1c/75/c9ba1f2971c618eb788d71212e03e582449db58500aec893cd003dd375d0/common_expression_language-0.10.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:425af498789ae891f4bdad1f978e2864a79fa0a4b8cd26ddaa47a48a22f9d311", size = 1554172, upload-time = "2026-09-15T08:28:06.427Z" }, + { url = "https://files.pythonhosted.org/packages/56/6a/cebf0f669ed69714197a2a9c8263ab333b4e7841f3447d593b2548332783/common_expression_language-0.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ef348991adf7df54cee0a2c1a642d5e4e2aa0239bb844719c6454e08d6bd1eaf", size = 1844978, upload-time = "2026-09-15T08:28:07.901Z" }, + { url = "https://files.pythonhosted.org/packages/07/1a/c151ba83f3308bfb3a8c66cb924ad2c660db35f36a25d400c6242febc6c3/common_expression_language-0.10.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ecc5b78d8136a4728667fc3f67aef8725993ab798a481cb7d615f895ebfd8428", size = 1767037, upload-time = "2026-09-15T08:28:09.335Z" }, + { url = "https://files.pythonhosted.org/packages/66/70/85045d857ced6637d4835d4e3c982ef4356d2864d816b8bb0d80c319ee21/common_expression_language-0.10.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:db90be33fc7918c3377dae8c862a3c150a285acbf29724197f0e53c5890c7d95", size = 2073349, upload-time = "2026-09-15T08:28:11.08Z" }, + { url = "https://files.pythonhosted.org/packages/5b/aa/83f1987e0545f68970f9385560c8d2622cba47321635d02d815077e0b44f/common_expression_language-0.10.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:943b6881841446beea827355ce10ef4594b66bd39d73e269b76edb696443395c", size = 1920583, upload-time = "2026-09-15T08:28:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/2c/c7/b16e80005750b2edf185a5e9f0913a1e63b96a4a16b801c19012d99ac7b8/common_expression_language-0.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0fe7d794056d99d5fca4fb7d99f640b2311a84d850e498507a4edd67244a79af", size = 1803408, upload-time = "2026-09-15T08:28:14.253Z" }, + { url = "https://files.pythonhosted.org/packages/ff/57/cbc74020da716b6439b270140229e8c691f1d35db22f23cb1e440840d485/common_expression_language-0.10.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:71bfd04a5011a51334a2d40b552942fdd41af2443c77a87aabd1842f5ea7d2fd", size = 1923154, upload-time = "2026-09-15T08:28:15.701Z" }, + { url = "https://files.pythonhosted.org/packages/97/97/4c191f773b3869e9cc9cdb242d0e142a6de0a48c7b0fd760675d64fed7fe/common_expression_language-0.10.0-cp311-cp311-win32.whl", hash = "sha256:5164b2b49b4f8e50cab9b4192fe6df75fe027d01f7f3c664b03b255f07fd431c", size = 1258504, upload-time = "2026-09-15T08:28:17.262Z" }, + { url = "https://files.pythonhosted.org/packages/5b/f5/900e87fb2b0a9c1e7e787a698c906b4fdf06e124758202944734a6382ccf/common_expression_language-0.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:c2484b50b2d6a8fa51b188d14b1c24555636564bd0045d2be1d0958b2a761908", size = 1427159, upload-time = "2026-09-15T08:28:18.803Z" }, + { url = "https://files.pythonhosted.org/packages/05/28/26d66e35380978baa6be4489ee9efe6fc1bbd4f034e8070951f1b0a06aa0/common_expression_language-0.10.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:ee7359e11da1c057cededf5c332f99c71c94a4e8fe7cf3fe14e9726479d2b6bf", size = 1595104, upload-time = "2026-09-15T08:28:20.177Z" }, + { url = "https://files.pythonhosted.org/packages/22/7a/a0ccb567968e762fdb2a8e79dac99167b6bad69b4568f1704e96699a8c1b/common_expression_language-0.10.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1def298c20eb1fa91a87f422cc2bf2a7568619ae56acae6b15609b2a29b5f66e", size = 1549075, upload-time = "2026-09-15T08:28:21.546Z" }, + { url = "https://files.pythonhosted.org/packages/d1/bf/02d7c5970bf79088163f5b628aca61e49a67fc9be65041f659e0022501bf/common_expression_language-0.10.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bd3f27cc10659267d6e74c204a74486f760e068b9352045303816b25e5bf06e7", size = 1840541, upload-time = "2026-09-15T08:28:23.093Z" }, + { url = "https://files.pythonhosted.org/packages/94/b7/e74982608079f46767f5a58cc0f932b4b157fa6e602c7072af49d150c567/common_expression_language-0.10.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:114663998d58a7d1e0a4a02c0e056304ae84acd4e52e1171e466abdd6279f526", size = 1764368, upload-time = "2026-09-15T08:28:24.798Z" }, + { url = "https://files.pythonhosted.org/packages/8f/2d/c299e7b6a93700e413f65e3597a3385a93a733f75b397857717b7cf7a267/common_expression_language-0.10.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:641c4a0eec9c89cee738f667e9d4544f564f761524ce3e4f57b178f4fbfa7784", size = 2069833, upload-time = "2026-09-15T08:28:26.289Z" }, + { url = "https://files.pythonhosted.org/packages/4e/68/4d2054ee8e218f87e64584c9e6e4357637efbe84a41e531b00708c01497a/common_expression_language-0.10.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4785d1c01cc8f786ff59ea26acee6f2b02d1a2e89e8d818daad4921de1e0dfed", size = 1918698, upload-time = "2026-09-15T08:28:27.628Z" }, + { url = "https://files.pythonhosted.org/packages/89/8d/70196c21038da4df0877362c6d6c293e2ce2cd6ef726920cf6415040068d/common_expression_language-0.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f8a18a28c53fa3c4fbb6ba46e50e66cbd555b3a5153920481dc4643509a6ad99", size = 1800127, upload-time = "2026-09-15T08:28:28.947Z" }, + { url = "https://files.pythonhosted.org/packages/87/b6/adc76f29cf63e23f12b5c7f269b19b029a0298d19ac4f75cdc4c6c6f08df/common_expression_language-0.10.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a6ea367d75be0ec7850f8ddbb8b540772a7f0da02625940c98f82de011379761", size = 1917063, upload-time = "2026-09-15T08:28:30.257Z" }, + { url = "https://files.pythonhosted.org/packages/30/f4/b010c969af49f0e4d0bd61595b0fb8e5a94bdd7a1ae49df7c2c6803cf153/common_expression_language-0.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:0d8c0b951675fc608eff2baa6c3c9f1d49ef3f014f6202ddfb25fb6ff9f2e228", size = 1424459, upload-time = "2026-09-15T08:28:31.759Z" }, + { url = "https://files.pythonhosted.org/packages/03/0c/37b561d94bb9040b64ad7caafb2fce42e595edab724f5ad995b3843b6ef2/common_expression_language-0.10.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5bc7fd00abd45769815a04dad545a28219ee973c426dc0b39aa6a8c82b97f173", size = 1594709, upload-time = "2026-09-15T08:28:33.117Z" }, + { url = "https://files.pythonhosted.org/packages/39/db/8f0dbbc7e566c233a112109992a093f7612150e459a66be5fa1609ba3dce/common_expression_language-0.10.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2c72d3b4da0ec5921df0d34e0ac8356d7eb466798327f5407c6daeffcf52c315", size = 1548426, upload-time = "2026-09-15T08:28:34.698Z" }, + { url = "https://files.pythonhosted.org/packages/a9/b1/4156b82d05335aea2e28e88f72aed90a31028a26f884066b521d2878a34c/common_expression_language-0.10.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:41c843d3d170c516520e95cfaca7a48df4a07d1c8c16fa9dd76f945b726a7095", size = 1839560, upload-time = "2026-09-15T08:28:36.246Z" }, + { url = "https://files.pythonhosted.org/packages/df/8f/522429743d7493b70b5f84a7301edd2d24724241b118a7df05523bf366fc/common_expression_language-0.10.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:346f43e55fe951de32ac7f5f9d7db22ac2d75fb7fe98afa27e004ee7f369e933", size = 1763815, upload-time = "2026-09-15T08:28:37.628Z" }, + { url = "https://files.pythonhosted.org/packages/7b/da/c4479957213f9816d3cfab13a5088721ca6964abe186a5ea650a13556076/common_expression_language-0.10.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:84c280e5d72f298c212f76530b76cde9b65bdf22cee7578f169c0aed2067b11b", size = 2067252, upload-time = "2026-09-15T08:28:39.335Z" }, + { url = "https://files.pythonhosted.org/packages/86/5c/8ba2db09a13733153d3cb1ab1e77416146df6b65796f08ac79650651dabe/common_expression_language-0.10.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:18bfe13195a507619129c34bfe9915c8553d272369416907733475567c72b51f", size = 1917727, upload-time = "2026-09-15T08:28:40.656Z" }, + { url = "https://files.pythonhosted.org/packages/27/aa/cceaef9b6b83ee0f29ecd008fb31cff5e10a667d4ab904acbf4564f8ad60/common_expression_language-0.10.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd7d7dc3168c6651a8436c2d83495f450ddc374326d2a3023097e83ddec505d5", size = 1799153, upload-time = "2026-09-15T08:28:42.188Z" }, + { url = "https://files.pythonhosted.org/packages/38/52/f6db15aaba2140de93be70556e58140cac1e2848512b536c5069873aeb92/common_expression_language-0.10.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:72b8b9893b0a6d09cf0eec98407cf36d7f32a9c53d22b5a0567e1b6c56f36a79", size = 1916393, upload-time = "2026-09-15T08:28:43.572Z" }, + { url = "https://files.pythonhosted.org/packages/68/8c/1ce2e42efe27d72a76c1eeaae21cd3c917e950732c07667a3f395ed3509b/common_expression_language-0.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:fe5192baab821a5f177df395b59b5ff55bce679944801dbc90855b32b39e06bb", size = 1423316, upload-time = "2026-09-15T08:28:45.032Z" }, + { url = "https://files.pythonhosted.org/packages/71/99/2d9627c5fc0e67a24da539b5caa03163e9c9183c41f1f21a4533c689de1a/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7968993ff66125a217abef387b8242d581117b97ed5857ff4e5784bce1a22aa1", size = 1849056, upload-time = "2026-09-15T08:29:14.646Z" }, + { url = "https://files.pythonhosted.org/packages/28/12/9e8d5f88f9cf4edfd6dbfd4a5d5bd3a6007b5b55a738556f58d61c1b830d/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2b0a603883c839fafbe1fe3a680dfb79184b4c189934137667be18331e8ec230", size = 1770768, upload-time = "2026-09-15T08:29:16.005Z" }, + { url = "https://files.pythonhosted.org/packages/8f/f6/caeebc701ae1fd42765ada89991939ce20e775747944de8bb64bd50679dc/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8b0e895747aaf19d1adecbcbf51949ab609a295464b11f9f4fbd63099b3e2fec", size = 2074479, upload-time = "2026-09-15T08:29:17.551Z" }, + { url = "https://files.pythonhosted.org/packages/73/1b/da2c6154935a0a3d7bacfa8afc64efeae32d09ad8fbf5bb1d0647c036e9d/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cd91c42eb839dd564bba9cba056c06be3dfee0eed494da6210d178f66108a773", size = 1921931, upload-time = "2026-09-15T08:29:19.107Z" }, + { url = "https://files.pythonhosted.org/packages/85/ff/02efa317a1c37b9c7cdeb457bd3b4d6029791279897ddb31d0a4a0a63609/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7ea8ca4d6d40beaee16e0c7302666c6deb26050ae05b53fe0b3fb51c3c49588e", size = 1807004, upload-time = "2026-09-15T08:29:20.465Z" }, + { url = "https://files.pythonhosted.org/packages/8b/28/c261b3e98875163ec29da831c27cc5487e9c912cc5e3a4fbd4e990b46c23/common_expression_language-0.10.0-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:21ba14368bbab920b768c4d5d5643e21a46f80a1a660c677dd2490ceab694923", size = 1925318, upload-time = "2026-09-15T08:29:21.843Z" }, +] + [[package]] name = "confection" version = "1.3.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ca/65/efd0fe8a936fc8ca2978cb7b82581fb20d901c6039e746a808f746b7647b/confection-1.3.3.tar.gz", hash = "sha256:f0f6810d567ff73993fe74d218ca5e1ffb6a44fb03f391257fc5d033546cbfaa", size = 54895, upload-time = "2026-03-24T18:45:24.331Z" } wheels = [ @@ -821,7 +869,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -1427,7 +1475,7 @@ name = "humanfriendly" version = "10.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "pyreadline3", marker = "sys_platform == 'win32'" }, + { name = "pyreadline3", marker = "python_full_version < '3.11' and sys_platform == 'win32'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cc/3f/2c29224acb2e2df4d2046e4c73ee2662023c58ff5b113c4c1adac0886c43/humanfriendly-10.0.tar.gz", hash = "sha256:6b0b831ce8f15f7300721aa49829fc4e83921a9a301cc7f606be6686a2288ddc", size = 360702, upload-time = "2021-09-17T21:40:43.31Z" } wheels = [ @@ -1647,13 +1695,13 @@ name = "langchain-classic" version = "1.0.8" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "langchain-core" }, - { name = "langchain-text-splitters" }, - { name = "langsmith" }, - { name = "pydantic" }, - { name = "pyyaml" }, - { name = "requests" }, - { name = "sqlalchemy" }, + { name = "langchain-core", marker = "python_full_version >= '3.11'" }, + { name = "langchain-text-splitters", marker = "python_full_version >= '3.11'" }, + { name = "langsmith", marker = "python_full_version >= '3.11'" }, + { name = "pydantic", marker = "python_full_version >= '3.11'" }, + { name = "pyyaml", marker = "python_full_version >= '3.11'" }, + { name = "requests", marker = "python_full_version >= '3.11'" }, + { name = "sqlalchemy", marker = "python_full_version >= '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/8d/65/6b5e8a7ff2f2968652c88a67dcecb925b9d8f0a0ce9458c76cd5a0dbd138/langchain_classic-1.0.8.tar.gz", hash = "sha256:ada0cc341a8a5b80fb24d73bdfaaeb849056ee2d8a41cc468355163fd3667484", size = 10557071, upload-time = "2026-06-10T21:27:54.866Z" } wheels = [ @@ -1668,18 +1716,18 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "aiohttp" }, - { name = "dataclasses-json" }, - { name = "httpx-sse" }, - { name = "langchain" }, - { name = "langchain-core" }, - { name = "langsmith" }, - { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" } }, - { name = "pydantic-settings" }, - { name = "pyyaml" }, - { name = "requests" }, - { name = "sqlalchemy" }, - { name = "tenacity" }, + { name = "aiohttp", marker = "python_full_version < '3.11'" }, + { name = "dataclasses-json", marker = "python_full_version < '3.11'" }, + { name = "httpx-sse", marker = "python_full_version < '3.11'" }, + { name = "langchain", marker = "python_full_version < '3.11'" }, + { name = "langchain-core", marker = "python_full_version < '3.11'" }, + { name = "langsmith", marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "pydantic-settings", marker = "python_full_version < '3.11'" }, + { name = "pyyaml", marker = "python_full_version < '3.11'" }, + { name = "requests", marker = "python_full_version < '3.11'" }, + { name = "sqlalchemy", marker = "python_full_version < '3.11'" }, + { name = "tenacity", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/83/49/2ff5354273809e9811392bc24bcffda545a196070666aef27bc6aacf1c21/langchain_community-0.3.31.tar.gz", hash = "sha256:250e4c1041539130f6d6ac6f9386cb018354eafccd917b01a4cff1950b80fd81", size = 33241237, upload-time = "2025-10-07T20:17:57.857Z" } wheels = [ @@ -1696,18 +1744,18 @@ resolution-markers = [ "python_full_version == '3.11.*'", ] dependencies = [ - { name = "aiohttp" }, - { name = "httpx-sse" }, - { name = "langchain-classic" }, - { name = "langchain-core" }, - { name = "langsmith" }, - { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "aiohttp", marker = "python_full_version >= '3.11'" }, + { name = "httpx-sse", marker = "python_full_version >= '3.11'" }, + { name = "langchain-classic", marker = "python_full_version >= '3.11'" }, + { name = "langchain-core", marker = "python_full_version >= '3.11'" }, + { name = "langsmith", marker = "python_full_version >= '3.11'" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" }, { name = "numpy", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "pydantic-settings" }, - { name = "pyyaml" }, - { name = "requests" }, - { name = "sqlalchemy" }, - { name = "tenacity" }, + { name = "pydantic-settings", marker = "python_full_version >= '3.11'" }, + { name = "pyyaml", marker = "python_full_version >= '3.11'" }, + { name = "requests", marker = "python_full_version >= '3.11'" }, + { name = "sqlalchemy", marker = "python_full_version >= '3.11'" }, + { name = "tenacity", marker = "python_full_version >= '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ea/0c/e3aca1f2b1c5b95f8b87cb2b6e81a6f20d538c07a128419dc01cef0617b6/langchain_community-0.4.2.tar.gz", hash = "sha256:a99308160d53d7e9b5965ee665e5173709914338210089fd5788ad724432c21e", size = 33268708, upload-time = "2026-05-22T19:42:59.374Z" } wheels = [ @@ -1780,7 +1828,7 @@ name = "langchain-text-splitters" version = "1.1.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "langchain-core" }, + { name = "langchain-core", marker = "python_full_version >= '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/26/9f/6c545900fefb7b00ddfa3f16b80d61338a0ec68c31c5451eeeab99082760/langchain_text_splitters-1.1.2.tar.gz", hash = "sha256:782a723db0a4746ac91e251c7c1d57fd23636e4f38ed733074e28d7a86f41627", size = 293580, upload-time = "2026-04-16T14:20:39.162Z" } wheels = [ @@ -2322,6 +2370,7 @@ dependencies = [ all = [ { name = "aiofiles" }, { name = "chainlit" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11'" }, { name = "fast-langdetect" }, { name = "fastapi" }, { name = "google-cloud-language" }, @@ -2344,6 +2393,9 @@ all = [ { name = "watchdog" }, { name = "yara-python" }, ] +cel = [ + { name = "common-expression-language", marker = "python_full_version >= '3.11'" }, +] chat-ui = [ { name = "chainlit" }, ] @@ -2390,6 +2442,7 @@ tracing = [ dev = [ { name = "aioresponses" }, { name = "chainlit" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11'" }, { name = "fast-langdetect" }, { name = "fastapi" }, { name = "inline-snapshot" }, @@ -2436,6 +2489,7 @@ test = [ test-integration = [ { name = "aioresponses" }, { name = "chainlit" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11'" }, { name = "fast-langdetect" }, { name = "fastapi" }, { name = "langchain" }, @@ -2468,6 +2522,8 @@ requires-dist = [ { name = "aiohttp-retry", specifier = ">=2.9.0" }, { name = "chainlit", marker = "extra == 'all'", specifier = ">=2.11.0,<3.0.0" }, { name = "chainlit", marker = "extra == 'chat-ui'", specifier = ">=2.11.0,<3.0.0" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11' and extra == 'all'", specifier = ">=0.10.0,<0.11.0" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11' and extra == 'cel'", specifier = ">=0.10.0,<0.11.0" }, { name = "dataclasses-json", specifier = ">=0.6.7,<0.7.0" }, { name = "fast-langdetect", marker = "extra == 'all'", specifier = ">=1.0.0" }, { name = "fast-langdetect", marker = "extra == 'multilingual'", specifier = ">=1.0.0" }, @@ -2531,12 +2587,13 @@ requires-dist = [ { name = "yara-python", marker = "extra == 'all'", specifier = ">=4.5.1,<5.0.0" }, { name = "yara-python", marker = "extra == 'jailbreak'", specifier = ">=4.5.1,<5.0.0" }, ] -provides-extras = ["tracing", "server", "sdd", "gcp", "jailbreak", "multilingual", "chat-ui", "eval", "all"] +provides-extras = ["tracing", "server", "sdd", "gcp", "jailbreak", "cel", "multilingual", "chat-ui", "eval", "all"] [package.metadata.requires-dev] dev = [ { name = "aioresponses", specifier = ">=0.7.6" }, { name = "chainlit", specifier = ">=2.11.0,<3.0.0" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11'", specifier = ">=0.10.0,<0.11.0" }, { name = "fast-langdetect", specifier = ">=1.0.0" }, { name = "fastapi", specifier = ">=0.103.0" }, { name = "inline-snapshot", specifier = ">=0.33.0,<0.34.0" }, @@ -2580,6 +2637,7 @@ test = [ test-integration = [ { name = "aioresponses", specifier = ">=0.7.6" }, { name = "chainlit", specifier = ">=2.11.0,<3.0.0" }, + { name = "common-expression-language", marker = "python_full_version >= '3.11'", specifier = ">=0.10.0,<0.11.0" }, { name = "fast-langdetect", specifier = ">=1.0.0" }, { name = "fastapi", specifier = ">=0.103.0" }, { name = "langchain", specifier = ">=0.2.14,<2.0.0" }, @@ -2786,12 +2844,12 @@ resolution-markers = [ "python_full_version < '3.11'", ] dependencies = [ - { name = "coloredlogs" }, - { name = "flatbuffers" }, - { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" } }, - { name = "packaging" }, - { name = "protobuf" }, - { name = "sympy" }, + { name = "coloredlogs", marker = "python_full_version < '3.11'" }, + { name = "flatbuffers", marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "packaging", marker = "python_full_version < '3.11'" }, + { name = "protobuf", marker = "python_full_version < '3.11'" }, + { name = "sympy", marker = "python_full_version < '3.11'" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/35/d6/311b1afea060015b56c742f3531168c1644650767f27ef40062569960587/onnxruntime-1.23.2-cp310-cp310-macosx_13_0_arm64.whl", hash = "sha256:a7730122afe186a784660f6ec5807138bf9d792fa1df76556b27307ea9ebcbe3", size = 17195934, upload-time = "2025-10-27T23:06:14.143Z" }, @@ -2828,11 +2886,11 @@ resolution-markers = [ "python_full_version == '3.11.*'", ] dependencies = [ - { name = "flatbuffers" }, - { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "flatbuffers", marker = "python_full_version >= '3.11'" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" }, { name = "numpy", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "packaging" }, - { name = "protobuf" }, + { name = "packaging", marker = "python_full_version >= '3.11'" }, + { name = "protobuf", marker = "python_full_version >= '3.11'" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/d4/e4/5353d7e09ced4a8f473f843223fc75d726b2b5519dcefc12f22a6c92852d/onnxruntime-1.27.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:8ba14a38c570087f3cdb8cfba33f7a38a1e826c1e5b29e17c28ceda0cc910016", size = 18416484, upload-time = "2026-06-15T22:43:43.894Z" }, @@ -3862,8 +3920,8 @@ name = "preshed" version = "3.0.13" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cymem" }, - { name = "murmurhash" }, + { name = "cymem", marker = "python_full_version < '3.13'" }, + { name = "murmurhash", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/43/75/fe6b7bbd0dea530a001b0e24c331b21a0be2786e402abf3c57f5dce43d4b/preshed-3.0.13.tar.gz", hash = "sha256:d75f718bbfd97e992f7827e0fa7faf6a91bdd9c922d5baa4b50d62731396cb89", size = 18338, upload-time = "2026-03-23T08:57:31.378Z" } wheels = [ @@ -3906,12 +3964,12 @@ name = "presidio-analyzer" version = "2.2.362" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "phonenumbers" }, - { name = "pydantic" }, - { name = "pyyaml" }, - { name = "regex" }, - { name = "spacy" }, - { name = "tldextract" }, + { name = "phonenumbers", marker = "python_full_version < '3.13'" }, + { name = "pydantic", marker = "python_full_version < '3.13'" }, + { name = "pyyaml", marker = "python_full_version < '3.13'" }, + { name = "regex", marker = "python_full_version < '3.13'" }, + { name = "spacy", marker = "python_full_version < '3.13'" }, + { name = "tldextract", marker = "python_full_version < '3.13'" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/2f/8a/d6cf4bddbb11d9c78f5c9342bbbcde4f90fe4e3c7de114a836ec4ed8a6cf/presidio_analyzer-2.2.362-py3-none-any.whl", hash = "sha256:4c36438924b1fcb4df92ea5cf2d8dc57508808e116b10923c983b8732aa07d90", size = 201084, upload-time = "2026-03-15T12:40:43.801Z" }, @@ -3922,7 +3980,7 @@ name = "presidio-anonymizer" version = "2.2.362" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cryptography" }, + { name = "cryptography", marker = "python_full_version < '3.13'" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/17/01/1c3404e8da63a979ae951b8822576bc0e52ca3a46a624ac9439786990c6d/presidio_anonymizer-2.2.362-py3-none-any.whl", hash = "sha256:b2d81542cb797360d18506a1aa73ea0dfacc44abca15248c81dd062f582e6b2b", size = 36603, upload-time = "2026-03-15T12:40:38.651Z" }, @@ -4725,7 +4783,7 @@ name = "requests-file" version = "3.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "requests" }, + { name = "requests", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/3c/f8/5dc70102e4d337063452c82e1f0d95e39abfe67aa222ed8a5ddeb9df8de8/requests_file-3.0.1.tar.gz", hash = "sha256:f14243d7796c588f3521bd423c5dea2ee4cc730e54a3cac9574d78aca1272576", size = 6967, upload-time = "2025-10-20T18:56:42.279Z" } wheels = [ @@ -5016,7 +5074,7 @@ name = "smart-open" version = "8.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "wrapt" }, + { name = "wrapt", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/29/3e/79fd5fd2375a8a500b9ec2f6a0762fc1ac33e35582d4a87483a78d19408f/smart_open-8.0.0.tar.gz", hash = "sha256:5a2008d60688bd3b33c52e2ef666d3c60cf956e73e215de8c7b242cf56fdd1b2", size = 61520, upload-time = "2026-06-27T16:28:11.894Z" } wheels = [ @@ -5046,27 +5104,27 @@ name = "spacy" version = "3.8.14" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "catalogue" }, - { name = "confection" }, - { name = "cymem" }, - { name = "jinja2" }, - { name = "murmurhash" }, - { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, + { name = "catalogue", marker = "python_full_version < '3.13'" }, + { name = "confection", marker = "python_full_version < '3.13'" }, + { name = "cymem", marker = "python_full_version < '3.13'" }, + { name = "jinja2", marker = "python_full_version < '3.13'" }, + { name = "murmurhash", marker = "python_full_version < '3.13'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" }, { name = "numpy", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.12.*'" }, - { name = "packaging" }, - { name = "preshed" }, - { name = "pydantic" }, - { name = "requests" }, - { name = "setuptools" }, - { name = "spacy-legacy" }, - { name = "spacy-loggers" }, - { name = "srsly" }, - { name = "thinc" }, - { name = "tqdm" }, - { name = "typer" }, - { name = "wasabi" }, - { name = "weasel" }, + { name = "packaging", marker = "python_full_version < '3.13'" }, + { name = "preshed", marker = "python_full_version < '3.13'" }, + { name = "pydantic", marker = "python_full_version < '3.13'" }, + { name = "requests", marker = "python_full_version < '3.13'" }, + { name = "setuptools", marker = "python_full_version < '3.13'" }, + { name = "spacy-legacy", marker = "python_full_version < '3.13'" }, + { name = "spacy-loggers", marker = "python_full_version < '3.13'" }, + { name = "srsly", marker = "python_full_version < '3.13'" }, + { name = "thinc", marker = "python_full_version < '3.13'" }, + { name = "tqdm", marker = "python_full_version < '3.13'" }, + { name = "typer", marker = "python_full_version < '3.13'" }, + { name = "wasabi", marker = "python_full_version < '3.13'" }, + { name = "weasel", marker = "python_full_version < '3.13'" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/4e/d5/9860449c9fbed97634fb974bbf7a128ae269f5e69f3d792a9679d2354141/spacy-3.8.14-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4a06e5cc029910eaa4a3c5a0a997f07fed6a41aba46b05da9f58643bc06fe8b9", size = 6625650, upload-time = "2026-03-29T10:40:07.13Z" }, @@ -5165,7 +5223,7 @@ name = "srsly" version = "2.5.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "catalogue" }, + { name = "catalogue", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/2b/db/f794f219a6c788b881252d2536a8c4a97d2bdaadc690391e1cb53d123d71/srsly-2.5.3.tar.gz", hash = "sha256:08f98dbecbff3a31466c4ae7c833131f59d3655a0ad8ac749e6e2c149e2b0680", size = 490881, upload-time = "2026-03-23T11:56:59.865Z" } wheels = [ @@ -5271,7 +5329,7 @@ name = "sympy" version = "1.14.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "mpmath" }, + { name = "mpmath", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" } wheels = [ @@ -5298,20 +5356,20 @@ name = "thinc" version = "8.3.13" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "blis" }, - { name = "catalogue" }, - { name = "confection" }, - { name = "cymem" }, - { name = "murmurhash" }, - { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or python_full_version >= '3.13'" }, + { name = "blis", marker = "python_full_version < '3.13'" }, + { name = "catalogue", marker = "python_full_version < '3.13'" }, + { name = "confection", marker = "python_full_version < '3.13'" }, + { name = "cymem", marker = "python_full_version < '3.13'" }, + { name = "murmurhash", marker = "python_full_version < '3.13'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.11.*'" }, { name = "numpy", version = "2.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.12.*'" }, - { name = "packaging" }, - { name = "preshed" }, - { name = "pydantic" }, - { name = "setuptools" }, - { name = "srsly" }, - { name = "wasabi" }, + { name = "packaging", marker = "python_full_version < '3.13'" }, + { name = "preshed", marker = "python_full_version < '3.13'" }, + { name = "pydantic", marker = "python_full_version < '3.13'" }, + { name = "setuptools", marker = "python_full_version < '3.13'" }, + { name = "srsly", marker = "python_full_version < '3.13'" }, + { name = "wasabi", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/13/46/76df95f2c327f9a9cef30c1523bf285627897097163584dcf5f77b2ebce2/thinc-8.3.13.tar.gz", hash = "sha256:68e658549fc1eb3ff92aed5147fcbb9c15d6e9cc0e623b4d0998d16522ffb4f9", size = 194640, upload-time = "2026-03-23T07:22:36.41Z" } wheels = [ @@ -5400,10 +5458,10 @@ name = "tldextract" version = "5.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "filelock" }, - { name = "idna" }, - { name = "requests" }, - { name = "requests-file" }, + { name = "filelock", marker = "python_full_version < '3.13'" }, + { name = "idna", marker = "python_full_version < '3.13'" }, + { name = "requests", marker = "python_full_version < '3.13'" }, + { name = "requests-file", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/65/7b/644fbbb49564a6cb124a8582013315a41148dba2f72209bba14a84242bf0/tldextract-5.3.1.tar.gz", hash = "sha256:a72756ca170b2510315076383ea2993478f7da6f897eef1f4a5400735d5057fb", size = 126105, upload-time = "2025-12-28T23:58:05.532Z" } wheels = [ @@ -5787,7 +5845,7 @@ name = "wasabi" version = "1.1.3" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "colorama", marker = "python_full_version < '3.13' and sys_platform == 'win32'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ac/f9/054e6e2f1071e963b5e746b48d1e3727470b2a490834d18ad92364929db3/wasabi-1.1.3.tar.gz", hash = "sha256:4bb3008f003809db0c3e28b4daf20906ea871a2bb43f9914197d540f4f2e0878", size = 30391, upload-time = "2024-05-31T16:56:18.99Z" } wheels = [ @@ -5921,15 +5979,15 @@ name = "weasel" version = "1.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "cloudpathlib" }, - { name = "confection" }, - { name = "httpx" }, - { name = "packaging" }, - { name = "pydantic" }, - { name = "smart-open" }, - { name = "srsly" }, - { name = "typer" }, - { name = "wasabi" }, + { name = "cloudpathlib", marker = "python_full_version < '3.13'" }, + { name = "confection", marker = "python_full_version < '3.13'" }, + { name = "httpx", marker = "python_full_version < '3.13'" }, + { name = "packaging", marker = "python_full_version < '3.13'" }, + { name = "pydantic", marker = "python_full_version < '3.13'" }, + { name = "smart-open", marker = "python_full_version < '3.13'" }, + { name = "srsly", marker = "python_full_version < '3.13'" }, + { name = "typer", marker = "python_full_version < '3.13'" }, + { name = "wasabi", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ce/e5/e272bb9a045105a1fdf4b798d8086f5932a178f4d738f17a74f5c9e0ae9a/weasel-1.0.0.tar.gz", hash = "sha256:7b129b44c90cc543b760532974ca1e4eb30dad2aa2026f57bdce66354ae610fc", size = 38682, upload-time = "2026-03-20T08:10:25.266Z" } wheels = [ From 0b6203c938c6a47e33f0b2f66eabdcca15b9489c Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Fri, 2 Oct 2026 16:39:38 -0700 Subject: [PATCH 2/6] fix: fail closed on any CEL expression failure and address review comments --- .../configure-rails/guardrail-catalog/cel.mdx | 32 +++++++++--- .../guardrail-catalog/tool-safety-check.mdx | 2 +- nemoguardrails/library/cel/actions.py | 20 ++++--- tests/test_cel_actions.py | 52 +++++++++++++++++-- 4 files changed, 87 insertions(+), 19 deletions(-) diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx index 07e1610777..85d2739223 100644 --- a/docs/configure-rails/guardrail-catalog/cel.mdx +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -35,8 +35,11 @@ To use the CEL rail, you should: tool_output: run_shell: expressions: - - 'args.command.matches("(?i)\\brm\\s+-[a-z]*r[a-z]*f")' - - 'has(args.cwd) && args.cwd.startsWith("/etc")' + - 'args.command.matches("(?i)\\brm\\s+-[a-z]*(r[a-z]*f|f[a-z]*r)")' + write_file: + expressions: + - 'args.path.startsWith("/etc/") || args.path.startsWith("/root/")' + - 'has(args.mode) && args.mode == "append" && args.path.endsWith(".bashrc")' tool_input: run_shell: expressions: @@ -51,6 +54,8 @@ To use the CEL rail, you should: per_tool: run_shell: - "cel check tool output" + write_file: + - "cel check tool output" tool_input: per_tool: run_shell: @@ -59,7 +64,7 @@ To use the CEL rail, you should: The `cel check tool output` flow evaluates the tool's `tool_output` expressions against each call to the tool, and the `cel check tool input` flow evaluates its `tool_input` expressions against each result the tool returns. A tool with no expressions configured for a direction is allowed. -3. Declare the tool, either in the main model's `parameters.tools` or per request in `options.llm_params.tools`. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. +3. Declare the tool in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. The `cel check tool output` and `cel check tool input` rails execute the [`evaluate_tool_output_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) and [`evaluate_tool_input_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) actions respectively. @@ -85,7 +90,7 @@ The rail evaluates every expression configured for the tool: - If any expression evaluates to `true`, the rail blocks the tool call or result. The block lists the expressions that matched in its `matched_expressions` metadata. The metadata never includes the checked arguments or content. - If no expression evaluates to `true`, the rail allows the tool call or result. -- If an expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail logs a warning and continues with the remaining expressions. A match from another expression still blocks. If no expression matched, the rail fails closed: the tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed and why. +- If any expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail fails closed, even if another expression matched. The tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed and why. ## Write Expressions @@ -93,7 +98,7 @@ Expressions use [CEL syntax](https://github.com/google/cel-spec/blob/master/doc/ When you write expressions, keep the following in mind: -- Guard optional arguments with `has()`, for example `has(args.cwd) && args.cwd.startsWith("/etc")`. Without the guard, a call that omits the argument fails to evaluate. +- Guard optional arguments with `has()`, for example `has(args.mode) && args.mode == "append"`. Without the guard, a call that omits the argument fails to evaluate. - If a tool can return either a string or a list of content blocks, check the type first, for example `type(result) == string && result.contains("password")`. - Each expression must evaluate to a boolean. An expression such as `args.command` that returns a string fails to evaluate. @@ -101,7 +106,7 @@ The rail checks each expression's syntax when the configuration loads, so a synt ## Complete Example -The following configuration blocks recursive forced deletes and writes under `/etc` in a `run_shell` tool, and blocks any result that contains a private key: +The following configuration blocks recursive forced deletes in `run_shell`, writes to `/etc` or `/root` and appends to `.bashrc` through `write_file`, and any `run_shell` result that contains a private key: ```yaml models: @@ -115,8 +120,11 @@ rails: tool_output: run_shell: expressions: - - 'args.command.matches("(?i)\\brm\\s+-[a-z]*r[a-z]*f")' - - 'has(args.cwd) && args.cwd.startsWith("/etc")' + - 'args.command.matches("(?i)\\brm\\s+-[a-z]*(r[a-z]*f|f[a-z]*r)")' + write_file: + expressions: + - 'args.path.startsWith("/etc/") || args.path.startsWith("/root/")' + - 'has(args.mode) && args.mode == "append" && args.path.endsWith(".bashrc")' tool_input: run_shell: expressions: @@ -125,12 +133,20 @@ rails: per_tool: run_shell: - "cel check tool output" + write_file: + - "cel check tool output" tool_input: per_tool: run_shell: - "cel check tool input" ``` + + +This configuration doesn't declare `run_shell` or `write_file`. Declare them in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. Otherwise `cel check tool output` blocks every call to them before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the declaration format. + + + ## Schema Validation `cel check tool output` runs behind `@tool_output_validation`: a call to a tool the model has not declared, or whose arguments violate the tool's declared JSON schema, is blocked before any expression is evaluated. `cel check tool input` has no such gate, matching regex detection's asymmetry between the two directions. diff --git a/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx b/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx index 14226f1de7..dd82127621 100644 --- a/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx +++ b/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx @@ -29,7 +29,7 @@ To use the tool safety check, you should: - The `type` is a unique identifier for the model that will be passed to the tool output and tool input rails as a parameter. `tools: [...]` above is a placeholder: replace it with `run_sql`'s actual function declaration and JSON schema, either here (static, applies to every request) or per request in `options.llm_params.tools`. `@tool_output_validation` blocks a call to a tool that isn't declared either way, before the judge ever runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. + The `type` is a unique identifier for the model that will be passed to the tool output and tool input rails as a parameter. `tools: [...]` above is a placeholder: replace it with `run_sql`'s actual function declaration and JSON schema, either here (static, applies to every request) or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. `@tool_output_validation` blocks a call to a tool that isn't declared either way, before the judge ever runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. diff --git a/nemoguardrails/library/cel/actions.py b/nemoguardrails/library/cel/actions.py index 4c47621598..3a32148447 100644 --- a/nemoguardrails/library/cel/actions.py +++ b/nemoguardrails/library/cel/actions.py @@ -26,11 +26,19 @@ log = logging.getLogger(__name__) +def _tool_options(config: RailsConfig, source: str, tool_name: str) -> Optional[CelOptions]: + """Return the tool's CEL options for *source*, or None when none are configured.""" + cel_config = config.rails.config.cel + if cel_config is None: + return None + return getattr(cel_config, source).get(tool_name) + + def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: Optional[CelOptions]) -> RailOutcome: """Evaluate all of a tool's CEL expressions, blocking when any is true. - A match blocks even if another expression fails. With no match, a failing expression - (an error or a non-boolean result) raises, so the rail fails closed. + A failing expression (an error or a non-boolean result) raises even if another expression + matched, so the rail fails closed. """ metadata: Dict[str, Any] = {"source": source} if options is None or not options.compiled_expressions: @@ -60,11 +68,11 @@ def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: O log.info("CEL expression matched: %s", expression) matched.append(expression) - if matched: - return RailOutcome.block(metadata={**metadata, "matched_expressions": matched}) if failures: details = "; ".join(f"{expression!r}: {type(error).__name__}: {error}" for expression, error in failures) raise RuntimeError(f"CEL expressions for tool {tool_name!r} failed: {details}") + if matched: + return RailOutcome.block(metadata={**metadata, "matched_expressions": matched}) return RailOutcome.allow(metadata=metadata) @@ -94,7 +102,7 @@ async def evaluate_tool_output_cel( raise ValueError("source must be 'tool_output'") tool_name = tool_call.function.name or tool_call.type - options = getattr(config.rails.config.cel, source).get(tool_name) + options = _tool_options(config, source, tool_name) variables = {"args": tool_call.function.arguments, "tool": tool_name} return _evaluate(source, tool_name, variables, options) @@ -124,6 +132,6 @@ async def evaluate_tool_input_cel( raise ValueError("source must be 'tool_input'") tool_name = tool_call.function.name or tool_call.type - options = getattr(config.rails.config.cel, source).get(tool_name) + options = _tool_options(config, source, tool_name) variables = {"result": tool_result.content, "args": tool_call.function.arguments, "tool": tool_name} return _evaluate(source, tool_name, variables, options) diff --git a/tests/test_cel_actions.py b/tests/test_cel_actions.py index f800da9ad7..4e4aeb7236 100644 --- a/tests/test_cel_actions.py +++ b/tests/test_cel_actions.py @@ -68,6 +68,16 @@ async def _check_result(config: RailsConfig, content, call: ToolCall = None): ) +# Expressions that compile but fail on a `run_shell` call with only `command` set, keyed by case, +# with the exception type the evaluation error names. +_EVALUATION_ERRORS = { + "argument_typo": ('args.comand == "rm"', "KeyError"), + "variable_typo": ('arg.command == "rm"', "RuntimeError"), + "function_typo": ('args.command.lowerAsci() == "rm"', "RuntimeError"), + "type_mismatch": ("args.command > 5", "TypeError"), +} + + class TestConfig: def test_expressions_compile_at_load(self): config = _config("tool_output", 'args.command == "ls"', "true") @@ -77,6 +87,22 @@ def test_syntax_error_fails_the_load(self): with pytest.raises(ValueError, match=r"Invalid CEL expression at index 1 \('args.command.matches\('\)"): _config("tool_output", "true", "args.command.matches(") + @pytest.mark.parametrize( + "expression", + ['args.command == "rm', "args.command ==", "args.command && && true"], + ids=["unterminated_string", "dangling_operator", "repeated_operator"], + ) + def test_other_syntax_errors_fail_the_load(self, expression): + with pytest.raises(ValueError, match="Invalid CEL expression at index 0"): + _config("tool_output", expression) + + @pytest.mark.parametrize( + "expression", [expression for expression, _ in _EVALUATION_ERRORS.values()], ids=_EVALUATION_ERRORS.keys() + ) + def test_only_syntax_is_checked_at_load(self, expression): + """Typos and type mismatches compile, and are caught only when evaluated (see TestFailures).""" + _config("tool_output", expression) + def test_missing_extra_names_it(self, monkeypatch): monkeypatch.setitem(sys.modules, "cel", None) with pytest.raises(ImportError, match=r"nemoguardrails\[cel\]"): @@ -114,6 +140,12 @@ async def test_tool_without_expressions_allows(self): outcome = await _check_call(_config("tool_output", "true", tool="other_tool"), _shell_call(command="rm")) assert outcome.decision is RailDecision.ALLOW + @pytest.mark.asyncio + async def test_null_cel_section_allows(self): + config = RailsConfig.from_content(config={"rails": {"config": {"cel": None}}}) + outcome = await _check_call(config, _shell_call(command="rm -rf /")) + assert outcome.decision is RailDecision.ALLOW + @pytest.mark.asyncio async def test_tool_name_is_available(self): outcome = await _check_call(_config("tool_output", 'tool == "run_shell"'), _shell_call(command="ls")) @@ -153,11 +185,17 @@ async def test_wrong_source_raises(self): class TestFailures: @pytest.mark.asyncio - async def test_match_blocks_despite_a_failing_expression(self): + @pytest.mark.parametrize("expression, error_type", _EVALUATION_ERRORS.values(), ids=_EVALUATION_ERRORS.keys()) + async def test_evaluation_error_fails_closed(self, expression, error_type): + with pytest.raises(RuntimeError) as excinfo: + await _check_call(_config("tool_output", expression), _shell_call(command="rm")) + assert f"{expression!r}: {error_type}: " in str(excinfo.value) + + @pytest.mark.asyncio + async def test_failing_expression_raises_despite_a_match(self): config = _config("tool_output", 'args.flags.exists(f, f == "--force")', 'args.command == "rm"') - outcome = await _check_call(config, _shell_call(command="rm")) - assert outcome.decision is RailDecision.BLOCK - assert outcome.metadata["matched_expressions"] == ['args.command == "rm"'] + with pytest.raises(RuntimeError, match="KeyError: 'flags'"): + await _check_call(config, _shell_call(command="rm")) @pytest.mark.asyncio async def test_failures_without_a_match_raise_listing_each(self): @@ -208,6 +246,12 @@ async def test_type_guard_skips_other_shapes(self): outcome = await _check_result(config, content) assert outcome.decision is RailDecision.ALLOW + @pytest.mark.asyncio + async def test_null_cel_section_allows(self): + config = RailsConfig.from_content(config={"rails": {"config": {"cel": None}}}) + outcome = await _check_result(config, "-----BEGIN RSA PRIVATE KEY-----") + assert outcome.decision is RailDecision.ALLOW + @pytest.mark.asyncio async def test_args_and_tool_come_from_the_answered_call(self): config = _config("tool_input", 'tool == "run_shell" && args.command.endsWith(".pem")') From 5bf42f1f429c5dba58c35abe71bd558a23e23ebd Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Fri, 2 Oct 2026 16:55:36 -0700 Subject: [PATCH 3/6] docs: note the server's current tools restriction --- docs/configure-rails/guardrail-catalog/cel.mdx | 4 ++-- docs/configure-rails/guardrail-catalog/tool-safety-check.mdx | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx index 85d2739223..4cee97921b 100644 --- a/docs/configure-rails/guardrail-catalog/cel.mdx +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -64,7 +64,7 @@ To use the CEL rail, you should: The `cel check tool output` flow evaluates the tool's `tool_output` expressions against each call to the tool, and the `cel check tool input` flow evaluates its `tool_input` expressions against each result the tool returns. A tool with no expressions configured for a direction is allowed. -3. Declare the tool in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. +3. Declare the tool in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration with `passthrough: true`. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. The `cel check tool output` and `cel check tool input` rails execute the [`evaluate_tool_output_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) and [`evaluate_tool_input_cel`](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/library/cel/actions.py) actions respectively. @@ -143,7 +143,7 @@ rails: -This configuration doesn't declare `run_shell` or `write_file`. Declare them in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. Otherwise `cel check tool output` blocks every call to them before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the declaration format. +This configuration doesn't declare `run_shell` or `write_file`. Declare them in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration with `passthrough: true`. Otherwise `cel check tool output` blocks every call to them before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the declaration format. diff --git a/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx b/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx index dd82127621..4fc716a908 100644 --- a/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx +++ b/docs/configure-rails/guardrail-catalog/tool-safety-check.mdx @@ -29,7 +29,7 @@ To use the tool safety check, you should: - The `type` is a unique identifier for the model that will be passed to the tool output and tool input rails as a parameter. `tools: [...]` above is a placeholder: replace it with `run_sql`'s actual function declaration and JSON schema, either here (static, applies to every request) or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server. `@tool_output_validation` blocks a call to a tool that isn't declared either way, before the judge ever runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. + The `type` is a unique identifier for the model that will be passed to the tool output and tool input rails as a parameter. `tools: [...]` above is a placeholder: replace it with `run_sql`'s actual function declaration and JSON schema, either here (static, applies to every request) or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration with `passthrough: true`. `@tool_output_validation` blocks a call to a tool that isn't declared either way, before the judge ever runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference. From 9990c645e6a116c0be61f3612a655ce165977800 Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Fri, 9 Oct 2026 15:30:14 -0700 Subject: [PATCH 4/6] fix: address CEL review comments --- .../configure-rails/guardrail-catalog/cel.mdx | 10 ++- nemoguardrails/library/cel/actions.py | 24 +++++-- nemoguardrails/library/cel/rail_config.py | 29 ++++---- schemas/rails_config.snapshot.json | 6 ++ .../test_per_tool_cel_rails_iorails.py | 30 ++++++++- tests/test_cel_actions.py | 67 ++++++++++++------- 6 files changed, 119 insertions(+), 47 deletions(-) diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx index 4cee97921b..9a77ef037c 100644 --- a/docs/configure-rails/guardrail-catalog/cel.mdx +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -90,7 +90,7 @@ The rail evaluates every expression configured for the tool: - If any expression evaluates to `true`, the rail blocks the tool call or result. The block lists the expressions that matched in its `matched_expressions` metadata. The metadata never includes the checked arguments or content. - If no expression evaluates to `true`, the rail allows the tool call or result. -- If any expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail fails closed, even if another expression matched. The tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed and why. +- If any expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail fails closed, even if another expression matched. The tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed with its error type. ## Write Expressions @@ -102,12 +102,18 @@ When you write expressions, keep the following in mind: - If a tool can return either a string or a list of content blocks, check the type first, for example `type(result) == string && result.contains("password")`. - Each expression must evaluate to a boolean. An expression such as `args.command` that returns a string fails to evaluate. -The rail checks each expression's syntax when the configuration loads, so a syntax error fails the load. A misspelled variable, field, or function, or a type mismatch, is detected only when the expression is evaluated, so test your expressions against representative tool calls and results before deploying them. +The rail checks each expression's syntax when the configuration loads, so a syntax error fails the load. Everything else is checked when an expression is evaluated: the CEL library the rail uses resolves variable and function names only at evaluation, and `args` and `result` hold the values of each tool call or result, so whether a field exists or has the expected type is known only then. A mistake found at evaluation makes the rail fail closed, so test your expressions against representative tool calls and results before deploying them. ## Complete Example The following configuration blocks recursive forced deletes in `run_shell`, writes to `/etc` or `/root` and appends to `.bashrc` through `write_file`, and any `run_shell` result that contains a private key: + + +The expressions on this page illustrate how the rail works; they are not complete security policies. Write and test expressions for the actual threats your tools face. + + + ```yaml models: - type: main diff --git a/nemoguardrails/library/cel/actions.py b/nemoguardrails/library/cel/actions.py index 3a32148447..218acaa24e 100644 --- a/nemoguardrails/library/cel/actions.py +++ b/nemoguardrails/library/cel/actions.py @@ -13,19 +13,29 @@ # See the License for the specific language governing permissions and # limitations under the License. +import functools import logging -from typing import Any, Dict, List, Optional, Tuple +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple from nemoguardrails import RailsConfig from nemoguardrails.actions import action from nemoguardrails.actions.rail_outcome import RailOutcome from nemoguardrails.guardrails.tool_schema import Tool, ToolResult, tool_output_validation -from nemoguardrails.library.cel.rail_config import CelOptions +from nemoguardrails.library.cel.rail_config import CelOptions, compile_expression from nemoguardrails.types import ToolCall +if TYPE_CHECKING: + from cel import Program + log = logging.getLogger(__name__) +@functools.lru_cache(maxsize=1024) +def _program(expression: str) -> "Program": + """The compiled program for *expression*, cached outside the config (see CelOptions.compile_expressions).""" + return compile_expression(expression) + + def _tool_options(config: RailsConfig, source: str, tool_name: str) -> Optional[CelOptions]: """Return the tool's CEL options for *source*, or None when none are configured.""" cel_config = config.rails.config.cel @@ -41,7 +51,7 @@ def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: O matched, so the rail fails closed. """ metadata: Dict[str, Any] = {"source": source} - if options is None or not options.compiled_expressions: + if options is None: log.debug("No CEL expressions configured for tool %r under source: %s", tool_name, source) return RailOutcome.allow(metadata=metadata) @@ -55,13 +65,12 @@ def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: O matched: List[str] = [] failures: List[Tuple[str, Exception]] = [] - for program, expression in zip(options.compiled_expressions, options.expressions): + for expression in options.expressions: try: - result = program.execute(context) + result = _program(expression).execute(context) if not isinstance(result, bool): raise TypeError(f"returned {type(result).__name__}, not bool") except Exception as e: - log.warning("CEL expression %r for tool %r failed: %s: %s", expression, tool_name, type(e).__name__, e) failures.append((expression, e)) continue if result: @@ -69,7 +78,8 @@ def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: O matched.append(expression) if failures: - details = "; ".join(f"{expression!r}: {type(error).__name__}: {error}" for expression, error in failures) + # Only the exception type: its message can quote the checked arguments or result. + details = "; ".join(f"{expression!r}: {type(error).__name__}" for expression, error in failures) raise RuntimeError(f"CEL expressions for tool {tool_name!r} failed: {details}") if matched: return RailOutcome.block(metadata={**metadata, "matched_expressions": matched}) diff --git a/nemoguardrails/library/cel/rail_config.py b/nemoguardrails/library/cel/rail_config.py index 1eaf3b8f0d..f8424ad461 100644 --- a/nemoguardrails/library/cel/rail_config.py +++ b/nemoguardrails/library/cel/rail_config.py @@ -15,9 +15,10 @@ from typing import TYPE_CHECKING, Dict, List, Optional +from pydantic import ConfigDict + from nemoguardrails.manifests.config_schema import ( Field, - PrivateAttr, RailConfigBaseModel, RailConfigSpec, model_validator, @@ -28,7 +29,7 @@ from cel import Program -def _compile(expression: str) -> "Program": +def compile_expression(expression: str) -> "Program": """Compile *expression*, importing the optional CEL library only when a config uses it.""" try: import cel @@ -43,34 +44,34 @@ def _compile(expression: str) -> "Program": class CelOptions(RailConfigBaseModel): """CEL expressions to check for one tool.""" + model_config = ConfigDict(extra="forbid") + expressions: List[str] = Field( - default_factory=list, + ..., + min_length=1, description="List of CEL expressions to evaluate, where any that evaluates to true is a match.", ) - _compiled_expressions: List["Program"] = PrivateAttr(default_factory=list) - @model_validator(mode="after") def compile_expressions(self) -> "CelOptions": - """Compile the expressions at config load time, so a syntax error fails the load.""" - compiled = [] + """Compile the expressions at config load time, so a syntax error fails the load. + + The compiled programs are not kept: a cel.Program cannot be deep-copied or pickled, which the + synchronous generate() and check() do to the config. + """ for i, expression in enumerate(self.expressions): try: - compiled.append(_compile(expression)) + compile_expression(expression) except ValueError as e: raise ValueError(f"Invalid CEL expression at index {i} ({expression!r}): {e}") from e - object.__setattr__(self, "_compiled_expressions", compiled) return self - @property - def compiled_expressions(self) -> List["Program"]: - """Return the compiled CEL programs.""" - return self._compiled_expressions - class CelConfig(RailConfigBaseModel): """Configuration for CEL expression checks.""" + model_config = ConfigDict(extra="forbid") + tool_output: Dict[str, CelOptions] = Field( default_factory=dict, description="Per-tool CEL expressions to check against a tool call, keyed by tool name.", diff --git a/schemas/rails_config.snapshot.json b/schemas/rails_config.snapshot.json index 70584836f9..3850a1b9c0 100644 --- a/schemas/rails_config.snapshot.json +++ b/schemas/rails_config.snapshot.json @@ -54,6 +54,7 @@ "type": "object" }, "CelConfig": { + "additionalProperties": false, "description": "Configuration for CEL expression checks.", "properties": { "tool_input": { @@ -77,6 +78,7 @@ "type": "object" }, "CelOptions": { + "additionalProperties": false, "description": "CEL expressions to check for one tool.", "properties": { "expressions": { @@ -84,10 +86,14 @@ "items": { "type": "string" }, + "minItems": 1, "title": "Expressions", "type": "array" } }, + "required": [ + "expressions" + ], "title": "CelOptions", "type": "object" }, diff --git a/tests/guardrails/test_per_tool_cel_rails_iorails.py b/tests/guardrails/test_per_tool_cel_rails_iorails.py index 7ae5a5eb0d..eae98ea0b1 100644 --- a/tests/guardrails/test_per_tool_cel_rails_iorails.py +++ b/tests/guardrails/test_per_tool_cel_rails_iorails.py @@ -23,8 +23,10 @@ import pytest import pytest_asyncio -from nemoguardrails.guardrails.iorails import INTERNAL_ERROR_MESSAGE, REFUSAL_MESSAGE +from nemoguardrails.guardrails.iorails import INTERNAL_ERROR_MESSAGE, REFUSAL_MESSAGE, IORails from nemoguardrails.imports import check_optional_dependency +from nemoguardrails.rails.llm.config import RailsConfig +from nemoguardrails.rails.llm.options import RailStatus, RailType from tests.guardrails.async_helpers import started_iorails from tests.guardrails.test_tool_rails_iorails import ( _inject_forbidden_transport, @@ -205,3 +207,29 @@ async def test_non_matching_expression_streams_through(self, result_iorails): chunks = await _collect(result_iorails.stream_async(_tool_conversation("no keys here"))) assert _stream_violation_chunks(chunks) == [] assert "The file has no key." in "".join(chunk for chunk in chunks if isinstance(chunk, str)) + + +class TestSynchronousCheck: + """The synchronous check() deep-copies the config, so it covers a config holding CEL expressions.""" + + def test_sync_check_blocks_a_matching_call(self): + iorails = IORails(RailsConfig.from_content(config=TOOL_OUTPUT_CONFIG)) + messages = [ + {"role": "user", "content": "clean up"}, + { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call_0", + "type": "function", + "function": {"name": "run_shell", "arguments": '{"command": "rm -rf /tmp"}'}, + } + ], + }, + ] + + result = iorails.check(messages, rail_types=[RailType.TOOL_CALL], tools=[RUN_SHELL_TOOL]) + + assert result.status == RailStatus.BLOCKED + assert result.rail == "cel check tool output" diff --git a/tests/test_cel_actions.py b/tests/test_cel_actions.py index 4e4aeb7236..0661f149bd 100644 --- a/tests/test_cel_actions.py +++ b/tests/test_cel_actions.py @@ -19,9 +19,12 @@ the same rail through IORails. """ +import copy +import pickle import sys import pytest +from pydantic import ValidationError from nemoguardrails import RailsConfig from nemoguardrails.actions.rail_outcome import RailDecision @@ -79,22 +82,39 @@ async def _check_result(config: RailsConfig, content, call: ToolCall = None): class TestConfig: - def test_expressions_compile_at_load(self): - config = _config("tool_output", 'args.command == "ls"', "true") - assert len(config.rails.config.cel.tool_output["run_shell"].compiled_expressions) == 2 + def test_config_can_be_deep_copied_and_pickled(self): + # The synchronous generate() and check() deep-copy the config, which a compiled cel.Program cannot survive. + config = _config("tool_output", 'args.command == "ls"') + assert copy.deepcopy(config).rails.config.cel == config.rails.config.cel + assert pickle.loads(pickle.dumps(config)).rails.config.cel == config.rails.config.cel - def test_syntax_error_fails_the_load(self): - with pytest.raises(ValueError, match=r"Invalid CEL expression at index 1 \('args.command.matches\('\)"): - _config("tool_output", "true", "args.command.matches(") + @pytest.mark.parametrize( + "cel_config", + [ + {"tool_output": {"run_shell": {"expresions": ["true"]}}}, + {"tool_ouput": {"run_shell": {"expressions": ["true"]}}}, + {"tool_output": {"run_shell": {"expressions": []}}}, + {"tool_output": {"run_shell": {}}}, + ], + ids=["misspelled_key", "misspelled_section", "empty_expressions", "missing_expressions"], + ) + def test_misconfiguration_fails_the_load(self, cel_config): + with pytest.raises(ValidationError): + RailsConfig.from_content(config={"rails": {"config": {"cel": cel_config}}}) @pytest.mark.parametrize( - "expression", - ['args.command == "rm', "args.command ==", "args.command && && true"], - ids=["unterminated_string", "dangling_operator", "repeated_operator"], + "expressions, index", + [ + (["true", "args.command.matches("], 1), + (['args.command == "rm'], 0), + (["args.command =="], 0), + (["args.command && && true"], 0), + ], + ids=["unclosed_call", "unterminated_string", "dangling_operator", "repeated_operator"], ) - def test_other_syntax_errors_fail_the_load(self, expression): - with pytest.raises(ValueError, match="Invalid CEL expression at index 0"): - _config("tool_output", expression) + def test_syntax_error_fails_the_load(self, expressions, index): + with pytest.raises(ValueError, match=f"Invalid CEL expression at index {index} "): + _config("tool_output", *expressions) @pytest.mark.parametrize( "expression", [expression for expression, _ in _EVALUATION_ERRORS.values()], ids=_EVALUATION_ERRORS.keys() @@ -108,10 +128,6 @@ def test_missing_extra_names_it(self, monkeypatch): with pytest.raises(ImportError, match=r"nemoguardrails\[cel\]"): CelOptions(expressions=["true"]) - def test_no_expressions_needs_no_cel(self, monkeypatch): - monkeypatch.setitem(sys.modules, "cel", None) - assert CelOptions().compiled_expressions == [] - class TestToolOutput: @pytest.mark.asyncio @@ -189,12 +205,12 @@ class TestFailures: async def test_evaluation_error_fails_closed(self, expression, error_type): with pytest.raises(RuntimeError) as excinfo: await _check_call(_config("tool_output", expression), _shell_call(command="rm")) - assert f"{expression!r}: {error_type}: " in str(excinfo.value) + assert str(excinfo.value).endswith(f"{expression!r}: {error_type}") @pytest.mark.asyncio async def test_failing_expression_raises_despite_a_match(self): config = _config("tool_output", 'args.flags.exists(f, f == "--force")', 'args.command == "rm"') - with pytest.raises(RuntimeError, match="KeyError: 'flags'"): + with pytest.raises(RuntimeError, match="KeyError"): await _check_call(config, _shell_call(command="rm")) @pytest.mark.asyncio @@ -204,14 +220,19 @@ async def test_failures_without_a_match_raise_listing_each(self): await _check_call(config, _shell_call(command="ls")) message = str(excinfo.value) assert message.startswith("CEL expressions for tool 'run_shell' failed: ") - assert "'args.flags.exists(f, f == \"--force\")': KeyError: 'flags'" in message - assert "'args.command': TypeError: returned str, not bool" in message + assert "'args.flags.exists(f, f == \"--force\")': KeyError" in message + assert "'args.command': TypeError" in message @pytest.mark.asyncio - async def test_failure_message_carries_no_argument_values(self): - config = _config("tool_output", "args.command > 5") + @pytest.mark.parametrize( + "expression", + ['args[args.command] == "x"', "args.command.matches(args.command)"], + ids=["dynamic_key", "argument_as_pattern"], + ) + async def test_failure_message_carries_no_argument_values(self, expression): + # Both make the CEL error itself quote the argument, which must not reach the message. with pytest.raises(RuntimeError) as excinfo: - await _check_call(config, _shell_call(command="SECRET_TOKEN")) + await _check_call(_config("tool_output", expression), _shell_call(command="SECRET_TOKEN[")) assert "SECRET_TOKEN" not in str(excinfo.value) From d1ca6a84e7c6d01246fb70e4251ab884a0ccbd08 Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Wed, 14 Oct 2026 14:54:54 -0700 Subject: [PATCH 5/6] fix(cel): fail closed when a tool has no CEL expressions Signed-off-by: Tanushriya Singh --- .../configure-rails/guardrail-catalog/cel.mdx | 1 + nemoguardrails/library/cel/actions.py | 20 ++++---- .../test_per_tool_cel_rails_iorails.py | 51 ++++++++++--------- tests/test_cel_actions.py | 28 ++++++---- 4 files changed, 57 insertions(+), 43 deletions(-) diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx index 9a77ef037c..7728239771 100644 --- a/docs/configure-rails/guardrail-catalog/cel.mdx +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -91,6 +91,7 @@ The rail evaluates every expression configured for the tool: - If any expression evaluates to `true`, the rail blocks the tool call or result. The block lists the expressions that matched in its `matched_expressions` metadata. The metadata never includes the checked arguments or content. - If no expression evaluates to `true`, the rail allows the tool call or result. - If any expression fails to evaluate, for example because it reads an argument the call did not include, or it returns a value that is not a boolean, the rail fails closed, even if another expression matched. The tool call or result is blocked and reported as a rail failure rather than a policy decision, and the logged error lists each expression that failed with its error type. +- If the tool has no expressions configured for the direction its flow runs in, the rail also fails closed, since there is nothing to validate the tool call or result against. ## Write Expressions diff --git a/nemoguardrails/library/cel/actions.py b/nemoguardrails/library/cel/actions.py index 218acaa24e..42d7b52515 100644 --- a/nemoguardrails/library/cel/actions.py +++ b/nemoguardrails/library/cel/actions.py @@ -36,24 +36,26 @@ def _program(expression: str) -> "Program": return compile_expression(expression) -def _tool_options(config: RailsConfig, source: str, tool_name: str) -> Optional[CelOptions]: - """Return the tool's CEL options for *source*, or None when none are configured.""" +def _tool_options(config: RailsConfig, source: str, tool_name: str) -> CelOptions: + """Return the tool's CEL options for *source*. + + Raises when none are configured, so the rail fails closed: a tool the CEL rail runs for + with no expressions to check cannot be validated. + """ cel_config = config.rails.config.cel - if cel_config is None: - return None - return getattr(cel_config, source).get(tool_name) + options = getattr(cel_config, source).get(tool_name) if cel_config is not None else None + if options is None: + raise RuntimeError(f"No CEL expressions configured for tool {tool_name!r} under rails.config.cel.{source}") + return options -def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: Optional[CelOptions]) -> RailOutcome: +def _evaluate(source: str, tool_name: str, variables: Dict[str, Any], options: CelOptions) -> RailOutcome: """Evaluate all of a tool's CEL expressions, blocking when any is true. A failing expression (an error or a non-boolean result) raises even if another expression matched, so the rail fails closed. """ metadata: Dict[str, Any] = {"source": source} - if options is None: - log.debug("No CEL expressions configured for tool %r under source: %s", tool_name, source) - return RailOutcome.allow(metadata=metadata) # Already imported when the expressions compiled at config load. import cel diff --git a/tests/guardrails/test_per_tool_cel_rails_iorails.py b/tests/guardrails/test_per_tool_cel_rails_iorails.py index eae98ea0b1..b1936ac04d 100644 --- a/tests/guardrails/test_per_tool_cel_rails_iorails.py +++ b/tests/guardrails/test_per_tool_cel_rails_iorails.py @@ -32,12 +32,13 @@ _inject_forbidden_transport, _inject_json_response, _inject_sse_stream, - _sse, _stream_violation_chunks, _text_payload, + _text_sse_lines, _tool_call_payload, _tool_call_sse_lines, ) +from tests.guardrails.tool_helpers import make_tool_conversation pytestmark = pytest.mark.skipif(not check_optional_dependency("cel"), reason="requires the 'cel' extra (Python 3.11+)") @@ -86,30 +87,11 @@ def _tool_output_config(*expressions: str) -> dict: MESSAGES = [{"role": "user", "content": "clean up the temp directory"}] -def _tool_conversation(result_content: str) -> list: - return [ - {"role": "user", "content": "show me the key"}, - { - "role": "assistant", - "content": None, - "tool_calls": [ - { - "id": "call_1", - "type": "function", - "function": {"name": "run_shell", "arguments": '{"command": "cat key.pem"}'}, - } - ], - }, - {"role": "tool", "tool_call_id": "call_1", "name": "run_shell", "content": result_content}, - ] - - -def _text_sse_lines(text: str) -> list: - chunks = [ - {"id": "chatcmpl-1", "choices": [{"index": 0, "delta": {"role": "assistant", "content": text}}]}, - {"id": "chatcmpl-1", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}, - ] - return [_sse(chunk) for chunk in chunks] + [b"data: [DONE]\n\n"] +def _tool_conversation(content: str) -> list: + """A run_shell call answered by a tool result carrying *content*.""" + return make_tool_conversation( + result_name="run_shell", tool_name="run_shell", arguments='{"command": "cat key.pem"}', content=content + ) async def _collect(stream) -> list: @@ -159,6 +141,17 @@ async def test_failing_expression_fails_closed_as_internal_error(self, failing_c result = await failing_call_iorails.generate_async(messages=MESSAGES) assert result == {"role": "assistant", "content": INTERNAL_ERROR_MESSAGE} + @pytest.mark.asyncio + async def test_tool_without_expressions_fails_closed_as_internal_error(self): + config = { + "models": [MAIN_MODEL], + "rails": {"tool_output": {"per_tool": {"run_shell": ["cel check tool output"]}}}, + } + async with started_iorails(config) as engine: + _inject_json_response(engine, _tool_call_payload("run_shell", '{"command": "ls"}')) + result = await engine.generate_async(messages=MESSAGES) + assert result == {"role": "assistant", "content": INTERNAL_ERROR_MESSAGE} + class TestStreamingPerToolCallCel: @pytest.mark.asyncio @@ -175,6 +168,14 @@ async def test_non_matching_expression_streams_through(self, call_iorails): chunks = await _collect(call_iorails.stream_async(MESSAGES)) assert _stream_violation_chunks(chunks) == [] + @pytest.mark.asyncio + async def test_failing_expression_streams_a_generic_error(self, failing_call_iorails): + _inject_sse_stream(failing_call_iorails, _tool_call_sse_lines("run_shell", ['{"command": "SECRET"}'])) + chunks = await _collect(failing_call_iorails.stream_async(MESSAGES)) + assert len(_stream_violation_chunks(chunks)) == 1 + payload = "".join(str(chunk) for chunk in chunks) + assert "args.flags" not in payload and "SECRET" not in payload + class TestNonStreamingPerToolResultCel: @pytest.mark.asyncio diff --git a/tests/test_cel_actions.py b/tests/test_cel_actions.py index 0661f149bd..d76f115e3c 100644 --- a/tests/test_cel_actions.py +++ b/tests/test_cel_actions.py @@ -152,15 +152,20 @@ async def test_no_match_allows(self): assert outcome.metadata == {"source": "tool_output"} @pytest.mark.asyncio - async def test_tool_without_expressions_allows(self): - outcome = await _check_call(_config("tool_output", "true", tool="other_tool"), _shell_call(command="rm")) - assert outcome.decision is RailDecision.ALLOW + async def test_tool_without_expressions_fails_closed(self): + with pytest.raises(RuntimeError, match="No CEL expressions configured for tool 'run_shell'"): + await _check_call(_config("tool_output", "true", tool="other_tool"), _shell_call(command="rm")) @pytest.mark.asyncio - async def test_null_cel_section_allows(self): + async def test_null_cel_section_fails_closed(self): config = RailsConfig.from_content(config={"rails": {"config": {"cel": None}}}) - outcome = await _check_call(config, _shell_call(command="rm -rf /")) - assert outcome.decision is RailDecision.ALLOW + with pytest.raises(RuntimeError, match="No CEL expressions configured for tool 'run_shell'"): + await _check_call(config, _shell_call(command="rm -rf /")) + + @pytest.mark.asyncio + async def test_has_guards_a_missing_argument(self): + config = _config("tool_output", 'has(args.mode) && args.mode == "append"') + assert (await _check_call(config, _shell_call(command="ls"))).decision is RailDecision.ALLOW @pytest.mark.asyncio async def test_tool_name_is_available(self): @@ -213,6 +218,11 @@ async def test_failing_expression_raises_despite_a_match(self): with pytest.raises(RuntimeError, match="KeyError"): await _check_call(config, _shell_call(command="rm")) + @pytest.mark.asyncio + async def test_has_on_a_missing_parent_fails_closed(self): + with pytest.raises(RuntimeError, match="KeyError"): + await _check_call(_config("tool_output", "has(args.a.b)"), _shell_call(command="ls")) + @pytest.mark.asyncio async def test_failures_without_a_match_raise_listing_each(self): config = _config("tool_output", 'args.flags.exists(f, f == "--force")', "args.command", 'args.command == "rm"') @@ -268,10 +278,10 @@ async def test_type_guard_skips_other_shapes(self): assert outcome.decision is RailDecision.ALLOW @pytest.mark.asyncio - async def test_null_cel_section_allows(self): + async def test_null_cel_section_fails_closed(self): config = RailsConfig.from_content(config={"rails": {"config": {"cel": None}}}) - outcome = await _check_result(config, "-----BEGIN RSA PRIVATE KEY-----") - assert outcome.decision is RailDecision.ALLOW + with pytest.raises(RuntimeError, match="No CEL expressions configured for tool 'run_shell'"): + await _check_result(config, "-----BEGIN RSA PRIVATE KEY-----") @pytest.mark.asyncio async def test_args_and_tool_come_from_the_answered_call(self): From 89e230590b28f715ca59d11bca478cad3dbeb815 Mon Sep 17 00:00:00 2001 From: Tanushriya Singh Date: Wed, 14 Oct 2026 15:12:06 -0700 Subject: [PATCH 6/6] docs(cel): describe fail-closed behavior for tools without expressions Signed-off-by: Tanushriya Singh --- docs/configure-rails/guardrail-catalog/cel.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configure-rails/guardrail-catalog/cel.mdx b/docs/configure-rails/guardrail-catalog/cel.mdx index 7728239771..5837ca5b61 100644 --- a/docs/configure-rails/guardrail-catalog/cel.mdx +++ b/docs/configure-rails/guardrail-catalog/cel.mdx @@ -62,7 +62,7 @@ To use the CEL rail, you should: - "cel check tool input" ``` - The `cel check tool output` flow evaluates the tool's `tool_output` expressions against each call to the tool, and the `cel check tool input` flow evaluates its `tool_input` expressions against each result the tool returns. A tool with no expressions configured for a direction is allowed. + The `cel check tool output` flow evaluates the tool's `tool_output` expressions against each call to the tool, and the `cel check tool input` flow evaluates its `tool_input` expressions against each result the tool returns. A tool without a CEL flow under `per_tool` is not checked by the rail, but a tool whose CEL flow has no expressions configured for its direction fails closed. 3. Declare the tool in the main model's `parameters.tools`, or per request: in `options.llm_params.tools` with the Python API, or in the `tools` field of a chat completions request to the server, which the server currently accepts only for non-streaming requests on a configuration with `passthrough: true`. `cel check tool output` blocks a call to an undeclared tool before any expression runs. Refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling) for the full declaration reference.