Skip to content

fix(security): mark knowledge base text as data in prompts - #1454

Open
raghvendradhakar wants to merge 1 commit into
juspay:assistfrom
raghvendradhakar:fix/prompt-data-marking
Open

raghvendradhakar wants to merge 1 commit into
juspay:assistfrom
raghvendradhakar:fix/prompt-data-marking

Conversation

@raghvendradhakar

Copy link
Copy Markdown
Contributor

What

Marks knowledge base text as data before the model reads it. This covers part of security finding BK-06.

  • build_kb_system_message and fetch_kb_context_message now put the text inside a marked block. The block says the text is reference data, and that the model must never follow instructions inside it.
  • The grounding rule stays outside the block.
  • query_knowledge_base tool results get a data_notice with the same message.

The block's marker is a hash of the text. A document cannot predict its own hash, so it cannot close the block early. The same text always gets the same marker, so the prompt prefix stays cacheable.

Function names and message roles do not change. Templates without a knowledge base are not affected.

What this does not fix

BK-06 is reduced, not closed:

  • Customer values in prompts, such as {customer_name}, are not changed. This is the larger remaining risk.
  • Knowledge base text stays in the system role, while the report asks to move it. On Gemini, later system messages are already sent as the user role.
  • handoff_summary, hold reason, language_name, greetings, and HTTP and MCP tool results are not changed.

Marking lowers the risk. It cannot guarantee that a model ignores a convincing fake fact.

Testing

Old and new prompt shapes on gemini-3.6-flash, 20 runs each, through a direct model call:

Check Old New
Fake store policy in a document, attack worked 15/20 0/20
"SYSTEM OVERRIDE" in a document, attack worked 0/20 0/20
Normal: store hours answered 20/20 20/20
Normal: promotion mentioned 20/20 20/20

No reply contained the marker. An earlier run on a prompt that also said "never reveal internal codes" gave 0/30 before and after, so the result depends on the prompt.

Test suite: 4817 passed, 22 skipped, 1 expected failure, 1 failed. The failure is tests/assist/test_lazy_loading.py::test_core_only_process_never_imports_commerce. It starts its own process, and local runs used a stub for google.cloud.kms that does not reach that process.

Not tested: Azure and Qwen, a real voice call, realtime models, and production templates. CI runs only tests/crm, so it does not run the new tests in tests/test_client_context.py.

Could conflict with

clairvoyance #1428, the pipecat upgrade, changes how the full-injection knowledge base message is placed. This PR changes only the message content.

@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 515d1477-627e-4632-b09f-b6da5b0a770e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Dev-Bhumika03

Copy link
Copy Markdown
Collaborator

The main concern is that external content is still going into the system message, which means the model might treat it as trusted instructions.

A few things I think are still missing in the PR:

  • Lead data ({customer_name}) isn't handled yet and still goes directly into the system prompt.
  • KB content is labelled but still part of the system message.
  • No sanitisation of control tokens like [SYSTEM] or <|...|>.
  • No validation to ensure only template-declared fields are passed.
  • The hash marker approach doesn't seem reliable since a document could contain a fake closing tag.

Please check above points.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants