Add child-safety guardrails (no content filtering right now)

[akkki007]

I went looking for content filtering in this repo and couldn't find any. Opening this partly to check whether that's deliberate, and partly to propose something if it isn't.

The prompts here are written for ages 5-12, but nothing checks what a child sends in, and nothing checks what the model sends back.

What's already there

The persona at the top of PROMPT_TEMPLATE does some work: "coding assistant built for kids", "focus on coding-related problems", "easy to understand for kids". It's a soft instruction to the model rather than an enforced check, but it's not nothing.

The catch is that it only applies to /ask. Looking at the other three:

• /ask-llm hands the question straight to the model. No template at all.
• /ask-llm-prompted lets the caller's custom_prompt replace the system prompt outright.
• /debug uses the code prompts, which never mention that the user is a child.

So a client with a valid API key can skip the child-safe framing entirely just by calling a different endpoint. And on the way out there's no check anywhere, on any endpoint. Whatever the model produces goes back to the child as-is.

Two smaller things I noticed while reading: there's no record kept when something does go wrong, so a maintainer would have no idea it happened, and /ask writes the child's question text into sugar_ai.log in the clear.

Why I think it's worth doing

Two things make this different from a general LLM app. Kids are much less likely to push back on an answer that's wrong or strange, especially one written in a deliberately warm and confident voice. And in a school, one bad output isn't a support ticket, it's a call from a parent.

Related: retrieval is broken right now (#154, #130, #88), so /ask isn't actually grounded in the docs. I ran it locally on the dev model and asked how to create a Pygame window. It told me Pygame was written by "Dennis Tecla" in 2000, for people working in C# and C++. All of it made up, and phrased in the friendly kid voice, which is what makes it bad. That's roughly the failure I'd want a guard to catch.

Proposed plan

Design constraints (these rule out most off-the-shelf options):

• Must work offline / CPU-only — schools have weak connectivity, so no cloud moderation API dependency.
• Fail closed — if the guard errors, return the safe fallback, never the raw output.
• Never scare or scold the child — redirect warmly ("Let's get back to your code! 🙂"), no lectures.
• Low added latency — /ask already makes 2 model calls.

Phase 1 — Input guard (app/guardrails.py, applied in the route layer so every endpoint gets it)

• Screen incoming text for: self-harm, violence, sexual content, bullying, and personal-info disclosure.
• Cap input length (question and code are currently unbounded query params).
• On a hit: return a kind redirect message; never reach the model.

Phase 2 — Output guard

• Screen generated text before it reaches the child; substitute a safe fallback on a hit.
• This is the phase that actually matters — small models are unpredictable, and the input guard cannot catch what the model invents on its own.

Phase 3 — Close the bypasses

• Apply the child-safety preamble to /ask-llm and /debug.
• Make the preamble immutable on /ask-llm-prompted — custom_prompt should extend it, never replace it. Activities can still customise tone and subject; they should not be able to switch off child safety.

Phase 4 — Privacy

• Redact PII (names, schools, addresses, emails) from logs.
• Add a prompt rule: never request or repeat a child's personal details.

Phase 5 — Visibility and tests

• Log flagged events by category only, not content, so maintainers can see patterns without storing children's messages.
• Add a fixture set of unsafe prompts that CI asserts are refused, so guardrails cannot silently regress.

Phases 1–3 are the meaningful safety win; 4–5 make it maintainable.

Not sure about

• Rules-based blocklist or a small local classifier like toxic-bert? Rules are fast and offline but crude; the model catches more but adds to startup, which matters on school hardware. Leaning towards rules first with a model behind it, but I don't have a strong view.
• Should flagged events show up in the existing /admin panel, or is that overkill?
• Is there existing Sugar Labs wording for this kind of refusal that I should match instead of writing my own?

Would rather get a maintainer's read on the direction before writing any code. I can pick up 1-3 if this seems reasonable.

[akkki007]

Hey @Noaman-Akhtar
Can you review the above plan for going abt guardrails ?
Lemme know if any changes

[saanidhi-git]

Hello @akkki007 and @Noaman-Akhtar,

I have reviewed the proposed 5-phase plan for implementing child-safety guardrails and would like to volunteer to work on this issue, starting with Phases 1 through 3.

Understanding the Requirements

The core objective is to ensure that all endpoints enforce child-safe interactions without relying on external cloud moderation APIs that could cause latency or connectivity issues in school environments. The system must fail closed, redirect gently without scolding, and maintain low overhead across the application.

Proposed Implementation Plan

My background involves engineering multi-tier AI pipelines and working extensively with Python-based API routing frameworks and LLM orchestration. To resolve this, I will implement the following:

1. Phases 1 & 2 (Input and Output Guards): I will implement a lightweight, rules-based moderation layer at the route level. This ensures all endpoints are protected uniformly without the heavy startup cost of local ML classifiers. I will enforce input length limits and implement a fail-closed mechanism that guarantees a safe, warm fallback message upon flagging.
2. Phase 3 (Closing Bypasses): I will restructure the prompt handling architecture so the child-safety persona is strictly enforced across /ask-llm and /debug. For /ask-llm-prompted, I will ensure custom_prompt can only append to, rather than overwrite, the foundational safety preamble.
3. Architecture for Future Phases: While focusing on delivering 1-3, I will structure the guardrail logic to easily support the redacted logging (Phase 4) and CI test fixtures (Phase 5) in subsequent updates.

Please let me know if you are open to assigning this to me, and if you have any immediate preferences regarding the rules-based blocklist structure before I begin drafting the pull request.