Skip to content

[Feature] Add an automated backend test suite #2

Description

@sjohnston1972

Background

NetBud's backend contains several pieces of subtle, failure-prone logic that run on every design request:

  • backend/app/jsonblock.py — extracting a JSON object from streamed LLM output, including a brace-counting truncation-recovery path for when the model runs out of tokens mid-object.
  • backend/app/rag/chroma_client.py_sources_match / _source_tokens, a 60%-token-overlap heuristic that decides whether a citation is verified (the "don't let the LLM grade its own homework" keystone).
  • backend/app/intake/extractor.py — customer-id regex + section normalization.

Problem / Goal

There is no automated test suite anywhere in the repo — no tests/ directory, no pytest (or any test dependency) in backend/requirements.txt, and no test files under frontend/. The only hint tests were ever intended is .pytest_cache/ in .gitignore.

That means the trickiest code (truncation recovery, fuzzy citation matching, JSON extraction fallbacks) has zero regression protection, and the SQLite-migration work in the sibling parent issue will land without a safety net.

Goal: stand up a backend test suite (pytest) and cover the highest-risk pure-logic modules first.

Where to look

  • backend/requirements.txt — no test deps today.
  • backend/app/jsonblock.py — pure functions, no I/O, ideal first target.
  • backend/app/rag/chroma_client.py:106-133_source_tokens, _sources_match are pure and testable without a live Chroma.
  • backend/app/intake/extractor.py:38-46, 113-128extract_customer_id_regex (pure) and extract_brief (needs the LLM router stubbed).
  • .gitignore — already ignores .pytest_cache/.

Suggested approach

  1. Add pytest (and pytest-asyncio for the async router paths) to a dev-dependency list, create a backend/tests/ package, and document cd backend && pytest in the README.
  2. Start with pure functions (no mocking): jsonblock, chroma_client._sources_match, extractor.extract_customer_id_regex.
  3. For anything that calls the LLM router or Chroma, inject/monkeypatch a fake so tests stay offline and deterministic.
  4. Keep tests fast and hermetic — no network, no real SQLite file (use sqlite:///:memory: where a DB is needed).

Sub-issues break this into: scaffolding, jsonblock tests, citation-matching tests, extractor tests.

Acceptance criteria

  • cd backend && pytest runs green with no network access.
  • The four modules above have meaningful coverage of their edge cases (not just happy path).
  • README documents how to run the tests.

Testing

cd backend && pytest -q from a clean checkout with .env absent should pass (tests must not require API keys).

Out of scope

  • Frontend (Vitest/Jest) tests — separate effort.
  • A CI pipeline / GitHub Actions workflow — can follow once tests exist.
  • Integration tests against a live Chroma, Ollama, or GNS3 server.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions