Evidence-Grounded Agentic Loop Platform
Every claim traces to evidence. If evidence is missing, SCOUT says so — it doesn't guess.
🚀 Runs with Llama 3 locally via Ollama • 📊 Real-time evidence tracking • 🔒 Tenant-safe from day one
SCOUT is a production-grade single-agent runtime that proves its answers. Every tool output becomes evidence. Every claim is validated. Every answer is grounded.
┌─────────────────────────────────────────────────────────────┐
│ │
│ User Query → Tool Calls → Evidence Pool → Grounded Answer │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────┐ ┌──────────┐ ┌────────┐ ┌─────────┐ │
│ │ Task │────▶│ LLM Call │─▶│ Score │────▶│ Verified│ │
│ └──────┘ └──────────┘ └────────┘ └─────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Most LLM agent frameworks give you a response and a prayer. They call tools, build context, and produce answers — but you have no way to prove why they're trustworthy.
The gap between "it runs" and "it's reliable" is where production agents die.
| Problem | SCOUT Solution |
|---|---|
| Hallucinated citations | Evidence validation against actual pool |
| Fabricated data | 5-tier grounding quality scoring |
| Unbounded loops | Typed state machine with budgets |
| Provider failures | Multi-provider fallback with fault injection |
| Multi-tenant security | Tenant isolation (404 on cross-tenant) |
flowchart TB
A[Client Request] --> B[Auth Middleware]
B --> C[Rate Limiter]
C --> D[Run Service]
D --> E[Loop Controller]
E --> F[Planner]
E --> G[Executor]
E --> H[Stop Controller]
G --> I[Evidence Store]
I --> J[Citation Parser]
J --> K[Grounding Validator]
F --> L[Primary Provider]
L -->|fail| M[Fallback Provider]
flowchart LR
A[Tool Output] --> B[EvidenceRecord]
B --> C[Content Hash]
B --> D[Source Reference]
C --> E[Score]
D --> E
E --> F[Citation Parser]
F --> G[Grounding Quality]
G --> H[Final Answer]
flowchart TD
A[LLM Call] --> B{Primary Provider}
B -->|Success| C[Response]
B -->|Rate Limit| D[Retry with Jitter]
B -->|Error| E[Fallback Manager]
D -->|Success| C
D -->|Exhausted| E
E -->|Success| C
E -->|All Failed| F[Output Repair]
F --> G[Final Response]
- Python 3.11+
- Ollama (for local LLM)
# Clone the repository
git clone https://github.com/shivamsingh-007/SCOUT-The-Agentic-Loop.git
cd SCOUT-The-Agentic-Loop
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -e ".[dev]"# Pull Llama 3 model
ollama pull llama3
# Run tests
pytest -q
# Start API server
uvicorn scout.api.app:create_app --factory --reload
# Create a run
curl -X POST http://localhost:8000/v1/runs \
-H "Content-Type: application/json" \
-d '{"task": "What is the capital of France?"}'Every tool output becomes evidence with:
- Content hashes for deduplication
- Source references for provenance
- Composite scoring (relevance + recency + quality)
| Tier | Description | Action |
|---|---|---|
| 🟢 Strong | Multiple strong citations | Proceed |
| 🟡 Adequate | Basic citations present | Proceed with note |
| 🟠 Weak | Limited evidence | Flag for review |
| 🔴 Speculative | Inconsistent evidence | Reject |
| ⚫ None | No evidence | Reject |
Primary Provider → Retry → Fallback → Output Repair
↓ ↓ ↓ ↓
Success Success Success Final Answer
- HMAC-based auth with constant-time comparison
- Per-tenant rate limiting (requests/min, requests/hour)
- Tenant isolation (404 on cross-tenant access)
- Error sanitization (no file paths or API keys in errors)
- Graceful shutdown with database cleanup
| Metric | Value | Status |
|---|---|---|
| Tests | 628 passing | ✅ |
| Type Checking | mypy strict, 0 errors | ✅ |
| Linting | Ruff, 0 errors | ✅ |
| API Isolation | Tenant-safe (404) | ✅ |
| Auth | Constant-time + HMAC | ✅ |
| Storage | SQLite hardened | ✅ |
| Metric | Result |
|---|---|
| Test Pass Rate | 100% |
| mypy Errors | 0 |
| Ruff Errors | 0 |
| Metric | Result |
|---|---|
| Tasks Run | 10/10 |
| Parse Success | 100% |
| Citation Validity | 70% |
| Avg Latency | 13.3s |
src/scout/
├── api/ # FastAPI server, auth, rate limiting
│ ├── routers/ # API endpoints
│ ├── services/ # Business logic
│ └── utils.py # Error sanitization
├── runtime/ # State machine, loop controller
│ ├── agent.py # Main agent class
│ ├── loop.py # Loop controller
│ └── verifier.py # Grounding verification
├── llm/ # LLM provider abstraction
│ ├── ollama.py # Ollama provider
│ ├── router.py # Provider routing
│ └── fallback.py # Fallback logic
├── tools/ # Tool protocol and implementations
├── grounding/ # Evidence tracking and scoring
│ ├── scorer.py # BM25 scoring
│ └── validator.py # Claim validation
├── memory/ # SQLite memory store
└── config.py # Typed configuration
# Run all tests
make test
# Run linting
make lint
# Run type checking
make typecheck
# Run all quality gates
make all
# Start development server
make serve- Ruff for linting and formatting
- mypy strict mode for type checking
- Pydantic v2 for data validation
- pytest for testing
| Phase | Status | Description |
|---|---|---|
| Phase 1-10 | ✅ Complete | Core platform |
| Phase 11 | 🔄 In Progress | Eval framework |
| Phase 12 | 📋 Planned | Dashboard UI |
| Phase 13 | 📋 Planned | Multi-provider |
| Phase 14 | 📋 Planned | Persistent memory |
See CONTRIBUTING.md for the full process.
# Quick start
git checkout -b feat/my-feature
make all
git commit -m "feat: add new feature"
git push origin feat/my-featureMIT License - see LICENSE
Built with ❤️ by Shivam Singh
