Skip to content
BranchesTags

Repository files navigation

Sightbox

Drop documents in. Get structured JSON out.

   ┌──────────────┐                    ┌──────────────┐
   │   DOCUMENTS  │                    │     JSON     │
   │              │      Sightbox      │              │
   │  PDF · image │  ───────────────►  │  { schema }  │
   │              │                    │     out      │
   └──────────────┘                    └──────────────┘
        in                                    out

Sightbox is a self-hosted document intelligence API. Upload images or PDFs into a project mailbox, and a vision LLM extracts structured data from each file using your prompt and optional JSON schema — asynchronously, with webhooks when results are ready.

Run locally with Ollama, or use OpenRouter for cloud models.

npm install && npm run setup && npm run dev

Open http://localhost:3001/admin — default API key: dev-api-key-change-me

Why Sightbox?

Most document-AI tools are either black-box SaaS or heavy ML pipelines you have to wire together yourself. Sightbox sits in the middle:

  • Async file queue with status tracking
  • Per-project prompt, JSON schema, provider, and model
  • Debug-first results — raw LLM output and errors are always saved
  • Local Ollama or cloud OpenRouter
  • SQLite out of the box

Upload a file, poll or webhook, done.

How it works

  ┌─────────────┐     POST /v1/files      ┌──────────────┐
  │  Your app   │ ──────────────────────► │   Sightbox   │
  │  or curl    │                         │   API        │
  └─────────────┘                         └──────┬───────┘
        ▲                                        │
        │                                        ▼
        │                                 ┌──────────────┐
        │   webhook or GET /v1/files/:id  │   Worker     │
        └─────────────────────────────────│   + Vision   │
                                          │   LLM        │
                                          └──────────────┘
  1. Create a project — set a prompt, optional JSON schema, provider (Ollama or OpenRouter), and model.
  2. Upload files — images, PDFs, or text. Sightbox stores them and queues processing.
  3. Get results — poll the API or register webhooks. Every LLM response is saved for inspection.

Each project is an isolated mailbox: its own files, schema, model config, and webhook endpoints.

Extraction flow

There is a single extraction path — no modes or pipelines:

  1. Build a prompt from the project prompt + JSON schema (schema is appended as reference text).
  2. Prepare the document (images for vision, text inline for text/*).
  3. Stream one chat completion to the configured provider.
  4. Parse JSON when possible; otherwise store the raw text. LLM errors are saved too, not discarded.

Result shape (success): parsed JSON object in structuredData.

Result shape (debug): when JSON is invalid or the LLM errors:

{
  "_llmError": "Provider returned error",
  "_errorDetail": { "code": 502, "message": "..." },
  "_raw": "...",
  "_validJson": false
}

The admin file view shows structured data, raw LLM output, and error details.

Quick start

Requirements: Node.js 20+, optional Ollama for local inference.

git clone https://github.com/illia-krlv/sightbox-ai.git
cd sightbox
npm install
npm run setup    # creates .env, SQLite DB, storage dirs
npm run dev

Try it with local Ollama (no API key)

ollama serve
ollama pull qwen3-vl:8b

In the admin UI, create a project with provider Ollama and model qwen3-vl:8b, then upload a PDF.

Use OpenRouter (cloud)

Add your key to .env:

OPENROUTER_API_KEY=sk-or-v1-...
OPENROUTER_DEFAULT_MODEL=nvidia/nemotron-nano-12b-v2-vl:free

In the admin UI, select provider OpenRouter and pick a vision-capable model.

Example

You can do everything below in the admin UI at http://localhost:3001/admin — create a project (prompt, schema, provider, model), upload files, and inspect results without writing curl commands. The API examples are for automation and integrations.

Create an invoice extractor:

curl -X POST http://localhost:3001/v1/projects \
  -H "X-API-Key: dev-api-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Invoice Extractor",
    "prompt": "Extract document type and key fields from this document.",
    "jsonSchema": {
      "type": "object",
      "required": ["documentType", "confidence"],
      "properties": {
        "documentType": {
          "enum": ["Invoice", "Receipt", "Purchase Order", "Other"]
        },
        "confidence": { "type": "number" }
      }
    },
    "modelConfig": {
      "provider": "ollama",
      "model": "qwen3-vl:8b"
    }
  }'

Upload a document:

curl -X POST http://localhost:3001/v1/files \
  -H "X-API-Key: dev-api-key-change-me" \
  -F "projectId=<PROJECT_ID>" \
  -F "file=@./invoice.pdf"

Response is 202 Accepted. Poll until status is COMPLETED:

curl http://localhost:3001/v1/files/<FILE_ID> \
  -H "X-API-Key: dev-api-key-change-me"

Project modelConfig

Field Description
provider "ollama" (default) or "openrouter"
model Vision model id (e.g. qwen3-vl:8b, nvidia/nemotron-nano-12b-v2-vl:free)
ollamaBaseUrl Optional. Defaults to OLLAMA_BASE_URL

Features

Vision LLM extraction

One streamed chat call per file via the official **ollama** or **@openrouter/sdk** clients.

  • Ollama — native /api/chat streaming with base64 images
  • OpenRouter — streaming chat completions with image_url parts

Results are not schema-validated server-side. The JSON schema is guidance for the model; whatever comes back is stored.

Schema tips (token efficiency):

  • Shorter property names in your schema mean fewer output tokens.
  • Prefer flat keys where possible (vendorName instead of nested vendor.name).
  • Sightbox compacts the schema in the prompt and assigns short key aliases automatically (e.g. documentType → dt). Stored structuredData always uses your original field names.
  • Set PROMPT_OPTIMIZATION_ENABLED=false to append the raw JSON Schema instead (legacy prompt format).

PDF handling

All PDF pages are rendered to PNG at 2× scale:

Pages Sent to the model
1–2 Stitched into a single vertical image
3+ One image per page

When IMAGE_OPTIMIZATION_ENABLED=true (default), images are resized to fit within IMAGE_MAX_WIDTH × IMAGE_MAX_HEIGHT and encoded as JPEG before sending to the model — fewer vision tokens and faster uploads. Set IMAGE_OPTIMIZATION_ENABLED=false to send originals unchanged.

Multiple projects, one server

Run separate extractors side by side — invoices in one mailbox, contracts in another — each with its own prompt, schema, provider, model, and webhooks.

Webhooks with delivery tracking

Register one or more webhook URLs per project. On COMPLETED or FAILED, Sightbox POSTs the result payload and records delivery status. Optional HMAC signing via X-Webhook-Signature.

Note: LLM-level failures (provider errors, bad JSON) still mark the file COMPLETED with debug fields saved. FAILED is reserved for infrastructure errors (e.g. unreadable file, worker crash before a result is stored).

Scale when you need to

  • **WORKER_CONCURRENCY** — process up to 32 files in parallel per worker
  • **RUN_MODE** — split API and worker into separate processes
  • Multiple worker replicas — optimistic claim + updateMany guard (see roadmap below)
WORKER_CONCURRENCY=8 npm run dev:worker   # prod example
RUN_MODE=api npm run dev:api              # API only, scale workers separately

High-volume roadmap (Phase A → B)

Priority Improvement Status Notes
Medium OpenRouter 429 retry + backoff Planned Concurrency cap + exponential backoff
Medium Adaptive worker poll interval when backlog > 0 Planned e.g. 500–1000 ms vs idle 2 s
Medium Graceful shutdown (drain in-flight jobs on SIGTERM) Planned Avoid mid-job kills on deploy
Low Async webhook delivery pool Planned Today webhooks run inline after extraction
Low LLM error-rate counter + last-job duration buffer Planned On-call signals without Prometheus yet

Failure modes to design for

Symptom Mitigation
Double-processing under load SKIP LOCKED claim (planned)
Queue grows overnight unnoticed Alert on /v1/metrics pendingFiles
OpenRouter rate limits Lower concurrency, retry with backoff
Files stuck in PROCESSING Stale reclaim cron (configurable)
Deploy kills active jobs Graceful drain (planned)

Deferred to Phase B+ (when SQLite/local disk becomes the bottleneck):

  • SQS / BullMQ (or similar) job queue
  • S3 (or object storage) + presigned upload
  • Dedicated webhook delivery workers
  • Auto-scaling on queue depth
  • Load test harness (e.g. k6)
  • OpenRouter circuit breaker on sustained 429/5xx

Phase A success signals

  1. One streamed LLM call per document (verify in logs/token usage).
  2. Two worker processes never double-claim the same file.
  3. /health and /v1/metrics expose backlog and active jobs.
  4. API and worker run as separate processes.
  5. Staging throughput ≥ ~15k/day with OpenRouter at concurrency 8 (measure before Phase B).

Admin UI

Browse projects, pick Ollama or OpenRouter, set the model, upload files, and inspect results at /admin. The file detail page shows structured JSON, raw LLM output, and error details.

Use cases

  • Invoice & receipt extraction — pull fields from PDFs and scans
  • Compliance document triage — extract type and fields from certs, forms, and statements
  • Mailroom automation — accept uploads via API, push structured results to your CRM
  • LLM prototyping — test prompts and schemas against real documents without building a pipeline
  • Privacy-sensitive workloads — keep everything on-prem with Ollama

Supported files

Type Handling
image/* Sent to vision model as base64 data URI
application/pdf All pages rendered to PNG (1–2 pages stitched, 3+ sent separately)
text/* Appended to the prompt as UTF-8

Default upload limit: 5 MB (MAX_UPLOAD_BYTES).

API reference

All /v1/* routes require X-API-Key or Authorization: Bearer <key>.

Method Endpoint Description
POST /v1/projects Create project
GET /v1/projects List projects
GET /v1/projects/:id Get project
PATCH /v1/projects/:id Update project
POST /v1/files Upload file (projectId + file)
GET /v1/files/:id Get file + result
GET /v1/files/:id/content Download stored file
DELETE /v1/files/:id Delete file and result
GET /v1/projects/:id/files List project files
POST /v1/projects/:id/webhooks Add webhook
GET /v1/metrics Queue depth and worker stats
GET /health Liveness probe

Configuration

See .env.example for the full list.

Variable Default Description
DATABASE_URL file:../data/file-processor.db SQLite path (no Docker needed)
API_KEYS dev-api-key-change-me Comma-separated API keys
WORKER_CONCURRENCY 1 Parallel files per worker (1–32)
MAX_UPLOAD_BYTES 5242880 Max upload size (5 MB)
RUN_MODE all all, api, or worker
OLLAMA_BASE_URL http://localhost:11434 Local Ollama endpoint
OLLAMA_DEFAULT_MODEL qwen3-vl:8b Default when project omits model
OPENROUTER_API_KEY — Required for OpenRouter provider
OPENROUTER_DEFAULT_MODEL see .env.example Default when project omits model

Deployment modes

RUN_MODE API + admin Worker
all ✓ ✓
api ✓ —
worker — ✓ (+ /health) 
npm run dev          # both (default)
npm run dev:api      # API only
npm run dev:worker   # worker only

Production database

SQLite works for development and moderate load. For heavier concurrency, optional MySQL is available:

docker compose -f docker-compose.mysql.yml up -d

Then switch provider in prisma/schema.prisma to mysql, set DATABASE_URL, and run npm run db:push.

Tech stack

  • Runtime — Node.js, TypeScript, Fastify
  • Database — Prisma + SQLite (MySQL optional)
  • LLM — Ollama (local) or @openrouter/sdk (cloud), streaming
  • PDF — pdf-to-img + sharp for page stitching
  • Storage — Local filesystem (./storage)

License

MIT — see LICENSE.

About

Self-hosted API for vision-LLM document extraction to structured JSON.

Topics

async-processingdocument-extractiondocument-processinginvoice-processingpdf-extractionreceipt-extractionself-hostedstructured-datavision-llm

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Contributors

Languages