Observatory

Full-Stack LLM Observability Platform

Ingest, trace, analyze, and optimize every interaction your LLM applications make — at any scale.

Getting Started · Architecture · Documentation · Contributing

Overview

Observatory is an open-source, full-stack observability platform built specifically for LLM-powered applications. It captures every span, token, tool call, agent step, and retrieval operation your AI systems produce — and surfaces that data through rich dashboards, real-time alerts, and cost analytics.

Whether you are running a single Anthropic-powered chatbot or a fleet of multi-agent LangGraph workflows, Observatory gives you end-to-end visibility into what your AI is doing, how much it costs, and where it's failing.

Features

Category	Capabilities
Telemetry Ingestion	Spans, logs, metrics, agent runs, LLM calls, tool calls, document retrievals, feature events
Infrastructure Tracing	HTTP/API traces, database query spans, worker/job spans
Agent Observability	Full step-by-step agent waterfall, state machine transitions, delegation trees
Cost Analytics	Per-model USD cost estimates, token-level breakdowns, historical trends
Security Detection	Post-ingest threat analysis, PII detection, severity-based alerting
Alerting	Rule-based alert engine (60s polling), Slack webhook routing
Model Registry	Track every model version, provider, pricing, and deprecation status
Deployments	Idempotent deployment intents for agents and models (preview/production)
Vector DB Tracking	Qdrant, Pinecone, ChromaDB — search latency, score distribution, embedding metrics
MCP Support	Model Context Protocol server over SSE for tool integration
Developer Integrations	GitHub App, GitLab, Linear, Slack
Multi-Cloud	AWS, Azure, Modal.com via Terraform

Architecture

Observatory follows a three-tier observability architecture separating the write path (ingest), control plane (PostgreSQL), and data plane (ClickHouse).

┌──────────────────────────────────────────────────────────────┐
│                     Client SDKs / Agents                     │
│          (sdk-node, sdk-py, LangGraph workflows)             │
└───────────────────────────┬──────────────────────────────────┘
                            │ HTTP
                            ▼
┌──────────────────────────────────────────────────────────────┐
│                  Fastify API Server (:3001)                   │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────────────┐ │
│  │  REST /v1/* │  │  tRPC /trpc  │  │  MCP SSE /v1/mcp    │ │
│  └──────┬──────┘  └──────┬───────┘  └──────────┬──────────┘ │
│         │  Ingest        │  Query               │ Tool Exec  │
└─────────┼────────────────┼──────────────────────┼────────────┘
          │                │                      │
    ┌─────▼──────┐   ┌─────▼──────┐        ┌─────▼──────┐
    │   Redis    │   │ClickHouse  │        │  PostgreSQL │
    │  (BullMQ)  │   │(Data Plane)│        │(Ctrl Plane) │
    └─────┬──────┘   └────────────┘        └────────────┘
          │
    ┌─────▼──────────────────────────────────────────────┐
    │              Telemetry Workers (BullMQ)             │
    │  - telemetry-ingest (ClickHouse batch inserts)      │
    │  - agent-execution-queue (LangGraph workflows)      │
    │  - deployment-intents (artifact management)         │
    └────────────────────────────────────────────────────┘

Data Flow

Write Path — Clients POST /v1/ingest/ with batched telemetry. The API validates payloads with Zod, applies rate limiting (10k req/min per project), and enqueues to Redis (BullMQ). Workers consume jobs and batch-insert into ClickHouse. Load shedding activates at queue depth > 50k.
Read Path — The dashboard queries via tRPC resolvers that hit ClickHouse for analytics (runs, waterfall, metrics, costs).
Control Plane — PostgreSQL (via Drizzle ORM) stores users, organizations, projects, API keys, alert rules, model registry, deployments, and integration credentials.
Agent Execution — Python LangGraph workflows execute via Modal.com or gRPC, with results stored back in both PostgreSQL and ClickHouse.

For an in-depth breakdown, see docs/ARCHITECTURE.md.

Tech Stack

Backend

Layer	Technology
HTTP Server	Fastify 4.26
RPC Protocol	tRPC 10.45
Language	TypeScript 5.4, Node.js ≥ 20
Schema Validation	Zod 3.22
ORM	Drizzle ORM 0.30
Job Queue	BullMQ 5.7 (Redis-backed)
Auth	JWT (`@fastify/jwt`), bcrypt 6
Logging	Pino 10
Metrics	`prom-client` 15 (Prometheus)
Email	Resend

Data

Layer	Technology
Control Plane DB	PostgreSQL 15 + pgvector
Analytics DB	ClickHouse (ReplacingMergeTree)
Cache / Queue	Redis 7
Object Storage	AWS S3 / Cloudflare R2

Frontend

Layer	Technology
Framework	Next.js (App Router)
Styling	(see `apps/web/`)

AI & Agents

Layer	Technology
LLM Providers	Anthropic SDK 0.80, OpenAI SDK 4.104
Agent Framework	LangGraph (Python)
Data Validation	Pydantic
MCP	`@modelcontextprotocol/sdk` 1.27
Vector DBs	Qdrant, Pinecone, ChromaDB

Infrastructure

Layer	Technology
Monorepo	Turborepo + pnpm workspaces
Containers	Docker + Docker Compose
IaC	Terraform (AWS, Azure, Modal)
Serverless Agents	Modal.com

Project Structure

openChain.app/software/
├── apps/
│   ├── api/                  # Fastify API server (TypeScript)
│   │   └── src/
│   │       ├── server.ts     # Entry point, plugin registration
│   │       ├── config.ts     # Env var schema (Zod)
│   │       ├── db/
│   │       │   ├── postgres/ # Drizzle schema + migrations
│   │       │   └── clickhouse/ # ClickHouse client + insert ops
│   │       └── modules/
│   │           ├── auth/     # JWT, API key middleware
│   │           ├── ingest/   # Telemetry ingestion routes + worker
│   │           ├── query/    # tRPC analytics resolvers
│   │           ├── agents/   # Agent execution jobs
│   │           ├── alerts/   # Alert rule engine
│   │           ├── deployment/ # Deployment intent lifecycle
│   │           ├── mcp/      # Model Context Protocol server
│   │           ├── security/ # Threat detection
│   │           ├── privacy/  # PII handling
│   │           ├── vectors/  # Vector DB analytics
│   │           ├── cost/     # LLM pricing & USD estimates
│   │           └── adapters/ # Provider adapters (Redis, vector DBs)
│   ├── web/                  # Next.js dashboard
│   └── agents/               # Python LangGraph agent workflows
│       └── runtime.py        # Agent orchestration entry point
│
├── packages/
│   ├── types/                # Shared TypeScript types & Zod schemas
│   ├── cli/                  # CLI tooling
│   ├── sdk-node/             # Node.js client SDK
│   └── sdk-py/               # Python client SDK
│
├── infra/
│   ├── clickhouse/           # ClickHouse init + schema configs
│   ├── cloudflare/           # Edge configs
│   └── docker/               # Dockerfiles
│
├── terraform/
│   ├── aws/                  # AWS deployment modules
│   ├── azure/                # Azure deployment
│   └── modal/                # Modal.com integration
│
├── docker-compose.yml        # Local dev stack
├── pnpm-workspace.yaml
├── turbo.json                # Turborepo pipeline
└── package.json

Getting Started

Prerequisites

Node.js ≥ 20.0.0
pnpm 9.1.0 — npm install -g pnpm@9.1.0
Docker + Docker Compose (for local databases)
Python ≥ 3.11 (for agent workflows)

Installation

# 1. Clone the repository
git clone https://github.com/noureddinle/Lschema.git
cd Lschema/software

# 2. Install all dependencies
pnpm install

# 3. Start local infrastructure (PostgreSQL, ClickHouse, Redis)
docker-compose up -d

# 4. Copy and configure environment variables
cp apps/api/.env.example apps/api/.env
# Edit apps/api/.env with your values (see Environment Variables below)

# 5. Push database schema
pnpm run db:push

Environment Variables

Create apps/api/.env with the following variables:

# ─── App ───────────────────────────────────────────────────────
NODE_ENV=development
PORT=3001
LOG_LEVEL=info

# ─── PostgreSQL (Control Plane) ────────────────────────────────
DATABASE_URL=postgresql://observatory:Qwerty123@localhost:5436/observatory

# ─── ClickHouse (Data Plane) ───────────────────────────────────
CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_USER=clickhouse
CLICKHOUSE_PASSWORD=Qwerty123
CLICKHOUSE_DB=clickhouse

# ─── Redis ─────────────────────────────────────────────────────
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_URL=redis://localhost:6379
MEMORY_CACHE_MAX_ITEMS=5000
MEMORY_CACHE_TTL_MS=60000

# ─── Object Storage (AWS S3 / Cloudflare R2) ───────────────────
R2_ACCOUNT_ID=your_account_id
R2_ACCESS_KEY_ID=your_access_key
R2_SECRET_ACCESS_KEY=your_secret_key
R2_BUCKET_NAME=observatory-artifacts

# ─── Authentication & Security ─────────────────────────────────
JWT_SECRET=your_jwt_secret_minimum_32_characters_long
ENCRYPTION_KEY=your_encryption_key_for_sensitive_data

# ─── Email ─────────────────────────────────────────────────────
RESEND_API_KEY=re_your_resend_api_key
EMAIL_FROM_ADDRESS=Observatory <noreply@yourdomain.com>
FRONTEND_URL=http://localhost:3000

# ─── Slack ─────────────────────────────────────────────────────
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxx/yyy/zzz

# ─── GitHub Integration ────────────────────────────────────────
GITHUB_APP_ID=your_github_app_id
GITHUB_APP_PRIVATE_KEY=your_github_app_private_key
GITHUB_WEBHOOK_SECRET=your_github_webhook_secret

# ─── Linear Integration ────────────────────────────────────────
LINEAR_WEBHOOK_SECRET=your_linear_webhook_secret

# ─── LLM Providers ─────────────────────────────────────────────
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

Security Note: Never commit .env files. Use secrets management (AWS Secrets Manager, HashiCorp Vault, etc.) in production.

Running Locally

# Start all services in watch mode (hot reload)
pnpm run dev

# Or start individually:
# API server
pnpm --filter api dev

# Web dashboard
pnpm --filter web dev

The API will be available at http://localhost:3001 and the dashboard at http://localhost:3000.

Useful dev commands:

# Explore the PostgreSQL database interactively
pnpm run db:studio

# Type check all packages
pnpm run typecheck

# Lint all packages
pnpm run lint

# Build for production
pnpm run build

Usage

Ingesting Telemetry

Send telemetry from your LLM application to the ingest endpoint. Authenticate with your project's API key.

Batch Ingest (Spans, Logs, Agent Runs, LLM Calls)

curl -X POST http://localhost:3001/v1/ingest/ \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "x-project-id: YOUR_PROJECT_ID" \
  -d '{
    "agent_runs": [
      {
        "run_id": "run_abc123",
        "goal": "Summarize quarterly earnings report",
        "outcome": "success",
        "total_tokens": 4821,
        "prompt_tokens": 3200,
        "completion_tokens": 1621,
        "estimated_cost_usd": 0.048,
        "duration_ms": 12400,
        "started_at": "2026-04-23T14:00:00Z",
        "ended_at": "2026-04-23T14:00:12.4Z"
      }
    ],
    "llm_calls": [
      {
        "call_id": "call_xyz789",
        "run_id": "run_abc123",
        "provider": "anthropic",
        "model": "claude-opus-4-5",
        "prompt_tokens": 3200,
        "completion_tokens": 1621,
        "duration_ms": 9800,
        "finish_reason": "stop",
        "status": "success",
        "estimated_cost_usd": 0.048
      }
    ],
    "spans": [],
    "logs": [],
    "metrics": []
  }'

Response: 202 Accepted — data is enqueued for processing.

API / Infrastructure Telemetry

# HTTP/API traces
POST /v1/ingest/api-telemetry

# Database & worker spans
POST /v1/ingest/infra-telemetry

Rate Limits: 10,000 requests/minute per project. The server sheds load automatically when the queue depth exceeds 50,000 jobs.

Querying Analytics

Observatory exposes a fully type-safe tRPC API for querying analytics data from ClickHouse.

import { createTRPCProxyClient, httpBatchLink } from '@trpc/client';

const trpc = createTRPCProxyClient({
  links: [
    httpBatchLink({
      url: 'http://localhost:3001/trpc',
      headers: { Authorization: `Bearer ${accessToken}` },
    }),
  ],
});

// Get all agent runs for a project
const runs = await trpc.getRuns.query({
  projectId: 'proj_abc',
  limit: 50,
  offset: 0,
});

// Get waterfall trace for a specific run
const waterfall = await trpc.getRunWaterfall.query({ runId: 'run_abc123' });

// Get aggregated global metrics
const metrics = await trpc.getGlobalMetrics.query({ projectId: 'proj_abc' });

// Get model performance metrics
const modelMetrics = await trpc.getModelMetrics.query({
  projectId: 'proj_abc',
  model: 'claude-opus-4-5',
});

Agent Execution

Deploy and execute LangGraph agent workflows via the deployment system.

# Create a deployment intent
curl -X POST http://localhost:3001/v1/deployments/ \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -H "Idempotency-Key: deploy_$(date +%s)" \
  -d '{
    "projectId": "proj_abc",
    "entrypoint": "agents/workflows/incident:run",
    "targetType": "agent",
    "environment": "production"
  }'

Deployment lifecycle: pending → queued → executing → success | failed

Documentation

All detailed documentation lives in the docs/ directory. Each major system has its own dedicated guide:

Document	Description
docs/DATABASE.md	Database — Full PostgreSQL schema reference, all ClickHouse tables with column types, Drizzle ORM usage, migrations, query patterns, and performance tuning
docs/AGENTS.md	Agents — Python LangGraph workflow architecture, all workflow types (incident, docs, summary), TypeScript executors, state schemas, and how to add new agents
docs/INGEST.md	Ingestion — Complete telemetry ingest pipeline, all data types with schemas, rate limiting, backpressure/load-shedding, validation, and client best practices
docs/AUTH.md	Authentication — JWT flow, token rotation, API key lookup with two-tier cache, RBAC roles, tier-based feature gating, integration auth, encryption
docs/QUEUE.md	Job Queues — BullMQ architecture, all queues and job types, worker configuration, retry logic, dead letter queues, backpressure, and monitoring
docs/ALERTS.md	Alerts — Alert engine polling, rule schema, all signal types and condition fields, example rules, Slack notifications, alert lifecycle
docs/SECURITY.md	Security — Post-ingest threat detection, all built-in detectors (injection, exfiltration, anomalous usage), PII module, severity levels, custom detectors
docs/INTEGRATIONS.md	Integrations — GitHub App (OAuth, webhooks), GitLab (token auth, client), Linear (issue creation, SDK), Slack (Block Kit messages)
docs/MCP.md	Model Context Protocol — MCP server implementation, SSE transport, all available tools, Claude Desktop/Cursor setup, adding new tools
docs/VECTORS.md	Vector Databases — Qdrant, Pinecone, ChromaDB adapters, retrieval telemetry schema, analytics queries, score quality tuning
docs/COST.md	Cost & Pricing — PricingService architecture, built-in provider pricing, cost calculation formula, analytics queries, optimization strategies
docs/CACHE.md	Caching — Two-tier LRU + Redis cache, API key lookup flow, rate limit counters, pricing cache, invalidation strategies, tuning
docs/SDK.md	SDKs — Node.js and Python SDK usage, batching, LangChain/OpenAI/Anthropic wrappers, LangGraph integration, testing patterns
docs/ARCHITECTURE.md	Architecture — Deep-dive system design, all data flows, component interactions, architectural decisions and trade-offs
docs/API.md	API Reference — All REST and tRPC endpoints with request/response examples
docs/DEPLOYMENT.md	Deployment — Local dev, Docker Compose, AWS/Azure Terraform, Modal.com agents, migrations, security hardening, scaling, monitoring

API Reference

Full API documentation is available in docs/API.md.

Quick Reference

Method	Endpoint	Description	Auth
`POST`	`/v1/auth/register`	Register a new user	Public
`POST`	`/v1/auth/login`	Login and get JWT tokens	Public
`POST`	`/v1/auth/refresh`	Refresh access token	Public
`POST`	`/v1/ingest/`	Batch telemetry ingest	API Key
`POST`	`/v1/ingest/api-telemetry`	HTTP/API traces	API Key
`POST`	`/v1/ingest/infra-telemetry`	DB/worker spans	API Key
`*`	`/trpc/*`	tRPC analytics queries	JWT
`GET`	`/v1/mcp/sse`	MCP SSE connection	Optional
`POST`	`/v1/mcp/message`	MCP tool execution	Optional
`POST`	`/v1/deployments/`	Create deployment intent	JWT
`GET`	`/v1/deployments/:id`	Get deployment status	JWT
`GET`	`/health`	Health check	Public

Deployment

Docker Compose (Local / Staging)

The included docker-compose.yml starts all required services:

docker-compose up -d

This starts:

Service	Port	Purpose
PostgreSQL 15 + pgvector	`5436`	Control plane database
ClickHouse Alpine	`8123` (HTTP), `9000` (native)	Analytics data plane
Redis 7 Alpine	`6379`	Job queue + cache

AWS (Production)

Terraform modules are provided for a full AWS deployment:

cd terraform/aws
terraform init
terraform plan -var-file="production.tfvars"
terraform apply

Provisions:

VPC with public/private subnets (10.0.0.0/16)
RDS PostgreSQL (db.t4g.medium)
Self-hosted ClickHouse
Auto-scaling Fastify API
Next.js dashboard (public subnets)

See docs/DEPLOYMENT.md for the full guide, including Azure and Modal.com deployments.

Production Checklist

Contributing

We welcome contributions! Please read CONTRIBUTING.md for:

Code of conduct
Development setup
Branch strategy and PR workflow
Coding standards (ESLint + Prettier)
How to add new telemetry types
How to add new integrations

Quick Contribution Guide

# 1. Fork and clone
git clone https://github.com/YOUR_USERNAME/Lschema.git

# 2. Create a feature branch
git checkout -b feat/your-feature-name

# 3. Make changes and verify
pnpm run typecheck
pnpm run lint
pnpm run build

# 4. Commit with conventional commits
git commit -m "feat(ingest): add support for multimodal token tracking"

# 5. Push and open a PR
git push origin feat/your-feature-name

License

This project is licensed under the MIT License — see the LICENSE file for details.

Built with care by Block and the open-source community.

Star the repo if Observatory helps you ship better AI systems.

Name		Name	Last commit message	Last commit date
Latest commit History 84 Commits
software		software
.gitignore		.gitignore
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

Observatory

Table of Contents

Overview

Features

Architecture

Data Flow

Tech Stack

Backend

Data

Frontend

AI & Agents

Infrastructure

Project Structure

Getting Started

Prerequisites

Installation

Environment Variables

Running Locally

Usage

Ingesting Telemetry

Batch Ingest (Spans, Logs, Agent Runs, LLM Calls)

API / Infrastructure Telemetry

Querying Analytics

Agent Execution

Documentation

API Reference

Quick Reference

Deployment

Docker Compose (Local / Staging)

AWS (Production)

Production Checklist

Contributing

Quick Contribution Guide

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages