AscendAI

Multi-provider AI orchestrator with MCP, RAG, and semantic memory. Built on Spring AI.

graph TB
    User["User"]

    subgraph "AscendAI Platform"
        Agent["ascend-agent<br/>REST API :9917<br/>Spring Boot · Java 21"]

        subgraph "MCP Tool Services"
            AudioScribe["ascend-audio-scribe<br/>:7017<br/>Audio Transcription"]
            Weather["ascend-weather-mcp<br/>:9998<br/>Weather Data"]
            WebHunter["ascend-web-hunter<br/>:7021<br/>Web Search"]
            PaddleOCR["ascend-ocr<br/>:7022<br/>OCR"]
        end

        Memory["AscendMemory<br/>:7020<br/>Semantic Memory"]
    end

    subgraph "AI Providers"
        Local["LM Studio<br/>(local, default)"]
        Cloud["OpenAI · Anthropic<br/>Gemini · MiniMax"]
    end

    subgraph "Data Layer"
        PG["PostgreSQL<br/>Chat · Metadata"]
        RD["Redis<br/>Chat cache"]
        QD["Qdrant<br/>Vector DB"]
        S3["Object Store<br/>Documents"]
    end

    User -->|"REST"| Agent
    Agent -->|"MCP"| AudioScribe
    Agent -->|"MCP"| Weather
    Agent -->|"MCP"| WebHunter
    Agent -->|"MCP"| PaddleOCR
    Agent -->|"REST"| Memory
    Agent -.-> Local
    Agent -.-> Cloud
    Agent --> PG
    Agent --> RD
    Agent --> QD
    Agent --> S3
    Memory --> QD

Why this exists

I built AscendAI because off-the-shelf orchestrators don't let you swap providers per request, run a privacy-respecting search backend you fully control, and persist semantic memory across sessions in one coherent platform. AscendAI does all three. Each prompt routes to the model you pick at call time (local LM Studio, OpenAI, Anthropic, Gemini, MiniMax). MCP tool servers handle audio, web, weather, and OCR. Long-term context lives in a Mem0-backed Qdrant store so conversations actually accumulate knowledge.

Features

Per-request provider routing. Pick LM Studio, OpenAI, Anthropic, Gemini, or MiniMax on every API call without restarts or config changes.
RAG pipeline with Qdrant. Thresholded soft-retrieval over ingested documents using provider-matched embedding dimensions (768 / 1536).
Semantic memory via Mem0. Long-lived, user-scoped memories searchable across sessions through the AscendMemory service.
MCP tool servers. First-class integrations for audio transcription (ascend-audio-scribe), web search (ascend-web-hunter + SearXNG), weather (ascend-weather-mcp), and OCR (ascend-ocr).
Document ingestion to object storage. Drop files (Markdown, PDF, DOCX) into a bucket and the pipeline parses them via Docling / Unstructured and indexes them automatically.
Hybrid chat history. Redis for the active context window, PostgreSQL for durable long-term archives and analytics.
Privacy-respecting web. SearXNG meta-search plus FlareSolverr for Cloudflare-protected pages, all self-hosted.

How it compares

The honest peer set is other deployable AI orchestration backends that bundle multi-provider routing, RAG, memory, and tools into one self-hosted service. Not chat UIs, not low-code workflow builders, not pure router proxies, not libraries. All four below are mature and well-known in this niche.

	AscendAI	R2R	Letta	Onyx	Quivr	LangChain
Shape	Deployable service	Deployable service	Deployable service	Deployable service	Deployable service	Library / framework
Stack	Java 21 / Spring AI	Python	Python	Python	Python	Python (TS port)
API-first (no UI shipped)	Yes	Yes	Yes (server on `:8283`)	UI bundled, API-driven	UI bundled, API exposed	N/A, you build it
Per-request provider switch	Built-in	Built-in	Built-in	Built-in	Built-in	Possible via chain rebuild
RAG over uploaded docs	Built-in (Qdrant + threshold)	Built-in (multimodal, hybrid, KGs)	Lighter, agent-state focused	Built-in (40+ connectors)	Built-in (pluggable stores)	Many backends, you wire it
Persistent semantic memory	Mem0 + Qdrant	Add-on	Native (OS-style hierarchical)	Add-on	Built-in	Roll-your-own
Tool integration model	MCP-native (Spring AI MCP client)	Function tools	Function tools	Function tools + connectors	Function tools	Tools + MCP via adapters
Single docker compose deploy	Yes	Yes	Yes	Yes	Yes	Bring-your-own

LangChain isn't strictly a peer. It's a framework, not a deployable service. It's in the table because it's the most likely thing readers reach for when they think "AI orchestration", and the honest answer is "if you're already wiring your own service in LangChain, you don't need AscendAI."

Where AscendAI is honestly distinctive

JVM-native. Every credible peer in this niche is Python or TypeScript. If you live in Spring Boot already, AscendAI drops in alongside the rest of your services without a polyglot deploy.
MCP-first tool model. Onyx and Letta do tool use; AscendAI is built around MCP from day one with multiple bundled MCP servers (audio, OCR, web search, weather). Add new tools by pointing the agent at another MCP server, no code changes.
Breadth of integration in one stack. RAG, semantic memory, MCP tools, multi-provider routing, hot / archive chat history. All present, no add-ons.

Where it loses

No UI. Onyx and Quivr ship one. AscendAI is a backend you put behind your own client.
Smaller community. All four peers above have more stars, more contributors, more battle testing.
RAG depth. R2R has a more sophisticated RAG pipeline (knowledge graphs, multimodal). AscendAI's RAG is solid but plain.
Memory depth. Letta's memory architecture is more advanced than the Mem0-based memory here.

If you're already happy in Python with R2R or Letta, you don't need this. AscendAI exists because I wanted these capabilities in a Spring-native deployment.

Demo

Send a prompt with per-request provider and model selection. The endpoint accepts multipart/form-data (so you can attach an optional image or document).

Bash:

curl -X POST http://localhost:9917/api/v1/ai/prompt \
  -H "X-User-Id: luksarna" \
  -F "prompt=Summarize my notes on Spring AI and MCP." \
  -F "provider=anthropic" \
  -F "model=claude-sonnet-4-6" \
  -F "embeddingProvider=lmstudio"

PowerShell 7+ (-Form supports multipart):

Invoke-RestMethod -Uri http://localhost:9917/api/v1/ai/prompt -Method Post -Headers @{ "X-User-Id" = "luksarna" } -Form @{ prompt = "Summarize my notes on Spring AI and MCP."; provider = "anthropic"; model = "claude-sonnet-4-6"; embeddingProvider = "lmstudio" }

Sample response (AiResponse: content plus an unwrapped Spring AI ChatResponseMetadata and the list of MCP tools invoked during the turn):

{
  "content": "Your notes describe AscendAI as a Spring AI orchestrator that routes prompts across providers and uses MCP for tool calls. Per-request model selection happens via /api/v1/ai/prompt; RAG runs over Qdrant collections (ascendai-768 / -1536); semantic memory is backed by Mem0…",
  "id": "msg_01ABcDEf…",
  "model": "claude-sonnet-4-6",
  "usage": { "promptTokens": 1842, "completionTokens": 312, "totalTokens": 2154 },
  "toolsUsed": ["ascend_memory_search", "web_search"]
}

Architecture

Two architecture entry points, depending on what you're after.

Monorepo architecture. System overview, service interactions, deployment, ADRs.
ascend-ai-agent arc42. Internals, component diagrams, module ADRs.

Module	Stack	Port	Role
ascend-ai-agent	Java 21 / Spring Boot	9917	API gateway, multi-provider AI, RAG, MCP client
ascend-audio-scribe	Python / FastMCP	7017	Audio transcription (Whisper / OpenAI / HF)
ascend-web-hunter	Python / FastMCP	7021	Web search and scraping via SearXNG
AscendMemory	Python / FastAPI	7020	Semantic memory (Mem0 + Qdrant)
ascend-weather-mcp	Java / Spring Boot	9998	Weather data MCP server
ascend-ocr	Python / FastMCP	7022	OCR service

Request flow

How a single prompt traverses the platform.

sequenceDiagram
    autonumber
    participant U as User
    participant A as ascend-agent
    participant R as Redis<br/>(short-term)
    participant M as AscendMemory<br/>(Mem0)
    participant Q as Qdrant<br/>(RAG)
    participant T as MCP Tools<br/>(Web/Audio/OCR/Weather)
    participant L as LLM Provider<br/>(per-request)
    participant P as PostgreSQL<br/>(long-term)

    U->>A: POST /api/v1/ai/prompt
    A->>R: Load chat window
    A->>M: Search semantic memory
    M->>Q: Vector search (memory collection)
    M-->>A: Top-k memories
    A->>Q: RAG retrieval (doc collection)
    Q-->>A: Top-k chunks (above threshold)
    A->>L: Prompt + memory + RAG + tool defs
    L-->>A: Tool call request (optional)
    A->>T: Invoke MCP tool
    T-->>A: Tool result
    A->>L: Tool result, then final answer
    L-->>A: Response + usage
    A->>R: Append turn
    A->>P: Persist transcript
    A->>M: Extract and store new memories
    A-->>U: AiResponse {content, metadata, toolsUsed}

Quick Start

Prerequisites

Docker Desktop
Java 21
PostgreSQL on 5432, Redis on 6379, Qdrant on 6333 / 6334, S3-compatible object storage on 9070 / 9071 (locally: Floci, admin / password)

Run it

1. Provision secrets.

Copy the example file and fill in the API keys you plan to use. .env is gitignored. Provider keys are optional individually. Leave a key blank and just don't pick that provider at request time.

Bash:

cp .env.example .env

PowerShell:

Copy-Item .env.example .env

One value is not optional. SEARXNG_SECRET must be filled in or the stack refuses to start, naming the missing variable. It has to be at least 32 characters, unique to this deployment, and not the literal ultrasecretkey.

2. Bring up the stack.

The main compose file pulls in compose.ascend-web-hunter.yaml via include:, so a single up brings up the full stack (ascend-agent + tool services + scrapper).

Bash:

docker compose up -d --build

PowerShell:

docker compose up -d --build

3. Ensure PostgreSQL has the ascend_ai database (user postgres, password local).

On first start the agent creates the knowledge-base bucket in the object store and initialises metadata tables. The API is then available at http://localhost:9917. Check the startup banner for live status of every dependency.

4. Optional: run the agent on the host.

For active development with hot reload and an attached debugger, run the agent on the host instead of in the container. Stop the container first (docker compose stop ascend-agent) so port 9917 is free.

Bash:

cd apps/ascend-agent

./gradlew bootRun

PowerShell:

cd apps/ascend-agent

./gradlew bootRun

5. Optional: point IntelliJ at each module's Python interpreter.

The committed .idea/ project opens with the right module layout, but a Python interpreter is registered per user, not per project, so a fresh clone has none. Open each of apps/ascend-audio-scribe, apps/ascend-web-hunter, apps/ascend-memory, and apps/ascend-ocr in turn; IntelliJ detects the module's own .venv/ at its root and offers to use it. Accept the offer once per module.

For advanced compose flags, per-service rebuilds, and production notes see docs/DEPLOYMENT.md. For document ingestion see docs/INGESTION.md.

Supported AI Providers

Per-request selection across the providers below. Models listed are the ones currently wired in application.yaml (chat default, memory extraction, and history compaction). Any model the provider accepts works at request time via the model form field; these are the values that ship with the agent.

OpenAI. gpt-4o (default), gpt-4o-mini (extraction + compaction).
Anthropic. claude-sonnet-4-5 (default), claude-3-5-haiku-20241022 (extraction), claude-haiku-4-5 (compaction).
Gemini. gemini-flash-latest (default), gemini-flash-lite-latest (extraction + compaction).
MiniMax. MiniMax-M2.7 (default + extraction + compaction).
LM Studio. meta-llama-3.1-8b-instruct (default, local).

Override per request with the provider and model form fields, or globally via the *_MODEL env vars listed in .env.example.

Configuration & Ports

AscendAI services

Each service ships both REST and MCP surfaces (except ascend-weather-mcp, MCP-only). The "Used by ascend-ai-agent via" column shows the actual transport ascend-ai-agent uses today. The other surface is available for direct external use.

Service	Port	Surfaces	Used by ascend-ai-agent via	Role
ascend-agent	`9917`	REST	(this is the agent)	API gateway and orchestrator. `POST /api/v1/ai/prompt` is the entry.
AscendMemory	`7020`	REST + MCP	REST	Semantic memory store (Mem0 + Qdrant). Search / insert per user.
ascend-audio-scribe	`7017`	REST + MCP	MCP (Streamable HTTP)	Speech-to-text (faster-whisper / OpenAI / HF / Audacity merge).
ascend-web-hunter	`7021`	REST + MCP	MCP (Streamable HTTP)	Web search + content extraction (SearXNG, Cloudflare, NoVNC).
ascend-ocr	`7022`	REST + MCP	MCP (Streamable HTTP)	Image OCR.
ascend-weather-mcp	`9998`	MCP only (SSE)	MCP (SSE)	Weather data tool (reference Spring AI MCP server).

Support services (in-stack, deployed via compose)

Service	Port	Default credentials	Role
SearXNG	`9020` (loopback)	`SEARXNG_SECRET`	Privacy-respecting meta-search; backend for ascend-web-hunter.
FlareSolverr	`8191` (loopback)	(none)	Cloudflare bypass proxy used by ascend-web-hunter.
ngrok (web-hunter)	(none)	`NGROK_AUTHTOKEN`	Public tunnel to ascend-web-hunter's NoVNC for remote CAPTCHA intervention.
Docling Serve	`5001`	(none)	PDF / DOCX to structured JSON (used by ingestion pipeline).
Unstructured API	`9080`	(none)	Generic document parsing fallback for ingestion.

Observability stack (in-stack, deployed via compose)

Full setup and usage in infra/observability/README.md.

Service	Port	Exposed	Role
Grafana	`7078` → `3000`	Browser UI	Dashboards + Explore. Anonymous `Viewer`; `admin` / `admin` to edit.
Prometheus	`7077` → `9090`	Browser UI	Scrapes metrics from the 6 services and Qdrant.
Loki	`3100`	Internal only	Log store; receives logs from Vector.
Tempo	(none)	Internal only	Trace store; receives traces from the OTel Collector.
Vector	(none)	Internal only	Tails the 6 app containers' Docker logs and ships them to Loki.
OTel Collector	(none)	Internal only	Receives OTLP traces from the services and exports them to Tempo.

External prerequisites (not in compose, managed / cloud in production)

Service	Port	Default credentials	Role
PostgreSQL	`5432`	`postgres` / `local`	Chat-history archive, ingestion metadata, user instructions.
Redis	`6379`	(none)	Short-term chat-history cache, session state.
Qdrant	`6333` / `6334`	(none)	Vector DB for RAG (`ascendai-768/1536`) and Mem0 memory.
Object storage (locally: Floci)	`9070` / `9071`	`admin` / `password`	S3-compatible object store for ingested documents. Port 9071 is the object-store web UI.

Documentation

Canonical index. Every doc the repo ships, in one place.

File	What's in it
docs/architecture/README.md	Monorepo architecture: system view, ADRs, deployment topology.
docs/architecture/arc42/01-introduction-and-goals.md	Arc42 entry point for the platform.
apps/ascend-agent/docs/architecture/arc42/01-introduction-and-goals.md	Arc42 for the agent internals.
docs/architecture/permission-aware-retrieval.md	How document access lists reach chunks and get enforced inside the vector search.
docs/DEPLOYMENT.md	Docker Compose recipes, image publishing, prod notes.
apps/ascend-web-hunter/deploy-standalone/README.md	Copy-and-run bundle for the web-search stack on a host of its own.
.github/workflows/README.md	CI and release workflows, image naming, registries, package visibility.
contracts/README.md	Agent to OCR Pact contract: who writes it, who verifies it, how to regenerate it.
docs/INGESTION.md	Upload flows for the RAG pipeline.
docs/TROUBLESHOOTING.md	Qdrant / object store / PostgreSQL / Redis reset recipes.
docs/OBSERVABILITY.md	Metrics, logs, traces - what is collected, dashboards, how to instrument.
infra/observability/README.md	Observability stack services (Grafana / Prometheus / Loki / Tempo / Vector / OTel), pipeline, and how to view logs.
docs/AGENT_TOOLING.md	Agent-standards import, OpenSpec workflow.
docs/AGENTS-UPDATE.md	Per-OS selective refresh of skills, subagents, and shipped docs.
docs/MCP_SETUP.md	How to configure the MCP servers wired into agent sessions.
apps/ascend-agent/e2e/README.md	End-to-end capability tests, fixtures, Bruno collection.
docs/E2E_COST.md	Token usage and per-provider pricing for the e2e suite; recalculate dollar cost when prices change.
docs/DEFECT_REGISTER.md	Every defect found across the stack: what's fixed with its commit, what's still open, what was declined.
AGENTS.md	Shared instructions for any AI coding agent operating in this repo.
.github/workflows/README.md	CI and Release workflow operator notes: secrets, bump convention, how to cut a release.

License

Released under the MIT License.

Name		Name	Last commit message	Last commit date
.agents		.agents
.claude		.claude
.codex		.codex
.github		.github
.idea		.idea
.kilo		.kilo
.opencode		.opencode
.run		.run
apps		apps
contracts		contracts
docs		docs
infra		infra
openspec		openspec
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.mcp.json		.mcp.json
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
compose.ascend-web-hunter.yaml		compose.ascend-web-hunter.yaml
compose.yaml		compose.yaml
opencode.json		opencode.json

Navigation Menu

Repository navigation

About

Repository files navigation

AscendAI

Table of Contents

Why this exists

Features

How it compares

Where AscendAI is honestly distinctive

Where it loses

Demo

Architecture

Request flow

Quick Start

Prerequisites

Run it

Supported AI Providers

Configuration & Ports

AscendAI services

Support services (in-stack, deployed via compose)

Observability stack (in-stack, deployed via compose)

External prerequisites (not in compose, managed / cloud in production)

Documentation

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

AscendAI

Table of Contents

Why this exists

Features

How it compares

Where AscendAI is honestly distinctive

Where it loses

Demo

Architecture

Request flow

Quick Start

Prerequisites

Run it

Supported AI Providers

Configuration & Ports

AscendAI services

Support services (in-stack, deployed via compose)

Observability stack (in-stack, deployed via compose)

External prerequisites (not in compose, managed / cloud in production)

Documentation

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages