Feature Description
LogTide currently runs as a single instance per service: one backend, one worker, one Postgres/TimescaleDB, one Redis, one ClickHouse or MongoDB when those profiles are enabled. There is no documented or supported way to scale the platform horizontally. This issue is a placeholder to evaluate which approaches make sense, layer by layer, and to capture the constraints that the multi-engine architecture and the per-org isolation model impose on any future design.
This is a research issue, not an implementation issue. The goal is to land on one or two recommended deployment shapes and then split the actual work into follow-ups.
Problem/Use Case
A self-hosted LogTide instance has two orthogonal axes of growth:
- Per-instance load. A single org or project ingests enough logs/spans/metrics that a single backend, a single worker, or a single storage node becomes the bottleneck.
- Multi-tenant load. The instance hosts many orgs, and one large or noisy tenant should not degrade the others. Some tenants may also have compliance or data-residency requirements that justify dedicated storage.
Right now neither axis has an answer beyond "give the box more cores". For a project that positions itself as a self-hosted alternative to large platforms, this is a credibility gap.
Current Architecture (Recap)
Layers that need an answer:
- Backend API (Fastify). Mostly stateless, except for SSE streams which today are tied to the process that accepts the connection.
- Worker (BullMQ). Already supports multiple instances natively via Redis, but never validated at scale and not documented as a supported deployment.
- Frontend. Stateless, trivial to replicate.
- Postgres / TimescaleDB. Single instance. Hypertables, continuous aggregates, retention and compression all live here.
- Redis. Single instance. Hosts queue, rate limiter state, sessions, and any pub/sub primitives we add.
- ClickHouse (optional profile). Single node today. Native sharding and replication available but unused.
- MongoDB (optional profile). Single node today. Replica set and sharding available but unused.
- Reservoir abstraction. The seam that makes the storage engine pluggable. This is the natural place to introduce tenant-aware routing.
Approaches to Evaluate
A. Stateless backend + shared infra (small step)
N backend instances behind a load balancer, N workers, shared Redis, shared storage. SSE either pinned via sticky sessions on the LB or fanned out through Redis pub/sub so any backend can serve any subscriber. Rate limiter must use Redis as the source of truth (verify current behavior). Internal logging keeps working because each replica points at the same backend host externally.
Pros: smallest delta from today, unblocks vertical CPU/RAM headroom on the API tier, validates the stateless contract.
Cons: storage and Redis remain single points of failure. Does not address the noisy-tenant problem.
B. Clustered storage and Redis HA (medium step)
Same as A, plus: Redis Sentinel or Cluster, TimescaleDB read replicas (or Citus-style scale-out for the pieces that support it), ClickHouse with native sharding and replication, MongoDB replica sets and sharding for the Mongo profile. Worker concurrency tuned per backend.
Pros: removes the obvious SPOFs, lets each engine play to its native scaling story.
Cons: significantly more operational surface. Each engine has a different HA model, so docs and validation multiply by the number of supported reservoirs. Probably out of scope for OSS defaults; more realistic as documented reference architectures.
C. Tenant-aware routing with per-org reservoirs (key option)
Use the reservoir abstraction as a routing layer: each org (or each "tier" of orgs) maps to its own storage backend connection. A large org could get a dedicated TimescaleDB or ClickHouse cluster; small orgs share a default pool. The mapping lives in a small control-plane table and is resolved at request time from the org context.
Pros: directly addresses noisy neighbors and data-residency requirements. Aligns with the existing multi-engine direction. Lets a single deployment scale by adding storage clusters rather than by scaling one giant cluster vertically.
Cons: requires a tenant-to-reservoir resolution path on every read and write; cache invalidation matters. Migrations and retention policies must be fan-out aware. Cross-org admin queries become harder. Dashboard widgets that aggregate across orgs need a federated query path or a separate analytics store.
D. Full distributed deployment (end-state)
Kubernetes-first deployment with Helm, autoscaling on the backend and worker tiers, clustered storage per engine, Redis Cluster, plus an optional dedicated reservoir per large tenant (combination of B and C). This is the shape that an enterprise self-hoster would actually run.
Pros: covers every scaling axis.
Cons: large investment, large docs surface, hard to validate without realistic load. Only worth pursuing once A and the routing seam from C are in place.
Cross-Cutting Considerations
- SSE. Decide between sticky sessions and Redis pub/sub fan-out. Sticky is simpler; pub/sub is more flexible and survives backend restarts better.
- Rate limiting. Confirm that the limiter state is in Redis, not in-process. If not, that is a prerequisite for A.
- Sessions and API keys. Already Redis-backed for sessions; verify there is no in-process cache that breaks under N replicas.
- Internal logging (dogfooding). Each replica should report under a stable service name with the instance id surfaced as metadata, not as part of the service name.
- BullMQ. Validate the assumption that adding worker replicas just works. Check for any job that assumes singleton execution (cron-like jobs, Sigma sync, retention).
- Migrations. With per-org reservoirs (option C), migrations have to fan out across all reservoirs. Today they assume one DB.
- Health checks and readiness. Need to expose per-instance state, not aggregate, so the LB can drain a replica cleanly.
- Observability under HA. The current admin dashboard assumes one instance. It either becomes a per-instance view or aggregates across replicas through a control-plane query.
Open Questions
- Does the rate limiter today share state across replicas, or is it process-local?
- Are there any singleton background jobs (Sigma sync, retention, alert rule evaluation) that would double-fire if the worker is replicated?
- Is the SSE layer ready for fan-out via Redis, or is it still a per-process EventEmitter?
- For option C, is the tenant-to-reservoir mapping a runtime config (resolved per request) or a deploy-time config (one binary per shard)?
- Which engines do we want to officially support in HA mode, and which stay as "best effort"?
Priority
Target Users
- Self-hosters running LogTide for more than a handful of services or orgs
- Teams with one large tenant whose load would otherwise drown the rest
- Operators with data-residency or compliance requirements that justify dedicated storage per org
Contribution
Feature Description
LogTide currently runs as a single instance per service: one backend, one worker, one Postgres/TimescaleDB, one Redis, one ClickHouse or MongoDB when those profiles are enabled. There is no documented or supported way to scale the platform horizontally. This issue is a placeholder to evaluate which approaches make sense, layer by layer, and to capture the constraints that the multi-engine architecture and the per-org isolation model impose on any future design.
This is a research issue, not an implementation issue. The goal is to land on one or two recommended deployment shapes and then split the actual work into follow-ups.
Problem/Use Case
A self-hosted LogTide instance has two orthogonal axes of growth:
Right now neither axis has an answer beyond "give the box more cores". For a project that positions itself as a self-hosted alternative to large platforms, this is a credibility gap.
Current Architecture (Recap)
Layers that need an answer:
Approaches to Evaluate
A. Stateless backend + shared infra (small step)
N backend instances behind a load balancer, N workers, shared Redis, shared storage. SSE either pinned via sticky sessions on the LB or fanned out through Redis pub/sub so any backend can serve any subscriber. Rate limiter must use Redis as the source of truth (verify current behavior). Internal logging keeps working because each replica points at the same backend host externally.
Pros: smallest delta from today, unblocks vertical CPU/RAM headroom on the API tier, validates the stateless contract.
Cons: storage and Redis remain single points of failure. Does not address the noisy-tenant problem.
B. Clustered storage and Redis HA (medium step)
Same as A, plus: Redis Sentinel or Cluster, TimescaleDB read replicas (or Citus-style scale-out for the pieces that support it), ClickHouse with native sharding and replication, MongoDB replica sets and sharding for the Mongo profile. Worker concurrency tuned per backend.
Pros: removes the obvious SPOFs, lets each engine play to its native scaling story.
Cons: significantly more operational surface. Each engine has a different HA model, so docs and validation multiply by the number of supported reservoirs. Probably out of scope for OSS defaults; more realistic as documented reference architectures.
C. Tenant-aware routing with per-org reservoirs (key option)
Use the reservoir abstraction as a routing layer: each org (or each "tier" of orgs) maps to its own storage backend connection. A large org could get a dedicated TimescaleDB or ClickHouse cluster; small orgs share a default pool. The mapping lives in a small control-plane table and is resolved at request time from the org context.
Pros: directly addresses noisy neighbors and data-residency requirements. Aligns with the existing multi-engine direction. Lets a single deployment scale by adding storage clusters rather than by scaling one giant cluster vertically.
Cons: requires a tenant-to-reservoir resolution path on every read and write; cache invalidation matters. Migrations and retention policies must be fan-out aware. Cross-org admin queries become harder. Dashboard widgets that aggregate across orgs need a federated query path or a separate analytics store.
D. Full distributed deployment (end-state)
Kubernetes-first deployment with Helm, autoscaling on the backend and worker tiers, clustered storage per engine, Redis Cluster, plus an optional dedicated reservoir per large tenant (combination of B and C). This is the shape that an enterprise self-hoster would actually run.
Pros: covers every scaling axis.
Cons: large investment, large docs surface, hard to validate without realistic load. Only worth pursuing once A and the routing seam from C are in place.
Cross-Cutting Considerations
Open Questions
Priority
Target Users
Contribution