Skip to content

Performance: deferred aggregation mode for batch/composite operations #95

Description

@mitselek

Problem

Every mutation (create entity, add property, delete) triggers a synchronous full re-aggregation (aggregateEntity()). For single operations this is fine, but composite flows that create multiple related entities pay the aggregation cost N times sequentially.

Our PoC's "create organization" flow creates 8 entities (org + 5 roles + member + ACL property) → 8 sequential aggregations → 2-3 seconds total.

Source Analysis

entity.js line ~420-430: after every insertOne to the property collection, aggregateEntity() runs synchronously. aggregateEntity() in aggregate.js does:

  1. Read all properties from property collection
  2. Resolve type definitions (DB query)
  3. Two-pass formula evaluation (may trigger further reads)
  4. Rights inheritance from parents (DB query per parent)
  5. Build search index
  6. replaceOne into entity collection

Each aggregation is ~200-400ms. For 8 sequential operations: ~1.6-3.2s of pure aggregation.

Suggestion: Deferred Aggregation

Add an optional skipAggregation flag (or X-Skip-Aggregation header) to mutation endpoints. When set, the property write completes but aggregation is skipped. The caller triggers a single aggregation at the end of the batch:

POST /entity (skipAggregation=true)  → create org (fast)
POST /entity (skipAggregation=true)  → create role 1 (fast)
POST /entity (skipAggregation=true)  → create role 2 (fast)
...
POST /entity/{orgId}/aggregate       → single aggregation for org
POST /entity/{role1Id}/aggregate     → aggregate role 1
...

This would reduce our 8 aggregations to a final batch of targeted aggregations, saving 1-2 seconds.

Alternative: insertMany for properties

A simpler win: entity.js ~line 421 uses sequential insertOne for properties. Using insertMany when creating an entity with multiple properties would reduce the DB round-trips.

Impact

This primarily benefits:

  • Entity creation with children (our org setup flow)
  • Bulk imports (CSV plugin)
  • Any operation that creates multiple related entities

(ER:Codd + Saavedra)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions