feat: collaboration persistence + collaborative content → per-document file storage (003+006) - #6193
Open
antst wants to merge 15 commits into
Open
feat: collaboration persistence + collaborative content → per-document file storage (003+006)#6193antst wants to merge 15 commits into
antst wants to merge 15 commits into
Conversation
…le & authZ Author the server slice's SpecKit sub-spec for epic 003-unify-collab-yjs (WS-E: persistence + lifecycle + authZ integration with the unified collaboration-service). Spec/design only — no implementation. Doc set under specs/003-collaboration-persistence/: spec, plan, research, data-model, quickstart, tasks, checklists/requirements — mirroring the collaboration-service sub-spec shape. Grounded in server's real code: - server is the persistence RESPONDER (legacy collab services call in), not a caller — inverts the naive T001/T002 reading. - blob lives inline today: Memo.content (bytea v2), Whiteboard.content (text JSON). - authZ: READ='read', collaborate='update-content'; authorizationPolicyId = AuthorizableEntity.authorizationId (AuthorizationPolicy.id). The authorization-evaluation-service is a SEPARATE Go repo reading server's DB; server hosts no /internal/auth/evaluate. -> confirms the collab authzeval adapter is correct. - delete cascade emit point: MemoService.deleteMemo / WhiteboardService.deleteWhiteboard. Self-analyze clean: 11 FRs / 7 SCs / 4 OPENs, all traced in tasks.md. Implementation gated on (a) OPEN-1/OPEN-3 answers and (b) the collab Wave-2 adapter freezing the unified wire contract.
Implement the server slice of epic 003-unify-collab-yjs (WS-E): the unified collaboration-service persistence/lifecycle/authZ integration, against the FROZEN cross-repo RabbitMQ contract (collaboration-service contract.go). Server is the RESPONDER; the blob never crosses the unified bus (server is a pure metadata/index store on the new path). Schema (S-T001): - Add contentPointer + blobStore columns to Memo + Whiteboard (+ interfaces, internal — not on the GraphQL schema). New BlobStoreKind enum. - Reversible migration 1781802081405-AddContentPointerAndBlobStore with inline-default back-fill (contentPointer=id, blobStore='inline'). Unified consumer (S-T002): - New collaboration-integration module: @MessagePattern collaboration-save / -fetch / -delete / -info + @EventPattern collaboration-contribution, routed by contentType, manual ack. Index-only (no blob on the bus). Structured replies, no exception leaks. fetch carries authorizationPolicyId (FR-005). - Index-only domain methods get/save/deleteCollaborationMetadata on Memo + Whiteboard services. New COLLABORATION_SERVICE queue connected in main.ts. - Legacy dialects retained for coexistence (removed at the big-bang cutover). Lifecycle (S-T003): - CollaborationLifecycleService emits document.deleted (+ optional created / access_changed) via a new COLLABORATION_SERVICE outbound client, at the deleteMemo / deleteWhiteboard cascade leaves (exactly-once, fire-and-forget). AuthZ (S-T004): surface the entity's own authorizationPolicyId; read=READ, collaborate=UPDATE_CONTENT — confirms the collab authzeval adapter. No evaluate endpoint in server (that service is a separate Go repo). Migration (S-T005): CollaborationMigrationService — a batched one-pass read of legacy memo (v2 base64) + whiteboard (decompressed JSON) content + policy id, flagging corrupt blobs instead of dropping them. Touched collaboration code at 100% line coverage; tsc + biome clean; build green. tasks.md S-T001..S-T006 marked done.
The collaboration-service room owns the content `version`: it bumps the value per persisted snapshot, sends it on `collaboration-save`, and adopts the stored value back on `collaboration-fetch` when it rehydrates. The previous server impl ignored the contract `version` and instead bumped the shared TypeORM `@VersionColumn`, echoing that counter back — so a reloaded room saw the server's value, not its own. Add a dedicated, nullable `contentVersion` integer column to the memo and whiteboard rows (TypeORM migration:generate), distinct from the inherited `@VersionColumn` (which keeps its optimistic-locking role). On `collaboration-save` persist the contract `version` into `contentVersion`; on `collaboration-fetch` return the stored `contentVersion` as the reply's `version`. The info/delete/contribution paths are unaffected. - memo/whiteboard entities + interfaces: add `contentVersion` - CollaborationMetadataUpdate: carry the room-owned `version` - save/get CollaborationMetadata: persist/return `contentVersion`, stop bumping `@VersionColumn` - integration service `save`: forward the contract `version` - tests: version round-trips (save N -> fetch N); latest of two increasing saves wins; @VersionColumn is never written by the collab path Refs workspace#003-unify-collab-yjs
- save: reject unknown contentType deterministically instead of defaulting to the whiteboard write path; validate before routing. - save/delete: return generic client-safe error replies; keep ids/raw causes in server logs only (no internal details over the bus). Drop the unreachable EntityNotFoundException branch in delete (the domain UPDATE is idempotent). - info: wrap the responder body in try/catch so a lookup failure normalizes to a deny like save/fetch/delete, never throwing on the bus. - lifecycle emit: subscribe to the ClientProxy.emit Observable to catch ASYNC broker failures (a sync try/catch misses them); drop e?.message from the log. - get/save/deleteCollaborationMetadata (memo + whiteboard): coerce NULL pointer/store to undefined; clear contentVersion on delete so a post-delete fetch can't round-trip a stale version; saveCollaborationMetadata now returns the CollaborationMetadata projection instead of a partial IMemo/IWhiteboard. - migration read: switch offset (skip/take) to keyset pagination over the id PK so the one-pass read can't skip/duplicate rows under concurrent writes.
…aths - Flip the spec/plan/quickstart/tasks/checklist status from 'spec/design only, blocked' to 'implemented, in review (PR open)'; OPEN-1/OPEN-3 gates resolved and the unified contract frozen. - Replace absolute /Users/antst/... paths with the repo root / symbolic repo names so the docs are portable. - Clarify the coverage gate as a ≥95% target on the touched diff (DEC-7), not a repo-wide CI threshold change. - Escape a literal pipe in the research.md encoding table (markdownlint MD056).
The fire-and-forget collaboration-contribution event handler could still throw from a non-not-found metadata lookup or a downstream reporter call (getCommunity*/getProfile), failing RMQ message handling. Wrap it in try/catch + error log to match save/fetch/delete/info, which all normalize failures. Covered by a new test asserting a downstream failure is swallowed, not thrown.
Replace the two legacy collab backends in the dev stack with the single
unified collaboration-service for end-to-end testing of 003-unify-collab-yjs.
- quickstart-services.yml: drop whiteboard-collaboration (4002) and
collaborative-document (4004); add one collaboration service
(ghcr.io/alkem-io/collaboration-service:pr-5, port 4006) wired for the
Alkemio topology (FANOUT_MODE=redis, METADATA_STORE=rabbitmq,
BLOB_STORE=file-service, AUTH_MODE=authzeval) on queue alkemio-collaboration.
- .build/traefik/http.yml: replace the legacy /api/private/ws +
/api/private/hocuspocus routers with a single collaboration router
(PathPrefix /collab -> collaboration:4006), reusing the oathkeeper
forwardAuth that injects the Authorization actor id consumed at the WS
handshake. The client UnifiedCollabProvider opens /collab/{documentId}.
- .env.docker: add COLLAB_FILE_SERVICE_STORAGE_BUCKET_ID /
COLLAB_FILE_SERVICE_AUTHORIZATION_ID placeholders (seeded per-DB UUIDs,
filled post-seed; blank => fail-fast, no half-configured blob store).
- docs/local-collab-testing.md: fresh-volume bring-up runbook (seed, extract
the file-service IDs via SQL, start server+client, verify healthz/metrics +
a /collab WS smoke, the RMQ save/fetch round-trip, register a user via
mailslurper, grant admin via direct SQL) plus assumptions/risks.
No migration on first boot (rooms materialize lazily). Authoring only.
Refs workspace#003-unify-collab-yjs
…tack The dev-stack swap rewired Traefik's /collab router but missed Oathkeeper's access rules, so /decisions returned 404 on /collab and forwardAuth propagated it (WS handshake never reached the service). Adds alkemio:api:private:ws:collab (mirrors the legacy ws handshake rule) → forwardAuth resolves the identity and the upgrade reaches collaboration:4006 (verified: HTTP 101).
…lab-yjs # Conflicts: # .build/traefik/http.yml # quickstart-services.yml
…tor-id header The develop OIDC cutover deleted the oathkeeper-auth-socket-io middleware the collaboration router referenced, so Traefik disabled the router (middleware does not exist). Switch it to the cutover's replacement: strip-client-alkemio- headers + alkemio-resolve, which injects the resolved identity into X-Alkemio-Actor-Id (same pattern as file-service). Set AUTH_TOKEN_HEADER= X-Alkemio-Actor-Id on the collaboration service so its WS handshake reads the actor id from the gateway's header instead of Authorization.
…-fetch
The collaboration-fetch reply (FetchOutputData) carried the document's own
authorizationPolicyId but not its storage bucket, so the collaboration-service
persisted every Yjs snapshot into a single flat platform bucket instead of each
document's own bucket.
Add storageBucketId to FetchOutputData and the CollaborationMetadata domain
projection, and resolve memo.profile.storageBucket.id /
whiteboard.profile.storageBucket.id in getCollaborationMetadata (extending the
relations/select to include profile: { storageBucket: { id } }). The responder
fetch() path forwards it per document for both memo and whiteboard.
authorizationPolicyId is unchanged. Responder + domain tests assert the correct
per-document storageBucketId for both memo and whiteboard.
…line content columns Lead storage slice of workspace#006-collab-content-unification. All real-time collaborative document content (memo + whiteboard) is now stored ONLY as a Yjs-V2 snapshot in the document's own storage bucket (counting toward the space quota); the inline `memo.content`/`whiteboard.content` columns are dropped and the whiteboard compression hooks removed. Derived memo text is computed from the snapshot via batched file-service reads. Content originates server-side at create (memo: markdownToYjsV2State; whiteboard: scene → Yjs) and is seeded into the room on first open. Storage + create-to-bucket (R2/R4/FR-005): - FileServiceAdapter.createSnapshotInBucket — uploads a Yjs-V2 snapshot to a document's bucket with NULL per-file authz, mirroring the collaboration-service BlobStore (store.go); returns the file id used as `contentPointer`. - FileServiceAdapter.getContentBatch — file-service POST /internal/file/content-batch (#52): N pointers → N blobs, order preserved, misses non-fatal. - MemoService/WhiteboardService create encode content → write snapshot to the document's bucket → set contentPointer/blobStore/contentVersion; no inline column. updateMemoContent/updateWhiteboardContent + template content-set route through the bucket (latest-only: prior snapshot file deleted). First-open seed (R4/FR-003): - collaboration-fetch carries the document's stored snapshot as base64 `content` (→ collab FetchReply.Content/Metadata.SeedContent) so a freshly created document materializes with its content; empty-on-create stays empty + editable (FR-010). Derived text (FR-006) + consumer migration (T007/T008): - Memo `markdown`/`content` GraphQL fields derive from the snapshot via a request-scoped batched DataLoader (MemoContentLoaderCreator) keyed by contentPointer. - Search ingest derives memo markdown via one batched content-batch read per page; whiteboard scene-text indexing dropped (whiteboards indexed by profile). - input-creator duplicate/export builders read content from the snapshot (async). - Legacy collaborative-document-integration + whiteboard-integration modules removed (superseded by the unified collaboration-integration); obsolete admin whiteboard content-reupload tool removed. Whiteboard scene ↔ Yjs (R1): - whiteboard.scene.to.yjs.v2.state — server-side scene JSON ↔ Yjs-V2 snapshot, a faithful port of the @alkemio/excalidraw-yjs-binding populateYDoc/exportSceneJSON + scene Y.Map schema. Ported (not the binding directly) because the published binding @33 bundles its own yjs and cannot be encodeStateAsUpdateV2-ed by the server's yjs instance (yjs#438); the wire format is byte-compatible. Up-front batch migration (R5/US6/FR-007): - CollaborationMigrationService.migrateAll — idempotent, resumable, dry-run-capable job: streams every legacy memo/whiteboard, encodes → writes the snapshot to its bucket → sets the pointer; skips already-migrated/empty rows; flags un-decodable content (never drops). Runs BEFORE the column drop in prod. Breaking (coupled to client-web at cutover): - TypeORM migration drops memo.content + whiteboard.content (up/down verified). - GraphQL Whiteboard.content output field removed; create-input scene seed kept. Gates: tsc + biome clean; full vitest suite green; migration up/down verified against a throwaway Postgres.
…inding (one source of truth) PR #6175 hand-ported populateYDoc/exportSceneJSON into the server because the binding bundled its own yjs (yjs#438 instance mismatch). That bug is fixed: the binding now resolves the consumer's yjs (peer-deduped to the server's single yjs@13.6.27 instance), so the server can consume it directly — eliminating the duplicate port and the scene-encoding drift between server and client-web. - add @alkemio/excalidraw-yjs-binding (pkg.pr.new @33) as a dependency; drop the fractional-indexing direct dep (the port's only other consumer — now transitive via the binding). Pin the binding's type-only @alkemio/excalidraw peer via a pnpm override so the prerelease-tagged editor version satisfies its >=0.18.0 range without pulling the editor into the runtime path. - reimplement whiteboardSceneToYjsV2State / whiteboardYjsV2StateToScene as thin wrappers over the binding (populateYDoc + Y.encodeStateAsUpdateV2 / applyUpdateV2 + exportSceneJSON), deleting ~280 lines of hand-ported scene-schema / order / migrate internals. Signatures unchanged — callers (whiteboard.service, collaboration-migration.service, input.creator.service) are untouched, as is the server-side embedded-media reupload glue that runs before encoding. - keep the empty/unparseable-scene resilience (parseScene guard) and add a parity test asserting a sample scene round-trips scene→V2 bytes→scene lossless on elements/appState/files through the binding. Single-yjs-instance verified end-to-end: populateYDoc(scene, new Y.Doc()) then the server's Y.encodeStateAsUpdateV2(doc) encodes clean (no "Unexpected content type"), and the binding's prod bundle imports only yjs + fractional-indexing at runtime. lint (tsc+biome) clean; full suite 6761 passing.
- CreateWhiteboardInput.sourceWhiteboardID: seed a new whiteboard from an existing whiteboard's stored Yjs-V2 snapshot (Save-as-Template / Duplicate). A live whiteboard's content is WS-only since 006, so the client can no longer read it to seed a template; it sends an empty placeholder and the server copies. - getWhiteboardContent reads the snapshot; rehomeSnapshotMedia re-homes embedded media into the new bucket via the snapshot's own files Y.Map (no Excalidraw JSON). - WhiteboardContent scalar + whiteboard content path are base64 Yjs-V2 throughout.
antst
requested review from
hero101,
techsmyth and
valentinyanakiev
as code owners
June 24, 2026 20:30
Contributor
|
Warning Review limit reached
More reviews will be available in 22 minutes and 27 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (145)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
|
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidated single PR for the server slice of the Yjs collaboration unification — 003 (unified persistence + lifecycle + authZ) + 006 (collaborative content → per-document file storage), as the final
develop…feat/006state (15 commits).Replaces #6171 (003) + #6175 (006).
Scope
Whiteboard.contentdropped from GraphQL;createWhiteboardserver-side copy viasourceWhiteboardID;getWhiteboardContent. Pairs with file-service (batched content-read) + client-web.