Skip to content

perf(qdrant): store collection data on disk - #231

Merged
jack-arturo merged 9 commits into
developfrom
perf/qdrant-on-disk-storage
Aug 28, 2026
Merged

perf(qdrant): store collection data on disk#231
jack-arturo merged 9 commits into
developfrom
perf/qdrant-on-disk-storage

Conversation

@jack-arturo

Copy link
Copy Markdown
Member

Summary

  • configure newly created Qdrant collections with disk-backed vectors, HNSW, and payloads
  • add an opt-in HNSW-on-disk restore option for migrating existing collections
  • document the destructive Qdrant-only migration, validation, rollback, and manual Railway resize boundary

Closes #225

Testing

  • /Users/jgarturo/Projects/OpenAI/automem/.venv/bin/black --check .
  • /Users/jgarturo/Projects/OpenAI/automem/.venv/bin/isort --check-only .
  • /Users/jgarturo/Projects/OpenAI/automem/.venv/bin/flake8 .
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /Users/jgarturo/Projects/OpenAI/automem/.venv/bin/pytest -q -m unit

Operational notes

Existing production collections are not modified at startup. The runbook requires an explicit backup/restore migration and leaves Railway volume resizing as a manual post-validation operation.

Breaking changes

None.

@jack-arturo jack-arturo added the babysit:active Babysit agent is actively working this PR label Aug 28, 2026
@jack-arturo jack-arturo added babysit:waiting-ci Babysit waiting for CI checks on current head babysit:active Babysit agent is actively working this PR and removed babysit:active Babysit agent is actively working this PR babysit:waiting-ci Babysit waiting for CI checks on current head labels Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d05a2d9cd4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/MONITORING_AND_BACKUPS.md Outdated
Comment thread docs/MONITORING_AND_BACKUPS.md
@jack-arturo

Copy link
Copy Markdown
Member Author

Babysit stopped — ci. The parity check fails independently of this Qdrant change: its log installs published @verygoodplugins/mcp-automem 0.15.0 and compares that stale stdio surface against the current API. The branch has no parity-file diff beyond merging current develop. A package release (or an explicit change to the parity/release policy) is required before this PR can satisfy CI; I am not widening #225 to make that release decision.

@jack-arturo jack-arturo added babysit:blocked Babysit stopped; unresolved blocker needs human and removed babysit:active Babysit agent is actively working this PR labels Aug 28, 2026
@jack-arturo jack-arturo added babysit:active Babysit agent is actively working this PR babysit:waiting-ci Babysit waiting for CI checks on current head and removed babysit:blocked Babysit stopped; unresolved blocker needs human babysit:active Babysit agent is actively working this PR labels Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db71f07096

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/MONITORING_AND_BACKUPS.md Outdated
@jack-arturo jack-arturo added babysit:active Babysit agent is actively working this PR babysit:waiting-ci Babysit waiting for CI checks on current head and removed babysit:waiting-ci Babysit waiting for CI checks on current head babysit:active Babysit agent is actively working this PR labels Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45e77f4922

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/MONITORING_AND_BACKUPS.md
@jack-arturo jack-arturo added babysit:active Babysit agent is actively working this PR and removed babysit:waiting-ci Babysit waiting for CI checks on current head labels Aug 28, 2026
@jack-arturo jack-arturo added babysit:blocked Babysit stopped; unresolved blocker needs human and removed babysit:active Babysit agent is actively working this PR labels Aug 28, 2026
@jack-arturo
jack-arturo merged commit 07d8704 into develop Aug 28, 2026
7 of 8 checks passed
@jack-arturo
jack-arturo deleted the perf/qdrant-on-disk-storage branch August 28, 2026 02:07
@jack-arturo jack-arturo removed the babysit:blocked Babysit stopped; unresolved blocker needs human label Aug 28, 2026
jack-arturo added a commit that referenced this pull request Aug 28, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.16.2](v0.16.1...v0.16.2)
(2026-08-28)


### Bug Fixes

* **backup:** page FalkorDB export by node id instead of deep SKIP
([#221](#221))
([009d58e](009d58e))
* **config:** preserve Ollama defaults and migration checks
([0bc0096](0bc0096))
* **docs:** complete embedding migration runbooks
([3468443](3468443))
* **docs:** correct re-embedding migration setup
([a8be394](a8be394))
* **enrichment:** circuit-break exhausted quota calls
([ea148e5](ea148e5))
* **enrichment:** circuit-break exhausted quota calls
([#229](#229))
([ad3ed27](ad3ed27))
* **enrichment:** preserve quota circuit during in-flight calls
([29f359d](29f359d))
* **mcp-sse:** emit the stored date in compact recall output
([#224](#224))
([42ba8b6](42ba8b6))
* **mcp:** align the parity harness with its own allowlist
([9ac5b4e](9ac5b4e))
* **mcp:** close false-failure and coverage gaps in the parity harness
([4cf9ef1](4cf9ef1))
* **mcp:** close partial connections when parity setup fails
([89346e9](89346e9))
* **mcp:** isolate parity fixtures per scenario and drop the false alarm
([fa458b0](fa458b0))
* **qdrant:** index the `type` payload field so recall filters stop
400ing
([c131f02](c131f02))
* **qdrant:** index the `type` payload field so recall filters stop
400ing ([#223](#223))
([10bcac7](10bcac7))
* **recovery:** honor configured graph and cloud migrations
([4dd1325](4dd1325))
* **stream:** emit live memory operations on GET /stream
([#227](#227))
([7a92ebd](7a92ebd))


### Performance Improvements

* **qdrant:** store collection data on disk
([#231](#231))
([07d8704](07d8704))
* **qdrant:** store new collection data on disk
([c97b5b1](c97b5b1))
* **qdrant:** support disk-backed HNSW restores
([b04b9e1](b04b9e1))


### Documentation

* add Qdrant on-disk implementation plan
([56da2b7](56da2b7))
* add Qdrant on-disk storage design
([e177b60](e177b60))
* align provider and maintenance guidance
([93afcff](93afcff))
* align provider and maintenance guidance
([#232](#232))
([c40d1d2](c40d1d2))
* **enrichment:** document quota circuit
([d9787cb](d9787cb))
* **mcp:** audit remote vs stdio transport parity
([92b98ec](92b98ec))
* **qdrant:** add on-disk migration runbook
([d05a2d9](d05a2d9))
* **qdrant:** harden on-disk migration runbook
([db71f07](db71f07))
* **qdrant:** limit migration to default topology
([9b31cde](9b31cde))
* **qdrant:** verify migration export before restore
([45e77f4](45e77f4))
* **readme:** lead benchmark section with neutral AMB numbers
([#219](#219))
([8ff266e](8ff266e))
* target feature PRs at develop
([#228](#228))
([bcef376](bcef376))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cost: Qdrant memory footprint is the main driver of ~$36/mo Railway spend — enable on-disk vector/HNSW storage

1 participant