Add Mnemosyne architecture design diagrams (S1 and S2) by palashkondekar-meesho · Pull Request #376 · Meesho/BharatMLStack

palashkondekar-meesho · 2026-05-27T18:08:32Z

Context:

This PR adds architectural design diagrams for Mnemosyne, documenting two distinct deployment strategies for the feature store system. These diagrams serve as visual references for understanding the system topology and component interactions.

Describe your changes:

Added two Excalidraw design documents:

mnemosyne-s1-deployable-per-shard.excalidraw: Illustrates the S1 architecture with shard-based deployment strategy
- Shows Databricks/PySpark job generating shard-specific files
- GCS versioned file storage
- Control plane managing onboarding, sizing, and cost
- Client SDK with scatter-gather pattern
- Per-shard deployable units with Envoy load balancing
- Multiple read replicas (pods) per shard with RocksDB engines
mnemosyne-s2-scattered-regions.excalidraw: Illustrates the S2 architecture with TiDB-like region-based strategy
- Similar data generation and storage pipeline
- Control plane with placement driver for region management
- Client SDK with region-map and redirect handling
- Scattered pod deployment where each pod hosts multiple non-contiguous regions
- Region replication factor across pods

Both diagrams document the complete data flow from generation through serving, including control plane interactions and client-side logic.

Testing:

N/A - These are design documentation artifacts (Excalidraw JSON files) with no executable code or tests.

Monitoring:

N/A - Design documentation only.

Rollback plan:

N/A - Design documentation only. Can be removed or updated without affecting system operation.

Checklist before requesting a review:

I have reviewed my own changes
Documentation has been added (design diagrams)

📂 Modules Affected:

docs (Documentation updates)

✅ Type of Change:

Documentation

https://claude.ai/code/session_01FM9sB54w7ry9GH7iU2qbqv

…orm LLD Excalidraw component diagrams for the two sharding strategies under review: - S1: deployable-per-shard (read-replicas, Envoy intra-shard LB) - S2: scattered regions (TiDB-like, Placement-Driver-managed) https://claude.ai/code/session_01FM9sB54w7ry9GH7iU2qbqv

coderabbitai · 2026-05-27T18:08:42Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8a53a148-1ee0-405c-b379-4796820b5728

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…-tenant, internal scaling) Companion to the Mnemosyne LLD. Shows the onboarding lane (admin -> API -> sizer -> tenant outputs), shared Control Plane + Producer/GCS, and three tenant lanes with per-tenant Client SDK and internal shard deployables (one with a scaled-up replica to illustrate horizontal scaling within a shard). https://claude.ai/code/session_01FM9sB54w7ry9GH7iU2qbqv

New py-sdk package mirroring spark_feature_push_client. Reads parquet, shards by crc32(key) % S using Spark SQL's crc32() (IEEE; same as Python binascii.crc32 and Go crc32.ChecksumIEEE), encodes per-entity payloads as persist.Data proto (reusing bharatml_commons), and writes per-shard sorted SST files via rocksdict.SstFileWriter inside mapPartitions, plus a _manifest.json with the schema + per-shard file/row/sha256 stats for the Control Plane to gate version promotion. Pipeline: project -> key + shard (Spark SQL) -> repartition by shard -> sortWithinPartitions(shard,key) -> mapPartitions writes SSTs. Output: <out>/<version_id>/<shard:05d>/data-<partition_uuid>.sst <out>/<version_id>/_manifest.json Includes a no-Spark smoke test that validates config parsing, key/shard computation, and persist.Data round-trip on the geo example config. https://claude.ai/code/session_01FM9sB54w7ry9GH7iU2qbqv

semgrep-sast-meesho · 2026-06-22T16:52:16Z

+dependencies = [
+    "pyspark>=3.4",
+    "protobuf>=4.21",
+    "rocksdict>=0.3",


Semgrep identified a blocking 🔴 issue in your code:
Dependency "$MATCH" uses a range operator. Pin to exact version with == or use a lockfile (e.g. uv.lock, pdm.lock, poetry.lock). Range pins allow auto-upgrades to compromised versions in CI.

To resolve this comment:

✨ Commit fix suggestion

Suggested change

"rocksdict>=0.3",

"rocksdict==0.3.0",

View step-by-step instructions

Replace the ranged dependency specifier for rocksdict with an exact version pin in pyproject.toml.
Change "rocksdict>=0.3" to "rocksdict==<tested-version>".

Pick the exact version that this project already uses and has been validated with, instead of allowing any newer release.
For example, if the current environment is using 0.3.x, pin that exact value as rocksdict==0.3.x.

Leave requires-python = ">=3.9" unchanged.
This rule applies to package dependencies, not the Python version constraint.

Alternatively, if your build process already uses a committed lockfile such as uv.lock, pdm.lock, or poetry.lock, keep the dependency resolution fixed there and make sure rocksdict is locked to one exact version.

💬 Ignore this finding

Reply with Semgrep commands to ignore this finding.

/fp <comment> for false positive

/ar <comment> for acceptable risk

/other <comment> for all other reasons

Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by pyproject-dependency-range-pin.

_{You can view more details about this finding in the Semgrep AppSec Platform.}

semgrep-sast-meesho · 2026-06-22T16:52:17Z

+requires-python = ">=3.9"
+dependencies = [
+    "pyspark>=3.4",
+    "protobuf>=4.21",


Semgrep identified a blocking 🔴 issue in your code:
Dependency "$MATCH" uses a range operator. Pin to exact version with == or use a lockfile (e.g. uv.lock, pdm.lock, poetry.lock). Range pins allow auto-upgrades to compromised versions in CI.

To resolve this comment:

✨ Commit fix suggestion

Suggested change

"protobuf>=4.21",

"protobuf==4.21.0",

View step-by-step instructions

Replace the ranged dependency with an exact version pin in pyproject.toml by changing protobuf>=4.21 to protobuf==<tested-version>.

Use the version already installed in this environment if it is the version you expect to ship, for example protobuf==<installed-version>, so the project does not auto-upgrade in CI.

Apply the same fix to the other ranged dependencies in this file if they are governed by the same rule, for example change pyspark>=3.4 to pyspark==<tested-version> and rocksdict>=0.3 to rocksdict==<tested-version>.

Leave requires-python = ">=3.9" unchanged, since this rule excludes Python version constraints.

Add a short comment above the dependency if helpful, such as # Pinned exactly to avoid CI pulling a newer unreviewed release.

💬 Ignore this finding

Reply with Semgrep commands to ignore this finding.

/fp <comment> for false positive

/ar <comment> for acceptable risk

/other <comment> for all other reasons

Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by pyproject-dependency-range-pin.

_{You can view more details about this finding in the Semgrep AppSec Platform.}

semgrep-sast-meesho · 2026-06-22T16:52:18Z

+description = "Mnemosyne WORM platform — Producer: Spark batch job that shards a parquet dataset by crc32(key) and emits per-shard RocksDB SST files."
+requires-python = ">=3.9"
+dependencies = [
+    "pyspark>=3.4",


Semgrep identified a blocking 🔴 issue in your code:
Dependency "$MATCH" uses a range operator. Pin to exact version with == or use a lockfile (e.g. uv.lock, pdm.lock, poetry.lock). Range pins allow auto-upgrades to compromised versions in CI.

To resolve this comment:

✨ Commit fix suggestion

Suggested change

"pyspark>=3.4",

"pyspark==3.4.0",

View step-by-step instructions

Change the dependency entry in pyproject.toml from a range pin to an exact pin by replacing "pyspark>=3.4" with "pyspark==<exact-version>".

Set <exact-version> to the specific PySpark release you have already approved for this project, for example the version currently installed in the dev environment from $ python -c "import pyspark; print(pyspark.__version__)".

Keep the requires-python = ">=3.9" line unchanged, because this rule only applies to package dependencies and not the Python version constraint.

Repeat the same change for the other ranged dependencies in this file, such as protobuf>=4.21 and rocksdict>=0.3, so each dependency uses == instead of >=. This prevents CI from silently pulling a newer package release than the one you intended to ship.

Alternatively, if your build tooling supports lockfiles and you want to keep version ranges in pyproject.toml, add a lockfile such as uv.lock, pdm.lock, or poetry.lock and make sure CI installs from that lockfile instead of resolving dependencies dynamically.

💬 Ignore this finding

Reply with Semgrep commands to ignore this finding.

/fp <comment> for false positive

/ar <comment> for acceptable risk

/other <comment> for all other reasons

Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by pyproject-dependency-range-pin.

_{You can view more details about this finding in the Semgrep AppSec Platform.}

semgrep-sast-meesho · 2026-06-22T16:52:20Z

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]


Semgrep identified a blocking 🔴 issue in your code:
Dependency "$MATCH" uses a range operator. Pin to exact version with == or use a lockfile (e.g. uv.lock, pdm.lock, poetry.lock). Range pins allow auto-upgrades to compromised versions in CI.

To resolve this comment:

✨ Commit fix suggestion

Suggested change

requires = ["setuptools>=61", "wheel"]

# Replace 61.0.0 with the exact setuptools version installed in your build environment if it differs.

requires = ["setuptools==61.0.0", "wheel"]

View step-by-step instructions

Change the build dependency in [build-system].requires from a range to an exact pin by replacing setuptools>=61 with setuptools==<tested-version>.

Keep wheel as-is only if your policy does not require every build dependency to be pinned; otherwise pin it too as wheel==<tested-version>.

Update the line so it looks like requires = ["setuptools==<tested-version>", "wheel"] or requires = ["setuptools==<tested-version>", "wheel==<tested-version>"].

Choose the exact setuptools version that is already installed and working in this environment, since setuptools is already available here. This avoids CI pulling a newer release automatically.

Alternatively, if your project manages build dependencies through a lockfile, add a lockfile supported by your packaging workflow and make sure the resolved setuptools version is fixed there.

💬 Ignore this finding

Reply with Semgrep commands to ignore this finding.

/fp <comment> for false positive

/ar <comment> for acceptable risk

/other <comment> for all other reasons

Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by pyproject-dependency-range-pin.

_{You can view more details about this finding in the Semgrep AppSec Platform.}

claude added 2 commits May 28, 2026 04:55

semgrep-sast-meesho Bot reviewed Jun 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Mnemosyne architecture design diagrams (S1 and S2)#376

Add Mnemosyne architecture design diagrams (S1 and S2)#376
palashkondekar-meesho wants to merge 3 commits into
developfrom
claude/feature-store-rust-rocksdb-NiKQc

palashkondekar-meesho commented May 27, 2026 •

edited by shanavasvb-meesho

Loading

Uh oh!

coderabbitai Bot commented May 27, 2026 •

edited

Loading

Review skipped

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,22 @@
		[build-system]
		requires = ["setuptools>=61", "wheel"]

	requires = ["setuptools>=61", "wheel"]
	# Replace 61.0.0 with the exact setuptools version installed in your build environment if it differs.
	requires = ["setuptools==61.0.0", "wheel"]

Uh oh!

Conversation

palashkondekar-meesho commented May 27, 2026 • edited by shanavasvb-meesho Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context:

Describe your changes:

Testing:

Monitoring:

Rollback plan:

Checklist before requesting a review:

📂 Modules Affected:

✅ Type of Change:

Uh oh!

coderabbitai Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

semgrep-sast-meesho Bot Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

palashkondekar-meesho commented May 27, 2026 •

edited by shanavasvb-meesho

Loading

coderabbitai Bot commented May 27, 2026 •

edited

Loading