spike: mongosh --from to query local files via an embedded engine - #2834
Draft
kmruiz wants to merge 4 commits into
Draft
spike: mongosh --from to query local files via an embedded engine#2834kmruiz wants to merge 4 commits into
kmruiz wants to merge 4 commits into
Conversation
Vendors the smongo Rust engine (smongo-engine) plus a minimal MongoDB wire-protocol server (OP_MSG/OP_QUERY) exposed to Node via a napi addon, under packages/smongo. The wire server reimplements just enough of smongo's PyO3 wire logic against smongo-engine so mongosh's driver can connect to the embedded engine over localhost TCP without a Python runtime. Build artifacts (packages/smongo/rust/target/, smongo-node.node) are gitignored.
Adds a --from <file> flag that starts the vendored smongo engine in-process, ingests CSV/JSON/NDJSON files into collections named after the file stem, and runs the shell (--eval or interactive) against it with no server. The engine is torn down and its temp directory removed on exit. - arg-parser: --from array option; rejects combining it with a connection string or --nodb - cli-repl: embedded/smongo.ts (engine lifecycle) + embedded/parse.ts (file parsing); _start() bypasses remote connect and ingests after connecting; close() stops the engine and cleans up - cli-repl: USAGE + i18n text, @mongosh/smongo dependency, webpack external for the native addon - workspace: register packages/smongo in package.json / mongosh.code-workspace
- arg-parser: --from single/multiple parsing and validation (rejects a connection string and --nodb); update the array-option list assertion - embedded/parse.spec.ts: pure-JS tests for CSV/JSON/NDJSON parsing and collection-name derivation (no native addon required) - embedded/smongo.spec.ts: native integration tests (gated on the addon) that ingest a CSV and query it over the wire - TEST.md: compile + run instructions and a feature/limitation summary
kmruiz
changed the title
handle_aggregate ran Collection::aggregate(), which passes no CollectionResolver, so $unionWith/$lookup/$graphLookup failed with '$unionWith requires a CollectionResolver'. Run the pipeline through DatabaseContext + aggregate_with_db_collection so cross-collection stages can resolve other ingested collections. Document the support in TEST.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
This is a PR left to DeepSeek while I was working on something more important. Likely the code is broken in several ways and doesn't follow best practices. The idea is just to understand how feasible is to embed smongo into mongosh for querying unstructured data locally.
What
Spike for the "mongosh as local db":
--fromingests local files into an in-process embedded engine and runs the shell against them — no server, no Docker.mongosh --from ./data.csv --eval 'db.data.find({})'How
packages/smongo/) with a minimal MongoDB wire server (OP_MSG/OP_QUERY) exposed to Node via a napi addon, so mongosh's driver connects over localhost TCP.--from <file>starts the engine, ingests each file intodb.<file-stem>(CSV/JSON/NDJSON), runs--evalor an interactive session, and tears the engine down on exit.--fromcombined with a remote connection string or--nodb.Notes / limitations
TEST.md).smongo-node.node,rust/target/) are gitignored.Full build + run instructions in
TEST.md.