feat: Bolt protocol endpoint + Cypher performance campaign#1449
Open
bplatz wants to merge 48 commits into
Open
feat: Bolt protocol endpoint + Cypher performance campaign#1449bplatz wants to merge 48 commits into
bplatz wants to merge 48 commits into
Conversation
Process-wide bounded LRU (FLUREE_CYPHER_AST_CACHE, default 512 entries) keyed on statement text, holding the pre-substitution AST. All five parse sites (read path, write plan, write lowering, RETURN planning) route through one choke point; params substitute into a per-request clone, so lowering still sees only concrete literals. Statements over 8 KiB are never cached, keeping unique bulk payloads from evicting the short hot statements the cache exists for. Also adds examples/cypher_phase_profile.rs, which times parse / clone / substitute / lower / end-to-end per statement shape to ground the lowered-IR-cache follow-up decision (see docs/design/bolt-adapter.md).
Pure, IO-free server-side Bolt implementation: PackStream encode/decode with hostile-input bounds (length caps, nesting depth), chunked message assembly (split chunks, NOOP keep-alives, size ceiling), 4-slot version negotiation (4.4 and 5.0-5.4, range expansion, manifest slots skipped), typed request/response messages, and the autocommit session state machine (HELLO/LOGON, RUN via caller-executed Turn::Execute, reactive PULL/DISCARD batching with has_more, FAILED->IGNORED->RESET recovery, single-entry ROUTE table). BEGIN answers a clear FAILURE per the support-matrix contract. See docs/design/bolt-adapter.md.
Feature-gated `bolt` (default off): --bolt-listen-addr / --bolt-default-db (+ [server.bolt] in the config file) bind a TCP listener alongside HTTP; each connection drives a fluree-db-bolt session against the same entry points the HTTP routes use — query_cypher_with_params for reads, the consensus submit path (with write-RETURN skolem planning) for writes. RUN dispatches on the new cypher_statement_is_write (AST-cache-backed). Result mapping shares the per-binding converter with cypher-json: format::cypher::table() now exposes the pre-flattening RDF-faithful cells (QueryResult::to_cypher_table), from which the Bolt glue maps PackStream values — xsd:decimal to Float (Neo4j parity, noted in the support matrix), temporal values as ISO strings in v1, node/rel/path shapes matching the JSON transport. The unauthenticated v1 listener refuses to start under data_auth_mode=required. End-to-end tests drive the real listener over TCP (handshake, 4.4/5.x sessions, params, PULL batching, write round-trips, FAILURE/RESET recovery); tests/bolt_driver_smoke.py is the official-python-driver compatibility bar, run manually.
parse 1-4us / lower 1-4us on benchmark-shaped statements: the lowered-IR cache the Bolt design doc left open is not warranted. Profiler defaults drop to 500 users / 100 iters with e2e capped at 20 rounds -- aggregate statements over large unindexed novelty run for minutes and the absolute e2e numbers are novelty-path-only anyway.
RETURN n now yields real Bolt Node structures: element_id = full IRI, numeric id = stable FNV-1a of the IRI, labels via the engine's cypher_name_from_iri rule (db:Node marker hidden), and properties fetched per subject at format time (SPOT scan over snapshot+overlay, cached per table walk; multi-valued predicates become lists, ref values stay IRI strings). Relationship values carry endpoints, type, and annotation properties when reified; paths map to Bolt's unique node/rel lists + walk indices. xsd:date/dateTime/time map to PackStream temporal structures, with the 4.4 legacy local-seconds DateTime and Local variants for timezone-less lexical forms. The typed surface is QueryResult::to_cypher_typed_table over a new format::cypher_typed cell model sharing column selection and the Cypher naming rule (fluree_db_query::eval::cypher_name_from_iri, now pub) with the JSON formatter. Hydration errors under a view policy rather than bypassing per-flake filtering.
Optimistic model, one commit per statement, atomic publish (fluree-db-api/src/cypher_txn.rs): BEGIN pins the cached head; each write statement lowers and stages against the transaction's private state (conditional MERGE probes included), builds a real commit via the pure build_commit, and advances the private state with StagedCommit::finalize_state — reads inside the transaction get read-your-writes, statement errors surface at RUN time and poison the transaction until RESET. COMMIT takes the ledger write lock, verifies the head is still the pinned base (t + commit id), writes the pending blobs (content-addressed, idempotent) and advances the head ref once: intermediate states are never observable while the commit chain keeps per-statement provenance. A moved base fails the whole transaction with CommitConflict, surfaced as Neo.TransientError.* so official drivers' managed transaction functions (execute_write/execute_query) retry — isolation is serializable-against-base, stronger than Neo4j's read-committed. Single-node deployments only: Raft/peer reject BEGIN. Session state machine grows TxReady/TxStreaming with qid-addressed result buffers (multiple open results per transaction), Turn::Begin/ Commit/Rollback/Reset lifecycle actions, and commit bookmarks (fluree:t:<t>). RESET now signals the glue so server-side transaction state drops with it.
Runtime config is the real gate — the listener binds only when bolt_listen_addr is set — and a default-off build feature meant release binaries could not speak Bolt at all. The feature stays declared for minimal builds.
docs/design/bolt-adapter.md was an implementation hand-off — sequencing, estimates, pre-implementation anchors, since-resolved questions — not durable knowledge. What lasts now lives where users and developers look: docs/api/bolt.md (protocol surface, value mappings, transaction semantics, implementation pointers) and docs/guides/bolt.md (enable, configure, driver examples, retry contract, troubleshooting). The transaction commit model stays documented at the code in cypher_txn.rs; code comments and the profiler example no longer cite the deleted doc.
--bolt-listen-addr / --bolt-default-db join the CLI's first-class server flags (previously reachable only via -- passthrough or the config file): threaded through foreground and background starts, preserved across restart via the child-arg merge, shown by --dry-run and by 'fluree server status' (recorded in server.meta.json). Child args skip the bolt flags when the user already passed them via -- passthrough so the child's arg parse never sees duplicates.
The typed table fetched each distinct node's properties with one awaited SPOT scan during the row walk — serial point reads that made data-heavy Bolt results 1.5-7x slower than the (non-hydrating) JSON transport. The engine has already produced the subject list by format time, so the fetches are independent: a prefetch pass now collects every hydration subject in the result (node refs, relationship reifiers, path nodes), sorts by subject for leaflet locality, and issues the reads with bounded concurrency (16) into a shared raw-flake cache; the row walk then renders from cache. Encoded bindings that materialize during the walk fall back to the single-fetch path unchanged.
CREATE ... RETURN n over Bolt returned the created entity as a bare skolem-id string while reads returned Node structures. The new write_return_typed_rows reconstructs the created-entity Sids from the skolem id (shared solution-count probe factored out of the JSON envelope path, which the HTTP route keeps unchanged) and hydrates them against the post-commit state through the same batched prefetch reads use — labels and properties included. Both the autocommit path and explicit transactions (CypherTxnWriteOutcome now carries the typed table) return them; RETURN of write expressions (n.id) remains a documented, cleanly-rejected v1 limit. Also resolves an unused-binding clippy failure that arrived with the rebased it_query_cypher.rs seek test — it would have failed CI's all-targets lint.
The select-map hydration path fetched each row's subject with one awaited SPOT scan — the same serial shape the Bolt typed table had. run_hydration_rows now bulk-fetches the top-level subjects of wildcard hydration columns before the row walk (sorted by subject, 16-way bounded concurrency) into a flake cache on HydrationCaches shared across rows, levels, and nested ref expansion; the expansion body reads through it. Prefetch and read-through route via fetch_subject_properties, so policy filtering and fuel charging are unchanged. Single-ledger only: dataset hydration routes roots to home-ledger views whose Sids share an encoding space, so it keeps the per-fetch path; explicit projections keep their narrowed reads.
The 625813c prefetch never fired on benchmark-shaped results: bare node vars come off the binary scan as EncodedSid bindings, which the subject collector skipped — every row silently fell back to the serial fetch, which is why data-heavy queries didn't move. The collector now materializes encoded subject refs (via a new graph-direct materialize_with_graph) before collecting. Profiling the engaged prefetch showed the second half of the story: the per-subject fetches are CPU-bound in-memory work (~18us of cursor setup + leaflet decode + dict resolves each), so buffer_unordered's cooperative concurrency ran them on one core. The prefetch now spawns chunked tasks across runtime workers. Also memoizes per-flake predicate-key and class-label decodes across the result (predicates repeat on every node). Local probe (200 rich nodes, indexed, RETURN n, release): typed table 4.0ms -> 2.1ms; prefetch 3.5ms -> 1.4ms. Probe kept as examples/bolt_node_emit_probe.rs. Remaining floor is the ~18us per-subject fetch itself — cutting that needs an engine-level batched leaflet pass (one sorted cursor sweep for many subjects), noted as the next lever.
Bench-verified root cause of the remaining RETURN n gap: node emit rendered every ref-valued flake as an IRI-string property, so each node dragged its full adjacency — one dict/arena IRI materialization per edge target server-side (literal objects are already inline in the leaflet) plus degree-scaled wire and driver-decode cost. A Neo4j Node carries scalar properties only; relationships surface by binding an edge or path var. Node and annotation property maps now skip Ref flakes. Probe (200 nodes x 10 edges, release): typed table 2.86 -> 2.35 ms server-side; the larger share of the bench win is the ~10 IRI strings per node no longer serialized and rebuilt by the driver. The SPOT probes still read ref flakes (unchanged, per design) — cutting that is the engine-level batched-sweep lever.
The per-subject point reads were the remaining node-emit floor (~18us each: cursor setup, leaflet re-decode, per-call overhead). The prefetch now has two lanes. Subjects that arrive with a raw s_id (EncodedSid off the binary scan) go through the existing gap-aware batched crawl (batched_lookup_subject_properties: sorted s_ids, one SPOT cursor sweep per contiguous chunk, each touched leaflet decoded once) and build nodes straight from (p_id, o_type, o_key) rows — ref-valued rows are skipped by o_type before any decode, so edge adjacency never touches the dictionary at all. The lane is gated on OverlayProvider::is_effectively_empty; live novelty, reifiers, path nodes, and Sid-only subjects keep the spawn-parallel per-subject fallback, so correctness never depends on the fast lane. Probe (200 rich nodes x 10 edges, indexed, release): typed table 2.35 -> 0.66 ms (~3.3us/node; 6x from the 4.0ms serial start). The probe now self-checks hydrated content, and a new wire-level integration test covers the batched lane on an indexed ledger (labels, scalars, Date structure, no-adjacency rule).
The bench showed ee4b597 flat at every result size — the batched lane never opened. Two compounding gates kept it shut whenever any write had landed since the last reindex: (1) the lane required OverlayProvider::is_effectively_empty, which is ledger-global, and (2) under live novelty the executor emits materialized Sid bindings rather than EncodedSid, so no subject carried the raw s_id the lane required. Mixed read/write suites therefore always took the per-subject fallback — identical numbers to the pre-batching code. The gate is now per-subject: one overlay SPOT walk collects the dirty-subject set, cached process-wide keyed on (content_version, g_id) — the contract content_version exists for. Clean subjects take the batched sweep even when their binding has no raw s_id (one reverse dict lookup, ~2us, versus an ~18us fallback read); dirty and novelty-only subjects keep the merge-correct fallback. A tracing::debug! reports the per-query lane split. Probe now covers the live-novelty regime: typed table 2.93 -> 0.65 ms/iter (identical to clean HEAD; 199 batched / 2 fallback), with correctness assertions that a novelty-touched subject renders merged truth, an untouched subject stays exact, and a novelty-only subject appears.
…ement Bolt now authenticates against the same identity pipeline as the HTTP data plane and enforces data policy from the verified identity. Auth (fluree-db-bolt + server glue): - HELLO (<=5.0) / LOGON (5.1+) auth maps surface as Turn::Authenticate; verification is async in the glue via the hoisted transport-agnostic verify_data_principal (the HTTP bearer extractor now wraps the same function). Schemes: bearer (data-plane JWT/JWS), basic as a token carrier (password holds the token, principal ignored), none. - data_auth_mode enforced per session (none ignores credentials, optional verifies when presented, required refuses anonymous), replacing the refuse-to-start startup guard. - Per-statement re-checks for long-lived sessions: token expiry (Neo.ClientError.Security.TokenExpired -> drivers re-auth) and ledger read/write scopes (DatabaseNotFound, existence-hiding like the HTTP routes' 404). LOGOFF drops the identity; RESET after a failed LOGON returns to the authentication state. Policy: - Reads (autocommit + inside explicit transactions) wrap the view with the session identity's in-ledger policy bindings, mirroring the HTTP Cypher route. No policy knobs over Bolt by design: the transport only authenticates; policy derives from the graph. - Typed-table node hydration is now policy-aware instead of failing closed: each raw-SPOT subject fetch filters through the view's QueryPolicyEnforcer (bulk class-cache populate + per-flake filter), the same enforcer the scan path uses. - Writes: identity flows into GovernanceOptions (f:modify enforcement via the consensus committer) and CommitOpts.identity (f:identity provenance). Explicit transactions pin governance at BEGIN, build the policy context against the transaction's private state per statement, and policy-wrap conditional MERGE probes. Tests: session-machine auth coverage in fluree-db-bolt; new bolt_auth_integration.rs covers required/optional modes, both protocol generations, basic-as-token-carrier, scope gating (autocommit and explicit transactions), per-statement expiry, LOGOFF re-auth, and two identities seeing different graphs from the same Cypher; api-level test pins typed-table hydration filtering under policy.
pattern_long/pattern_cycle ran 2.2-2.8x Neo4j because every value-only bound relationship variable added a per-hop OPTIONAL annotation probe, gated only on f:reifiesSubject being in the dictionary — which the built-in vocab satisfies on effectively every ledger, so the probes always ran, annotations or not. The gate is now evidence-based with three conservative tiers: dictionary absence (certain no), index property stats showing f:reifiesSubject facts (per-snapshot, sync), and a cached overlay PSOT walk answering whether novelty carries any reifies fact (keyed on content_version; callers that cannot supply the overlay stay conservative). LoweringContext grows reified_edges_possible (default true); the read path, query_cypher_ast, and the conditional-write probes supply their view's overlay, explain stays conservative. Property-reading edge vars keep annotation-only lowering unchanged. Probe (200 users, indexed, no annotations): 2-hop bound-edge pattern 3.04 -> 0.54 ms (~5.6x). The new gate-tier integration test also exposed a pre-existing Bolt mapping gap: a reified edge binds its variable to the annotation subject, which the typed table rendered as a Node. Subject cells are now reifier-aware in both hydration lanes — f:reifies* rows make a Relationship (endpoints + type from the reified triple, reifier IRI as element_id, annotation scalars as properties); the tier test pins a novelty-only annotation binding through the reopened gate.
shortestPath ran one-to-two awaited range reads per node per hop (cursor setup + overlay merge per call), which is why it sat 7-15x behind Memgraph and scaled with graph size. Both search modes (bidirectional Single, layered allShortestPaths) now expand whole frontier levels through a new frontier module: base edges come from the gap-aware batched index sweeps (PSOT refs for typed hops, SPOT rows for wildcard, OPST inbound for the reverse side), and novelty is applied as an edge delta built from ONE netted overlay walk (cached on content_version) — asserts extend the expansion, retractions mask base rows, so the fast path is merge-correct rather than gated. Traversal state is PathNode (raw u64 subject ids for persisted nodes, Sids only for novelty-only nodes), cutting per-visit hashing and clone cost; Sids materialize only for emitted paths. Also unifies the two hardcoded traversal caps behind FLUREE_PATH_MAX_VISITED (read once, default raised 10k/100k -> 1M), which unblocks wide traversals like benchgraph expansion_4 at 100k+ nodes; the cap doc states the memory tradeoff. Probe (200 nodes, wildcard *..15): 1.52 -> 0.54 ms/iter; the win grows with scale since per-node overhead multiplied by visited count is what it removes. New regression test pins the delta semantics: a novelty-added shortcut wins over the indexed route, and retracted indexed edges stop being traversable.
The level expansion materialized a whole level's (parent, neighbor) pairs before insertion — frontier x out-degree x 48B, which at the 1M visited cap on a degree-30 graph is ~1.4GB transient. expand_with now processes the frontier in 8192-node slices and streams edges through a visitor (Break stops a level early, e.g. a bidirectional meet), so peak transient memory is slice x degree (~15MB) regardless of the cap. The bidirectional side maps now carry (predecessor, depth) so the min_hops check at a frontier meet needs no mid-stream reconstruction.
The level-batched shortestPath frontier regressed 4-5x on the real benchmark graph (pokec medium: 4.77 -> 23ms) while winning on the toy probe. The probe was doubly unrepresentative: clustered subject ids (no scan amplification) and, once made scattered, an unreachable endpoint pair (exhaustive search, where batching wins). The bench measures reachable pairs, where a bidirectional search meets after touching a few hundred nodes — an early-terminating search must not read whole levels ahead of the meet, at any batch size. Lesson recorded: batched level expansion fits exhaustive traversals (expansion_4-style reachability — already at Memgraph parity on per-node reads); per-node point reads fit early-exit searches. shortest_path.rs returns to the proven implementation (0.375 ms on a bench-shaped reachable pair vs 8-49ms for the batched variants); the frontier module is removed with it (git keeps it for a future property-path attempt measured against representative shapes). Kept, both bench-confirmed or latent-bug fixes: - visited-cap unification behind FLUREE_PATH_MAX_VISITED (expansion_4 completes at medium, dead even with Memgraph, rows exact); - gap-aware chunking in the batched lookups (scattered subject sets become near-point-seeks instead of dragging one scan across the id space between them); - per-chunk membership filtering in all three batched lookups: cursor batches are leaflet-granular and spill past a chunk's range, and the global membership set collected spillover rows twice — duplicated rows for boundary subjects (surfaced as doubled node properties at >1000-subject hydrations; latent for the stats consumers). The probe seeds a scattered 20k-node graph (multiplicative-hash edges) with a reachable shortestPath pair so future traversal work measures against representative shapes.
Node hydration allocated a fresh String per property key, label, and IRI per node — despite the hydrator already memoizing every name as Arc<str> — and then the PackStream encoder copied each one again into Value::String. On a 20k-node RETURN n that was ~180k redundant allocations per result. - CypherNode/CypherRelationship/CypherPath carry Arc<str> for iris, labels, property keys, and type names; the hydrator's memoized names now flow through by refcount instead of by copy. - The Bolt codec's Value::String and MapValue keys hold Arc<str>, so node/relationship/path structures move the shared strings straight onto the wire encoder — the string is allocated once per distinct name per result, not once per use. - IriCompactor::decode_sid_shared: decode to Arc<str>, returning the Sid's own name (refcount bump, zero work) for the empty and overflow namespaces and for any namespace whose prefix is empty — the hot path if Cypher identifiers move to namespace 0. Probe (20k nodes, 8 props): typed hydration 87 -> 73 ms/iter; the server-side PackStream encode additionally stops copying every key/label/IRI per node.
Cypher labels, relationship types, and property keys now resolve to bare names in namespace 0 (the pre-registered empty prefix) instead of being forced under http://example.org/. Cypher users get durable, readable identifiers with zero configuration; the planner keeps full Sid-gating (stats lookups, batched join lanes) because bare names encode to Sid(0, name) at lowering on both read and write paths. RDF compatibility is now the opt-in: a ledger default context or per-view context with @vocab restores IRI-prefixed resolution, covered by new API-level and server-level tests. Bare JSON-LD terms land in the same namespace, so Cypher and JSON-LD converge on identical names. Bare-name lowering only fires when no vocab is configured and the identifier contains no colon, so scheme-ful strings (rdf:type, backticked IRIs) keep strict encode semantics. Policy identities are unaffected: identity resolution uses strict encoding, so identities remain scheme-ful IRIs/DIDs.
Gap-aware chunking (split at subject-id gaps > 64) shattered scattered hydration sets into near-point-seeks — hundreds of ~8 us cursor descents per node-emit crawl. The benchmark it was meant to help measured it as a ~2x per-row regression instead (pokec node-emit 32 -> 61-126 us/row), and its original beneficiary (the batched shortestPath frontier) is already reverted. Span/size-bounded chunking returns for all three lookups; the per-chunk membership filtering stays — chunk boundaries can still share a leaflet, so global-set filtering would double-collect spillover rows.
The wildcard subject-flake cache arm (5717392) matched on 'anything but K=0/K=1', so narrow K>=2 explicit projections fetched every predicate unfiltered — silently paying the dict-touch and per-row fuel the predicate_filter exists to save. The cache read-through now applies to true wildcard levels only; narrowed projections keep their filtered SPOT reads.
End-to-end proof that colon-free backticked names (slash, hash, space, embedded @) round-trip whole through write and read — canonical_split only splits when a colon is present, so no namespace is ever minted for them — while colon-containing backticked names take the RDF escape hatch (namespace split + registration) and still round-trip. Document the placement rule in the Cypher names section.
Cursor batches are leaflet-granular, and the batched subject-property lookup tested every row against a membership set — a single-node hydration paid a full-leaflet scan (~16k contains-checks, ~120 us) to collect one subject's ~20 rows. SPOT's primary sort key is s_id, so the consumer now sorted-merges the wanted id list against the batch with galloping binary search, visiting only each subject's contiguous run. Spillover rows for a neighboring chunk's subjects are excluded by construction, preserving the no-double-collect invariant the membership set provided. Single-node typed hydration drops 135 -> 34 us (in-process single_vertex_read 0.24 -> 0.14 ms); scattered mid-size sets (the node-emit shape) skip the same per-leaflet full scans. Dense whole-graph hydrations are unchanged (decode/build dominates). PSOT/OPST lookups keep the membership set — s_id only ascends within a predicate/object run there, so the merge precondition doesn't hold. New examples/cypher_point_floor_profile.rs measures the fixed per-query floor (parse/plan/exec/format) on indexed point lookups.
Every cursor open paid an open+mmap(+munmap) syscall cycle per leaf, and the batched-star SPOT walk and group-count fast paths read whole leaf files into fresh Vecs per query — together the dominant fixed cost of an indexed point lookup (sample profile: __open/read/stat were the top real frames under exec). Leaf files are immutable content-addressed CAS artifacts, so the mapping is now created once and shared via a LeafMmap entry in the existing moka leaflet cache (single-flight, error-transparent, keyed by xxh3(leaf CID)). Mmap does not retain the file descriptor, so cached mappings cost address space, not fds; the weigher counts mappings at len/64 (file-backed reclaimable pages, not heap) which bounds total mapped space at 64x the cache budget. Whole-leaf readers switch to get_leaf_bytes_shared, which serves the mapping for local leaves and falls back to the owned read for remote ones. Indexed floor queries (release, in-process, 20k users): single-vertex 0.22 -> 0.07 ms, bound-property scalar 0.13 -> 0.08 ms, 2-hop bound edge 0.67 -> 0.44 ms, shortestPath 0.40 -> 0.25 ms. Combined with the galloping batched crawl, the fixed per-query floor is roughly halved.
…efix filter_batch binary-searched only the LEADING bound sort column, so an OPST bound-object point probe (o_type pinned = every ref row) still linear-scanned nearly the whole leaflet per probe, and SPOT s+p probes scanned the subject's full row run. The seek now cascades: within the rows equal on key column k, rows are sorted by key column k+1, so each successively bound filter column narrows by two more partition_points (Const columns pass through or prove the range empty). Rows outside the range differ on a bound prefix column, so output is identical to the full scan; the per-row filter still validates the range. This makes bound point probes leaflet-size-independent on every index order and every query surface. Indexed floor (release, in-process, 20k users): single-vertex 0.072 -> 0.027 ms, scalar prop 0.082 -> 0.039 ms, shortestPath 0.25 -> 0.085 ms (200k users: 138 -> 77 ms — per-node probes are the shortestPath cost model, so it gains everywhere the search touches). The floor profiler gains a shortest_path statement and a PROBE_ONLY filter.
The bidirectional BFS paid Sid-space costs at every visited node: a range_with_overlay probe (cursor + full Flake/Sid materialization), string-hashed visited maps, and a dictionary re-encode per probe. At bench scale that made shortestPath the widest remaining gap (18x at large), scaling with visited-set size. The search now runs in raw s_id space when the view allows it (binary store present, no active policy, summarizable overlay): frontier levels expand through the batched lookups — one galloping sweep per side per level instead of a probe per node, neighbors taken directly from IRI_REF o_keys — and state lives in FxHashMaps keyed by u64. The dictionary is touched only at the endpoints and to materialize the final path. Overlay correctness is per-node: subjects/ref-objects the overlay touches (retracts stamp both sides) are collected once per overlay version into id-space dirty sets (LRU-cached), and dirty or novelty-only nodes expand through the existing Sid fallback, so novelty shortcuts, novelty retracts of base edges, re-asserted edges, and novelty-only endpoints all resolve exactly (new end-to-end test). Views the lane can't serve keep the Sid search unchanged. Supporting changes in the batched lookups: the galloping merge now covers all three (PSOT rows are s_id-sorted after the cursor's p_id filter pins the leading key; OPST gallops o_key inside the binary-searched IRI_REF o_type run), and gap-splitting returns to the chunker at the measured break-even (4096 ids) — wide-gap scattered sets stop sweeping every leaflet in their span, while dense sets keep long shared scans (node-emit unchanged, ~62 ms/20k dense rows). shortestPath (release, in-process): 20k users 0.085 ms (parity with the per-node lane), 200k users 138 -> 6.4 ms.
Comments and test fixtures referenced external benchmark workloads and their query names; describe the query shapes on their own terms instead. Protocol-required compatibility references (driver agent string, wire-format parity, spec links) are unchanged.
Bare `MATCH (n) RETURN count(n), count(n.age)` and the min/max/avg/sum family folded from index directories only at HEAD; any uncommitted novelty made the whole operator decline to a full-graph flake scan (linear in graph size — seconds-to-minutes for a single count once a large indexed ledger carried post-index writes). Add an overlay lane to fast_whole_graph_agg: keep the O(directory) base counts and reconcile them with a bounded O(novelty) pass. The subject universe is the base SPOT lead-group count plus the overlay's net subject delta (per touched subject: a surviving assertion means it exists; new vs base-present decides +1/0; a retraction-only base-present subject is a possible deletion and declines to the exact pipeline). Each predicate-scoped fold (rows, distinct subjects/objects, min/max/avg/sum) runs an overlay-merging cursor, linear in the predicate's rows rather than the graph's. Covers the whole-graph anchor's scalar folds; class anchors and histograms under novelty still decline. In-process (dirty indexed ledger): count(n),count(n.age) 200k 1744ms -> 0.5ms; count(n) 1562ms -> 0.02ms; holds at 1M base (2.3ms) and 5000-flake novelty (9ms). Results verified exact against the WITH-barrier pipeline across new/updated/deleted subjects. Also fixes a clippy semicolon lint in batched_lookup for_each_subject_run.
`MATCH p = shortestPath(...) WHERE all(x IN nodes(p) WHERE <pred>)` was post-filtered: Fluree found the unconstrained shortest path, then checked the predicate, returning EMPTY whenever that single path violated it. Neo4j/openCypher evaluate the predicate during the search and return the shortest path whose nodes all qualify — so on any graph with a longer qualifying route Fluree gave a wrong (empty) answer. Absorb a trailing `all(x IN nodes(p) WHERE …)` filter into the ShortestPath pattern (new `node_filter`) at lower time when the predicate references only the iteration var, and evaluate it per node during the Sid-lane bidirectional BFS: endpoints must qualify and only qualifying nodes enter the frontiers, so the search returns the shortest qualifying path. Multiple such filters AND-combine. The raw-id lane declines when a node filter is present (it never materializes a node Sid). Qualification reuses the post-filter's member resolution (eval_single_node_predicate), so a pushed check is identical to the original `all(...)`. Correctness only; the filtered case runs the slower Sid lane (batched raw-id filtering to close the perf gap is a follow-up).
The batched subject probe read every leaflet spanned by the probe set's [min_subject, max_subject] key range, decoding all columns and walking the predicate run before discarding leaflets that held none of the wanted subjects. For a hash-scattered neighbor set (e.g. multi-hop Cypher expansion) that range covers nearly the whole predicate partition, so the probe decoded hundreds of leaflets to match a handful of subjects. Add a directory-level skip: for a predicate-homogeneous leaflet the first_key/last_key bound its subject interval, so a leaflet holding no wanted subject is declined before the column decode. Also short-circuit the p-run linear scan on homogeneous leaflets, whose run is the whole leaflet by definition. Both guard on p_const == Some(p_id); the subject skip additionally excludes history replay. expansion_1 2.52ms -> 0.073ms, expansion_1_filter 3.30ms -> 0.076ms, pattern_long 1.18ms -> 0.131ms on a 200k-user indexed knows-graph; results unchanged, full cypher/query/sparql suites green.
The per-graph-stats shortcut for COUNT(DISTINCT ?p) was disabled because the incremental index build persisted delta-only per-graph property stats (base entries dropped, net-zero churn pinned at count 0), making the in-memory count wrong after any incremental index. That indexer defect is now fixed — the incremental build carries base per-graph property stats forward — and covered by regression tests, so the shortcut is exact again. Restore distinct_predicates_from_graph_stats in the DistinctPosition:: Predicates arm, keeping the PSOT p_const scan as the fallback when stats are absent (older index). The scan-hidden-predicate gate above the arm is unchanged, so the shortcut only runs when its count equals the scan. Recovers the fast lane on the number-of-predicates query (few predicates: ~0.5ms vs ~1.4ms general distinct scan). New regression test drives the delta-only trigger without a reifier so the shortcut arm is actually exercised, and pins its count to the general pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Bolt protocol endpoint (Neo4j drivers → openCypher), moves Cypher to bare namespace-0 names by default.
Bolt protocol server
fluree-db-boltprotocol crate: PackStream codec + session state machine; listener influree-db-server(autocommit + explicitBEGIN/COMMIT/ROLLBACK),Node/Relationship/Path+ temporal structures, write-RETURNentities as typed nodes.boltfeature on by default (listener binds only whenbolt_listen_addris set); first-class CLI flags onserver run/start/restart; reference + guide docs.Bare namespace-0 names as the Cypher default
Cypher labels/types/property keys now live under namespace 0 (the empty prefix) instead of being forced under
http://example.org/: zero-config durable names for LPG users, planner keeps full Sid-gating, and bare JSON-LD terms converge on the same names. RDF compat is the opt-in via@context/@vocab(ledger default or per-view). Placement rule is locked by test: no colon → whole name verbatim (never namespace-split); colon-containing backticked names are the RDF escape hatch.Performance campaign
Read/serve path, all query surfaces (SPARQL and JSON-LD ride the same substrate):
fs::readcopies were the dominant fixed cost of an indexed point lookup.Arc<str>strings through the encode path; adjacency no longer inlined into property maps.s_idspace with batched frontier expansion (one galloping sweep per side per level); dictionary touched only at endpoints + final path. Per-node overlay dirty-split keeps novelty shortcuts/retracts/re-asserts and novelty-only endpoints exact (end-to-end test); views the lane can't serve (policy, unsummarizable overlay) keep the Sid search.Fixes surfaced along the way
to_t < index max_t) so probes stay on the replay-correct SPOT sidecar lane. (The underlying rebuild-path hole — omitted empty partitions — predates this branch and remains open.)Validation
bolt_node_emit_probe,cypher_point_floor_profile(per-phase floor decomposition,PROBE_ONLY/PROBE_USERSfilters).