diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5b68712..126f070 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -57,6 +57,14 @@ jobs: - `issundb-rest`: IssunDB with a REST API server - `issundb-mcp`: IssunDB with an MCP server + ### A Note on the Windows ARM64 Build + + The `windows-arm64` binaries are built without the `hnsw` feature, because the C SIMD + library behind it does not compile for that target. Vector search there is an exact + scan: it returns the true nearest neighbors rather than approximate ones, so recall is + higher, but the search is linear in the number of vectors instead of sublinear and + quantization is ignored. Every other platform uses the HNSW index. Nothing else differs. + ### Container Image The same release is published as a container image to GHCR: @@ -95,7 +103,19 @@ jobs: runner: windows-11-arm archive_ext: zip content_type: application/zip - cflags: "-DNK_TARGET_HASWELL=0 -DNK_TARGET_SKYLAKE=0 -DNK_TARGET_ICELAKE=0 -DNK_TARGET_GENOA=0 -DNK_TARGET_DIAMOND=0 -DNK_TARGET_SAPPHIRE=0 -DNK_TARGET_SAPPHIREAMX=0 -DNK_TARGET_GRANITEAMX=0 -DNK_TARGET_TURIN=0 -DNK_TARGET_ALDER=0 -DNK_TARGET_SIERRA=0" + # `usearch` pulls in `numkong`, whose C dispatch files fail to compile for + # aarch64-pc-windows-msvc: `winnt.h` reports "No Target Architecture", meaning the + # compiler it is driven with defines none of the `_M_*` architecture macros. The + # `-DNK_TARGET_*=0` flags that used to be set here were a no-op, since numkong's own + # build script already sets every one of them to 0 on this target. + # + # So this target is built without `hnsw`, which drops usearch and numkong from the + # graph entirely and selects the pure-Rust exact vector index. That index returns the + # true nearest neighbors under the same distance conventions, so results are exact + # rather than approximate; what it gives up is the sublinear search and quantization. + # Remove this once numkong builds here, and the target goes back to HNSW with no other + # change. + build_flags: "--no-default-features --features lmdb" steps: - name: Checkout repository uses: actions/checkout@v4 @@ -120,10 +140,7 @@ jobs: run: echo "RUSTFLAGS=-C link-arg=-Wl,-ld_classic" >> $GITHUB_ENV - name: Build release binaries - run: cargo build --release -p issundb-cli -p issundb-rest -p issundb-mcp - env: - CFLAGS_aarch64_pc_windows_msvc: ${{ matrix.cflags || '' }} - CFLAGS_aarch64-pc-windows-msvc: ${{ matrix.cflags || '' }} + run: cargo build --release -p issundb-cli -p issundb-rest -p issundb-mcp ${{ matrix.build_flags || '' }} - name: Package release asset (Unix) if: runner.os != 'Windows' diff --git a/AGENTS.md b/AGENTS.md index bd5bfce..e5cc651 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,10 @@ Quick examples: This map describes the current structure and the target decoupled crate boundaries. Do not invent modules that do not yet exist, but do place new modules according to this map. +An entry says what a module owns and where a new thing belongs. It does not explain how the module works: that lives in the crate's own `AGENTS.md`, +named at the end of this section, and a public method's contract lives under Component APIs. Keep it that way when you edit. This section had grown a +second copy of both, so the same rule was stated in three places and the copies had begun to disagree. + - `crates/issundb-core/`: storage engine. Public surface is `Graph` and the schema types. - `src/bin/gen_testdata.rs`: the `gen_testdata` binary that regenerates the versioned LMDB storage-format snapshot (works with `make testdata`). - `src/schema.rs`: `NodeId`, `EdgeId`, `LabelId`, `TypeId`, `AdjEntry`, `NodeRecord`, and `EdgeRecord`. `NodeRecord` holds `labels: Vec` @@ -79,66 +83,33 @@ modules according to this map. - `src/graph/node.rs`: node CRUD (`add_node`, `get_node`, `update_node`, `delete_node`). - `src/graph/edge.rs`: edge CRUD and adjacency (`add_edge`, `get_edge`, `delete_edge`, `out_neighbors`, `in_neighbors`, `node_has_relationships`). - `src/graph/index.rs`: label and type indexes, property indexes, constraints, and property scan methods. - - `src/graph/stats.rs`: high-order cardinality statistics and the data-graph schema for the optimizer. Owns the `(label, type)` edge-frequency - table behind `estimate_expand_fanout` and the realized `(src_label, type, dst_label)` triples behind `estimate_expand_fanout_to` and - `schema_has_edge`, built by one pass over `label_idx` and one over `out_adj` (neither decodes a record, since a `NodeRecord` or `EdgeRecord` - decode also copies a property blob this table never reads) and cached against the committed-write generation. Nothing builds it as a side effect - of a query, and the generation check gates use rather than refresh: a table from an earlier generation is ignored, never trusted. The two fan-out - estimates are advisory and fall back to the global average without it, so only `Graph::materialize_edge_statistics` builds it. `schema_has_edge` - is not advisory (the optimizer drops rows on a negative), so it never depends on the table existing: with no current table it probes `label_idx` - and the adjacency directly under `SCHEMA_PROBE_BUDGET`, settling on the first matching edge and reporting `None` when the budget runs out. + - `src/graph/stats.rs`: high-order cardinality statistics and the data-graph schema for the optimizer, behind `estimate_expand_fanout`, + `estimate_expand_fanout_to`, and `schema_has_edge`. What may build the table, and what a negative means, are in the crate guide under + "Schema Statistics" and with those methods under Component APIs. - `src/graph/fts_mod.rs`: full-text search index lifecycle and FTS storage primitives. - `src/graph/vector.rs`: vector byte storage helpers. - `src/graph/algo.rs`: public algorithm dispatch methods and internal traversal helpers. - `src/graph/kernels/`: graph algorithm implementations over the CSR snapshot, split by family: `traversal.rs`, `analytics.rs`, `paths.rs`, and - `flow.rs`. Almost every kernel reads the snapshot and nothing else, so one gate (`Graph::with_snapshot`) covers them; a kernel needing a per-edge - property the snapshot does not carry reads that property from storage per call. `label_propagation` is the one exception, walking - `all_neighbors` per node per iteration instead, which is why it needs no gate and is far more expensive than its siblings. Where a result's sequence is observable the kernel fixes it deliberately: - a traversal reports reached nodes in ascending dense (so ascending node id) order, each frontier is sorted, and Brandes accumulates over sources - and predecessors in that order, so a betweenness total is reproducible rather than merely close. No depth-first kernel recurses over graph structure, - because a stack overflow aborts the process instead of returning an error; `dfs` is the sole exception, bounded by its `u8` hop count. + `flow.rs`. The one freshness gate they share, the ordering guarantees, and the no-recursion rule are in the crate guide under + "Algorithm Kernels". - `src/graph/txn.rs`: `ReadTxn` and `WriteTxn` delegation impls and transaction tests. - - `src/csr.rs`: in-memory CSR snapshot (outgoing arrays plus a transposed incoming view with per-edge type and edge ids), rebuilt in the - background and swapped via `arc-swap`. Also owns the `GraphDelta` buffer captured on the write path (whose only consumers are the property - column caches) and the `write_gen`/`snapshot_gen` generation counters that drive on-demand CSR refresh. The adjacency is read from `out_adj`, whose 20-byte - `AdjEntry` carries every field the arrays hold, already grouped by source in ascending key order, so the build decodes no `EdgeRecord` and copies - no property blob. Entries go straight into the flat arrays, counting each row as it goes: a per-node `Vec` staged first, as this once did, cost - one allocation per node and left 3.3 GB resident for 620 MB of live arrays on a 1 M-node, 13.9 M-edge graph, because a million freed small chunks - are holes the allocator cannot return. Because `DUPSORT` orders duplicates by their raw little-endian bytes, each row is then reordered by - ascending edge id, which is the order every consumer has always seen and which `load_weights` binary-searches. The per-entry `edge_weight` is - `Option`, built only by `build_weighted`, since a weight lives in the edge's property blob and only Dijkstra reads one. - - `src/columns.rs`: in-memory property columns for the read path. One typed column (`Int`, `Float`, `Bool`, dictionary-encoded `Str`, or the - exact-semantics `Json` fallback) per node property, built lazily from one full node scan and kept fresh by a post-commit delta (node deletion - forces a rebuild). Read through `Graph::node_prop_json`. Also owns the lazily computed per-property statistics (`PropStats`: bounds, an - equi-depth histogram, and the most common values) that back the selectivity estimates, invalidated by the post-commit patch. Which readers may - cause the build is deliberate, because the build is one full scan: a gather larger than `SMALL_GATHER_MAX` does, a smaller one is served straight - from storage (`should_serve_directly`), and the advisory statistics never do (`with_existing_mut` rather than `with_fresh`). + - `src/csr.rs`: the in-memory CSR snapshot, its transposed incoming view, the `GraphDelta` write buffer, and the generation counters that drive + on-demand refresh. How it is built from `out_adj`, why no per-node `Vec` is staged, and the row-ordering rule are in the crate guide under + "CSR Snapshot Vs. LMDB Adjacency". + - `src/columns.rs`: in-memory typed property columns for the read path, plus the per-property statistics behind the selectivity estimates. Which + readers cause the one full scan that builds them is in the crate guide under "In-memory Property Columns". - `src/histogram.rs`: equi-depth histogram over property values with equality and range selectivity estimates; backs `PropStats`. Nothing here is persisted. - - `src/threads.rs`: the one resolution of the thread budget every parallel consumer shares (`threads::resolve`). Precedence is the programmatic - override from `set_thread_count`, then `ISSUNDB_NUM_THREADS`, then `OMP_NUM_THREADS`, then the machine's parallelism, clamped to `MAX_THREADS`. - Both the counting kernels' scoped threads and the analytics passes that split over nodes or sources resolve through it (`Graph::kernel_threads`), - so the one knob has one meaning and two overlapping passes cannot each claim the whole machine. `OMP_NUM_THREADS` is honored because setting it is - how a caller caps parallelism process-wide, including this repository's own `test` and `coverage` targets. + - `src/threads.rs`: the one resolution of the thread budget every parallel consumer shares (`threads::resolve`). Precedence, the clamp, and why + `OMP_NUM_THREADS` is honored are in the crate guide under "Thread Count". - `src/storage/memory.rs`: the in-memory storage backend, second implementor of the contract in `storage/mod.rs`. Byte-ordered `BTreeMap` tables with `BTreeSet` duplicate values, copy-on-write transactions over `ArcSwap`, and a single writer lock. It is what a target with no libc compiles, and it is - what holds the storage seam to something: the whole suite runs against it (893 tests across core, vector, text, retrieval, and cypher). + what holds the storage seam to something: the whole suite runs against it, across core, vector, text, retrieval, and cypher. - `src/error.rs`: `Error` enum; all storage and serialization errors unify here. `Error::Storage` carries `storage::StorageError`, which is the selected backend's error type, so the variant is `heed::Error` on a default build and unchanged from before the backend split. - `crates/issundb-cypher/`: Cypher parser, AST, logical planner, physical planner, optimizer, and executor. - - `src/parser.rs`: Cypher parser built with the `chumsky` parser-combinator library (with a Pratt parser for operator-precedence expressions), - covering MATCH (including inline relationship property maps and multi-label node patterns such as `(n:A:B)`), WHERE, RETURN, CREATE, SET - (property and label assignment), REMOVE (label and property), and DELETE/DETACH DELETE over arbitrary expression targets. An iterative - token-stream scan (`scan_nesting`) rejects genuinely pathological input (thousands of levels) with a parse error before any AST is built. - Realistic deep input is kept safe by running on large stacks: a deep parse runs on a dedicated large-stack thread, and a query whose nesting - exceeds `SMALL_STACK_EXEC_BUDGET_KB` has its execution dispatched to a large-stack thread by `execute_with_procedures`. Shallow queries, the - common case, parse and execute inline on the caller stack. - Building the combinator graph costs more than consuming the tokens, and for a small query more than executing it, so the executor's entry point - (`parse_with_exec_depth`) serves repeated query text from a bounded thread-local cache of `Arc` and returns the same allocation. - Parsing reads no graph state, no parameters, and no clock, so the cached outcome (including a parse error, and including the nesting-depth - rejection) is always valid and the cache needs no invalidation. `parse` is the uncached entry point: it always does the work, which keeps the - `parse` benchmarks a regression guard on the parser itself. Query text over `PARSE_CACHE_MAX_QUERY_LEN` is parsed but not stored, so many large - unique statements cannot grow the cache by their length. + - `src/parser.rs`: Cypher parser built with the `chumsky` parser-combinator library, with a Pratt parser for operator precedence. The nesting + scan, the large-stack dispatch, and the parse cache are in the crate guide under "Parser Structure Rules". - `src/ast.rs`: AST node types. - `src/plan/`: logical planner, physical planner, optimizer, and statistics helpers. - `src/procedure.rs`: the `ProcedureRegistry` a caller passes to `query_with_procedures`, plus the argument and yield types a procedure sees. @@ -147,21 +118,8 @@ modules according to this map. - `src/exec/mod.rs`: public entry points (`execute`, `explain`), shared type definitions, and tests. - `src/exec/read.rs`: `execute_physical` and read-path helpers (`evaluate_where`, `evaluate_sort_key`, `json_to_prop_value`, `filter_over_expand_batch`, and `multiway_join_rows`, the last shared by the materializing and streaming `MultiwayJoin` paths). - - `src/exec/vectorized.rs`: columnar fast path for the final projection or aggregation over a linear chain of up to `MAX_VEC_HOPS` directed - single hops. A structural recognizer matches `[Limit]? [Sort]? [Distinct]? Project [Aggregate]? Stage* (Expand(directed single hop) - Stage*){0,MAX_VEC_HOPS} Leaf` with single-property expressions, executing column-at-a-time (bulk expansion via `Graph::node_props_json_table` - and group-by-code aggregation via `Graph::node_prop_group_codes`). A multi-hop chain is recognized only when every hop carries a distinct - relationship type, so relationship uniqueness is vacuous; a repeated type or a chain longer than `MAX_VEC_HOPS` falls back. A non-distinct - `count` over the terminal variable that feeds no group key collapses the final hop (`execute_collapsed_count`). The collapse counts each source's - qualifying neighbors through `Graph::typed_neighbor_counts`, so the final hop costs no triple per traversed edge and no hash lookup per edge: a - terminal filter that is a label test goes straight into the spec, and a terminal property comparison is resolved into a `neighbor_allow` set by - running those exact stages over the label's whole node set (`resolve_terminal_allow`). That resolution is gated on the sources' `adjacency_span` - reaching half the label count, so a selective hop over a large label keeps the expansion fallback instead of paying for a full label pass, and it - is speculative: it evaluates predicates over a superset of the real neighbors, so a stage that errors there declines to the fallback rather than - raising. Two shapes route to the fallback regardless: a multi-type hop, because `Expand::rel_type` carries the raw pattern text (`"F|G"`) and the - kernel resolves one registered type; and a stale snapshot with at most `STALE_POINT_EXPAND_MAX` sources (`Graph::prefers_point_expansion`), because - the kernel would rebuild the whole snapshot where the fallback serves those sources from per-source adjacency. The recognizer sees through a `Distinct` because the caller deduplicates. Any unrecognized shape falls back to the row pipeline, so - correctness never depends on the recognizer. + - `src/exec/vectorized.rs`: columnar fast path for a final projection or aggregation over a linear chain of directed single hops. Exactly which + shapes it accepts, and the two that always fall back, are in the crate guide under "Vectorized Aggregate and Columnar Fast Path". - `src/exec/factorize.rs`: `FactorizedRecordGroup` (shared `Arc` prefix plus per-row extensions) and `filter_refs_in_expr`. - `src/exec/expr.rs`: expression evaluation (`evaluate_expr`, `eval_binary_op`, `eval_arithmetic`, `eval_function_call`). - `src/exec/write.rs`: mutation execution (`execute_create`, `execute_set`, `execute_delete`, `execute_merge`). @@ -169,14 +127,10 @@ modules according to this map. node property lookups are already served by the always-on auto-index; a relationship `CREATE INDEX` provisions the property index. - `src/exec/copy.rs`: bulk data administration execution (`COPY ... FROM`, `EXPORT DATABASE`, and `IMPORT DATABASE`). - `src/exec/row.rs`: the positional row representation (`SlotRow` and `SlotSchema`) the row pipeline binds variables through. -- `crates/issundb-vector/`: vector index abstraction, vector metadata, vector storage integration, and vector search APIs. The index itself sits behind - `backend.rs`, which selects one at compile time from the `hnsw` feature: on by default it is `usearch`, the workspace's only C++ dependency, and with - `--no-default-features` it is an exact scan in pure Rust. The fallback is not a stub. It returns the true nearest neighbors under the same distance - conventions (`exact_distance`, shared with the rescore pass), so the crate's whole suite passes either way; what it gives up is the sublinear query and - `quantization`, which it ignores because it keeps the raw `f32`. The feature is forwarded by every crate that reaches this one (`issundb-cypher`, - `issundb-retrieval`, and the `issundb` facade), and the workspace declarations of those three carry `default-features = false` so that - `--no-default-features` on the facade actually reaches the bottom of the graph rather than being re-enabled by a sibling. Verify a change to this - plumbing with `cargo tree -p issundb --no-default-features | grep usearch`, which must print nothing. +- `crates/issundb-vector/`: vector index abstraction, vector metadata, vector storage integration, and vector search APIs. The index sits behind + `backend.rs`, selected at compile time from the default-on `hnsw` feature: `usearch`, the workspace's only C++ dependency, or a pure-Rust exact scan. + The fallback is exact rather than a stub, which is what lets one suite prove both. The feature plumbing that makes `--no-default-features` actually + reach the bottom of the graph is under Architecture Constraints; the rest is in the crate guide under "The Backend Seam". - `crates/issundb-text/`: text query APIs and ranking. Tokenization and the inverted-index storage are *not* here: they live in `issundb-core` (`graph/fts_mod.rs` and `storage/fts.rs`), because the write path is in core and the FTS postings are maintained inside the same write transaction as the node record (`index_node_for_label` on insert and update, `delete_node_fts` on delete). A tokenizer in this crate could not @@ -199,42 +153,18 @@ modules according to this map. `issundb`; uses `tokio`. See its Component APIs entry for the tool surface and the Host-header allowlist. - `crates/issundb-py/`: Python bindings via PyO3. Exposes the `IssunDB` class. Depends only on `issundb`. - `crates/issundb-wasm/`: browser bindings, exposing one `Playground` type that owns a single `Graph`. Depends only on `issundb`, and is the only crate - built for `wasm32-unknown-unknown`. It is what proves the storage-backend seam and the pure-Rust kernels actually hold: the module is built - `--no-default-features`, so storage is the in-memory backend and the vector index is the exact scan, and a regression that reintroduces an LMDB or C++ - dependency below the facade breaks this build rather than going unnoticed. Do not add `--features hnsw` to that build, which reads like it selects the - index and in fact selects `usearch`: the wasm build then fails compiling `cxx`. `make playground-check` is where that was caught, so keep the flags in - the `WASM_BUILD` variable rather than repeating them per target. Every method returns a JSON string, so the boundary carries one - type in both directions instead of a second serialization contract. The methods are split into a private logic layer returning `Result<_, String>` and a - thin exported layer that converts to `JsError`, because constructing a `JsError` calls a wasm-bindgen import that panics off-target, and without the - split none of it could be covered by `cargo test`. Reading all of a node's properties decodes the stored msgpack blob directly, as the REST node route - does, since every read-path method on `Graph` takes the property names to fetch and an inspector cannot know them. -- `web/`: the playground page that loads that module: `index.html`, `app.js`, `demos.js`, and `style.css`, with the generated module in the gitignored - `web/pkg/`. Vanilla ES modules with no build step, and no library is fetched from a network, so the Cypher highlighter and the force-directed layout are - written in `app.js` rather than pulled from one. The page's only external request is the Google Fonts link for Inter and JetBrains Mono, which is the same - request `theme.font` in `mkdocs.yml` already makes for the same two families; the size scale is the reference playground's, in rem against a 1rem body. It is served under the MkDocs site and styled to match it: the custom properties at the top of - `style.css` are Material for MkDocs' own tokens, copied from the built `palette.*.min.css` for this site's palette, and the scheme is carried on - `data-md-color-scheme` with Material's `default` and `slate` values. A `theme.palette` change in `mkdocs.yml` means updating that block from a fresh - `make docs` build rather than from the Material Design palette, since MkDocs derives its primary from the named color instead of using it directly. `demos.js` holds the example catalog, the - Setup panel's six sample graphs, and the sidebar's procedure reference, all of which are Cypher inside a JavaScript file and therefore invisible to every - Rust test; `make playground-check` runs all three through the compiled module and fails on an error, which is how a wrong procedure signature is caught. - A sample graph is the only place a dataset lives: every example queries whatever is loaded rather than creating its own, each category names the graph it - queries through `sample` and the label that proves it is loaded through `requiresLabel`, and the check seeds that graph before running the category. The one - exception is the Cypher basics lesson on `CREATE`, which writes two nodes. - Selecting an example or a sample loads it into the editor without running it, since running a `CREATE` on click wrote to the database before the statement - had been read and a second click silently duplicated its data; the full-text and vector examples keep their post-statement step by holding the selected - example until the run. The sample graphs carry no comments, being data rather than documentation. `app.js` also holds a Cypher formatter, whose casing rule - is narrower than the highlighter's keyword set on purpose: uppercasing every word in that set rewrote `issundb.shortestPath` and the case-sensitive yield - fields `index` and `count`. It must not be able to change what a query means, which is checked by running every string in `demos.js` before and after - formatting and comparing the rows. The procedure reference is written out by hand - because the engine cannot enumerate its own procedures, so that check is the only thing keeping it from drifting; it treats `ProcedureNotFound` as a failure - even for the two retrieval entries whose empty-index error it tolerates, since a rename is exactly what that error reports. - `docs/hooks/playground_links.py` is the MkDocs hook putting a "Run in the playground" link under a Cypher block in `docs/` marked ``, - carrying the block as `q` and the page's earlier marked blocks as `s`. The marker is opt-in because most documented Cypher cannot run in the playground - (a query parameter, a CLI script, or embeddings the seeded graph lacks), so marking a block asserts that it runs, and `make playground-check` executes - every marked block and fails one returning no rows. - See `web/README.md` for the three build targets and what the browser configuration gives up (no persistence, one thread, - no `backup`/`restore`, and a 16 MB stack set by a link argument in `.cargo/config.toml` because the 1 MB default is also the engine's inline-execution - budget). + built for `wasm32-unknown-unknown`. It is what proves the storage-backend seam and the pure-Rust kernels hold: the module is built + `--no-default-features`, so a regression that reintroduces an LMDB or C++ dependency below the facade breaks this build rather than going unnoticed. + Do not add `--features hnsw` to that build, which reads like it selects the index and in fact selects `usearch`, whereupon the wasm build fails + compiling `cxx`. The binding conventions and the build flags are in `web/README.md`. +- `web/`: the playground page that loads that module: `index.html`, `app.js`, `worker.js`, `format.js`, `demos.js`, and `style.css`, with the generated + module in the gitignored `web/pkg/`. Vanilla ES modules with no build step and no library fetched from a network, served under the MkDocs site and + styled to match it. The engine runs in `worker.js` and the page reaches it only by message, so a query never blocks the tab; cancelling therefore + terminates the worker and replays the sample plus the setup log, since a WebAssembly call has no interruption point and the graph dies with the + thread. `demos.js` holds the example catalog, the sample graphs, and the procedure and function references, all of which are Cypher inside a + JavaScript file and therefore invisible to every Rust test; `make playground-check` runs them all, plus the formatter round trip, and fails on an + error. That check is the only thing keeping the hand-written references from drifting. Everything else about the page, including the theme tokens, + the graph view, sharing, and what the browser build gives up, is in `web/README.md`. - `crates/issundb-examples/`: standalone example programs. These depend only on `issundb`. - `crates/*/benches/`: crate-local Criterion benchmark targets (storage and write throughput, Cypher parsing and execution plus LSQB Q1-Q9 and OLTP reads, vector search, full-text search, and hybrid retrieval plus GraphRAG). @@ -267,9 +197,10 @@ modules according to this map. on their version string. Consolidating is a manifest change nobody has made yet, so do not read the current layout as the intended one. - `Makefile`: developer workflow entry points. - Directory-scoped guides: `crates/issundb-core/AGENTS.md`, `crates/issundb-cypher/AGENTS.md`, `crates/issundb-text/AGENTS.md`, and - `crates/issundb-vector/AGENTS.md` carry crate-specific rules that this file does not repeat (LMDB lifetime rules, the query pipeline stages, the - tokenization order, the HNSW lock ordering). Read the one covering the crate being changed, and update it in the same patch when its subject changes: - being unreferenced from here is what let several of them drift behind the code. + `crates/issundb-vector/AGENTS.md` carry the crate-specific rules this file does not repeat: LMDB lifetime rules, the write-lock contract, the CSR + freshness gate, the property columns, the schema statistics, the query pipeline stages, what the vectorized recognizer accepts, the tokenization + order, and the HNSW lock ordering. Read the one covering the crate being changed, and update it in the same patch when its subject changes: being + unreferenced from here is what let several of them drift behind the code. `web/README.md` plays the same role for the playground page. ## Testing Layout Rules @@ -333,6 +264,13 @@ modules according to this map. committed state through a separate read transaction while its own write transaction is still open, and a single reader-writer lock deadlocks on exactly that. The in-memory backend does not persist, so a reopen sees an empty graph; the handful of tests whose premise is reopen or backup are gated on the `lmdb` feature and say so. +- `issundb-cli`, `issundb-rest`, and `issundb-mcp` relay `lmdb` and `hnsw` to the facade rather than naming them on the dependency, the way `issundb-wasm` + already did, so a binary can be built without the C vector index. That is not hypothetical: `usearch` pulls in `numkong`, whose C dispatch files fail to + compile for `aarch64-pc-windows-msvc` with `winnt.h` reporting "No Target Architecture", so `release.yml` builds that one target with + `--no-default-features --features lmdb` and its binaries use the exact-scan index. Do not put `features = ["lmdb", "hnsw"]` back on those dependencies: + it makes the feature unselectable from the command line, which is what had to be undone to get that target building. +- The `hnsw` feature is forwarded the same way, and each intermediate workspace declaration carries `default-features = false` for the same reason. + Verify a change to that plumbing with `cargo tree -p issundb --no-default-features | grep usearch`, which must print nothing. - The `lmdb` feature is forwarded by every crate between the facade and core, and each of their workspace declarations carries `default-features = false`, or a sibling silently re-enables LMDB for the whole graph. Verify a change to that plumbing with `cargo tree -p issundb --no-default-features | grep lmdb`, which must print nothing. Note that a whole-workspace `--no-default-features` build does *not* select the in-memory backend, because `issundb-cli` and the other @@ -374,11 +312,8 @@ Lower-level crates must not know about higher-level crates. The central coordination type. All graph operations go through `Graph`; do not call `Storage` directly from outside `issundb-core`. `Graph::open(path: &Path, map_size_gb: usize) -> Result` is the only constructor. -Node and edge CRUD, accessors, and registry lookups have self-describing signatures; read them from the source rather than this file. Methods: -`add_node`, `add_node_multi`, `get_node`, `update_node`, `delete_node`, `add_label`, `remove_label`, `node_labels`, `add_edge`, `get_edge`, -`update_edge`, `delete_edge`, `out_neighbors`, `in_neighbors`, `node_has_relationships`, `nodes_by_label`, `edges_by_type`, `all_nodes`, `label_name`, -`type_name`, `list_node_indexes_and_constraints`, `list_edge_indexes_and_constraints`, `node_count_by_label`, `edge_count_by_type`, -`put_vector_bytes`, `vector_bytes`, and `rebuild_csr`. +Node and edge CRUD, accessors, and registry lookups have self-describing signatures; read them from the source rather than this file. Only the +methods below carry behavior the signature does not show. The read-path and statistics methods carry non-obvious semantics: @@ -407,7 +342,8 @@ The read-path and statistics methods carry non-obvious semantics: property's equi-depth histogram. - `estimate_equality_selectivity(prop, val) -> Result, Error>`: estimated fraction of non-null values equal to `val`, exact for the most common values and histogram-estimated otherwise; both feed the optimizer's selectivity-aware `Filter` plan weight. -- Those three readers are advisory, and none of them builds the property columns: each also returns `None` when the columns do not exist yet, leaving +- Those three readers are advisory, and none of them builds the property columns (`with_existing_mut` rather than `with_fresh`): each also returns + `None` when the columns do not exist yet, leaving the caller on its default plan weight or declining to prune. Forcing a build for them made the first query mentioning any property pay one full node scan (measured at roughly 1.3 seconds on an 800 K-node graph), which was the dominant cold-start latency, and the answer only weights a choice. A caller that needs statistics on a cold graph must materialize the columns first, and `Graph::materialize_property_columns` is how: no small read @@ -452,10 +388,8 @@ The read-path and statistics methods carry non-obvious semantics: environment variable (0 restores default behavior, resolved by `threads::resolve`). There is no pool to configure: each pass resolves the budget when it starts and spawns scoped threads for its own duration, so the call stores the value, takes effect on the next pass, and cannot fail. -Graph algorithms have self-describing signatures over `NodeId` and `EdgeId`: `bfs`, `bfs_multi_source`, `expand_bulk`, `dfs`, `shortest_path`, `all_paths`, `all_shortest_paths`, -`longest_path`, `shortest_path_top_k`, `page_rank`, `connected_components`, `strongly_connected_components`, `detect_cycle`, `label_propagation`, -`degree_centrality`, `betweenness_centrality`, `harmonic_centrality`, `spanning_forest`, `maximum_flow`, and `all_neighbors`. Several carry behavior -worth pinning: +The graph algorithms are the public methods of `graph/algo.rs`, with signatures over `NodeId` and `EdgeId` that read for themselves. Several carry +behavior a signature cannot show, and those are pinned here: - `shortest_path_dijkstra(src, dst) -> Result, Error>`: edge weight is the first present of the `weight`, `cost`, `capacity`, or `cap` property, default `1.0`; the source is fixed, so unlike `shortest_path_top_k` and `spanning_forest` this method takes no weight-property @@ -476,6 +410,24 @@ worth pinning: redistributed, so ranks do not sum to 1; `tests/oracle.rs` compares against NetworkX over a corpus restricted to graphs with no dangling nodes for exactly that reason. The accumulation reads the incoming rows, so each output entry is a sum over one node's in-edges, which is what makes the pass parallel over disjoint output chunks and independent of the worker count. +- The algorithms ported from Graphina (`closeness_centrality`, `eigenvector_centrality`, `katz_centrality`, `clustering_coefficient`, `louvain`, and + `link_prediction_score`) each had to take a position on parallel edges, because Graphina's graph type is simple and collapses them, so its + implementations carry no answer to inherit. There are three rules, and which one applies follows from what the score means rather than from taste. + `eigenvector_centrality` and `katz_centrality` count every edge, like `page_rank`, since each edge is a distinct path for influence to flow along. + `clustering_coefficient` and `link_prediction_score` count *distinct* neighbors, like `degree_centrality`: both are ratios over neighbor sets, and the + coefficient in particular is bounded by 1, so counting a parallel edge twice would push it above its own maximum. `louvain` weights an edge by its + multiplicity, because modularity is defined over edge weight and a pair joined five times genuinely is more strongly tied. +- `closeness_centrality() -> Result, Error>`: Wasserman-Faust closeness, `(reachable / total_distance) * (reachable / (n - 1))` over + hop distances, sharing the per-source breadth-first pass of `harmonic_centrality`. The second factor is what keeps the score usable on a disconnected + graph, where a plain reciprocal mean distance would rank a node in a two-node component above a well-connected node in a large one. +- `eigenvector_centrality(iterations, tolerance)` and `katz_centrality(alpha, beta, iterations, tolerance)`: power iterations that are bounded rather than + fallible. Each stops early on convergence and otherwise returns the estimate after the iteration budget, following `page_rank` rather than erroring on a + slowly converging graph. Katz needs `alpha` below the reciprocal of the largest eigenvalue, which is a property of the data; above it the series diverges + and the bounded loop returns a large, meaningless, finite answer. +- `louvain() -> Result, Error>`: modularity optimization with level coarsening, naming each community after the smallest node id it + contains, as `connected_components` does. It is the one analytics pass that is deliberately serial: local moving is order-dependent by construction, so + splitting it over workers would make the partition depend on the worker count. It is a strict upgrade on `label_propagation` for quality and separates + communities joined by a few edges, which label propagation merges. - `count_triangle_cycles(spec: &TriangleCountSpec) -> Result`: assignment count of the directed triangle pattern `(a)-[t1]->(b)-[t2]->(c)-[t3]->(a)` with optional per-hop relationship types and per-variable labels, following Cypher MATCH row semantics including relationship uniqueness; the Cypher optimizer lowers grouping-free `count` aggregates over that pattern to this kernel via the `TriangleCount` @@ -508,7 +460,10 @@ Vector search crate. Owns vector index abstractions, vector metadata, vector sto - `VectorGraphExt::reindex_vector_index(opts) -> Result<(), VectorError>`: changes the metric or quantization on a populated graph and rebuilds the index from the persisted embeddings. The stored vectors are raw, metric-agnostic f32, so they re-index under any metric; this is O(n) and is an administrative operation, not a concurrent one. -- `VectorGraphExt::upsert_vector(n, v) -> Result<(), VectorError>` +- `VectorGraphExt::upsert_vector(n, v) -> Result<(), VectorError>`: rejects a node that does not exist with `VectorError::NodeNotFound`. Node ids are + handed out monotonically, so a vector accepted ahead of its node is not inert: the next node allocated that id inherits it and answers a search at + distance zero having never been embedded, which nothing downstream can detect. `remove_vector` stays permissive, so a database written before the + check can still be cleaned up. - Searching a graph with no stored embeddings returns `VectorError::EmptyIndex` rather than an empty hit list, so a caller can distinguish "no semantic matches" from "there is nothing to search". The Cypher `VectorTopK` operator maps that error to zero rows, keeping MATCH semantics. - `VectorGraphExt::remove_vector(n) -> Result<(), VectorError>`: removes the embedding from both memory and storage. diff --git a/Cargo.lock b/Cargo.lock index 54ddaa3..2031954 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1409,7 +1409,7 @@ checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" [[package]] name = "issundb" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "criterion", "gherkin", @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "issundb-cli" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "arrow-array", "arrow-schema", @@ -1444,7 +1444,7 @@ dependencies = [ [[package]] name = "issundb-core" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "arc-swap", @@ -1468,7 +1468,7 @@ dependencies = [ [[package]] name = "issundb-cypher" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "arrow-array", @@ -1494,7 +1494,7 @@ dependencies = [ [[package]] name = "issundb-examples" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb", "rmp-serde", @@ -1504,7 +1504,7 @@ dependencies = [ [[package]] name = "issundb-mcp" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "anyhow", "axum", @@ -1522,7 +1522,7 @@ dependencies = [ [[package]] name = "issundb-py" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb", "pyo3", @@ -1532,7 +1532,7 @@ dependencies = [ [[package]] name = "issundb-rest" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "anyhow", "axum", @@ -1553,7 +1553,7 @@ dependencies = [ [[package]] name = "issundb-retrieval" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "arrow-array", @@ -1569,7 +1569,7 @@ dependencies = [ [[package]] name = "issundb-text" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "arrow-array", "criterion", @@ -1584,7 +1584,7 @@ dependencies = [ [[package]] name = "issundb-vector" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "arrow-array", "criterion", @@ -1600,7 +1600,7 @@ dependencies = [ [[package]] name = "issundb-wasm" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb", "rmp-serde", diff --git a/Cargo.toml b/Cargo.toml index f630c01..d5d1c97 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -8,7 +8,7 @@ exclude = ["benchmarks/ladybugdb-compare"] [workspace.package] -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" edition = "2024" authors = ["Hassan Abedi "] license = "MIT OR Apache-2.0" @@ -18,12 +18,12 @@ rust-version = "1.85.0" [workspace.dependencies] -issundb = { path = "crates/issundb", version = "0.1.0-alpha.20", default-features = false } -issundb-core = { path = "crates/issundb-core", version = "0.1.0-alpha.20", default-features = false } -issundb-vector = { path = "crates/issundb-vector", version = "0.1.0-alpha.20", default-features = false } -issundb-text = { path = "crates/issundb-text", version = "0.1.0-alpha.20", default-features = false } -issundb-retrieval = { path = "crates/issundb-retrieval", version = "0.1.0-alpha.20", default-features = false } -issundb-cypher = { path = "crates/issundb-cypher", version = "0.1.0-alpha.20", default-features = false } +issundb = { path = "crates/issundb", version = "0.1.0-alpha.21", default-features = false } +issundb-core = { path = "crates/issundb-core", version = "0.1.0-alpha.21", default-features = false } +issundb-vector = { path = "crates/issundb-vector", version = "0.1.0-alpha.21", default-features = false } +issundb-text = { path = "crates/issundb-text", version = "0.1.0-alpha.21", default-features = false } +issundb-retrieval = { path = "crates/issundb-retrieval", version = "0.1.0-alpha.21", default-features = false } +issundb-cypher = { path = "crates/issundb-cypher", version = "0.1.0-alpha.21", default-features = false } heed = "0.22" colored = "3.1.1" byteorder = "1" diff --git a/Makefile b/Makefile index 757c29a..3e50b9c 100644 --- a/Makefile +++ b/Makefile @@ -281,6 +281,7 @@ playground-build: check-wasm-bindgen check-wasm-stack ## Build the browser modul @echo "Building issundb-wasm for wasm32-unknown-unknown (in-memory storage, exact vector index)..." @$(WASM_BUILD) @echo "Generating the JavaScript glue into $(PLAYGROUND_DIR)/pkg..." + @rm -rf $(PLAYGROUND_DIR)/pkg @wasm-bindgen $(WASM_ARTIFACT) --out-dir $(PLAYGROUND_DIR)/pkg --target web --no-typescript @cp docs/assets/logo.svg $(PLAYGROUND_DIR)/logo.svg @ls -l $(PLAYGROUND_DIR)/pkg @@ -289,6 +290,7 @@ playground-build: check-wasm-bindgen check-wasm-stack ## Build the browser modul playground-check: check-wasm-bindgen check-wasm-stack ## Run every playground demo through the compiled module @echo "Building the module for Node..." @$(WASM_BUILD) + @rm -rf $(PLAYGROUND_NODE_PKG) @wasm-bindgen $(WASM_ARTIFACT) --out-dir $(PLAYGROUND_NODE_PKG) --target nodejs --no-typescript @echo "Running the demo catalog..." @node $(SCRIPTS_DIR)/check_playground.mjs @@ -298,7 +300,9 @@ playground-serve: ## Serve the playground at http://localhost:$(PLAYGROUND_PORT) @test -f $(PLAYGROUND_DIR)/pkg/issundb_wasm.js || \ { echo "No module in $(PLAYGROUND_DIR)/pkg. Run 'make playground-build' first."; exit 1; } @echo "Serving $(PLAYGROUND_DIR) at http://localhost:$(PLAYGROUND_PORT) (Ctrl-C to stop)..." - @python3 -m http.server $(PLAYGROUND_PORT) --directory $(PLAYGROUND_DIR) + @python3 -c 'import functools, http.server as h; \ + C = type("C", (h.SimpleHTTPRequestHandler,), {"end_headers": lambda s: (s.send_header("Cache-Control", "no-store"), h.SimpleHTTPRequestHandler.end_headers(s))}); \ + h.test(HandlerClass=functools.partial(C, directory="$(PLAYGROUND_DIR)"), port=$(PLAYGROUND_PORT))' # An exported RUSTFLAGS replaces the `[target.wasm32-unknown-unknown] rustflags` in # .cargo/config.toml rather than merging with it, which drops the 16 MB stack the inline diff --git a/README.md b/README.md index 9e2da03..33605ad 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ To use IssunDB in your Rust project, add the dependency to your `Cargo.toml`: ```toml [dependencies] -issundb = "0.1.0-alpha.20" +issundb = "0.1.0-alpha.21" serde_json = "1.0" ``` diff --git a/benchmarks/ladybugdb-compare/Cargo.lock b/benchmarks/ladybugdb-compare/Cargo.lock index ea05ebf..e8956a9 100644 --- a/benchmarks/ladybugdb-compare/Cargo.lock +++ b/benchmarks/ladybugdb-compare/Cargo.lock @@ -847,7 +847,7 @@ dependencies = [ [[package]] name = "issundb" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb-core", "issundb-cypher", @@ -859,7 +859,7 @@ dependencies = [ [[package]] name = "issundb-core" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "arc-swap", @@ -879,7 +879,7 @@ dependencies = [ [[package]] name = "issundb-cypher" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "arrow-array", @@ -902,7 +902,7 @@ dependencies = [ [[package]] name = "issundb-retrieval" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "ahash", "issundb-core", @@ -913,7 +913,7 @@ dependencies = [ [[package]] name = "issundb-text" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb-core", "roaring", @@ -922,7 +922,7 @@ dependencies = [ [[package]] name = "issundb-vector" -version = "0.1.0-alpha.20" +version = "0.1.0-alpha.21" dependencies = [ "issundb-core", "parking_lot", diff --git a/crates/issundb-cli/Cargo.toml b/crates/issundb-cli/Cargo.toml index 91e52c2..418300a 100644 --- a/crates/issundb-cli/Cargo.toml +++ b/crates/issundb-cli/Cargo.toml @@ -15,8 +15,13 @@ name = "issundb-cli" path = "src/main.rs" doc = false +[features] +default = ["lmdb", "hnsw"] +lmdb = ["issundb/lmdb"] +hnsw = ["issundb/hnsw"] + [dependencies] -issundb = { workspace = true, features = ["lmdb", "hnsw"] } +issundb = { workspace = true } colored.workspace = true rmp-serde.workspace = true serde_json = "1" diff --git a/crates/issundb-core/AGENTS.md b/crates/issundb-core/AGENTS.md index db6ff94..4d3ec17 100644 --- a/crates/issundb-core/AGENTS.md +++ b/crates/issundb-core/AGENTS.md @@ -147,6 +147,10 @@ the freshness path. - A kernel that needs a per-edge property the snapshot does not carry reads it from storage per call. That is deliberate for the weight-*property* algorithms (`spanning_forest`, `shortest_path_top_k`, `maximum_flow`), which take the property name as an argument: there is no fixed key to preload. +- The snapshot is published through `arc-swap` and paired with two counters, `write_gen` and `snapshot_gen`, whose comparison is the freshness + condition every consumer tests. The write path also fills a `GraphDelta` buffer, whose only consumers are the property column caches; the CSR itself + is rebuilt rather than patched. + ## In-memory Property Columns `columns.rs` holds a typed, in-memory columnar view of scalar properties used as the hot read path for property gathers and aggregations. @@ -155,6 +159,9 @@ It is derived from LMDB, like the CSR snapshot, and follows the same write-LMDB- - `PropColumns` stores one typed column per property (Int, Float, Bool, dict-encoded Str, or a JSON fallback) over a dense `id -> index` map. `NodeSource` and `EdgeSource` implement `ColumnSource`, so nodes and edges share one generic store; `Graph` holds `prop_columns: ColumnsCache` and `edge_columns: ColumnsCache`. +- A column is `Int`, `Float`, `Bool`, a dictionary-encoded `Str`, or the exact-semantics `Json` fallback, and single-property reads come through + `Graph::node_prop_json`. The per-property statistics (`PropStats`: bounds, an equi-depth histogram, and the most common values) are computed lazily + beside the columns and invalidated by the same post-commit patch. - `ColumnsCache` builds lazily from one full `scan_all`, but a read does not necessarily cause that build, and the distinction is deliberate. A request for at most `SMALL_GATHER_MAX` entities is served as point reads straight from storage while the columns are absent (`should_serve_directly`), because building every column is one full scan and that is the wrong answer to a request for a handful of entities. Those diff --git a/crates/issundb-core/src/graph/algo.rs b/crates/issundb-core/src/graph/algo.rs index 6620596..0d7d190 100644 --- a/crates/issundb-core/src/graph/algo.rs +++ b/crates/issundb-core/src/graph/algo.rs @@ -1048,6 +1048,68 @@ impl Graph { self.with_snapshot(|snap| self.betweenness_centrality_kernel(snap)) } + /// Computes the closeness centrality for all nodes, in the Wasserman-Faust form + /// that stays meaningful on a disconnected graph. See + /// [`Graph::closeness_centrality_kernel`]. + pub fn closeness_centrality(&self) -> Result, Error> { + self.with_snapshot(|snap| self.closeness_centrality_kernel(snap)) + } + + /// Computes the eigenvector centrality for all nodes by power iteration. + /// + /// Bounded rather than fallible: it stops early once the L2 change falls below + /// `tolerance` and otherwise returns the estimate after `iterations` rounds. See + /// [`Graph::eigenvector_centrality_kernel`]. + pub fn eigenvector_centrality( + &self, + iterations: u32, + tolerance: f64, + ) -> Result, Error> { + self.with_snapshot(|snap| self.eigenvector_centrality_kernel(snap, iterations, tolerance)) + } + + /// Computes the Katz centrality for all nodes. + /// + /// `alpha` must be below the reciprocal of the largest eigenvalue or the series + /// diverges; see [`Graph::katz_centrality_kernel`] for what happens if it is not. + pub fn katz_centrality( + &self, + alpha: f64, + beta: f64, + iterations: u32, + tolerance: f64, + ) -> Result, Error> { + self.with_snapshot(|snap| { + self.katz_centrality_kernel(snap, alpha, beta, iterations, tolerance) + }) + } + + /// Computes the local clustering coefficient for all nodes, reading the graph as + /// undirected over distinct neighbors. See + /// [`Graph::clustering_coefficient_kernel`]. + pub fn clustering_coefficient(&self) -> Result, Error> { + self.with_snapshot(|snap| self.clustering_coefficient_kernel(snap)) + } + + /// Detects communities by the Louvain method, returning the community of every + /// node. The community id is the smallest node id it contains, and only the + /// induced partition is contractual. See [`Graph::louvain_kernel`]. + pub fn louvain(&self) -> Result, Error> { + self.with_snapshot(|snap| self.louvain_kernel(snap)) + } + + /// Scores how likely `a` and `b` are to become connected, under one of the + /// neighborhood heuristics. A node the snapshot does not know scores zero rather + /// than erroring. See [`Graph::link_prediction_kernel`]. + pub fn link_prediction_score( + &self, + a: NodeId, + b: NodeId, + metric: LinkPredictionMetric, + ) -> Result { + self.with_snapshot(|snap| Ok(self.link_prediction_kernel(snap, a, b, metric))) + } + /// Computes the strongly connected components (SCC) of the graph using Tarjan's algorithm. pub fn strongly_connected_components(&self) -> Result, Error> { self.with_snapshot(|snap| self.strongly_connected_components_kernel(snap)) diff --git a/crates/issundb-core/src/graph/kernels/analytics.rs b/crates/issundb-core/src/graph/kernels/analytics.rs index 156ea66..164478b 100644 --- a/crates/issundb-core/src/graph/kernels/analytics.rs +++ b/crates/issundb-core/src/graph/kernels/analytics.rs @@ -2,6 +2,317 @@ use super::*; use super::traversal::{UNREACHED, expand_frontier}; +/// Below this L2 norm the eigenvector iteration has collapsed and the next rescale +/// would divide by roughly zero, so the kernel reports the uniform distribution. +const EIGENVECTOR_MIN_NORM: f64 = 1e-10; + +/// Hard cap on Louvain coarsening levels. +/// +/// Each level strictly reduces the node count or the pass stops, so a graph cannot +/// need more than `log2(n)` of them and this is unreachable in practice. It exists so +/// that a defect in the merge condition cannot turn into a non-terminating query. +const LOUVAIN_MAX_LEVELS: usize = 32; + +/// One level of the Louvain hierarchy: an undirected weighted graph in CSR form. +/// +/// Every edge appears in both endpoints' rows, so a scan over all rows visits each +/// edge twice. Self-loops are held apart from the rows so the neighbor scan never has +/// to test for them, and because modularity counts a self-loop twice in a node's +/// degree but never as a link to another community. +struct LouvainLevel { + row_ptr: Vec, + col: Vec, + weight: Vec, + self_loop: Vec, + /// Sum of incident edge weights per node, with a self-loop counted twice. + degree: Vec, + /// `2m`, the sum of every node's degree. Constant across levels, since + /// coarsening moves weight around without creating or destroying it. + total: f64, +} + +impl LouvainLevel { + fn len(&self) -> usize { + self.degree.len() + } + + fn neighbors(&self, u: usize) -> impl Iterator + '_ { + (self.row_ptr[u]..self.row_ptr[u + 1]).map(|k| (self.col[k], self.weight[k])) + } + + /// Project the directed multigraph in `snap` onto an undirected weighted graph. + /// + /// The weight between two nodes is the number of edges joining them in either + /// direction, so parallel edges strengthen a connection rather than collapsing. + /// That is the reading modularity wants, and unlike the clustering coefficient + /// there is no bound for it to violate: a pair joined by five edges genuinely is + /// more strongly tied than a pair joined by one. + /// + /// A node's out-row and in-row together list every edge incident to it exactly + /// once, except a self-loop, which appears in both. Self-loops are therefore + /// counted from the out-row alone. + fn from_snapshot(snap: &CsrSnapshot) -> Self { + let n = snap.dense_to_id.len(); + let mut row_ptr = Vec::with_capacity(n + 1); + let mut col: Vec = Vec::new(); + let mut weight: Vec = Vec::new(); + let mut self_loop = vec![0.0f64; n]; + let mut degree = vec![0.0f64; n]; + + // `stamp[v] == u + 1` marks v as already collected for u, and `acc[v]` holds + // the running multiplicity, so each row is built in one pass without a map. + let mut stamp = vec![0usize; n]; + let mut acc = vec![0.0f64; n]; + let mut seen: Vec = Vec::new(); + + row_ptr.push(0); + for u in 0..n { + seen.clear(); + for k in snap.row_ptr[u]..snap.row_ptr[u + 1] { + let v = snap.col_idx[k]; + if v as usize == u { + self_loop[u] += 1.0; + continue; + } + if stamp[v as usize] != u + 1 { + stamp[v as usize] = u + 1; + acc[v as usize] = 0.0; + seen.push(v); + } + acc[v as usize] += 1.0; + } + for k in snap.in_row_ptr[u]..snap.in_row_ptr[u + 1] { + let v = snap.in_col_idx[k]; + if v as usize == u { + // Already counted from the out-row. + continue; + } + if stamp[v as usize] != u + 1 { + stamp[v as usize] = u + 1; + acc[v as usize] = 0.0; + seen.push(v); + } + acc[v as usize] += 1.0; + } + + let mut incident = 0.0f64; + // Sorted so a row's order depends on the graph and not on scan order, + // which keeps the floating-point accumulation below reproducible. + seen.sort_unstable(); + for &v in &seen { + col.push(v); + weight.push(acc[v as usize]); + incident += acc[v as usize]; + } + degree[u] = incident + 2.0 * self_loop[u]; + row_ptr.push(col.len()); + } + + let total = degree.iter().sum(); + Self { + row_ptr, + col, + weight, + self_loop, + degree, + total, + } + } + + /// Louvain's first phase: repeatedly move each node into the neighboring + /// community that most increases modularity, until a full sweep moves nobody. + /// + /// Returns the community index of every node, not yet renumbered. + /// + /// The move gain for node `i` joining community `c` is + /// `w(i, c) - sigma_tot(c) * k_i / 2m`, dropping the terms that are equal for + /// every candidate. `i` is removed from its own community first, so staying put + /// is evaluated on the same footing as leaving. + /// + /// This phase is deliberately serial. The outcome depends on the order nodes are + /// visited, so parallelizing it would make the result depend on the worker count, + /// which every other kernel here avoids. Ascending dense order is ascending node + /// id, so the partition is reproducible run to run. + fn local_moving(&self) -> Vec { + let n = self.len(); + let mut community: Vec = (0..n as u32).collect(); + let mut sigma_tot = self.degree.clone(); + if self.total <= 0.0 { + return community; + } + + // `stamp`/`link` accumulate the weight from the current node into each + // neighboring community without clearing an n-sized buffer per node. + let mut stamp = vec![usize::MAX; n]; + let mut link = vec![0.0f64; n]; + let mut candidates: Vec = Vec::new(); + + for _ in 0..LOUVAIN_MAX_LEVELS { + let mut moved = false; + for u in 0..n { + let origin = community[u]; + let k_u = self.degree[u]; + + candidates.clear(); + for (v, w) in self.neighbors(u) { + let c = community[v as usize] as usize; + if stamp[c] != u { + stamp[c] = u; + link[c] = 0.0; + candidates.push(c as u32); + } + link[c] += w; + } + + // Detach `u` before scoring, so its own degree does not appear in the + // penalty term of the community it is sitting in. + sigma_tot[origin as usize] -= k_u; + + let gain_of = |c: u32| -> f64 { + let to_c = if stamp[c as usize] == u { + link[c as usize] + } else { + 0.0 + }; + to_c - sigma_tot[c as usize] * k_u / self.total + }; + + let mut best = origin; + let mut best_gain = gain_of(origin); + for &c in &candidates { + let gain = gain_of(c); + // The tie-break on the smaller index is what makes the result + // independent of the order candidates were discovered in, which + // follows CSR row order rather than anything meaningful. + if gain > best_gain || (gain == best_gain && c < best) { + best = c; + best_gain = gain; + } + } + + sigma_tot[best as usize] += k_u; + community[u] = best; + if best != origin { + moved = true; + } + } + if !moved { + break; + } + } + + community + } + + /// Louvain's second phase: contract each community into a single node. + /// + /// Communities are renumbered by the ascending dense index of their smallest + /// member, so the coarse graph's node order is a deterministic function of the + /// fine one. Returns the coarse level and the renumbered assignment. + fn coarsen(&self, community: &[u32]) -> (LouvainLevel, Vec) { + let n = self.len(); + let mut renumbered = vec![u32::MAX; n]; + let mut next_id = 0u32; + let mut assignment = vec![0u32; n]; + for u in 0..n { + let c = community[u] as usize; + if renumbered[c] == u32::MAX { + renumbered[c] = next_id; + next_id += 1; + } + assignment[u] = renumbered[c]; + } + + let groups = next_id as usize; + let mut self_loop = vec![0.0f64; groups]; + let mut degree = vec![0.0f64; groups]; + // Degree is carried over rather than recomputed. A community's incident weight + // is exactly the sum of its members' degrees, and summing them avoids any + // chance of the coarse graph disagreeing with the fine one about `2m`. + for (u, &group) in assignment.iter().enumerate() { + let c = group as usize; + degree[c] += self.degree[u]; + self_loop[c] += self.self_loop[u]; + } + + // Group members by community up front. Rescanning every node once per + // community instead would make coarsening quadratic, which on a graph that + // resolves into many small communities is the whole cost of the algorithm. + let mut member_ptr = vec![0usize; groups + 1]; + for &c in &assignment { + member_ptr[c as usize + 1] += 1; + } + for c in 0..groups { + member_ptr[c + 1] += member_ptr[c]; + } + let mut members = vec![0u32; n]; + let mut cursor = member_ptr.clone(); + for (u, &c) in assignment.iter().enumerate() { + members[cursor[c as usize]] = u as u32; + cursor[c as usize] += 1; + } + + let mut stamp = vec![usize::MAX; groups]; + let mut acc = vec![0.0f64; groups]; + let mut seen: Vec = Vec::new(); + let mut row_ptr = Vec::with_capacity(groups + 1); + let mut col: Vec = Vec::new(); + let mut weight: Vec = Vec::new(); + + row_ptr.push(0); + for c in 0..groups { + seen.clear(); + for &u in &members[member_ptr[c]..member_ptr[c + 1]] { + for (v, w) in self.neighbors(u as usize) { + let d = assignment[v as usize] as usize; + if d == c { + // Each intra-community edge is walked from both endpoints, so + // halving turns the doubled total into the self-loop weight. + self_loop[c] += w / 2.0; + continue; + } + if stamp[d] != c { + stamp[d] = c; + acc[d] = 0.0; + seen.push(d as u32); + } + acc[d] += w; + } + } + seen.sort_unstable(); + for &d in &seen { + col.push(d); + weight.push(acc[d as usize]); + } + row_ptr.push(col.len()); + } + + let total = self.total; + ( + LouvainLevel { + row_ptr, + col, + weight, + self_loop, + degree, + total, + }, + assignment, + ) + } +} + +/// Every neighbor of `u` in both directions, parallel edges and self-loops included. +/// +/// The caller dedups, because the two rows overlap whenever a pair is joined in both +/// directions and either row alone can hold a pair twice. +fn undirected_neighbors(snap: &CsrSnapshot, u: usize) -> impl Iterator + '_ { + snap.col_idx[snap.row_ptr[u]..snap.row_ptr[u + 1]] + .iter() + .chain(snap.in_col_idx[snap.in_row_ptr[u]..snap.in_row_ptr[u + 1]].iter()) + .copied() +} + /// Out-degree of every dense index, counting parallel edges separately. /// /// This is the row length, so an edge added twice between the same pair counts @@ -586,6 +897,491 @@ impl Graph { Ok(labels) } + + /// Closeness centrality in the Wasserman-Faust form: for each node, + /// `(reachable / total_distance) * (reachable / (n - 1))`, where `reachable` is + /// the number of other nodes it can reach and `total_distance` is the sum of the + /// hop distances to them. + /// + /// Distance is hop count over outgoing edges rather than a weighted path length, + /// the same convention [`Graph::harmonic_centrality_kernel`] uses, whose + /// per-source breadth-first pass this shares. The two differ only in what they + /// accumulate: harmonic sums `1 / hop` and so needs no reachability correction, + /// while closeness sums the distances and does. + /// + /// The Wasserman-Faust factor is what makes the score usable on a disconnected + /// graph. Plain `reachable / total_distance` is a reciprocal mean distance, so a + /// node in a two-node component scores the maximum while a well-connected node in + /// a large component scores less; scaling by the fraction of the graph it reaches + /// removes that inversion. On a connected graph the factor is 1 and the score + /// reduces to `(n - 1) / total_distance`. A node that reaches nothing scores zero. + /// + /// Parallel edges cannot change the result, since a second edge between the same + /// pair does not shorten a hop distance. + pub(in crate::graph) fn closeness_centrality_kernel( + &self, + snap: &CsrSnapshot, + ) -> Result, Error> { + let n = snap.dense_to_id.len(); + if n == 0 { + return Ok(HashMap::new()); + } + + let per_source = n.saturating_add(snap.col_idx.len()); + let threads = self.parallel_threads(n.saturating_mul(per_source)); + let centrality = map_dense_range(n, threads, |lo, slice| { + let mut levels = vec![UNREACHED; n]; + let mut next = Vec::new(); + for (offset, value) in slice.iter_mut().enumerate() { + let src = lo + offset; + levels.fill(UNREACHED); + levels[src] = 0; + let mut frontier = vec![src as u32]; + let mut total_distance = 0.0f64; + let mut reachable = 0usize; + for hop in 1.. { + expand_frontier(snap, &frontier, hop, &mut levels, &mut next); + if next.is_empty() { + break; + } + total_distance += next.len() as f64 * f64::from(hop); + reachable += next.len(); + std::mem::swap(&mut frontier, &mut next); + } + *value = if reachable > 0 && n > 1 { + (reachable as f64 / total_distance) * (reachable as f64 / (n as f64 - 1.0)) + } else { + 0.0 + }; + } + }); + + Ok(snap + .dense_to_id + .iter() + .enumerate() + .map(|(d, &id)| (id, centrality[d])) + .collect()) + } + + /// Eigenvector centrality by power iteration over the CSR snapshot. + /// + /// Each iteration computes `x[j] = sum over edges i -> j of x[i]` and rescales to + /// unit L2 norm, so a node is important when the nodes pointing at it are. Like + /// [`Graph::page_rank`] the accumulation reads the *incoming* rows, which is what + /// keeps the pass parallel over disjoint output chunks and independent of the + /// worker count. + /// + /// Parallel edges each contribute, matching PageRank rather than the + /// distinct-neighbor rule of [`Graph::degree_centrality`]. Two edges from `i` to + /// `j` pass `i`'s score to `j` twice, which is the multigraph reading of "how much + /// influence flows along this connection". + /// + /// Iteration is bounded and does not fail. It stops early once the L2 change falls + /// below `tolerance`, and otherwise returns the estimate after `iterations` + /// rounds. That follows `page_rank`, which also runs a fixed budget, and it is the + /// right choice for a database procedure: refusing to answer because a graph + /// converges slowly is worse than answering approximately and saying so. A + /// degenerate operator, meaning no edges at all or a vector that collapses to + /// zero, yields the uniform `1 / n` distribution rather than a division by zero. + /// + /// Scores are reported as magnitudes scaled to sum to `n`. An eigenvector's sign + /// and length are arbitrary, so only the ratios between nodes carry meaning. + pub(in crate::graph) fn eigenvector_centrality_kernel( + &self, + snap: &CsrSnapshot, + iterations: u32, + tolerance: f64, + ) -> Result, Error> { + let n = snap.dense_to_id.len(); + if n == 0 { + return Ok(HashMap::new()); + } + + let uniform = |value: f64| -> HashMap { + snap.dense_to_id.iter().map(|&id| (id, value)).collect() + }; + if snap.col_idx.is_empty() { + return Ok(uniform(1.0 / n as f64)); + } + + let threads = self.kernel_threads(n.saturating_add(snap.col_idx.len())); + let mut x = vec![1.0 / (n as f64).sqrt(); n]; + let mut next = vec![0.0f64; n]; + + for _ in 0..iterations { + { + let previous = &x; + fill_dense_range(&mut next, threads, move |lo, slice| { + for (offset, value) in slice.iter_mut().enumerate() { + let j = lo + offset; + let mut sum = 0.0f64; + for k in snap.in_row_ptr[j]..snap.in_row_ptr[j + 1] { + sum += previous[snap.in_col_idx[k] as usize]; + } + *value = sum; + } + }); + } + + let norm = next.iter().map(|v| v * v).sum::().sqrt(); + if norm < EIGENVECTOR_MIN_NORM { + return Ok(uniform(1.0 / n as f64)); + } + let mut delta = 0.0f64; + for (current, raw) in x.iter_mut().zip(next.iter()) { + let normalized = raw / norm; + let step = normalized - *current; + delta += step * step; + *current = normalized; + } + if delta.sqrt() < tolerance { + break; + } + } + + // The orientation of an eigenvector is arbitrary, so report magnitudes, and + // rescale to sum to `n` so the numbers do not shrink as the graph grows. + let total: f64 = x.iter().map(|v| v.abs()).sum(); + if total > 0.0 { + for value in x.iter_mut() { + *value = value.abs() * n as f64 / total; + } + } + + Ok(snap + .dense_to_id + .iter() + .enumerate() + .map(|(d, &id)| (id, x[d])) + .collect()) + } + + /// Katz centrality by the fixed-point iteration `x = alpha * A^T x + beta`. + /// + /// A node's score is a sum over all walks that reach it, with a walk of length `k` + /// attenuated by `alpha^k`, plus the constant `beta` every node receives for + /// existing. Compared with eigenvector centrality this gives a node with no + /// incoming edges a non-zero score, which is why it behaves better on the directed + /// acyclic shapes where eigenvector centrality collapses. + /// + /// `alpha` must be below the reciprocal of the largest eigenvalue for the series + /// to converge, and that bound is a property of the data rather than something + /// this method can check cheaply. A value above it diverges; the bounded iteration + /// then returns a large but finite estimate instead of looping, and the scores are + /// meaningless. A safe default is well under `1 / max_degree`. + /// + /// Parallel edges each contribute, matching [`Graph::page_rank`] and eigenvector + /// centrality, since every edge is a distinct walk. Iteration is bounded and does + /// not fail, on the same reasoning as eigenvector centrality. + pub(in crate::graph) fn katz_centrality_kernel( + &self, + snap: &CsrSnapshot, + alpha: f64, + beta: f64, + iterations: u32, + tolerance: f64, + ) -> Result, Error> { + let n = snap.dense_to_id.len(); + if n == 0 { + return Ok(HashMap::new()); + } + + let threads = self.kernel_threads(n.saturating_add(snap.col_idx.len())); + let mut x = vec![beta; n]; + let mut next = vec![0.0f64; n]; + + for _ in 0..iterations { + { + let previous = &x; + fill_dense_range(&mut next, threads, move |lo, slice| { + for (offset, value) in slice.iter_mut().enumerate() { + let j = lo + offset; + let mut sum = 0.0f64; + for k in snap.in_row_ptr[j]..snap.in_row_ptr[j + 1] { + sum += previous[snap.in_col_idx[k] as usize]; + } + *value = alpha * sum + beta; + } + }); + } + + let mut delta = 0.0f64; + for (current, raw) in x.iter_mut().zip(next.iter()) { + let step = raw - *current; + delta += step * step; + *current = *raw; + } + if delta.sqrt() < tolerance { + break; + } + } + + Ok(snap + .dense_to_id + .iter() + .enumerate() + .map(|(d, &id)| (id, x[d])) + .collect()) + } + + /// Score how likely two nodes are to become connected, by one of the classic + /// neighborhood heuristics. + /// + /// The neighborhood is undirected and distinct, matching + /// [`Graph::clustering_coefficient`]: a pair joined by three edges is one + /// neighbor, direction is ignored, and a node is never its own neighbor. Every + /// metric here is a statement about *who* two nodes both know, so multiplicity + /// would double-count one relationship as evidence of several. + /// + /// A node absent from the snapshot scores zero rather than erroring, which is the + /// same choice [`Graph::typed_neighbor_counts`] makes: these are per-row scoring + /// functions, and failing a whole query because one row names a node that has + /// since been deleted is worse than scoring it zero. + /// + /// Cost is the two nodes' degrees, plus the degrees of their shared neighbors for + /// the two weighted metrics. + pub(in crate::graph) fn link_prediction_kernel( + &self, + snap: &CsrSnapshot, + a: NodeId, + b: NodeId, + metric: LinkPredictionMetric, + ) -> f64 { + let (Some(&da), Some(&db)) = (snap.id_to_dense.get(&a), snap.id_to_dense.get(&b)) else { + return 0.0; + }; + + let neighborhood = |u: usize| -> Vec { + let mut set: Vec = undirected_neighbors(snap, u) + .filter(|&v| v as usize != u) + .collect(); + set.sort_unstable(); + set.dedup(); + set + }; + let na = neighborhood(da as usize); + let nb = neighborhood(db as usize); + + if metric == LinkPredictionMetric::PreferentialAttachment { + return na.len() as f64 * nb.len() as f64; + } + + // Both sides are sorted, so the intersection is a merge rather than a hash + // probe per element. + let mut shared: Vec = Vec::new(); + let (mut i, mut j) = (0usize, 0usize); + while i < na.len() && j < nb.len() { + match na[i].cmp(&nb[j]) { + std::cmp::Ordering::Less => i += 1, + std::cmp::Ordering::Greater => j += 1, + std::cmp::Ordering::Equal => { + shared.push(na[i]); + i += 1; + j += 1; + } + } + } + + let degree_of = |w: u32| -> usize { + let mut set: Vec = undirected_neighbors(snap, w as usize) + .filter(|&v| v != w) + .collect(); + set.sort_unstable(); + set.dedup(); + set.len() + }; + + match metric { + LinkPredictionMetric::CommonNeighbors => shared.len() as f64, + LinkPredictionMetric::Jaccard => { + let union = na.len() + nb.len() - shared.len(); + if union == 0 { + 0.0 + } else { + shared.len() as f64 / union as f64 + } + } + LinkPredictionMetric::AdamicAdar => shared + .iter() + .filter_map(|&w| { + let degree = degree_of(w); + // `ln(1)` is zero, so a neighbor of degree one has no defined + // weight; contributing nothing is the conventional reading. + (degree > 1).then(|| 1.0 / (degree as f64).ln()) + }) + .sum(), + LinkPredictionMetric::ResourceAllocation => shared + .iter() + .map(|&w| { + let degree = degree_of(w); + if degree == 0 { + 0.0 + } else { + 1.0 / degree as f64 + } + }) + .sum(), + // Handled above, before the intersection is built. + LinkPredictionMetric::PreferentialAttachment => unreachable!(), + } + } + + /// Community detection by the Louvain method. + /// + /// Two phases alternate until neither changes anything. The first moves each node + /// into whichever neighboring community most increases modularity; the second + /// contracts every community into one node and repeats on the smaller graph, which + /// is what lets the method find communities larger than a single neighborhood. + /// + /// The graph is read as undirected and weighted by edge multiplicity, so a pair + /// joined by three edges is three times as strongly tied as a pair joined by one. + /// That is the opposite of the distinct-neighbor rule + /// [`Graph::clustering_coefficient`] needs, and it is right for the same underlying + /// reason: modularity compares observed against expected edge weight, so weight is + /// the quantity it is defined over, and there is no bound for multiplicity to + /// break. Self-loops contribute to a node's degree, as modularity requires, but + /// never pull it toward another community. + /// + /// The community id is the smallest *node id* in the community, matching + /// [`Graph::connected_components`]. Only the induced partition is contractual, so + /// compare membership rather than depending on the numbering. + /// + /// Unlike the other analytics passes this one is serial. Local moving is + /// order-dependent by construction, so splitting it over workers would make the + /// partition depend on the worker count; visiting nodes in ascending id order + /// instead makes the result reproducible. + pub(in crate::graph) fn louvain_kernel( + &self, + snap: &CsrSnapshot, + ) -> Result, Error> { + let n = snap.dense_to_id.len(); + if n == 0 { + return Ok(HashMap::new()); + } + + let mut level = LouvainLevel::from_snapshot(snap); + // Maps every original node to its community in the current level. + let mut membership: Vec = (0..n as u32).collect(); + + for _ in 0..LOUVAIN_MAX_LEVELS { + let community = level.local_moving(); + let (coarse, assignment) = level.coarsen(&community); + // Nothing merged, so further levels would repeat this one unchanged. + if coarse.len() == level.len() { + break; + } + for slot in membership.iter_mut() { + *slot = assignment[*slot as usize]; + } + level = coarse; + if level.len() == 1 { + break; + } + } + + // Name each community after the smallest node id it contains. Dense indices + // ascend with node id, so the first member encountered is the smallest. + let mut label = vec![None; level.len()]; + for (dense, &group) in membership.iter().enumerate() { + label[group as usize].get_or_insert(snap.dense_to_id[dense]); + } + + Ok(membership + .iter() + .enumerate() + .map(|(dense, &group)| { + let id = snap.dense_to_id[dense]; + (id, label[group as usize].unwrap_or(id)) + }) + .collect()) + } + + /// Local clustering coefficient: for each node, the fraction of its neighbor pairs + /// that are themselves connected. + /// + /// The graph is read as undirected here, so a node's neighborhood is the union of + /// its out- and in-neighbors, and a neighbor pair counts as connected when an edge + /// runs between them in either direction. Directed clustering has several + /// competing definitions and no default worth guessing at; the undirected + /// coefficient is the one every other tool means by the name. + /// + /// Neighbors are *distinct*, following [`Graph::degree_centrality`] rather than + /// PageRank. That is not a preference: the coefficient is a ratio bounded by 1, and + /// counting a parallel edge twice inflates the numerator past its denominator and + /// produces scores above 1. Self-loops are excluded for the same reason. + /// + /// A node with fewer than two distinct neighbors scores zero, since it has no pair + /// that could be connected. + /// + /// Cost is the sum over nodes of the degrees of their neighbors, so a hub makes + /// this expensive in a way the linear passes are not. + pub(in crate::graph) fn clustering_coefficient_kernel( + &self, + snap: &CsrSnapshot, + ) -> Result, Error> { + let n = snap.dense_to_id.len(); + if n == 0 { + return Ok(HashMap::new()); + } + + let per_node = snap.col_idx.len().saturating_mul(2) / n.max(1); + let threads = self.parallel_threads(n.saturating_mul(per_node.max(1))); + let coefficients = map_dense_range(n, threads, |lo, slice| { + // `member[v] == token` marks v as a neighbor of the node being scored, and + // `seen[v] == token` dedups one neighbor's own adjacency. Stamping avoids + // clearing an n-sized buffer per node; both counters only ever increase, + // so a stale stamp can never be mistaken for a current one. + let mut member = vec![0u64; n]; + let mut seen = vec![0u64; n]; + let mut node_token = 0u64; + let mut pair_token = 0u64; + let mut neighbors: Vec = Vec::new(); + + for (offset, value) in slice.iter_mut().enumerate() { + let u = lo + offset; + node_token += 1; + neighbors.clear(); + for v in undirected_neighbors(snap, u) { + if v as usize != u && member[v as usize] != node_token { + member[v as usize] = node_token; + neighbors.push(v); + } + } + + let k = neighbors.len(); + if k < 2 { + *value = 0.0; + continue; + } + + // Each connected pair inside the neighborhood is counted once from + // each of its two endpoints, so this ordered total is exactly twice + // the number of pairs. That cancels the 2 in the usual + // `2 * pairs / (k * (k - 1))`, leaving the plain ratio below. + let mut ordered_links = 0u64; + for &a in &neighbors { + pair_token += 1; + for b in undirected_neighbors(snap, a as usize) { + let bi = b as usize; + if b != a && member[bi] == node_token && seen[bi] != pair_token { + seen[bi] = pair_token; + ordered_links += 1; + } + } + } + + *value = ordered_links as f64 / (k as f64 * (k as f64 - 1.0)); + } + }); + + Ok(snap + .dense_to_id + .iter() + .enumerate() + .map(|(d, &id)| (id, coefficients[d])) + .collect()) + } } #[cfg(test)] @@ -877,4 +1673,251 @@ mod tests { ); } } + + /// On the directed path a -> b -> c the Wasserman-Faust score is computable by + /// hand: a reaches two nodes at total distance 3 for `(2/3) * (2/2)`, b reaches + /// one at distance 1 for `(1/1) * (1/2)`, and c reaches nothing. + /// + /// The middle value is the one that pins the reachability factor. Without it b + /// would score 1.0, beating a, purely because its single reachable node is + /// adjacent. + #[test] + fn closeness_scales_by_the_fraction_of_the_graph_reached() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(nodes[0], nodes[1], "E", &()).unwrap(); + g.add_edge(nodes[1], nodes[2], "E", &()).unwrap(); + + let scores = g.closeness_centrality().unwrap(); + assert!((scores[&nodes[0]] - 2.0 / 3.0).abs() < 1e-12); + assert!((scores[&nodes[1]] - 0.5).abs() < 1e-12); + assert_eq!(scores[&nodes[2]], 0.0); + } + + /// A second edge between the same pair cannot shorten a hop, so it must not move + /// a closeness score. + #[test] + fn closeness_ignores_parallel_edges() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(nodes[0], nodes[1], "E", &()).unwrap(); + g.add_edge(nodes[1], nodes[2], "E", &()).unwrap(); + let before = g.closeness_centrality().unwrap(); + + g.add_edge(nodes[0], nodes[1], "E", &()).unwrap(); + assert_eq!(g.closeness_centrality().unwrap(), before); + } + + /// Every node of a directed cycle is structurally identical, so eigenvector + /// centrality must give them equal scores, and the sum-to-n normalization makes + /// each exactly 1. + #[test] + fn eigenvector_is_uniform_on_a_directed_cycle() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..4).map(|_| g.add_node("N", &()).unwrap()).collect(); + for i in 0..4 { + g.add_edge(nodes[i], nodes[(i + 1) % 4], "E", &()).unwrap(); + } + + let scores = g.eigenvector_centrality(100, 1e-10).unwrap(); + for node in &nodes { + assert!((scores[node] - 1.0).abs() < 1e-9, "{}", scores[node]); + } + } + + /// With no edges the operator is degenerate and the kernel reports the uniform + /// distribution rather than dividing by a zero norm. + #[test] + fn eigenvector_on_an_edgeless_graph_is_uniform() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + + let scores = g.eigenvector_centrality(100, 1e-10).unwrap(); + for node in &nodes { + assert!((scores[node] - 1.0 / 3.0).abs() < 1e-12); + } + } + + /// Unlike eigenvector centrality, Katz gives a source with no incoming edges a + /// non-zero score, because every node collects `beta` for existing. On the path + /// a -> b -> c with alpha 0.1 and beta 1 the fixed point is a = 1, b = 1.1, and + /// c = 1.11. + #[test] + fn katz_gives_every_node_the_beta_floor() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(nodes[0], nodes[1], "E", &()).unwrap(); + g.add_edge(nodes[1], nodes[2], "E", &()).unwrap(); + + let scores = g.katz_centrality(0.1, 1.0, 200, 1e-12).unwrap(); + assert!((scores[&nodes[0]] - 1.0).abs() < 1e-9); + assert!((scores[&nodes[1]] - 1.1).abs() < 1e-9); + assert!((scores[&nodes[2]] - 1.11).abs() < 1e-9); + } + + /// In a triangle every node's two neighbors are joined, so the coefficient is 1; + /// in a star the center's neighbors are joined to nothing, so it is 0. Direction + /// must not matter, which the triangle's one-way edges check. + #[test] + fn clustering_coefficient_reads_the_graph_as_undirected() { + let (_dir, g) = open_tmp(); + let tri: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(tri[0], tri[1], "E", &()).unwrap(); + g.add_edge(tri[1], tri[2], "E", &()).unwrap(); + g.add_edge(tri[2], tri[0], "E", &()).unwrap(); + + let scores = g.clustering_coefficient().unwrap(); + for node in &tri { + assert!((scores[node] - 1.0).abs() < 1e-12, "{}", scores[node]); + } + + let (_dir2, star) = open_tmp(); + let hub = star.add_node("N", &()).unwrap(); + let spokes: Vec = (0..3).map(|_| star.add_node("N", &()).unwrap()).collect(); + for spoke in &spokes { + star.add_edge(hub, *spoke, "E", &()).unwrap(); + } + assert_eq!(star.clustering_coefficient().unwrap()[&hub], 0.0); + } + + /// The coefficient is a ratio bounded by 1, so the neighborhood must be counted + /// over *distinct* neighbors. Counting a parallel edge twice inflates the + /// numerator past the denominator and yields a score above 1, which is the whole + /// reason this kernel does not follow PageRank's parallel-edge rule. + #[test] + fn clustering_coefficient_stays_bounded_under_parallel_edges() { + let (_dir, g) = open_tmp(); + let tri: Vec = (0..3).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(tri[0], tri[1], "E", &()).unwrap(); + g.add_edge(tri[1], tri[2], "E", &()).unwrap(); + g.add_edge(tri[2], tri[0], "E", &()).unwrap(); + // Duplicate every edge, and add a reversed one, so each pair is multiply and + // bidirectionally connected. + g.add_edge(tri[0], tri[1], "E", &()).unwrap(); + g.add_edge(tri[1], tri[0], "E", &()).unwrap(); + g.add_edge(tri[1], tri[2], "E", &()).unwrap(); + + for (node, score) in g.clustering_coefficient().unwrap() { + assert!((score - 1.0).abs() < 1e-12, "node {node} scored {score}"); + } + } + + /// Three cliques joined by one edge each is the standard shape Louvain must get + /// right and label propagation often does not: the bridges are too weak to justify + /// merging, so the partition has to be the three cliques. + #[test] + fn louvain_separates_cliques_joined_by_single_edges() { + let (_dir, g) = open_tmp(); + let groups: Vec> = (0..3) + .map(|_| (0..5).map(|_| g.add_node("N", &()).unwrap()).collect()) + .collect(); + for group in &groups { + add_clique(&g, group); + } + g.add_edge(groups[0][0], groups[1][0], "E", &()).unwrap(); + g.add_edge(groups[1][0], groups[2][0], "E", &()).unwrap(); + + let mut expected: Vec> = groups + .iter() + .map(|group| { + let mut sorted = group.clone(); + sorted.sort_unstable(); + sorted + }) + .collect(); + expected.sort(); + assert_eq!(partition(&g.louvain().unwrap()), expected); + } + + /// Nodes in different weakly connected components can never be in one community, + /// since no sequence of moves could ever increase modularity by joining them. + /// This is the invariant that holds on every graph, so it is the one to check on + /// a shape with no obvious hand-computable answer. + #[test] + fn louvain_never_merges_disconnected_components() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..12).map(|_| g.add_node("N", &()).unwrap()).collect(); + // Two components with an irregular internal shape. + for (a, b) in [(0, 1), (1, 2), (2, 3), (3, 0), (0, 2), (4, 5)] { + g.add_edge(nodes[a], nodes[b], "E", &()).unwrap(); + } + for (a, b) in [(6, 7), (7, 8), (8, 9), (9, 10), (10, 11), (11, 6)] { + g.add_edge(nodes[a], nodes[b], "E", &()).unwrap(); + } + + let communities = g.louvain().unwrap(); + let components = g.connected_components().unwrap(); + for (a, b) in communities.keys().flat_map(|&a| { + communities + .keys() + .filter(move |&&b| b > a) + .map(move |&b| (a, b)) + }) { + if communities[&a] == communities[&b] { + assert_eq!( + components[&a], components[&b], + "{a} and {b} share a community across components" + ); + } + } + } + + /// The community id is the smallest node id in the community, matching the + /// connected-components convention, and every node must be assigned exactly once. + #[test] + fn louvain_names_a_community_after_its_smallest_member() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..6).map(|_| g.add_node("N", &()).unwrap()).collect(); + add_clique(&g, &nodes[..3]); + add_clique(&g, &nodes[3..]); + + let communities = g.louvain().unwrap(); + assert_eq!(communities.len(), nodes.len()); + for part in partition(&communities) { + let smallest = *part.iter().min().unwrap(); + for node in &part { + assert_eq!(communities[node], smallest); + } + } + } + + /// The partition must not depend on how many times the pass is run, which is the + /// determinism the serial local-moving phase exists to provide. + #[test] + fn louvain_is_reproducible() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..10).map(|_| g.add_node("N", &()).unwrap()).collect(); + add_clique(&g, &nodes[..4]); + add_clique(&g, &nodes[4..]); + g.add_edge(nodes[0], nodes[4], "E", &()).unwrap(); + + let first = g.louvain().unwrap(); + for _ in 0..4 { + assert_eq!(g.louvain().unwrap(), first); + } + } + + /// An edgeless graph has no moves available, so every node stays alone. + #[test] + fn louvain_leaves_isolated_nodes_alone() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..4).map(|_| g.add_node("N", &()).unwrap()).collect(); + + let communities = g.louvain().unwrap(); + for node in &nodes { + assert_eq!(communities[node], *node); + } + } + + /// A self-loop is not a neighbor pair and must not enter the neighborhood, or a + /// node with one real neighbor plus a loop would look like it had two. + #[test] + fn clustering_coefficient_excludes_self_loops() { + let (_dir, g) = open_tmp(); + let nodes: Vec = (0..2).map(|_| g.add_node("N", &()).unwrap()).collect(); + g.add_edge(nodes[0], nodes[0], "E", &()).unwrap(); + g.add_edge(nodes[0], nodes[1], "E", &()).unwrap(); + + assert_eq!(g.clustering_coefficient().unwrap()[&nodes[0]], 0.0); + } } diff --git a/crates/issundb-core/src/graph/mod.rs b/crates/issundb-core/src/graph/mod.rs index a448bf3..554538b 100644 --- a/crates/issundb-core/src/graph/mod.rs +++ b/crates/issundb-core/src/graph/mod.rs @@ -51,6 +51,33 @@ pub enum DegreeDirection { Both, } +/// Which score [`Graph::link_prediction_score`] computes for a pair of nodes. +/// +/// All five read the graph as undirected over distinct neighbors, the same +/// neighborhood [`Graph::clustering_coefficient`] uses, so a pair joined by several +/// edges is one neighbor and direction never matters. A higher score means the pair +/// is more likely to become connected. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)] +pub enum LinkPredictionMetric { + /// How many neighbors the two nodes share. + CommonNeighbors, + /// Shared neighbors over the size of the combined neighborhood, so a pair of + /// low-degree nodes is not penalized against a pair of hubs. Zero when neither + /// node has a neighbor. + Jaccard, + /// Shared neighbors weighted by `1 / ln(degree)`, so a neighbor that everyone + /// shares counts for little. A shared neighbor of degree one contributes nothing, + /// since `ln(1)` is zero and the term is undefined rather than large. + AdamicAdar, + /// Shared neighbors weighted by `1 / degree`, which penalizes popular neighbors + /// harder than Adamic-Adar does. + ResourceAllocation, + /// The product of the two degrees, on the theory that busy nodes attract more + /// edges. This one ignores shared neighbors entirely, so it scores pairs that + /// have nothing in common. + PreferentialAttachment, +} + /// Pattern description for [`Graph::count_triangle_cycles`]: the directed /// cycle `(a)-[t1]->(b)-[t2]->(c)-[t3]->(a)` with an optional relationship /// type per hop and an optional label per node variable. `None` means diff --git a/crates/issundb-core/src/graph/node.rs b/crates/issundb-core/src/graph/node.rs index c8a6462..c3b585d 100644 --- a/crates/issundb-core/src/graph/node.rs +++ b/crates/issundb-core/src/graph/node.rs @@ -273,6 +273,24 @@ impl Graph { } } + /// Whether a node with this id exists. + /// + /// A key probe, not a read: [`Graph::get_node`] decodes the record, which also copies the + /// node's whole property blob, and a caller asking only whether the id is live should not pay + /// for that. This is the check a writer runs per item, so the difference is the point. + pub fn node_exists(&self, id: NodeId) -> Result { + let rtxn = self.storage.env.read_txn()?; + self.node_exists_impl(&rtxn, id) + } + + pub(crate) fn node_exists_impl( + &self, + txn: &crate::storage::RoTxn, + id: NodeId, + ) -> Result { + Ok(self.storage.nodes.get(txn, &id)?.is_some()) + } + /// Update the properties of an existing node. The node's label is unchanged. /// /// # Deadlock warning diff --git a/crates/issundb-core/src/lib.rs b/crates/issundb-core/src/lib.rs index c0a4867..f602e90 100644 --- a/crates/issundb-core/src/lib.rs +++ b/crates/issundb-core/src/lib.rs @@ -9,8 +9,8 @@ pub(crate) mod threads; pub use error::Error; pub use graph::{ - DegreeDirection, Graph, GroupedDegreeSpec, NeighborCountSpec, PathCountSpec, ReadTxn, - TriangleCountSpec, WriteTxn, + DegreeDirection, Graph, GroupedDegreeSpec, LinkPredictionMetric, NeighborCountSpec, + PathCountSpec, ReadTxn, TriangleCountSpec, WriteTxn, }; pub use schema::{ DirectedNeighborEntry, EdgeId, EdgeRecord, LabelId, Language, NeighborEntry, NodeId, diff --git a/crates/issundb-core/src/threads.rs b/crates/issundb-core/src/threads.rs index 2c81d95..f63c577 100644 --- a/crates/issundb-core/src/threads.rs +++ b/crates/issundb-core/src/threads.rs @@ -9,6 +9,10 @@ /// Upper bound on threads any single pass will use, so a misconfigured value /// cannot spawn an unbounded pool. +/// +/// Absent on a threadless target for the same reason [`resolve_from_lazy`] is: nothing +/// there can reach a clamp that only the resolved paths apply. +#[cfg(any(not(target_family = "wasm"), test))] pub(crate) const MAX_THREADS: usize = 64; /// Resolve the thread count for a parallel pass. @@ -71,6 +75,12 @@ fn resolve_from( /// [`resolve_from`] with the machine count behind a closure, so a configured /// value never pays for measuring the machine. +/// +/// Compiled out on a threadless target, where [`resolve`] answers 1 before any of this +/// precedence is reachable. The gate is the fix rather than an `allow(dead_code)`, +/// because a blanket allow here would also hide a genuinely orphaned resolver later; +/// `test` is included so the precedence stays covered wherever the suite runs. +#[cfg(any(not(target_family = "wasm"), test))] fn resolve_from_lazy( programmatic: i32, issundb_env: Option<&str>, diff --git a/crates/issundb-cypher/AGENTS.md b/crates/issundb-cypher/AGENTS.md index 2f4fe06..af659b1 100644 --- a/crates/issundb-cypher/AGENTS.md +++ b/crates/issundb-cypher/AGENTS.md @@ -27,6 +27,8 @@ Keep each concern in its own file. Do not call `Graph` methods from `parser.rs`, ## Parser Structure Rules +The grammar covers multi-label node patterns such as `(n:A:B)` and inline relationship property maps. + The parser is built with the `chumsky` parser-combinator library, in two phases. Phase 1 lexes the query text into a token stream. Phase 2 builds the combinator graph, with operator precedence expressed through chumsky's Pratt parser (`chumsky::pratt::{infix, left, postfix, prefix}`) rather than a hand-written descent chain. @@ -43,6 +45,9 @@ hand-written descent chain. genuinely pathological input before any AST is built, a deep parse runs on a dedicated large-stack thread, and a query whose nesting exceeds `SMALL_STACK_EXEC_BUDGET_KB` has its execution dispatched to a large-stack thread by `execute_with_procedures`. +Query text longer than `PARSE_CACHE_MAX_QUERY_LEN` is parsed but not cached, so many large unique statements cannot grow the parse cache by their +length. + ## AST Immutability Policy - All AST node types derive `Clone` and `PartialEq`. They are produced once by the parser and treated as read-only thereafter. @@ -152,6 +157,25 @@ of every referenced `(variable, property)` column per query rather than a point test that asserts byte-identical columns and records against the row pipeline (`assert_matches_row_path` and the `*_matches_row_path` tests in `exec/vectorized.rs`); add one whenever you widen `recognize`. +### What `recognize` Accepts + +The structural pattern is `[Limit]? [Sort]? [Distinct]? Project [Aggregate]? Stage* (Expand(directed single hop) Stage*){0,MAX_VEC_HOPS} Leaf` with +single-property expressions, executed column-at-a-time: bulk expansion through `Graph::node_props_json_table` and group-by-code aggregation through +`Graph::node_prop_group_codes`. A multi-hop chain qualifies only when every hop carries a distinct relationship type, which makes relationship uniqueness +vacuous; a repeated type, or a chain longer than `MAX_VEC_HOPS`, falls back. The recognizer sees through a `Distinct` because the caller deduplicates. + +A non-distinct `count` over the terminal variable that feeds no group key collapses the final hop (`execute_collapsed_count`), counting each source's +qualifying neighbors through `Graph::typed_neighbor_counts` so the last hop costs no triple per traversed edge and no hash lookup per edge. A terminal +filter that is a label test goes into the spec directly; a terminal property comparison is resolved into a `neighbor_allow` set by running those exact +stages over the label's whole node set (`resolve_terminal_allow`). That resolution is gated on the sources' `adjacency_span` reaching half the label +count, so a selective hop over a large label keeps the expansion fallback rather than paying for a full label pass, and it is speculative: it evaluates +predicates over a superset of the real neighbors, so a stage that errors there declines to the fallback rather than raising. + +Two shapes route to the fallback whatever else holds. A multi-type hop does, because `Expand::rel_type` carries the raw pattern text (`"F|G"`) while +the kernel resolves one registered type. And a stale snapshot with at most `STALE_POINT_EXPAND_MAX` sources does +(`Graph::prefers_point_expansion`), because the kernel would rebuild the whole snapshot where the fallback serves those sources from per-source +adjacency. + **Group-key identity invariant** (binds both executors): grouping by a bare node or edge variable (`Expr::Prop(var, "")`) keys on the element id, not its materialized property bag, and the group row keeps the `Node` or `Edge` binding rather than a materialized `Scalar`. The row pipeline's `aggregate_all` fold and the vectorized aggregate both depend on this. Serializing a whole node to a JSON object per input row to build the group key diff --git a/crates/issundb-cypher/src/builtin_procs.rs b/crates/issundb-cypher/src/builtin_procs.rs index 3d62e1d..e31deca 100644 --- a/crates/issundb-cypher/src/builtin_procs.rs +++ b/crates/issundb-cypher/src/builtin_procs.rs @@ -44,6 +44,17 @@ const PAGE_RANK_ITERATIONS: u32 = 20; const PAGE_RANK_DAMPING: f32 = 0.85; /// Default label-propagation iteration cap. const LABEL_PROP_ITERATIONS: usize = 20; +/// Default iteration cap for the power-iteration centralities (eigenvector and Katz). +/// Both stop early once they converge, so this only bounds the pathological case. +const POWER_ITERATION_MAX: u32 = 100; +/// Default convergence threshold on the L2 change between successive iterations. +const POWER_ITERATION_TOLERANCE: f64 = 1e-6; +/// Default Katz attenuation. Convergence needs `alpha` below the reciprocal of the +/// largest eigenvalue, which depends on the data, so the default is deliberately +/// small enough to be safe on the graphs a caller is likely to try it on first. +const KATZ_ALPHA: f64 = 0.1; +/// Default Katz baseline, the score every node receives before any walk reaches it. +const KATZ_BETA: f64 = 1.0; /// Build the concrete [`Procedure`] for a built-in `issundb.*` name by running it /// against `graph`. @@ -75,12 +86,19 @@ pub fn build(graph: &Graph, name: &str, args: &[Value]) -> Result Result, String> { let parameterized = matches!( name, - "issundb.pageRank" | "issundb.degree" | "issundb.labelPropagation" + "issundb.pageRank" + | "issundb.degree" + | "issundb.labelPropagation" + | "issundb.eigenvector" + | "issundb.katz" ); let parameterless = matches!( name, "issundb.betweenness" | "issundb.harmonic" + | "issundb.closeness" + | "issundb.clusteringCoefficient" + | "issundb.louvain" | "issundb.connectedComponents" | "issundb.wcc" | "issundb.stronglyConnectedComponents" @@ -127,6 +145,49 @@ fn build_algorithm(graph: &Graph, name: &str, args: &[Value]) -> Result ( + "score", + float_rows(graph.closeness_centrality().map_err(proc_err)?.into_iter()), + ), + "issundb.clusteringCoefficient" => ( + "score", + float_rows( + graph + .clustering_coefficient() + .map_err(proc_err)? + .into_iter(), + ), + ), + "issundb.eigenvector" => { + let iterations = + cfg_usize(name, cfg, "iterations", POWER_ITERATION_MAX as usize)? as u32; + let tolerance = cfg_f64(name, cfg, "tolerance", POWER_ITERATION_TOLERANCE)?; + ( + "score", + float_rows( + graph + .eigenvector_centrality(iterations, tolerance) + .map_err(proc_err)? + .into_iter(), + ), + ) + } + "issundb.katz" => { + let alpha = cfg_f64(name, cfg, "alpha", KATZ_ALPHA)?; + let beta = cfg_f64(name, cfg, "beta", KATZ_BETA)?; + let iterations = + cfg_usize(name, cfg, "iterations", POWER_ITERATION_MAX as usize)? as u32; + let tolerance = cfg_f64(name, cfg, "tolerance", POWER_ITERATION_TOLERANCE)?; + ( + "score", + float_rows( + graph + .katz_centrality(alpha, beta, iterations, tolerance) + .map_err(proc_err)? + .into_iter(), + ), + ) + } "issundb.degree" => { let direction = parse_degree_direction(name, cfg)?; ( @@ -164,6 +225,10 @@ fn build_algorithm(graph: &Graph, name: &str, args: &[Value]) -> Result ( + "communityId", + int_rows(graph.louvain().map_err(proc_err)?.into_iter()), + ), _ => unreachable!("name was checked against the built-in sets above"), }; @@ -424,6 +489,17 @@ fn cfg_f32(proc: &str, cfg: Option<&Value>, key: &str, default: f32) -> Result, key: &str, default: f64) -> Result { + match cfg_field(proc, cfg, key)? { + None => Ok(default), + Some(Value::Number(n)) => n + .as_f64() + .ok_or_else(|| cfg_type_err(proc, key, "a number")), + Some(_) => Err(cfg_type_err(proc, key, "a number")), + } +} + /// Read an optional string configuration field (`None` when absent). fn cfg_opt_string(proc: &str, cfg: Option<&Value>, key: &str) -> Result, String> { match cfg_field(proc, cfg, key)? { @@ -621,8 +697,22 @@ fn build_communities( let cfg = args.first(); let max_iterations = cfg_usize(name, cfg, "maxIterations", LABEL_PROP_ITERATIONS)?; let top = cfg_opt_usize(name, cfg, "topPerCommunity")?; - - let communities = graph.label_propagation(max_iterations).map_err(proc_err)?; + // Label propagation stays the default so an existing query keeps its partition. + // Louvain usually separates better, particularly where communities are joined by + // a few edges, which is the case label propagation tends to merge. + let algorithm = cfg_opt_string(name, cfg, "algorithm")?; + + let communities = match algorithm.as_deref() { + None | Some("labelPropagation") => { + graph.label_propagation(max_iterations).map_err(proc_err)? + } + Some("louvain") => graph.louvain().map_err(proc_err)?, + Some(other) => { + return Err(format!( + "{name}() algorithm must be 'labelPropagation' or 'louvain', got '{other}'" + )); + } + }; let ranks = graph .page_rank(PAGE_RANK_ITERATIONS, PAGE_RANK_DAMPING) .map_err(proc_err)?; @@ -750,6 +840,120 @@ mod tests { assert_eq!(res.records.len(), 3); } + /// The four centralities ported from Graphina all yield the same + /// `(nodeId, score)` shape as the existing ones, so a caller can swap between + /// them without changing the surrounding query. + #[test] + fn ported_centralities_yield_a_scored_row_per_node() { + let (_d, g) = triangle(); + let params = HashMap::new(); + for call in [ + "issundb.closeness()", + "issundb.clusteringCoefficient()", + "issundb.eigenvector()", + "issundb.katz()", + ] { + let res = execute( + &g, + &format!("CALL {call} YIELD nodeId, score RETURN nodeId, score"), + ¶ms, + ) + .unwrap_or_else(|e| panic!("{call} failed: {e}")); + assert_eq!(res.columns, vec!["nodeId".to_string(), "score".to_string()]); + assert_eq!(res.records.len(), 3, "{call}"); + } + } + + /// Every node of a directed triangle reaches both others at total distance 3, so + /// closeness is `(2/3) * (2/2)` for all three. This pins the value through the + /// procedure layer, not just the kernel. + #[test] + fn closeness_reports_the_expected_value_through_cypher() { + let (_d, g) = triangle(); + let params = HashMap::new(); + let res = execute( + &g, + "CALL issundb.closeness() YIELD score RETURN score ORDER BY score", + ¶ms, + ) + .unwrap(); + for record in &res.records { + let score = record.values[0].as_f64().unwrap(); + assert!((score - 2.0 / 3.0).abs() < 1e-12, "{score}"); + } + } + + /// The power-iteration centralities take a configuration map, and an unknown or + /// mistyped field must be rejected rather than silently ignored. + #[test] + fn power_iteration_centralities_accept_configuration() { + let (_d, g) = triangle(); + let params = HashMap::new(); + let res = execute( + &g, + "CALL issundb.katz({alpha: 0.05, beta: 2.0, iterations: 50}) \ + YIELD nodeId, score RETURN count(*) AS c", + ¶ms, + ) + .unwrap(); + assert_eq!(res.records[0].values[0], serde_json::json!(3)); + + let err = execute( + &g, + "CALL issundb.eigenvector({tolerance: 'nope'}) YIELD nodeId RETURN nodeId", + ¶ms, + ) + .unwrap_err(); + assert!(format!("{err}").contains("tolerance"), "{err}"); + } + + /// Louvain yields the same `(nodeId, communityId)` shape as label propagation, and + /// on a directed triangle both must put all three nodes together. + #[test] + fn louvain_yields_a_community_per_node() { + let (_d, g) = triangle(); + let params = HashMap::new(); + let res = execute( + &g, + "CALL issundb.louvain() YIELD nodeId, communityId RETURN nodeId, communityId", + ¶ms, + ) + .unwrap(); + assert_eq!( + res.columns, + vec!["nodeId".to_string(), "communityId".to_string()] + ); + assert_eq!(res.records.len(), 3); + let ids: std::collections::HashSet<_> = + res.records.iter().map(|r| r.values[1].clone()).collect(); + assert_eq!(ids.len(), 1, "a triangle is one community"); + } + + /// `communities` can be driven by either algorithm, and an unknown name is an + /// error rather than a silent fallback to the default. + #[test] + fn communities_selects_its_algorithm() { + let (_d, g) = triangle(); + let params = HashMap::new(); + for algorithm in ["labelPropagation", "louvain"] { + let res = execute( + &g, + &format!("CALL issundb.communities({{algorithm: '{algorithm}'}}) YIELD nodeId"), + ¶ms, + ) + .unwrap_or_else(|e| panic!("{algorithm}: {e}")); + assert_eq!(res.records.len(), 3, "{algorithm}"); + } + + let err = execute( + &g, + "CALL issundb.communities({algorithm: 'infomap'}) YIELD nodeId", + ¶ms, + ) + .unwrap_err(); + assert!(format!("{err}").contains("infomap"), "{err}"); + } + #[test] fn standalone_connected_components_projects_outputs() { let (_d, g) = triangle(); diff --git a/crates/issundb-cypher/src/exec/expr.rs b/crates/issundb-cypher/src/exec/expr.rs index 48437fa..f20dcbe 100644 --- a/crates/issundb-cypher/src/exec/expr.rs +++ b/crates/issundb-cypher/src/exec/expr.rs @@ -4,6 +4,7 @@ use chrono::{ Datelike, Duration, NaiveDate, NaiveDateTime, NaiveTime, Offset, TimeZone, Timelike, Weekday, }; use chrono_tz::Tz; +use issundb_core::LinkPredictionMetric; use issundb_vector::VectorGraphExt; use std::cell::{Cell, RefCell}; use std::rc::Rc; @@ -1522,6 +1523,66 @@ fn eval_set_metric( }) } +/// Resolve a link-prediction argument to a node id. +/// +/// A node value resolves to its id and an integer to itself, so both +/// `issundb.link.jaccard(a, b)` over matched nodes and `issundb.link.jaccard(1, 2)` +/// over literal ids work. `null` resolves to `None` so the caller can propagate +/// Cypher null semantics rather than raising. +fn resolve_node_arg( + graph: &Graph, + path: &B, + name: &str, + expr: &Expr, + params: &HashMap, +) -> Result, String> { + match evaluate_expr(graph, path, expr, params)? { + serde_json::Value::Null => Ok(None), + serde_json::Value::Object(map) + if map.get("__type__").and_then(|t| t.as_str()) == Some("__Node__") => + { + map.get("id") + .and_then(|v| v.as_u64()) + .map(Some) + .ok_or_else(|| format!("{name}() node argument has no id")) + } + serde_json::Value::Number(n) => n + .as_u64() + .map(Some) + .ok_or_else(|| format!("{name}() node id must be a non-negative integer")), + _ => Err(format!("{name}() arguments must be nodes or node ids")), + } +} + +/// Evaluate one of the `issundb.link.*` neighborhood link-prediction scores. +/// +/// These are functions rather than procedures because a `CALL` evaluates its +/// arguments against no bindings and runs once per statement, so it could never see +/// the `a` and `b` of a `MATCH`. A pairwise score has to run per row. +fn eval_link_metric( + graph: &Graph, + path: &B, + name: &str, + args: &[Expr], + params: &HashMap, + metric: LinkPredictionMetric, +) -> Result { + if args.len() != 2 { + return Err(format!("{name}() requires exactly 2 arguments")); + } + let a = resolve_node_arg(graph, path, name, &args[0], params)?; + let b = resolve_node_arg(graph, path, name, &args[1], params)?; + match (a, b) { + (Some(a), Some(b)) => { + let score = graph + .link_prediction_score(a, b, metric) + .map_err(|e| format!("{name}(): {e}"))?; + Ok(serde_json::Value::from(score)) + } + _ => Ok(serde_json::Value::Null), + } +} + pub(super) fn eval_function_call( graph: &Graph, path: &B, @@ -1636,6 +1697,50 @@ pub(super) fn eval_function_call( "issundb.similarity.jaccard" => { eval_set_metric(graph, path, name, args, params, jaccard_similarity, |s| s) } + // Neighborhood link prediction, distinct from the `similarity` family above: + // those compare two value sets a caller supplies, these compare the two + // nodes' neighborhoods in the graph. `name` has already been lowercased, so + // these arms carry no camel case even though a query may write it. + "issundb.link.commonneighbors" => eval_link_metric( + graph, + path, + name, + args, + params, + LinkPredictionMetric::CommonNeighbors, + ), + "issundb.link.jaccard" => eval_link_metric( + graph, + path, + name, + args, + params, + LinkPredictionMetric::Jaccard, + ), + "issundb.link.adamicadar" => eval_link_metric( + graph, + path, + name, + args, + params, + LinkPredictionMetric::AdamicAdar, + ), + "issundb.link.resourceallocation" => eval_link_metric( + graph, + path, + name, + args, + params, + LinkPredictionMetric::ResourceAllocation, + ), + "issundb.link.preferentialattachment" => eval_link_metric( + graph, + path, + name, + args, + params, + LinkPredictionMetric::PreferentialAttachment, + ), "issundb.similarity.overlap" => { eval_set_metric(graph, path, name, args, params, overlap_similarity, |s| s) } diff --git a/crates/issundb-cypher/src/exec/mod.rs b/crates/issundb-cypher/src/exec/mod.rs index 57d4bdd..bce745e 100644 --- a/crates/issundb-cypher/src/exec/mod.rs +++ b/crates/issundb-cypher/src/exec/mod.rs @@ -139,7 +139,9 @@ pub fn execute_with_procedures( // the parser flags such depth, run the whole statement on a large-stack // thread. The statement clock is thread-local, so it is installed inside the // worker, not on the caller. Shallow queries (the common case) execute inline. - if exec_needs_large_stack { + // The same threadless-target exception the parser makes, for the same reason: there is no + // worker to dispatch to, and the 16 MiB wasm stack is already the mitigation. + if exec_needs_large_stack && !cfg!(target_family = "wasm") { // Both the statement clock and the row-pipeline-only switch are // thread-local, and a fresh thread starts from neither this thread's // setting nor an installed clock, so both are installed inside the worker. @@ -1097,6 +1099,96 @@ mod tests { ); } + /// The `issundb.link.*` family scores how likely two nodes are to become + /// connected, from their neighborhoods in the graph. They are functions rather + /// than procedures precisely so they can see `a` and `b` bound by a `MATCH`, + /// which a `CALL` cannot: its arguments are evaluated against no bindings. + /// + /// The fixture is a bowtie. `a` and `b` share the two hub neighbors `h1` and + /// `h2`, so every value below is computable by hand. + #[test] + fn link_prediction_scalar_functions() { + let params = HashMap::new(); + let dir = TempDir::new().unwrap(); + let graph = Graph::open(dir.path(), 1).unwrap(); + execute( + &graph, + "CREATE (a:P {n:'a'}), (b:P {n:'b'}), (h1:P {n:'h1'}), (h2:P {n:'h2'}), (x:P {n:'x'}), + (a)-[:K]->(h1), (a)-[:K]->(h2), + (b)-[:K]->(h1), (b)-[:K]->(h2), + (h1)-[:K]->(x)", + ¶ms, + ) + .unwrap(); + graph.rebuild_csr().unwrap(); + + let pair = |f: &str| -> f64 { + let q = format!("MATCH (a:P {{n:'a'}}), (b:P {{n:'b'}}) RETURN {f}(a, b) AS s"); + execute(&graph, &q, ¶ms).unwrap().records[0].values[0] + .as_f64() + .unwrap() + }; + + // a and b share exactly h1 and h2. + assert_eq!(pair("issundb.link.commonNeighbors"), 2.0); + // Neighborhoods are both {h1, h2}, so intersection and union are both 2. + assert!((pair("issundb.link.jaccard") - 1.0).abs() < 1e-12); + // Degrees are 2 and 2, so the product is 4. + assert_eq!(pair("issundb.link.preferentialAttachment"), 4.0); + // h1 is joined to a, b, and x, so degree 3; h2 to a and b, so degree 2. + let expected_ra = 1.0 / 3.0 + 1.0 / 2.0; + assert!((pair("issundb.link.resourceAllocation") - expected_ra).abs() < 1e-12); + let expected_aa = 1.0 / 3.0f64.ln() + 1.0 / 2.0f64.ln(); + assert!((pair("issundb.link.adamicAdar") - expected_aa).abs() < 1e-12); + + // Node ids work as well as node values, and null propagates. + let scalar = |q: &str| -> serde_json::Value { + execute(&graph, q, ¶ms).unwrap().records[0].values[0].clone() + }; + assert_eq!( + scalar("RETURN issundb.link.commonNeighbors(null, 1) AS s"), + serde_json::Value::Null + ); + // A node that does not exist shares nothing rather than failing the query. + assert_eq!( + scalar("RETURN issundb.link.commonNeighbors(0, 999999) AS s") + .as_f64() + .unwrap(), + 0.0 + ); + } + + /// The point of a function over a procedure: it runs once per row, so a single + /// query can rank many candidate pairs against one anchor. + #[test] + fn link_prediction_scores_every_row() { + let params = HashMap::new(); + let dir = TempDir::new().unwrap(); + let graph = Graph::open(dir.path(), 1).unwrap(); + execute( + &graph, + "CREATE (a:P {n:'a'}), (b:P {n:'b'}), (c:P {n:'c'}), (h:P {n:'h'}), + (a)-[:K]->(h), (b)-[:K]->(h), (c)-[:K]->(h), (a)-[:K]->(b)", + ¶ms, + ) + .unwrap(); + graph.rebuild_csr().unwrap(); + + let res = execute( + &graph, + "MATCH (a:P {n:'a'}), (other:P) WHERE other.n <> 'a' + RETURN other.n AS n, issundb.link.commonNeighbors(a, other) AS score + ORDER BY score DESC, n", + ¶ms, + ) + .unwrap(); + assert_eq!(res.records.len(), 3); + // b and c both share h with a; h shares b and c with a through its own edges. + for record in &res.records { + assert!(record.values[1].as_f64().unwrap() >= 1.0); + } + } + /// Run `setup` then `query`, returning the single scalar value of the one expected row. fn agg_scalar(setup: &[&str], query: &str) -> serde_json::Value { let params = HashMap::new(); diff --git a/crates/issundb-cypher/src/parser.rs b/crates/issundb-cypher/src/parser.rs index 26ddb47..113c919 100644 --- a/crates/issundb-cypher/src/parser.rs +++ b/crates/issundb-cypher/src/parser.rs @@ -3549,6 +3549,11 @@ fn is_known_function(name: &str) -> bool { | "issundb.distance.euclidean" | "issundb.similarity.jaccard" | "issundb.similarity.overlap" + | "issundb.link.commonneighbors" + | "issundb.link.jaccard" + | "issundb.link.adamicadar" + | "issundb.link.resourceallocation" + | "issundb.link.preferentialattachment" ) } @@ -5823,7 +5828,14 @@ fn parse_uncached(cypher: &str) -> Result<(Statement, bool), CypherError> { let inline = nesting.bracket_case <= INLINE_BRACKET_CASE_DEPTH && nesting.op <= INLINE_OP_DEPTH && nesting.union <= INLINE_UNION_DEPTH; - let stmt = if inline { + // A target with no threads has nowhere to hand the work, so it parses inline whatever the + // depth. That is not a silent downgrade of the guarantee: `.cargo/config.toml` raises the wasm + // stack to 16 MiB precisely because this path cannot spawn, which is sixteen times the stack + // the inline thresholds are calibrated against. Refusing instead would make every query past + // the inline budget unparseable in a browser, which is worse than parsing it on a stack chosen + // for the purpose. A genuinely pathological input is still rejected before here, by + // `scan_nesting`. + let stmt = if inline || cfg!(target_family = "wasm") { run()? } else { std::thread::scope(|scope| { diff --git a/crates/issundb-mcp/Cargo.toml b/crates/issundb-mcp/Cargo.toml index 794e1c8..d1f54ce 100644 --- a/crates/issundb-mcp/Cargo.toml +++ b/crates/issundb-mcp/Cargo.toml @@ -15,8 +15,13 @@ name = "issundb-mcp" path = "src/main.rs" doc = false +[features] +default = ["lmdb", "hnsw"] +lmdb = ["issundb/lmdb"] +hnsw = ["issundb/hnsw"] + [dependencies] -issundb = { workspace = true, features = ["lmdb", "hnsw"] } +issundb = { workspace = true } rmcp = { version = "=0.11", features = [ "server", "macros", diff --git a/crates/issundb-py/tests/test_search.py b/crates/issundb-py/tests/test_search.py index 1531e96..547f248 100644 --- a/crates/issundb-py/tests/test_search.py +++ b/crates/issundb-py/tests/test_search.py @@ -47,3 +47,36 @@ def test_search_dropped_index_raises(db): # IndexNotFound, which crosses the boundary as a RuntimeError. with pytest.raises(RuntimeError): db.text_search("quick", "Article", "body", 10) + + +def test_upsert_vector_rejects_a_node_that_does_not_exist(db): + """An embedding may only be given to a node that exists. + + Regression: it used to be accepted. Node ids are handed out monotonically, so + a vector written ahead of its node was inherited by the next node created with + that id, which then answered a search at distance zero having never been + embedded. Nothing downstream could detect it, because a stored vector carries + no evidence of who it was meant for. + """ + alice = db.add_node("Person", json.dumps({"name": "Alice"})) + db.upsert_vector(alice, [1.0, 0.0]) + + future = alice + 1 + with pytest.raises(RuntimeError, match=f"node {future} does not exist"): + db.upsert_vector(future, [0.0, 1.0]) + + # Bob takes that id and must own no embedding. + bob = db.add_node("Person", json.dumps({"name": "Bob"})) + assert bob == future + hits = json.loads(db.vector_search([0.0, 1.0], 5)) + assert all(h["node"] != bob for h in hits), ( + f"a node that was never embedded must not appear in a vector search: {hits}" + ) + + +def test_remove_vector_for_a_deleted_node_is_allowed(db): + """Removal stays permissive, so a vector whose node is gone can still be cleaned up.""" + node = db.add_node("Doc", json.dumps({"title": "x"})) + db.upsert_vector(node, [1.0, 0.0]) + db.delete_node(node) + db.remove_vector(node) diff --git a/crates/issundb-rest/Cargo.toml b/crates/issundb-rest/Cargo.toml index ab72d71..c44a327 100644 --- a/crates/issundb-rest/Cargo.toml +++ b/crates/issundb-rest/Cargo.toml @@ -15,8 +15,13 @@ name = "issundb-rest" path = "src/main.rs" doc = false +[features] +default = ["lmdb", "hnsw"] +lmdb = ["issundb/lmdb"] +hnsw = ["issundb/hnsw"] + [dependencies] -issundb = { workspace = true, features = ["lmdb", "hnsw"] } +issundb = { workspace = true } axum = "0.8.9" tokio = { version = "1", features = ["full"] } serde = { version = "1", features = ["derive"] } diff --git a/crates/issundb-vector/src/error.rs b/crates/issundb-vector/src/error.rs index c7ccbd0..4804612 100644 --- a/crates/issundb-vector/src/error.rs +++ b/crates/issundb-vector/src/error.rs @@ -22,6 +22,11 @@ pub enum VectorError { #[error("invalid vector configuration: {0}")] InvalidConfig(String), + #[error( + "node {0} does not exist: create the node before giving it an embedding, or the vector is left for whichever node is later allocated that id" + )] + NodeNotFound(u64), + #[error("underlying storage error: {0}")] Storage(#[from] issundb_core::Error), } diff --git a/crates/issundb-vector/src/index.rs b/crates/issundb-vector/src/index.rs index 07a97dc..8bdff00 100644 --- a/crates/issundb-vector/src/index.rs +++ b/crates/issundb-vector/src/index.rs @@ -394,6 +394,17 @@ impl VectorGraphExt for Graph { #[instrument(skip(self, v), fields(node = %n, dims = v.len()))] fn upsert_vector(&self, n: NodeId, v: &[f32]) -> Result<(), VectorError> { + // Reject an embedding for an id no node holds. Node ids are handed out monotonically, so a + // vector written ahead of its node is not inert: the next node allocated that id inherits + // it and answers a search at distance zero, having never been embedded. Nothing downstream + // could detect that, because a stored vector carries no evidence of who it was meant for. + // + // The cost is one key probe per upsert, inside a call that already opens a write + // transaction and rebuilds an index entry. `remove_vector` stays permissive on purpose, so + // a database that already holds such a vector can still be cleaned up. + if !self.node_exists(n)? { + return Err(VectorError::NodeNotFound(n)); + } let bytes = encode_vector(v)?; // Validate against (and update) the in-memory index BEFORE persisting to // LMDB. `upsert` rejects empty or dimension-mismatched embeddings, so @@ -740,6 +751,69 @@ mod tests { (dir, graph) } + /// An embedding for an id no node holds is refused. + /// + /// Regression: it used to be accepted, and because node ids are handed out monotonically, the + /// next node created with that id inherited it. The node below is never embedded and yet + /// answered a search at distance zero, with no error at any layer. + #[test] + fn a_vector_for_a_node_that_does_not_exist_is_refused() { + let (_dir, graph) = open_tmp(); + let alice = graph + .add_node("Person", &json!({ "name": "Alice" })) + .unwrap(); + graph.upsert_vector(alice, &[1.0, 0.0]).unwrap(); + + // The very next id, which no node holds yet. + let future = alice + 1; + let err = graph.upsert_vector(future, &[0.0, 1.0]).unwrap_err(); + assert!( + matches!(err, VectorError::NodeNotFound(id) if id == future), + "expected NodeNotFound, got {err:?}" + ); + + // Bob takes that id and must own no embedding. + let bob = graph.add_node("Person", &json!({ "name": "Bob" })).unwrap(); + assert_eq!(bob, future); + let hits = graph.vector_search(&[0.0, 1.0], 5).unwrap(); + assert!( + hits.iter().all(|h| h.node != bob), + "a node that was never embedded must not appear in a vector search: {hits:?}" + ); + } + + /// The rejection happens before anything is written, so a refused upsert leaves neither an + /// index entry nor a stored vector behind. Checking after the fact is what a partial write + /// would defeat. + #[test] + fn a_refused_vector_reaches_neither_the_index_nor_storage() { + let (_dir, graph) = open_tmp(); + let real = graph.add_node("N", &json!({})).unwrap(); + graph.upsert_vector(real, &[1.0, 0.0]).unwrap(); + + assert!(graph.upsert_vector(real + 99, &[0.0, 1.0]).is_err()); + let stored = graph.vector_bytes().unwrap(); + assert!( + stored.iter().all(|(id, _)| *id != real + 99), + "the refused vector must not be in storage: {:?}", + stored.iter().map(|(id, _)| *id).collect::>() + ); + let hits = graph.vector_search(&[0.0, 1.0], 5).unwrap(); + assert_eq!(hits.len(), 1, "only the one real embedding: {hits:?}"); + assert_eq!(hits[0].node, real); + } + + /// Removal stays permissive, which is the escape hatch for a database written before the check + /// existed: a caller has to be able to delete a vector whose node is already gone. + #[test] + fn removing_a_vector_for_a_missing_node_is_not_an_error() { + let (_dir, graph) = open_tmp(); + let node = graph.add_node("N", &json!({})).unwrap(); + graph.upsert_vector(node, &[1.0, 0.0]).unwrap(); + graph.delete_node(node).unwrap(); + graph.remove_vector(node).unwrap(); + } + #[test] fn metric_from_str_is_case_insensitive_with_alias() { assert_eq!( diff --git a/crates/issundb/src/lib.rs b/crates/issundb/src/lib.rs index 651dd96..301ae54 100644 --- a/crates/issundb/src/lib.rs +++ b/crates/issundb/src/lib.rs @@ -32,8 +32,8 @@ // method whose argument type cannot be named from here is not callable. pub use issundb_core::{ DegreeDirection, DirectedNeighborEntry, EdgeId, EdgeRecord, Error, Graph, GroupedDegreeSpec, - LabelId, Language, NeighborCountSpec, NeighborEntry, NodeId, NodeRecord, PathCountSpec, - PropValue, ReadTxn, TriangleCountSpec, TypeId, WeightedPath, WriteTxn, + LabelId, Language, LinkPredictionMetric, NeighborCountSpec, NeighborEntry, NodeId, NodeRecord, + PathCountSpec, PropValue, ReadTxn, TriangleCountSpec, TypeId, WeightedPath, WriteTxn, }; pub use issundb_cypher::{ CypherError, CypherType, Procedure, ProcedureRegistry, QueryResult, Record, diff --git a/docs/api-reference.md b/docs/api-reference.md index 47fc464..b1b6050 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -177,6 +177,23 @@ Pathfinding, network centrality, and connectivity algorithms run over the in-mem Computes the betweenness centrality score for all nodes. - `harmonic_centrality() -> Result, Error>` Computes the harmonic centrality score for all nodes. +- `closeness_centrality() -> Result, Error>` + Computes closeness centrality in the Wasserman-Faust form, which scales by the fraction of the graph a node reaches and so stays + meaningful on a disconnected graph. +- `eigenvector_centrality(iterations: u32, tolerance: f64) -> Result, Error>` + Computes eigenvector centrality by power iteration, stopping early on convergence. Scores are magnitudes scaled to sum to the node + count. +- `katz_centrality(alpha: f64, beta: f64, iterations: u32, tolerance: f64) -> Result, Error>` + Computes Katz centrality, which gives every node the `beta` baseline and attenuates each walk by `alpha`. Convergence needs `alpha` + below the reciprocal of the largest eigenvalue. +- `clustering_coefficient() -> Result, Error>` + Computes the local clustering coefficient, reading the graph as undirected over distinct neighbors. +- `louvain() -> Result, Error>` + Detects communities by the Louvain method. The community id is the smallest node id in the community, and only the induced partition + is contractual. +- `link_prediction_score(a: NodeId, b: NodeId, metric: LinkPredictionMetric) -> Result` + Scores how likely two nodes are to become connected, by common neighbors, Jaccard, Adamic-Adar, resource allocation, or preferential + attachment. ### Connectivity and Flow @@ -210,6 +227,9 @@ The index is configured through `VectorIndexOptions`, which holds a `VectorMetri - `VectorGraphExt::reindex_vector_index(opts: VectorIndexOptions) -> Result<(), VectorError>` Changes the metric and quantization settings and rebuilds the index from the persisted embeddings. - `VectorGraphExt::upsert_vector(n: NodeId, v: &[f32]) -> Result<(), VectorError>` + Stores the embedding for an existing node. A node that does not exist is rejected with + `VectorError::NodeNotFound`, because node ids are handed out monotonically and a vector written + ahead of its node would be inherited by whichever node is later allocated that id. Associates a float vector embedding with a node. - `VectorGraphExt::remove_vector(n: NodeId) -> Result<(), VectorError>` Removes the embedding for a node from both the index and storage. @@ -302,8 +322,23 @@ Graph data science procedures can be executed through the query interface using - `CALL issundb.betweenness()` and `CALL issundb.harmonic()` yield `(nodeId, score)`. Both take no arguments. - `CALL issundb.degree({direction})` yields `(nodeId, score)`, where `direction` is `'IN'`, `'OUT'`, or `'BOTH'` (the default). - `CALL issundb.connectedComponents()` (alias `issundb.wcc`) and `CALL issundb.stronglyConnectedComponents()` (alias `issundb.scc`) yield `(nodeId, componentId)`. +- `CALL issundb.closeness()` and `CALL issundb.clusteringCoefficient()` yield `(nodeId, score)`. Both take no arguments. +- `CALL issundb.eigenvector({iterations, tolerance})` and `CALL issundb.katz({alpha, beta, iterations, tolerance})` yield `(nodeId, score)`. Both stop early on convergence and never fail on a slow graph; the configuration map is optional. - `CALL issundb.labelPropagation({maxIterations})` yields `(nodeId, communityId)`. -- `CALL issundb.communities({maxIterations, topPerCommunity})` yields `(communityId, nodeId, rank)`, partitioning by label propagation and ranking each community by PageRank. +- `CALL issundb.louvain()` yields `(nodeId, communityId)`. It separates communities joined by a few edges, which label propagation tends to merge. +- `CALL issundb.communities({maxIterations, topPerCommunity, algorithm})` yields `(communityId, nodeId, rank)`, ranking each community by PageRank. The `algorithm` field selects `'labelPropagation'` (the default) or `'louvain'`. + +### Link Prediction + +Pairwise scores are scalar functions rather than procedures, because a `CALL` evaluates its arguments against no bindings and runs once per +statement, so it can never see the two nodes a `MATCH` bound. Each takes two nodes or node ids, reads the neighborhood as undirected over +distinct neighbors, and returns null when either argument is null. + +- `issundb.link.commonNeighbors(a, b)` counts the neighbors the two nodes share. +- `issundb.link.jaccard(a, b)` divides shared neighbors by the size of the combined neighborhood. +- `issundb.link.adamicAdar(a, b)` weights each shared neighbor by `1 / ln(degree)`; a shared neighbor of degree one contributes nothing. +- `issundb.link.resourceAllocation(a, b)` weights each shared neighbor by `1 / degree`. +- `issundb.link.preferentialAttachment(a, b)` multiplies the two degrees, ignoring shared neighbors entirely. ### Pathfinding diff --git a/docs/cypher.md b/docs/cypher.md index e15509a..064dca6 100644 --- a/docs/cypher.md +++ b/docs/cypher.md @@ -81,6 +81,8 @@ The temporal constructors `date`, `time`, `localtime`, `datetime`, `localdatetim - `SET n = {map}` and `SET n += {map}`: assign properties individually with `SET n.prop = value`. - `CALL { ... }` subqueries and `EXISTS { ... }` subqueries: `CALL` is procedure invocation only, and `exists()` is a scalar null check. - `shortestPath(...)` and `allShortestPaths(...)` pattern functions: use the `shortest_path` and `all_shortest_paths` methods on the `Graph` API instead. +- Pattern predicates in `WHERE` (`WHERE (a)-[:KNOWS]->(b)`, `WHERE NOT (a)-->(b)`). A pattern is matched, not tested, so express the positive case as + an additional `MATCH` and the negative case as an anti-join: `OPTIONAL MATCH (a)-[r:KNOWS]->(b) WITH a, b, r WHERE r IS NULL`. - Map projections (`n{.name, .age}`). - `MANDATORY MATCH`. diff --git a/docs/getting-started.md b/docs/getting-started.md index 7d8b8b4..d5117b8 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -5,7 +5,7 @@ It explains prerequisites, building the engine from source, and using the comman ## Prerequisites -Compiling IssunDB and its native dependencies requires Rust 1.85.0 or later, along with the following system tools: +Compiling IssunDB and its dependencies needs Rust 1.85.0 or later, along with the following system tools: - Build tools: a C/C++ compiler (such as Clang or GCC), which compiles the bundled LMDB sources and the vector index. @@ -120,7 +120,7 @@ To use IssunDB as an embedded database in a Rust project, add the `issundb` libr ```toml [dependencies] -issundb = "0.1.0-alpha.20" # Match the version published on crates.io +issundb = "0.1.0-alpha.21" # Match the version published on crates.io serde_json = "1.0" # Used to construct property maps ``` diff --git a/scripts/check_playground.mjs b/scripts/check_playground.mjs index f6dee17..19266c4 100644 --- a/scripts/check_playground.mjs +++ b/scripts/check_playground.mjs @@ -13,9 +13,10 @@ const pkg = process.env.PLAYGROUND_PKG ?? join(here, "..", "target", "playground const require = createRequire(import.meta.url); const { Playground } = require(join(pkg, "issundb_wasm.js")); -const { DEMO_CATEGORIES, PROCEDURES, SAMPLE_GRAPHS, SAMPLE_SOCIAL } = await import( +const { DEMO_CATEGORIES, FUNCTIONS, PROCEDURES, SAMPLE_GRAPHS, SAMPLE_SOCIAL } = await import( join(here, "..", "web", "demos.js") ); +const { formatCypher } = await import(join(here, "..", "web", "format.js")); // The graph each category's examples query, since none of them builds its own data any more. const sampleById = new Map(SAMPLE_GRAPHS.map((sample) => [sample.id, sample.cypher])); @@ -151,6 +152,93 @@ for (const proc of PROCEDURES) { } } +// The function catalog, checked exactly as the procedures are. A function is called in an +// expression rather than through CALL, so its snippet is an ordinary query, and the same loop +// works: what matters is that the name resolves and the snippet runs. An `UnknownFunction` is the +// drift this catches, the way `ProcedureNotFound` is above. +console.log("\nFunction reference"); + +let fnChecked = 0; +let fnFailures = 0; + +for (const fn of FUNCTIONS) { + fnChecked += 1; + const p = new Playground(); + try { + p.query(SAMPLE_SOCIAL); + const result = JSON.parse(p.query(fn.snippet)); + if (result.rows.length === 0) { + fnFailures += 1; + console.log(` FAIL ${fn.name.padEnd(38)} returned no rows`); + } else { + console.log(` ok ${fn.name.padEnd(38)} ${result.rows.length} row(s)`); + } + } catch (e) { + fnFailures += 1; + console.log(` FAIL ${fn.name.padEnd(38)} ${String(e.message ?? e).split("\n")[0]}`); + } +} + +// The formatter, which rewrites a query in the editor at the press of a button and must therefore +// be incapable of changing what that query means. Casing is the sharp edge: uppercasing every word +// the highlighter treats as a keyword once rewrote `issundb.shortestPath` and the case-sensitive +// yield fields `index` and `count`. Nothing but this loop stands between such a rule and the page. +// +// Every catalog string is run twice on two fresh databases, once as written and once formatted, and +// the two results must agree. A string that errors as written is skipped rather than failed: the +// other passes above own that verdict, and the two entries needing embeddings error by design. +console.log("\nFormatter round trip"); + +const formatterCorpus = [ + ...SAMPLE_GRAPHS.map((s) => [`sample:${s.id}`, s.cypher]), + ...DEMO_CATEGORIES.flatMap((c) => + c.demos.map((d, i) => [`demo:${c.label}#${i + 1}`, d.cypher]).filter(([, q]) => q), + ), + ...PROCEDURES.map((p) => [`proc:${p.name}`, p.snippet]), + ...FUNCTIONS.map((f) => [`fn:${f.name}`, f.snippet]), +]; + +let fmtChecked = 0; +let fmtFailures = 0; +let fmtSkipped = 0; + +// Row order is not guaranteed by a query without ORDER BY, and formatting cannot change it anyway, +// so the comparison sorts. What it is looking for is a changed row *set*. +const rowsOf = (cypher) => { + const p = new Playground(); + p.query(SAMPLE_SOCIAL); + const result = JSON.parse(p.query(cypher)); + return JSON.stringify([result.columns, [...result.rows].map((r) => JSON.stringify(r)).sort()]); +}; + +for (const [label, cypher] of formatterCorpus) { + let before; + try { + before = rowsOf(cypher); + } catch { + fmtSkipped += 1; + continue; + } + fmtChecked += 1; + const formatted = formatCypher(cypher); + let after; + try { + after = rowsOf(formatted); + } catch (e) { + fmtFailures += 1; + console.log(` FAIL ${label.padEnd(38)} formatted query errors: ${String(e.message ?? e).split("\n")[0]}`); + continue; + } + if (before !== after) { + fmtFailures += 1; + console.log(` FAIL ${label.padEnd(38)} formatting changed the result`); + } +} +console.log( + ` ${fmtChecked - fmtFailures}/${fmtChecked} strings unchanged by formatting` + + (fmtSkipped ? `, ${fmtSkipped} skipped (error as written)` : ""), +); + // Cypher blocks in `docs/` marked ``, which `docs/hooks/playground_links.py` // turns into a "Run in the playground" link. The marker is a claim that the block runs against the // seeded sample graph, and nothing else checks it, so an example edited into a parameter or a @@ -203,6 +291,9 @@ console.log( `${procChecked - procFailures}/${procChecked} procedures ok` + (procFailures ? `, ${procFailures} failed` : ""), ); +console.log( + `${fnChecked - fnFailures}/${fnChecked} functions ok` + (fnFailures ? `, ${fnFailures} failed` : ""), +); console.log( `${docChecked - docFailures}/${docChecked} marked doc blocks ok` + (docFailures ? `, ${docFailures} failed` : ""), @@ -211,4 +302,6 @@ console.log( `${SAMPLE_GRAPHS.length - sampleFailures}/${SAMPLE_GRAPHS.length} sample graphs ok` + (sampleFailures ? `, ${sampleFailures} failed` : ""), ); -process.exit(failures + procFailures + docFailures + sampleFailures ? 1 : 0); +process.exit( + failures + procFailures + fnFailures + fmtFailures + docFailures + sampleFailures ? 1 : 0, +); diff --git a/web/README.md b/web/README.md index 6276f3e..436900d 100644 --- a/web/README.md +++ b/web/README.md @@ -27,14 +27,14 @@ statement it depends on. The Setup panel offers five, each small enough to read at once and shaped so that one part of the engine has something to say about it: -| Sample | What it is | Examples that query it | -|---|---|---| -| Social network | Weighted acquaintances. Seeded on load. | Cypher basics, Graph algorithms, Query planning, Vector search | -| Article corpus | Documents, topics, and citations. | Full-text search, GraphRAG | -| Knowledge graph | Researchers, labs, papers, and the concepts those mention. | Knowledge graph | -| Org chart | A reporting tree, so variable-length hops, shortest path, and a numeric range scan. | | -| Transport network | Routes carrying a weight, a cost, and a capacity, so a weighted path differs from a shortest one. | | -| Retail co-purchase | Customers and products, so grouped counts, a price range scan, and a co-purchase join. | | +| Sample | What it is | Examples that query it | +|--------------------|---------------------------------------------------------------------------------------------------|----------------------------------------------------------------| +| Social network | Weighted acquaintances. Seeded on load. | Cypher basics, Graph algorithms, Query planning, Vector search | +| Article corpus | Documents, topics, and citations. | Full-text search, GraphRAG | +| Knowledge graph | Researchers, labs, papers, and the concepts those mention. | Knowledge graph | +| Org chart | A reporting tree, so variable-length hops, shortest path, and a numeric range scan. | | +| Transport network | Routes carrying a weight, a cost, and a capacity, so a weighted path differs from a shortest one. | | +| Retail co-purchase | Customers and products, so grouped counts, a price range scan, and a co-purchase join. | | This is the only place a dataset comes from. Every example queries whatever graph is loaded rather than creating one, which is what keeps the two panels from being two lists of datasets: an earlier @@ -67,8 +67,8 @@ make playground-serve # http://localhost:8000 A module cannot be loaded over `file://`, so the page has to be served over HTTP. Any static server works; `make playground-serve` uses Python's. -`make playground-check` runs every sample graph, demo, and procedure snippet in `demos.js` through -the compiled module and fails on an error. All three are Cypher inside a JavaScript file, which no +`make playground-check` runs every sample graph, demo, procedure, and function snippet in +`demos.js` through the compiled module and fails on an error. All three are Cypher inside a JavaScript file, which no Rust test can see, so this is what keeps a button, a preset, or a reference entry from silently breaking. The procedure half also rejects `ProcedureNotFound` specifically, which is the failure a rename produces. @@ -133,13 +133,15 @@ and native controls from the light palette on both schemes. - `index.html`: the page. The inline script in `` applies the stored scheme before first paint, so a dark-theme visitor never sees a white flash. -- `app.js`: everything the page does. The Cypher highlighter and the force-directed layout - are written here rather than pulled from a library, so the page loads nothing it does not - contain. -- `demos.js`: the demo catalog and the procedure reference, both checked by +- `app.js`: everything the page does except run the engine. The Cypher highlighter, the + autocomplete, the plan tree, and the force-directed layout are written here rather than pulled + from a library, so the page loads nothing it does not contain. +- `worker.js`: the engine. It owns the wasm module and the graph, and the page reaches it only by + message, so a query never blocks the tab. See "Cancelling a Query" for what that costs. +- `demos.js`: the demo catalog and the procedure and function references, all checked by `make playground-check`. Each demo category carries a `docs` link into the surrounding documentation; those are relative to `/playground/`, so they break if a heading they anchor to - is renamed. The procedure list is written out by hand because the engine cannot enumerate its + is renamed. Both reference lists are written out by hand because the engine cannot enumerate its own procedures, which is why the check runs each snippet. - `style.css`: light and dark themes over one set of custom properties. - `logo.svg`: the header logo and the favicon, copied from `docs/assets/logo.svg` by @@ -162,11 +164,39 @@ bounded to a factor of eight in and four out, or a few scrolls leave an empty ca way the graph went. A redraw returns the view to the whole canvas, so a query is never answered into a frame computed for different data. +## How a Category Knows Its Data + +A demo category names the graph it queries through `sample` and the label that proves that graph is loaded through `requiresLabel`, so the Examples +panel can tell you to load the right sample instead of running a query against the wrong one. `make playground-check` seeds that graph before running +the category, which is what lets every example query whatever is loaded rather than creating its own data. + +## The Binding Layer + +`crates/issundb-wasm` exposes one `Playground` type and every method returns a JSON string, so the boundary carries one type in both directions rather +than a second serialization contract. The methods are split into a private logic layer returning `Result<_, String>` and a thin exported layer that +converts to `JsError`, because constructing a `JsError` calls a wasm-bindgen import that panics off-target; without the split none of it could be +covered by `cargo test`. The build flags live in the `WASM_BUILD` variable in the Makefile rather than being repeated per target, since +`--features hnsw` reads like it selects the index and in fact selects `usearch`, which fails to compile `cxx` for wasm. + +## Cancelling a Query + +The engine runs in `worker.js`, so a long query leaves the page responsive and a Cancel button +appears beside Execute. Pressing it terminates the worker, which is the only way to stop a +WebAssembly call: there is no interruption point for a message to be handled at, so nothing short of +killing the thread will do. + +The graph lives in the worker and therefore dies with it. Cancelling is still recoverable rather +than destructive, because the page immediately starts a fresh worker, re-seeds the sample, and +replays every statement it recorded as setup, which is the same log a share link carries. A query +that only read loses nothing. One that had already written is reapplied from that log, so anything +typed straight into the editor and run is restored, while a write made by a statement the page never +saw is not. + ## What the Footer Reports The left end reads `This playground app is powered by IssunDB (0.1.0-alpha.20; develop@1f938); ...`. The version is the crate's, and `develop@1f938` is the branch and short commit the module was built -from. The right end reads `6 nodes and 9 edges · 32 KB in use, 19.9 MB heap`. +from. The right end reads `| 6 nodes and 9 edges | 32 KB in use | 19.9 MB heap |`. `in use` is live bytes the engine has allocated and not freed, counted by a `GlobalAlloc` wrapper the `issundb-wasm` crate installs and read through `Playground.memoryBytes()`. `heap` is the WebAssembly @@ -227,9 +257,11 @@ A word is uppercased only if the clause-phrase scan recognized it or it is in a operators, and never when it follows a `.`, a `:`, or an `AS`. A property that happens to spell a clause is left where it is for the same reason: `RETURN n.set` is one line, not two. -Because the pass cannot change what a query means, that is testable, and it is tested: every Cypher -string in `demos.js` is run before and after formatting on identical fresh databases, and the -columns and rows compared. That check is what found all three casing bugs above. +Because the pass cannot change what a query means, that is testable, and `make playground-check` +tests it: every Cypher string in the catalog is run on two fresh databases, once as written and once +formatted, and the column and row sets compared. Reintroducing any of the three casing bugs above +fails it. The formatter lives in `format.js` rather than `app.js` so the checker can import it, +since `app.js` touches the DOM at import time and a Node process has none. ## Links from the Documentation diff --git a/web/app.js b/web/app.js index 7e370b0..372c4b3 100644 --- a/web/app.js +++ b/web/app.js @@ -2,70 +2,139 @@ // the two web fonts are the page's only external request. One `Playground` for the tab's lifetime, // so data accumulates across queries the // way it would in an embedded database; "Reset data" replaces it. +// +// The engine itself runs in `worker.js`, so everything here that reaches it is asynchronous. -import init, { Playground } from "./pkg/issundb_wasm.js"; -import { DEMO_CATEGORIES, PROCEDURES, SAMPLE_GRAPHS } from "./demos.js"; +import {DEMO_CATEGORIES, FUNCTIONS, PROCEDURES, SAMPLE_GRAPHS} from "./demos.js"; +import {formatCypher} from "./format.js"; const $ = (id) => document.getElementById(id); +// --------------------------------------------------------------------------- +// Engine +// --------------------------------------------------------------------------- + +let worker = null; +let nextCall = 0; +const pending = new Map(); + +function spawnWorker() { + worker = new Worker(new URL("./worker.js", import.meta.url), {type: "module"}); + worker.onmessage = ({data: {id, ok, value, error}}) => { + const entry = pending.get(id); + if (!entry) return; + pending.delete(id); + if (ok) entry.resolve(value); + else entry.reject(new Error(error)); + }; + // A worker whose module fails to evaluate never reaches its message handler, so without this + // every call would stay pending and the page would sit on the loading spinner rather than saying + // what went wrong. + worker.onerror = (e) => { + const reason = new Error(e.message || "the engine worker failed to start"); + for (const entry of pending.values()) entry.reject(reason); + pending.clear(); + }; +} + +function call(op, ...args) { + return new Promise((resolve, reject) => { + const id = ++nextCall; + pending.set(id, {resolve, reject}); + worker.postMessage({id, op, args}); + }); +} + +const engine = { + boot: () => call("boot"), + reset: () => call("reset"), + query: (cypher) => call("query", cypher), + explain: (cypher) => call("explain", cypher), + stats: () => call("stats"), + graphSnapshot: () => call("graphSnapshot"), + createTextIndex: (label, property) => call("createTextIndex", label, property), + textSearch: (query, k) => call("textSearch", query, k), + upsertVector: (id, vector) => call("upsertVector", id, vector), + vectorSearch: (vector, k) => call("vectorSearch", vector, k), + memory: () => call("memory"), +}; + +const CANCELLED = "The query was cancelled."; + +// Terminating the worker is the only way to stop a running query, because a wasm call has no +// interruption point for a message to be handled at. The graph lives in the worker, so it dies with +// it, and the replay below is what makes cancelling recoverable rather than destructive: the sample +// is re-seeded and every statement the page recorded as setup is applied again. A query that only +// read loses nothing; one that had already written is reapplied from `setupLog`, which is the same +// log a share link carries. +async function cancelRunningQuery() { + worker.terminate(); + for (const entry of pending.values()) entry.reject(new Error(CANCELLED)); + pending.clear(); + spawnWorker(); + await engine.boot(); + await engine.query(currentSample().cypher); + if (setupLog.length > 0) await engine.query(setupLog.join(";\n")); +} + // --------------------------------------------------------------------------- // Cypher highlighting // --------------------------------------------------------------------------- const KEYWORDS = new Set( - `match optional where return create merge set remove delete detach with unwind + `match optional where return create merge set remove delete detach with unwind order by skip limit distinct as and or xor not in starts ends contains is null true false asc ascending desc descending union all call yield on constraint index explain profile case when then else end exists count collect sum avg min max foreach load csv from headers using periodic commit drop unique assert require for scalar single any none shortestpath allshortestpaths copy export import database` - .split(/\s+/) - .filter(Boolean), + .split(/\s+/) + .filter(Boolean), ); const TOKEN = new RegExp( - [ - "(\\/\\/[^\\n]*)", - "(\\/\\*[\\s\\S]*?\\*\\/)", - "('(?:[^'\\\\]|\\\\.)*'|\"(?:[^\"\\\\]|\\\\.)*\")", - "(\\$[A-Za-z_]\\w*)", - "(:[A-Za-z_]\\w*)", - "(\\b\\d+\\.?\\d*(?:[eE][-+]?\\d+)?\\b)", - "([A-Za-z_]\\w*)(?=\\s*\\()", - "([A-Za-z_][\\w.]*)", - "([-=<>|*+\\/%!,.;{}\\[\\]()]+)", - ].join("|"), - "g", + [ + "(\\/\\/[^\\n]*)", + "(\\/\\*[\\s\\S]*?\\*\\/)", + "('(?:[^'\\\\]|\\\\.)*'|\"(?:[^\"\\\\]|\\\\.)*\")", + "(\\$[A-Za-z_]\\w*)", + "(:[A-Za-z_]\\w*)", + "(\\b\\d+\\.?\\d*(?:[eE][-+]?\\d+)?\\b)", + "([A-Za-z_]\\w*)(?=\\s*\\()", + "([A-Za-z_][\\w.]*)", + "([-=<>|*+\\/%!,.;{}\\[\\]()]+)", + ].join("|"), + "g", ); const esc = (s) => - s.replace(/&/g, "&").replace(//g, ">"); + s.replace(/&/g, "&").replace(//g, ">"); function highlight(src) { - let out = ""; - let last = 0; - for (const m of src.matchAll(TOKEN)) { - out += esc(src.slice(last, m.index)); - last = m.index + m[0].length; - const cls = m[1] || m[2] ? "com" - : m[3] ? "str" - : m[4] ? "lbl" - : m[5] ? "lbl" - : m[6] ? "num" - : m[7] ? (KEYWORDS.has(m[7].toLowerCase()) ? "kw" : "fn") - : m[8] ? (KEYWORDS.has(m[8].toLowerCase()) ? "kw" : null) - : m[9] ? "op" - : null; - out += cls ? `${esc(m[0])}` : esc(m[0]); - } - return out + esc(src.slice(last)); + let out = ""; + let last = 0; + for (const m of src.matchAll(TOKEN)) { + out += esc(src.slice(last, m.index)); + last = m.index + m[0].length; + const cls = m[1] || m[2] ? "com" + : m[3] ? "str" + : m[4] ? "lbl" + : m[5] ? "lbl" + : m[6] ? "num" + : m[7] ? (KEYWORDS.has(m[7].toLowerCase()) ? "kw" : "fn") + : m[8] ? (KEYWORDS.has(m[8].toLowerCase()) ? "kw" : null) + : m[9] ? "op" + : null; + out += cls ? `${esc(m[0])}` : esc(m[0]); + } + return out + esc(src.slice(last)); } // --------------------------------------------------------------------------- // State // --------------------------------------------------------------------------- -let db = null; +let ready = false; let lastResult = null; let sim = null; @@ -98,16 +167,16 @@ const editor = $("editor"); const highlightEl = $("highlight"); function syncScroll() { - const backdrop = highlightEl.parentElement; - backdrop.scrollTop = editor.scrollTop; - backdrop.scrollLeft = editor.scrollLeft; + const backdrop = highlightEl.parentElement; + backdrop.scrollTop = editor.scrollTop; + backdrop.scrollLeft = editor.scrollLeft; } function syncHighlight() { - // The trailing newline stops the backdrop's last line from collapsing, which would let - // the two panes disagree by one line height at the bottom. - highlightEl.innerHTML = highlight(editor.value) + "\n"; - syncScroll(); + // The trailing newline stops the backdrop's last line from collapsing, which would let + // the two panes disagree by one line height at the bottom. + highlightEl.innerHTML = highlight(editor.value) + "\n"; + syncScroll(); } // This build keeps the graph in memory, so a reload starts from the seeded sample either way. @@ -117,274 +186,245 @@ function syncHighlight() { const MAX_STORED_EDITOR = 100000; function storeEditor() { - try { - if (editor.value.length > MAX_STORED_EDITOR) localStorage.removeItem(EDITOR_KEY); - else localStorage.setItem(EDITOR_KEY, editor.value); - } catch { - // Storage being unavailable only costs the restore. - } + try { + if (editor.value.length > MAX_STORED_EDITOR) localStorage.removeItem(EDITOR_KEY); + else localStorage.setItem(EDITOR_KEY, editor.value); + } catch { + // Storage being unavailable only costs the restore. + } } function readStoredEditor() { - try { - return localStorage.getItem(EDITOR_KEY) ?? ""; - } catch { - return ""; - } + try { + return localStorage.getItem(EDITOR_KEY) ?? ""; + } catch { + return ""; + } } function setQuery(text) { - editor.value = text; - syncHighlight(); - storeEditor(); - editor.focus(); + editor.value = text; + syncHighlight(); + storeEditor(); + editor.focus(); } // Debounced rather than written per keystroke, since every write serializes the whole buffer. let storeTimer = null; editor.addEventListener("input", () => { - syncHighlight(); - // The follow-up belongs to the example that was loaded, so editing the query retires it. - pendingDemo = null; - clearTimeout(storeTimer); - storeTimer = setTimeout(storeEditor, 400); + syncHighlight(); + // The follow-up belongs to the example that was loaded, so editing the query retires it. + pendingDemo = null; + clearTimeout(storeTimer); + storeTimer = setTimeout(storeEditor, 400); + openCompletions(); }); // Scrolling only moves the backdrop. Re-running the highlighter per scroll event rebuilt // the whole document's markup on every frame of a drag. editor.addEventListener("scroll", syncScroll); editor.addEventListener("keydown", (e) => { - if (e.key === "Enter" && (e.metaKey || e.ctrlKey)) { - e.preventDefault(); - run(); - return; - } - // Shift-Alt-F, the shortcut an editor is expected to answer to. - if (e.altKey && e.shiftKey && (e.key === "F" || e.key === "f")) { - e.preventDefault(); - formatEditor(); - return; - } - if (e.key === "Tab") { - e.preventDefault(); - const { selectionStart: a, selectionEnd: b, value } = editor; - editor.value = value.slice(0, a) + " " + value.slice(b); - editor.selectionStart = editor.selectionEnd = a + 2; - syncHighlight(); - } + // The popup owns these keys while it is open, or Enter would run the query instead of accepting + // the highlighted completion and Escape would do nothing. + if (completionOpen()) { + if (e.key === "ArrowDown" || e.key === "ArrowUp") { + e.preventDefault(); + moveCompletion(e.key === "ArrowDown" ? 1 : -1); + return; + } + if (e.key === "Enter" || e.key === "Tab") { + e.preventDefault(); + acceptCompletion(); + return; + } + if (e.key === "Escape") { + e.preventDefault(); + closeCompletions(); + return; + } + } + + if (e.key === "Enter" && (e.metaKey || e.ctrlKey)) { + e.preventDefault(); + run(); + return; + } + // Shift-Alt-F, the shortcut an editor is expected to answer to. + if (e.altKey && e.shiftKey && (e.key === "F" || e.key === "f")) { + e.preventDefault(); + formatEditor(); + return; + } + // Ctrl-Space asks for completions where the prefix rules below would not have offered any. + if (e.key === " " && (e.ctrlKey || e.metaKey)) { + e.preventDefault(); + openCompletions(true); + return; + } + if (e.key === "Tab") { + e.preventDefault(); + const {selectionStart: a, selectionEnd: b, value} = editor; + editor.value = value.slice(0, a) + " " + value.slice(b); + editor.selectionStart = editor.selectionEnd = a + 2; + syncHighlight(); + } }); // --------------------------------------------------------------------------- -// Formatting +// Autocomplete // --------------------------------------------------------------------------- -// Clause phrases that begin a line, longest first so `ON CREATE SET` is recognized before the `SET` -// inside it. -const CLAUSE_PHRASES = [ - ["ON", "CREATE", "SET"], - ["ON", "MATCH", "SET"], - ["OPTIONAL", "MATCH"], - ["DETACH", "DELETE"], - ["ORDER", "BY"], - ["UNION", "ALL"], - ["MATCH"], - ["WHERE"], - ["WITH"], - ["RETURN"], - ["SKIP"], - ["LIMIT"], - ["CREATE"], - ["MERGE"], - ["SET"], - ["REMOVE"], - ["DELETE"], - ["UNWIND"], - ["CALL"], - ["YIELD"], - ["UNION"], - ["FOREACH"], -]; - -// The clauses whose comma-separated items are patterns rather than expressions. Breaking after each -// comma there turns a long line into a readable list of paths; doing it in RETURN would scatter a -// projection over as many lines as it has columns. -const PATTERN_CLAUSES = new Set(["CREATE", "MERGE"]); - -// Deliberately much narrower than the highlighter's keyword set. Uppercasing everything that set -// contains rewrote `issundb.shortestPath` to `issundb.SHORTESTPATH`, and the yield fields `index` -// and `count` to `INDEX` and `COUNT`, all three of which are case-sensitive names rather than -// syntax. So only operators are listed here, and a clause word is uppercased because the phrase -// scan recognized it as one, not because it appears in a list. Function names are left alone: an -// aggregate is conventionally lowercase, and `all(` is not the `ALL` of `UNION ALL`. -const FORMAT_UPPERCASE = new Set([ - "and", - "or", - "xor", - "not", - "in", - "is", - "null", - "true", - "false", - "distinct", - "as", - "asc", - "desc", - "ascending", - "descending", - "starts", - "ends", - "contains", -]); - -const FORMAT_TOKEN = new RegExp( - [ - "(\\/\\/[^\\n]*|\\/\\*[\\s\\S]*?\\*\\/)", - "('(?:[^'\\\\]|\\\\.)*'|\"(?:[^\"\\\\]|\\\\.)*\")", - "([A-Za-z_]\\w*)", - "(\\s+)", - "([^\\s])", - ].join("|"), - "g", -); +// Two characters before anything is offered unprompted, since one letter matches most of the +// catalog and a popup over every keystroke is noise. Ctrl-Space overrides the rule. +const COMPLETION_MIN_PREFIX = 2; +const COMPLETION_LIMIT = 12; + +let completions = []; +let completionAt = 0; +let completionStart = 0; + +const completionOpen = () => !$("ac").hidden; + +// The word under the caret, taken as the characters a Cypher name can contain. A leading colon is +// part of it so `:Pers` completes to a label rather than being read as an empty prefix. +function prefixBeforeCaret() { + const caret = editor.selectionStart; + let start = caret; + while (start > 0 && /[A-Za-z0-9_.]/.test(editor.value[start - 1])) start -= 1; + if (start > 0 && editor.value[start - 1] === ":") start -= 1; + return {start, text: editor.value.slice(start, caret)}; +} -// Line breaking and keyword casing, and nothing else. Spacing within a line is left as written apart -// from collapsing runs of whitespace, because re-spacing would have to know that the `-` in -// `-[:KNOWS]->` and the `*` in `[r*1..3]` are not binary operators. That restraint is what makes the -// pass safe to run on any query: it cannot change what the query means. -function formatCypher(src) { - const tokens = [...src.matchAll(FORMAT_TOKEN)].map((m) => ({ - comment: m[1], - string: m[2], - word: m[3], - space: m[4], - other: m[5], - text: m[0], - })); - - // A bracket depth per token, so a clause word inside a pattern or a map is not mistaken for the - // start of a line, and the index of every word, so a phrase can be matched by lookahead. - let depth = 0; - const words = []; - tokens.forEach((token, i) => { - token.depth = depth; - if (token.other && "([{".includes(token.other)) depth += 1; - if (token.other && ")]}".includes(token.other)) depth -= 1; - if (token.word) words.push(i); - }); - - const previousWordOf = (index) => { - for (let j = index - 1; j >= 0; j -= 1) { - if (tokens[j].space || tokens[j].comment) continue; - return tokens[j]; +// The schema half is live: it comes from the same `stats` call the schema panel renders, so a label +// created by the query you just ran is offered by the next keystroke. +function completionPool() { + const pool = []; + for (const label of Object.keys(lastStats?.label_counts ?? {})) { + pool.push({text: `:${label}`, kind: "label"}); } - return null; - }; - - const nextNonSpaceOf = (index) => { - for (let j = index + 1; j < tokens.length; j += 1) { - if (tokens[j].space) continue; - return tokens[j]; + for (const type of Object.keys(lastStats?.type_counts ?? {})) { + pool.push({text: `:${type}`, kind: "type"}); } - return null; - }; - - // `n.set` and `:Match` are names. Guarding the phrase scan and not only the casing is what stops - // `RETURN n.set` from being broken across two lines at the property. - const isQualifiedName = (index) => { - const previous = previousWordOf(index); - if (previous && (previous.other === "." || previous.other === ":")) return true; - return Boolean(previous && previous.word && previous.word.toLowerCase() === "as"); - }; - - const upperOf = (index) => (index === undefined ? "" : tokens[index].word.toUpperCase()); - const breakAt = new Set(); - const consumed = new Set(); - const phraseWords = new Set(); - words.forEach((i, w) => { - if (consumed.has(i) || tokens[i].depth !== 0 || isQualifiedName(i)) return; - const phrase = CLAUSE_PHRASES.find((candidate) => - candidate.every((word, k) => upperOf(words[w + k]) === word), - ); - if (!phrase) return; - breakAt.add(i); - tokens[i].clause = phrase.join(" "); - phrase.forEach((_, k) => phraseWords.add(words[w + k])); - for (let k = 1; k < phrase.length; k += 1) consumed.add(words[w + k]); - }); - - function shouldUppercase(index) { - if (phraseWords.has(index)) return true; - if (!FORMAT_UPPERCASE.has(tokens[index].word.toLowerCase())) return false; - if (isQualifiedName(index)) return false; - // A word the phrase scan did not claim, followed by an open parenthesis, is a function name - // rather than an operator. A clause keyword is exempt, since `MATCH (` is still a clause. - const next = nextNonSpaceOf(index); - return !(next && next.other === "("); - } - - let out = ""; - let atLineStart = true; - let pendingSpace = false; - let clause = ""; - - const newline = () => { - if (!atLineStart) out += "\n"; - atLineStart = true; - pendingSpace = false; - }; - - tokens.forEach((token, i) => { - if (token.space) { - pendingSpace = out.length > 0; - return; + for (const entry of REFERENCE) { + pool.push({text: entry.name, kind: entry.kind === "function" ? "fn" : "proc"}); } + for (const keyword of KEYWORDS) pool.push({text: keyword.toUpperCase(), kind: "kw"}); + return pool; +} - // A comment runs to the end of its line, so it has to keep one to itself or it would swallow - // whatever the formatter put after it. - if (token.comment) { - newline(); - out += token.text; - out += "\n"; - atLineStart = true; - return; +function rankCompletions(prefix) { + const needle = prefix.toLowerCase(); + const scored = []; + for (const item of completionPool()) { + const haystack = item.text.toLowerCase(); + const at = haystack.indexOf(needle); + if (at < 0) continue; + // A prefix match is what the typist meant; a match in the middle is a fallback, which is what + // makes `jacc` reach `issundb.link.jaccard` without burying the keywords that start with it. + scored.push({...item, rank: at === 0 ? 0 : 1, length: item.text.length}); } + scored.sort((a, b) => a.rank - b.rank || a.length - b.length || a.text.localeCompare(b.text)); + return scored.slice(0, COMPLETION_LIMIT); +} - if (breakAt.has(i)) { - newline(); - clause = token.clause; +// Measured against the highlight backdrop rather than a second hidden mirror. Its text is the +// editor's text exactly, and it already carries the same font, padding, and scroll offset, so a +// range inside it lands where the caret is drawn. +function caretPoint() { + const index = editor.selectionStart; + const walker = document.createTreeWalker(highlightEl, NodeFilter.SHOW_TEXT); + let seen = 0; + let node = walker.nextNode(); + while (node) { + const length = node.nodeValue.length; + if (seen + length >= index) { + const range = document.createRange(); + range.setStart(node, index - seen); + range.collapse(true); + const rect = range.getBoundingClientRect(); + const box = $("editor-box").getBoundingClientRect(); + return {x: rect.left - box.left, y: rect.bottom - box.top}; + } + seen += length; + node = walker.nextNode(); } + return null; +} - if (pendingSpace && !atLineStart) out += " "; - pendingSpace = false; - - if (token.word) { - out += shouldUppercase(i) ? token.word.toUpperCase() : token.word; - atLineStart = false; - return; +function openCompletions(forced = false) { + const {start, text} = prefixBeforeCaret(); + if (!forced && text.length < COMPLETION_MIN_PREFIX) return closeCompletions(); + const matches = rankCompletions(text); + if (matches.length === 0) return closeCompletions(); + + completions = matches; + completionAt = 0; + completionStart = start; + renderCompletions(); + + const point = caretPoint(); + const host = $("ac"); + host.hidden = false; + if (point) { + host.style.left = `${Math.max(4, point.x)}px`; + host.style.top = `${point.y + 4}px`; } +} - if (token.other === ";" && token.depth === 0) { - out += ";\n"; - atLineStart = true; - clause = ""; - return; - } +function renderCompletions() { + $("ac").innerHTML = completions + .map( + (item, i) => + `
` + + `${item.kind}` + + `${esc(item.text)}
`, + ) + .join(""); +} - if (token.other === "," && token.depth === 0 && PATTERN_CLAUSES.has(clause)) { - out += ",\n" + " ".repeat(clause.length + 1); - atLineStart = true; - return; - } +function moveCompletion(step) { + completionAt = (completionAt + step + completions.length) % completions.length; + renderCompletions(); + $("ac").querySelector(".ac-row.on")?.scrollIntoView({block: "nearest"}); +} - out += token.text; - atLineStart = false; - }); +function acceptCompletion() { + const chosen = completions[completionAt]; + if (!chosen) return closeCompletions(); + const caret = editor.selectionStart; + const before = editor.value.slice(0, completionStart); + const after = editor.value.slice(caret); + // A procedure or function is always followed by an argument list, so the parentheses come with + // it and the caret lands between them. + const call = chosen.kind === "proc" || chosen.kind === "fn"; + const inserted = call ? `${chosen.text}()` : chosen.text; + editor.value = before + inserted + after; + const caretAt = before.length + inserted.length - (call ? 1 : 0); + editor.selectionStart = editor.selectionEnd = caretAt; + closeCompletions(); + syncHighlight(); + storeEditor(); +} - return out.replace(/[ \t]+$/gm, "").trim(); +function closeCompletions() { + $("ac").hidden = true; + completions = []; } +$("ac").addEventListener("mousedown", (e) => { + // Ahead of blur, or the popup would close before the click registered. + e.preventDefault(); + const row = e.target.closest(".ac-row"); + if (!row) return; + completionAt = Number(row.dataset.i); + acceptCompletion(); +}); + +editor.addEventListener("blur", closeCompletions); +editor.addEventListener("click", closeCompletions); + + // --------------------------------------------------------------------------- // Status and results // --------------------------------------------------------------------------- @@ -393,96 +433,181 @@ function formatCypher(src) { // where it can be several lines long and can carry the did-you-mean hint; the banner only says // that the run failed. `kind` is "", "busy", "ok", or "err". function setStatus(kind, text) { - const banner = $("status"); - banner.className = kind ? `banner ${kind}` : "banner"; - banner.innerHTML = - kind === "busy" ? `${esc(text)}` : esc(text); + const banner = $("status"); + banner.className = kind ? `banner ${kind}` : "banner"; + banner.innerHTML = + kind === "busy" ? `${esc(text)}` : esc(text); } function setMeta(text) { - $("result-meta").textContent = text; + $("result-meta").textContent = text; } const plural = (n, word) => `${n} ${word}${n === 1 ? "" : "s"}`; function showPane(name) { - for (const tab of document.querySelectorAll(".tab")) { - tab.setAttribute("aria-selected", String(tab.dataset.pane === name)); - } - for (const pane of document.querySelectorAll(".pane")) { - pane.classList.toggle("on", pane.id === `pane-${name}`); - } - if (name === "graph") { - if (snapshotStale) loadSnapshot(); - drawGraph(); - } + for (const tab of document.querySelectorAll(".tab")) { + tab.setAttribute("aria-selected", String(tab.dataset.pane === name)); + } + for (const pane of document.querySelectorAll(".pane")) { + pane.classList.toggle("on", pane.id === `pane-${name}`); + } + if (name === "graph") { + if (snapshotStale) loadSnapshot().then(drawGraph); + else drawGraph(); + } } for (const tab of document.querySelectorAll(".tab")) { - tab.addEventListener("click", () => showPane(tab.dataset.pane)); + tab.addEventListener("click", () => showPane(tab.dataset.pane)); } // One long property would otherwise set the width of its whole column. The full value stays // reachable through the tooltip, the JSON tab, and both downloads. function clip(text) { - if (text.length <= MAX_CELL_CHARS) return esc(text); - const shown = `${esc(text.slice(0, MAX_CELL_CHARS))}…`; - if (text.length > MAX_TITLE_CHARS) return shown; - return `${shown}`; + if (text.length <= MAX_CELL_CHARS) return esc(text); + const shown = `${esc(text.slice(0, MAX_CELL_CHARS))}…`; + if (text.length > MAX_TITLE_CHARS) return shown; + return `${shown}`; } function cell(value) { - if (value === null || value === undefined) return 'null'; - if (typeof value === "string") return `${clip(value)}`; - if (typeof value === "number") return `${value}`; - if (typeof value === "boolean") return `${value}`; - return `${clip(JSON.stringify(value))}`; + if (value === null || value === undefined) return 'null'; + if (typeof value === "string") return `${clip(value)}`; + if (typeof value === "number") return `${value}`; + if (typeof value === "boolean") return `${value}`; + return `${clip(JSON.stringify(value))}`; } function renderTable(result) { - const pane = $("pane-table"); - setMeta(`${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.`); - if (result.columns.length === 0) { + const pane = $("pane-table"); + setMeta(`${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.`); + if (result.columns.length === 0) { + pane.innerHTML = + '
The statement returned no columns. Writes report nothing unless the statement ends in RETURN.
'; + return; + } + if (result.rows.length === 0) { + pane.innerHTML = `
No rows. The query is valid and matched nothing.
`; + return; + } + const shown = result.rows.slice(0, MAX_TABLE_ROWS); + const head = result.columns.map((c) => `${esc(c)}`).join(""); + const body = shown + .map( + (row, i) => + `${i + 1}${row.map((v) => `${cell(v)}`).join("")}`, + ) + .join(""); + const capped = + result.rows.length > shown.length + ? `
Showing the first ${MAX_TABLE_ROWS} of ${result.rows.length} rows. ` + + `The CSV and JSON downloads include every row.
` + : ""; pane.innerHTML = - '
The statement returned no columns. Writes report nothing unless the statement ends in RETURN.
'; - return; - } - if (result.rows.length === 0) { - pane.innerHTML = `
No rows. The query is valid and matched nothing.
`; - return; - } - const shown = result.rows.slice(0, MAX_TABLE_ROWS); - const head = result.columns.map((c) => `${esc(c)}`).join(""); - const body = shown - .map( - (row, i) => - `${i + 1}${row.map((v) => `${cell(v)}`).join("")}`, - ) - .join(""); - const capped = - result.rows.length > shown.length - ? `
Showing the first ${MAX_TABLE_ROWS} of ${result.rows.length} rows. ` + - `The CSV and JSON downloads include every row.
` - : ""; - pane.innerHTML = - `${head}${body}
` + capped; + `${head}${body}
` + capped; } // The JSON tab is one string too, so it takes the same cap. Values are not clipped here, since // this is the pane a reader opens to see a value the table clipped. function renderJson(result) { - const shown = result.rows.slice(0, MAX_TABLE_ROWS); - const note = - result.rows.length > shown.length - ? `// Showing the first ${MAX_TABLE_ROWS} of ${result.rows.length} rows. The JSON download includes every row.\n` - : ""; - $("pane-json").innerHTML = `
${esc(note + JSON.stringify(shown, null, 2))}
`; + const shown = result.rows.slice(0, MAX_TABLE_ROWS); + const note = + result.rows.length > shown.length + ? `// Showing the first ${MAX_TABLE_ROWS} of ${result.rows.length} rows. The JSON download includes every row.\n` + : ""; + $("pane-json").innerHTML = `
${esc(note + JSON.stringify(shown, null, 2))}
`; +} + +// --------------------------------------------------------------------------- +// Query plan +// --------------------------------------------------------------------------- + +// What each operator says about how the query will run. The engine's plan text names the operator +// first on every line, so the name alone is enough to classify it, and an operator missing from +// here still renders with no badge rather than breaking the tree. +// +// `kernel` is the interesting one: those operators mean the query stopped being a row pipeline and +// became a single pass over the adjacency arrays, which is usually the difference between +// milliseconds and seconds. Nothing else in the page ever showed that. +const PLAN_ROLES = { + PathCount: "kernel", + GroupedDegree: "kernel", + TriangleCount: "kernel", + ExpandIntersect: "kernel", + VectorTopK: "kernel", + NodeIndexScan: "index", + NodeRangeScan: "index", + NodeByIdSeek: "index", + CorrelatedIndexSeek: "index", + LabelScan: "scan", + AllNodesScan: "scan", + HashJoin: "join", + MultiwayJoin: "join", + Expand: "expand", + ExpandInto: "expand", +}; + +const ROLE_TITLE = { + kernel: "Runs as a counting kernel over the adjacency arrays, not as a row pipeline", + index: "Seeks an index instead of scanning", + scan: "Reads every node carrying the label", + join: "Joins two branches", + expand: "Walks the adjacency one hop", + pruned: "Provably empty: the optimizer proved this hop returns nothing", +}; + +function parsePlan(text) { + const rows = []; + for (const line of text.split("\n")) { + if (!line.trim()) continue; + const depth = (line.match(/^ */)[0].length / 2) | 0; + const [, op, detail = ""] = line.trim().match(/^(\S+)\s*(.*)$/); + // A `Limit` with a zero count is the type-inference pass reporting that it proved the pattern + // unsatisfiable, which is worth saying out loud rather than leaving as an odd-looking bound. + const role = /^Limit\b/.test(op) && /\bcount=0\b/.test(detail) ? "pruned" : PLAN_ROLES[op]; + rows.push({depth, op, detail, role}); + } + return rows; +} + +function renderPlan(text) { + const rows = parsePlan(text); + const host = $("pane-plan"); + if (rows.length === 0) { + host.innerHTML = '
No plan: this statement is not a query.
'; + return; + } + + const body = rows + .map(({depth, op, detail, role}) => { + const badge = role ? `` : ""; + return ( + `
  • ${badge}` + + `${esc(op)}` + + (detail ? ` ${esc(detail)}` : "") + + "
    • " + ); + }) + .join(""); + + const fast = [...new Set(rows.filter((r) => r.role === "kernel").map((r) => r.op))]; + const pruned = rows.some((r) => r.role === "pruned"); + let summary = `${plural(rows.length, "operator")}.`; + if (fast.length > 0) summary += ` Lowered to ${fast.join(", ")}.`; + if (pruned) summary += " One branch was proved empty and will not run."; + + host.innerHTML = + `
    ${esc(summary)}
    ` + + `
      ${body}
    ` + + `
    Plan as text` + + `
    ${esc(text)}
    `; } function showError(message) { - $("pane-table").innerHTML = `
    ${esc(message)}
    `; - setMeta("No results."); - showPane("table"); + $("pane-table").innerHTML = `
    ${esc(message)}
    `; + setMeta("No results."); + showPane("table"); } // --------------------------------------------------------------------------- @@ -496,98 +621,122 @@ let busy = false; const MAY_WRITE = /\b(CREATE|MERGE|SET|DELETE|DETACH|REMOVE|COPY|IMPORT|DROP)\b/i; async function run(mode = "run") { - // The disabled Run button does not cover the keyboard shortcut, the Explain button, or a - // demo click, so two runs could interleave and the slower one's panes would win. - if (!db || busy) return; - const cypher = editor.value.trim(); - if (!cypher) return; - - busy = true; - $("run").disabled = true; - setStatus("busy", "Running…"); - // Execution is synchronous inside the module, so this is the only chance the browser gets - // to paint the disabled button before the thread blocks. - await new Promise((r) => setTimeout(r, 0)); - - try { - if (mode === "explain") { - const plan = db.explain(cypher); - $("pane-plan").innerHTML = `
    ${esc(plan)}
    `; - setStatus("ok", "Plan generated."); - setMeta("Physical plan. The query was not executed."); - showPane("plan"); - remember(cypher); - return; - } - - const started = performance.now(); - const result = JSON.parse(db.query(cypher)); - const wall = performance.now() - started; - lastResult = result; - - renderTable(result); - renderJson(result); + // The disabled Run button does not cover the keyboard shortcut, the Explain button, or a + // demo click, so two runs could interleave and the slower one's panes would win. + if (!ready || busy) return; + const cypher = editor.value.trim(); + if (!cypher) return; + + setBusy(true); + setStatus("busy", "Running…"); try { - $("pane-plan").innerHTML = `
    ${esc(db.explain(cypher))}
    `; - } catch { - $("pane-plan").innerHTML = - '
    No plan: this statement is not a query.
    '; - } + if (mode === "explain") { + renderPlan(await engine.explain(cypher)); + setStatus("ok", "Plan generated."); + setMeta("Physical plan. The query was not executed."); + showPane("plan"); + remember(cypher); + return; + } - const multi = - result.statement_count > 1 - ? ` ${result.statement_count} statements ran; this is the last one's result.` - : ""; - setStatus("ok", "Query finished."); - setMeta( - `${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.` + - ` Query took ${result.elapsed_ms.toFixed(2)} ms` + - ` (${wall.toFixed(1)} ms including the round trip).${multi}`, - ); - showPane("table"); - remember(cypher); - // `stats` is a full node scan and an adjacency walk. A read-only statement cannot change - // what it reports, so only a statement that might have written is worth rescanning for. - const mayWrite = MAY_WRITE.test(cypher); - if (mayWrite) { - refreshSchema(); - rememberSetup(cypher); - } - await refreshGraph(mayWrite); - renderFooter(); + const started = performance.now(); + const result = JSON.parse(await engine.query(cypher)); + const wall = performance.now() - started; + lastResult = result; - if (pendingDemo) { - const demo = pendingDemo; - pendingDemo = null; - if (demo.embed) embedLabel(demo.embed); - if (demo.thenQuery) { - if (demo.textIndex) db.createTextIndex(demo.textIndex[0], demo.textIndex[1]); - await runThenQuery(demo.thenQuery); - } else if (demo.textSearch) { - await runTextDemo(demo); - } else if (demo.vectors) { - await runVectorDemo(demo.vectors); - } - } - } catch (e) { - // Cleared, or the export buttons would hand back the previous query's rows while the - // table shows this one's error. - lastResult = null; - const message = String(e.message ?? e); - showError(message + procedureHint(cypher, message)); - setStatus("err", "Query failed."); - } finally { - busy = false; - $("run").disabled = false; - } + renderTable(result); + renderJson(result); + + try { + renderPlan(await engine.explain(cypher)); + } catch { + $("pane-plan").innerHTML = + '
    No plan: this statement is not a query.
    '; + } + + const multi = + result.statement_count > 1 + ? ` ${result.statement_count} statements ran; this is the last one's result.` + : ""; + setStatus("ok", "Query finished."); + setMeta( + `${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.` + + ` Query took ${result.elapsed_ms.toFixed(2)} ms` + + ` (${wall.toFixed(1)} ms including the round trip).${multi}`, + ); + showPane("table"); + remember(cypher); + // `stats` is a full node scan and an adjacency walk. A read-only statement cannot change + // what it reports, so only a statement that might have written is worth rescanning for. + const mayWrite = MAY_WRITE.test(cypher); + if (mayWrite) { + await refreshSchema(); + rememberSetup(cypher); + } + await refreshGraph(mayWrite); + await renderFooter(); + + if (pendingDemo) { + const demo = pendingDemo; + pendingDemo = null; + if (demo.embed) await embedLabel(demo.embed); + if (demo.thenQuery) { + if (demo.textIndex) await engine.createTextIndex(demo.textIndex[0], demo.textIndex[1]); + await runThenQuery(demo.thenQuery); + } else if (demo.textSearch) { + await runTextDemo(demo); + } else if (demo.vectors) { + await runVectorDemo(demo.vectors); + } + } + } catch (e) { + // Cleared, or the export buttons would hand back the previous query's rows while the + // table shows this one's error. + lastResult = null; + const message = String(e.message ?? e); + if (message === CANCELLED) { + showError( + `${CANCELLED} The engine was restarted and the ${esc(currentSample().label)} sample` + + " re-seeded, because a WebAssembly call cannot be interrupted any other way.", + ); + setStatus("", "Cancelled."); + } else { + showError(message + procedureHint(cypher, message)); + setStatus("err", "Query failed."); + } + } finally { + setBusy(false); + } } +// Cancel is only reachable while a query is in flight, and the restart it performs is itself a +// series of engine calls, so the button stays disabled until they finish. +function setBusy(value) { + busy = value; + $("run").disabled = value; + $("cancel").disabled = !value; + $("cancel").hidden = !value; +} + +$("cancel").addEventListener("click", async () => { + if (!busy) return; + $("cancel").disabled = true; + setStatus("busy", "Cancelling…"); + try { + await cancelRunningQuery(); + await refreshSchema(); + await refreshGraph(); + } catch (e) { + setStatus("err", `The engine could not be restarted: ${String(e.message ?? e)}`); + } +}); + $("run").addEventListener("click", () => run()); $("explain").addEventListener("click", () => run("explain")); $("clear").addEventListener("click", () => { - setQuery(""); - setStatus("", "Editor cleared."); + setQuery(""); + setStatus("", "Editor cleared."); }); // Generous for a query and small enough that the highlighter, which rebuilds the whole document's @@ -597,57 +746,57 @@ const MAX_LOADED_FILE = 512 * 1024; $("load").addEventListener("click", () => $("load-file").click()); $("load-file").addEventListener("change", async (e) => { - const file = e.target.files?.[0]; - // Cleared so choosing the same file twice fires the event again. - e.target.value = ""; - if (!file) return; - if (file.size > MAX_LOADED_FILE) { - setStatus("err", `${file.name} is ${Math.round(file.size / 1024)} KB; the editor takes 512 KB.`); - return; - } - try { - setQuery(await file.text()); - pendingDemo = null; - setStatus("", `Loaded ${file.name}. Press Execute Query to run it.`); - } catch { - setStatus("err", `${file.name} could not be read.`); - } + const file = e.target.files?.[0]; + // Cleared so choosing the same file twice fires the event again. + e.target.value = ""; + if (!file) return; + if (file.size > MAX_LOADED_FILE) { + setStatus("err", `${file.name} is ${Math.round(file.size / 1024)} KB; the editor takes 512 KB.`); + return; + } + try { + setQuery(await file.text()); + pendingDemo = null; + setStatus("", `Loaded ${file.name}. Press Execute Query to run it.`); + } catch { + setStatus("err", `${file.name} could not be read.`); + } }); $("download").addEventListener("click", () => { - const cypher = editor.value; - if (!cypher.trim()) { - setStatus("", "Nothing to download: the editor is empty."); - return; - } - download("issundb-query.cypher", "text/plain;charset=utf-8", cypher); - setStatus("", "Saved issundb-query.cypher."); + const cypher = editor.value; + if (!cypher.trim()) { + setStatus("", "Nothing to download: the editor is empty."); + return; + } + download("issundb-query.cypher", "text/plain;charset=utf-8", cypher); + setStatus("", "Saved issundb-query.cypher."); }); function formatEditor() { - const before = editor.value; - if (!before.trim()) return; - const after = formatCypher(before); - if (after === before) { - setStatus("", "Already formatted."); - return; - } - setQuery(after); - setStatus("", "Formatted."); + const before = editor.value; + if (!before.trim()) return; + const after = formatCypher(before); + if (after === before) { + setStatus("", "Already formatted."); + return; + } + setQuery(after); + setStatus("", "Formatted."); } $("format").addEventListener("click", formatEditor); // Loaded into the editor rather than executed, so the statement is read before it writes. Running // it on a database that already holds the sample adds a second copy, which the caption says. $("load-sample").addEventListener("click", () => { - const sample = currentSample(); - setQuery(sample.cypher); - pendingDemo = null; - setStatus( - "", - `Loaded the ${sample.label} sample. Press Execute Query to create it.` + - " Running it on a database that already has it adds a second copy.", - ); + const sample = currentSample(); + setQuery(sample.cypher); + pendingDemo = null; + setStatus( + "", + `Loaded the ${sample.label} sample. Press Execute Query to create it.` + + " Running it on a database that already has it adds a second copy.", + ); }); // --------------------------------------------------------------------------- @@ -659,38 +808,38 @@ $("load-sample").addEventListener("click", () => { const MAX_HISTORY_ITEMS = 10; function readHistory() { - try { - const stored = JSON.parse(localStorage.getItem(HISTORY_KEY) ?? "[]"); - return Array.isArray(stored) ? stored.slice(0, MAX_HISTORY_ITEMS) : []; - } catch { - return []; - } + try { + const stored = JSON.parse(localStorage.getItem(HISTORY_KEY) ?? "[]"); + return Array.isArray(stored) ? stored.slice(0, MAX_HISTORY_ITEMS) : []; + } catch { + return []; + } } function remember(cypher) { - const history = readHistory().filter((q) => q !== cypher); - history.unshift(cypher); - try { - localStorage.setItem(HISTORY_KEY, JSON.stringify(history.slice(0, MAX_HISTORY_ITEMS))); - } catch { - // A browser refusing storage must not fail the query. - } - renderHistory(); + const history = readHistory().filter((q) => q !== cypher); + history.unshift(cypher); + try { + localStorage.setItem(HISTORY_KEY, JSON.stringify(history.slice(0, MAX_HISTORY_ITEMS))); + } catch { + // A browser refusing storage must not fail the query. + } + renderHistory(); } function renderHistory() { - const history = readHistory(); - $("history").innerHTML = history.length - ? "" - : '
    Queries you run appear here.
    '; - for (const cypher of history) { - const button = document.createElement("button"); - button.className = "hist"; - button.textContent = cypher.replace(/\s+/g, " ").slice(0, 70); - button.title = cypher; - button.addEventListener("click", () => setQuery(cypher)); - $("history").append(button); - } + const history = readHistory(); + $("history").innerHTML = history.length + ? "" + : '
    Queries you run appear here.
    '; + for (const cypher of history) { + const button = document.createElement("button"); + button.className = "hist"; + button.textContent = cypher.replace(/\s+/g, " ").slice(0, 70); + button.title = cypher; + button.addEventListener("click", () => setQuery(cypher)); + $("history").append(button); + } } // --------------------------------------------------------------------------- @@ -703,9 +852,9 @@ function renderHistory() { const setupLog = []; function rememberSetup(cypher) { - // The log is replayed as one semicolon-separated statement, so a statement that already ends - // in one would contribute an empty statement between two real ones. - setupLog.push(cypher.replace(/;\s*$/, "")); + // The log is replayed as one semicolon-separated statement, so a statement that already ends + // in one would contribute an empty statement between two real ones. + setupLog.push(cypher.replace(/;\s*$/, "")); } // --------------------------------------------------------------------------- @@ -713,58 +862,79 @@ function rememberSetup(cypher) { // --------------------------------------------------------------------------- const procedureNames = new Set(PROCEDURES.flatMap((p) => [p.name, p.aka].filter(Boolean))); +const functionNames = new Set(FUNCTIONS.map((f) => f.name)); + +// A procedure yields rows and a function returns a value, so the two are listed apart rather than +// merged behind one word that would be wrong for half of them. +const REFERENCE = [ + ...PROCEDURES.map((entry) => ({...entry, kind: "procedure"})), + ...FUNCTIONS.map((entry) => ({...entry, kind: "function"})), +]; + +let referenceKind = "all"; function renderProcedures(filter = "") { - const needle = filter.trim().toLowerCase(); - const host = $("proc-list"); - const matches = PROCEDURES.filter( - (proc) => - !needle || - `${proc.name} ${proc.aka ?? ""} ${proc.args} ${proc.yields} ${proc.summary}` - .toLowerCase() - .includes(needle), - ); - host.replaceChildren(); - if (matches.length === 0) { - host.innerHTML = '
    No procedure matches.
    '; - return; - } - for (const proc of matches) { - const button = document.createElement("button"); - button.className = "proc"; - // The signature is in the tooltip rather than the row. In a sidebar this narrow a form like - // `issundb.pageRank([{iterations, damping}])` wraps mid-identifier, which is harder to scan - // than the name alone, and clicking inserts the call anyway. - const signature = `${proc.name}(${proc.args})`; - button.title = proc.aka - ? `${signature}\n\n${proc.summary}\n\nAlso registered as ${proc.aka}.` - : `${signature}\n\n${proc.summary}`; - const name = document.createElement("span"); - name.className = "nm"; - name.textContent = proc.name; - const yields = document.createElement("span"); - yields.className = "yd"; - yields.textContent = `yields ${proc.yields}`; - button.append(name, yields); - button.addEventListener("click", () => setQuery(proc.snippet)); - host.append(button); - } + const needle = filter.trim().toLowerCase(); + const host = $("proc-list"); + const matches = REFERENCE.filter( + (entry) => + (referenceKind === "all" || entry.kind === referenceKind) && + (!needle || + `${entry.name} ${entry.aka ?? ""} ${entry.args} ${entry.yields} ${entry.summary}` + .toLowerCase() + .includes(needle)), + ); + host.replaceChildren(); + if (matches.length === 0) { + host.innerHTML = '
    Nothing matches.
    '; + return; + } + for (const entry of matches) { + const button = document.createElement("button"); + button.className = "proc"; + // The signature is in the tooltip rather than the row. In a sidebar this narrow a form like + // `issundb.pageRank([{iterations, damping}])` wraps mid-identifier, which is harder to scan + // than the name alone, and clicking inserts the call anyway. + const signature = `${entry.name}(${entry.args})`; + button.title = entry.aka + ? `${signature}\n\n${entry.summary}\n\nAlso registered as ${entry.aka}.` + : `${signature}\n\n${entry.summary}`; + const name = document.createElement("span"); + name.className = "nm"; + name.textContent = entry.name; + const yields = document.createElement("span"); + yields.className = "yd"; + yields.textContent = entry.kind === "function" ? `returns ${entry.yields}` : `yields ${entry.yields}`; + button.append(name, yields); + button.addEventListener("click", () => setQuery(entry.snippet)); + host.append(button); + } +} + +for (const chip of document.querySelectorAll("#proc-kind .chip")) { + chip.addEventListener("click", () => { + referenceKind = chip.dataset.kind; + for (const other of document.querySelectorAll("#proc-kind .chip")) { + other.setAttribute("aria-pressed", String(other === chip)); + } + renderProcedures($("proc-search").value); + }); } // Iterative over two rows, so the whole matrix is never held. function editDistance(a, b) { - let previous = Array.from({ length: b.length + 1 }, (_, j) => j); - for (let i = 1; i <= a.length; i += 1) { - const current = [i]; - for (let j = 1; j <= b.length; j += 1) { - current[j] = - a[i - 1] === b[j - 1] - ? previous[j - 1] - : 1 + Math.min(previous[j - 1], previous[j], current[j - 1]); + let previous = Array.from({length: b.length + 1}, (_, j) => j); + for (let i = 1; i <= a.length; i += 1) { + const current = [i]; + for (let j = 1; j <= b.length; j += 1) { + current[j] = + a[i - 1] === b[j - 1] + ? previous[j - 1] + : 1 + Math.min(previous[j - 1], previous[j], current[j - 1]); + } + previous = current; } - previous = current; - } - return previous[b.length]; + return previous[b.length]; } // `ProcedureNotFound` says the name is wrong without saying what was meant, and the difference is @@ -772,24 +942,27 @@ function editDistance(a, b) { // suggestion is exactly as complete as the catalog is; a procedure missing from it gets no hint // rather than a wrong one. function procedureHint(cypher, message) { - if (!/ProcedureNotFound/.test(message)) return ""; - for (const token of new Set(cypher.match(/\bissundb\.[A-Za-z_][\w.]*/g) ?? [])) { - if (procedureNames.has(token)) continue; - let best = null; - // Further than three edits apart the suggestion is noise rather than a correction. - let bestDistance = 4; - for (const name of procedureNames) { - const distance = editDistance(token.toLowerCase(), name.toLowerCase()); - if (distance < bestDistance) { - bestDistance = distance; - best = name; - } - } - if (best) return `\n\nThere is no ${token}. Did you mean ${best}?`; - } - return ""; + if (!/ProcedureNotFound/.test(message)) return ""; + for (const token of new Set(cypher.match(/\bissundb\.[A-Za-z_][\w.]*/g) ?? [])) { + if (procedureNames.has(token)) continue; + let best = null; + // Further than three edits apart the suggestion is noise rather than a correction. + let bestDistance = 4; + for (const name of procedureNames) { + const distance = editDistance(token.toLowerCase(), name.toLowerCase()); + if (distance < bestDistance) { + bestDistance = distance; + best = name; + } + } + if (best) return `\n\nThere is no ${token}. Did you mean ${best}?`; + } + return ""; } +// The count comes from the catalog rather than the markup, which carried a stale 13 for as long as +// the catalog had more than that. +$("proc-search").placeholder = `Search ${PROCEDURES.length} procedures…`; $("proc-search").addEventListener("input", (e) => renderProcedures(e.target.value)); // --------------------------------------------------------------------------- @@ -808,175 +981,175 @@ let pendingDemo = null; // once that graph is loaded. The label a category names is how the page tells, out of the schema it // already has, and says so instead of leaving an empty table to be read as a fault. function graphIsLoaded(category) { - if (!category?.requiresLabel) return true; - return Boolean(lastStats?.label_counts?.[category.requiresLabel]); + if (!category?.requiresLabel) return true; + return Boolean(lastStats?.label_counts?.[category.requiresLabel]); } const sampleLabel = (id) => SAMPLE_GRAPHS.find((sample) => sample.id === id)?.label ?? id; function selectDemo(index) { - const category = DEMO_CATEGORIES[activeCategory]; - const demo = category?.demos[index]; - if (!demo) return; - for (const other of document.querySelectorAll(".demo")) { - other.classList.toggle("active", Number(other.dataset.index) === index); - } - setQuery(demo.cypher); - pendingDemo = - demo.textSearch || demo.vectors || demo.thenQuery || demo.embed ? demo : null; - - if (!graphIsLoaded(category)) { + const category = DEMO_CATEGORIES[activeCategory]; + const demo = category?.demos[index]; + if (!demo) return; + for (const other of document.querySelectorAll(".demo")) { + other.classList.toggle("active", Number(other.dataset.index) === index); + } + setQuery(demo.cypher); + pendingDemo = + demo.textSearch || demo.vectors || demo.thenQuery || demo.embed ? demo : null; + + if (!graphIsLoaded(category)) { + setStatus( + "", + `Loaded "${demo.label}". It queries the ${sampleLabel(category.sample)} graph, which is not` + + " in the database: press Reset Graph to load it, then Execute Query.", + ); + return; + } setStatus( - "", - `Loaded "${demo.label}". It queries the ${sampleLabel(category.sample)} graph, which is not` + - " in the database: press Reset Graph to load it, then Execute Query.", + "", + demo.explain + ? `Loaded "${demo.label}". Press Explain to see the plan.` + : `Loaded "${demo.label}". Press Execute Query to run it.`, ); - return; - } - setStatus( - "", - demo.explain - ? `Loaded "${demo.label}". Press Explain to see the plan.` - : `Loaded "${demo.label}". Press Execute Query to run it.`, - ); } function renderCategory() { - const category = DEMO_CATEGORIES[activeCategory]; - // The picker follows the category, so loading the graph these examples want is one press of Reset - // Graph rather than a hunt through the list. Nothing runs here: this moves a dropdown. - const wanted = SAMPLE_GRAPHS.findIndex((sample) => sample.id === category.sample); - if (wanted >= 0) { - activeSample = wanted; - $("sample-graph").value = String(wanted); - } - const host = $("demo-buttons"); - host.replaceChildren(); - category.demos.forEach((demo, i) => { - const button = document.createElement("button"); - button.className = "demo"; - button.textContent = demo.label; - button.dataset.index = String(i); - button.title = demo.desc; - button.addEventListener("click", () => selectDemo(i)); - host.append(button); - }); - const link = $("category-docs"); - if (category.docs) { - link.href = category.docs; - link.textContent = `Read more: ${category.label}`; - link.hidden = false; - } else { - link.hidden = true; - } + const category = DEMO_CATEGORIES[activeCategory]; + // The picker follows the category, so loading the graph these examples want is one press of Reset + // Graph rather than a hunt through the list. Nothing runs here: this moves a dropdown. + const wanted = SAMPLE_GRAPHS.findIndex((sample) => sample.id === category.sample); + if (wanted >= 0) { + activeSample = wanted; + $("sample-graph").value = String(wanted); + } + const host = $("demo-buttons"); + host.replaceChildren(); + category.demos.forEach((demo, i) => { + const button = document.createElement("button"); + button.className = "demo"; + button.textContent = demo.label; + button.dataset.index = String(i); + button.title = demo.desc; + button.addEventListener("click", () => selectDemo(i)); + host.append(button); + }); + const link = $("category-docs"); + if (category.docs) { + link.href = category.docs; + link.textContent = `Read more: ${category.label}`; + link.hidden = false; + } else { + link.hidden = true; + } } function renderDemos() { - const select = $("demo-category"); - DEMO_CATEGORIES.forEach((category, i) => { - const option = document.createElement("option"); - option.value = String(i); - option.textContent = category.label; - select.append(option); - }); - select.addEventListener("change", () => { - activeCategory = Number(select.value); + const select = $("demo-category"); + DEMO_CATEGORIES.forEach((category, i) => { + const option = document.createElement("option"); + option.value = String(i); + option.textContent = category.label; + select.append(option); + }); + select.addEventListener("change", () => { + activeCategory = Number(select.value); + renderCategory(); + }); renderCategory(); - }); - renderCategory(); } async function runTextDemo(demo) { - const [label, property] = demo.textIndex; - try { - db.createTextIndex(label, property); - const { hits } = JSON.parse(db.textSearch(demo.textSearch, 10)); - const rows = []; - for (const hit of hits) { - const title = JSON.parse(db.query(`MATCH (a) WHERE id(a) = ${hit.node} RETURN a.title`)); - rows.push([hit.node, title.rows[0]?.[0] ?? null, Number(hit.score.toFixed(4)), hit.property]); - } - lastResult = { columns: ["node", "title", "bm25", "field"], rows }; - renderTable(lastResult); - setStatus("ok", "Full-text search finished."); - setMeta( - `${plural(rows.length, "hit")}, 4 columns.` + - ` BM25 over the ${label}.${property} index for "${demo.textSearch}".`, - ); - showPane("table"); - } catch (e) { - showError(String(e.message ?? e)); - } + const [label, property] = demo.textIndex; + try { + await engine.createTextIndex(label, property); + const {hits} = JSON.parse(await engine.textSearch(demo.textSearch, 10)); + const rows = []; + for (const hit of hits) { + const title = JSON.parse(await engine.query(`MATCH (a) WHERE id(a) = ${hit.node} RETURN a.title`)); + rows.push([hit.node, title.rows[0]?.[0] ?? null, Number(hit.score.toFixed(4)), hit.property]); + } + lastResult = {columns: ["node", "title", "bm25", "field"], rows}; + renderTable(lastResult); + setStatus("ok", "Full-text search finished."); + setMeta( + `${plural(rows.length, "hit")}, 4 columns.` + + ` BM25 over the ${label}.${property} index for "${demo.textSearch}".`, + ); + showPane("table"); + } catch (e) { + showError(String(e.message ?? e)); + } } // Places each node of a label on a circle, so "nearest" has a meaning the table can be checked // against by eye. A node id is a u64, which wasm-bindgen takes as a BigInt. Returns the ids with // their captions, so the search that follows can name its hits. -function embedLabel(spec) { - const label = spec.label ?? "Person"; - const caption = spec.caption ?? "name"; - const rows = JSON.parse( - db.query(`MATCH (n:${label}) RETURN id(n) AS id, n.${caption} AS caption ORDER BY id`), - ).rows; - rows.forEach(([id], i) => { - const angle = (i / Math.max(rows.length, 1)) * Math.PI * 2; - db.upsertVector(BigInt(id), new Float32Array([Math.cos(angle), Math.sin(angle), 0.25])); - }); - return { label, rows }; +async function embedLabel(spec) { + const label = spec.label ?? "Person"; + const caption = spec.caption ?? "name"; + const rows = JSON.parse( + await engine.query(`MATCH (n:${label}) RETURN id(n) AS id, n.${caption} AS caption ORDER BY id`), + ).rows; + for (const [i, [id]] of rows.entries()) { + const angle = (i / Math.max(rows.length, 1)) * Math.PI * 2; + await engine.upsertVector(BigInt(id), new Float32Array([Math.cos(angle), Math.sin(angle), 0.25])); + } + return {label, rows}; } async function runVectorDemo(spec) { - try { - const { label, rows } = embedLabel(spec); - if (rows.length === 0) { - showError(`No ${label} nodes to embed. Run the example's own CREATE first.`); - return; + try { + const {label, rows} = await embedLabel(spec); + if (rows.length === 0) { + showError(`No ${label} nodes to embed. Run the example's own CREATE first.`); + return; + } + const {hits} = JSON.parse( + await engine.vectorSearch(new Float32Array([1, 0, 0.25]), Math.min(5, rows.length)), + ); + const captions = new Map(rows.map(([id, caption]) => [id, caption])); + lastResult = { + columns: ["rank", "node", "label", "distance"], + rows: hits.map((h, i) => [ + i + 1, + h.node, + captions.get(h.node) ?? null, + Number(h.distance.toFixed(5)), + ]), + }; + renderTable(lastResult); + setStatus("ok", "Vector search finished."); + setMeta( + `${plural(hits.length, "neighbour")}, 4 columns.` + + ` Exact search over ${plural(rows.length, `${label} embedding`)} for [1, 0, 0.25].`, + ); + showPane("table"); + } catch (e) { + showError(String(e.message ?? e)); } - const { hits } = JSON.parse( - db.vectorSearch(new Float32Array([1, 0, 0.25]), Math.min(5, rows.length)), - ); - const captions = new Map(rows.map(([id, caption]) => [id, caption])); - lastResult = { - columns: ["rank", "node", "label", "distance"], - rows: hits.map((h, i) => [ - i + 1, - h.node, - captions.get(h.node) ?? null, - Number(h.distance.toFixed(5)), - ]), - }; - renderTable(lastResult); - setStatus("ok", "Vector search finished."); - setMeta( - `${plural(hits.length, "neighbour")}, 4 columns.` + - ` Exact search over ${plural(rows.length, `${label} embedding`)} for [1, 0, 0.25].`, - ); - showPane("table"); - } catch (e) { - showError(String(e.message ?? e)); - } } // A query that needs embeddings or a text index in place before it can run, so it cannot be part of // the example's own statement. `issundb.retrieve.hybrid` is the case this exists for. async function runThenQuery(cypher) { - try { - const result = JSON.parse(db.query(cypher)); - lastResult = result; - renderTable(result); - renderJson(result); - setStatus("ok", "Query finished."); - setMeta( - `${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.` + - ` Query took ${result.elapsed_ms.toFixed(2)} ms, after the example put its index and` - + " embeddings in place.", - ); - showPane("table"); - await refreshGraph(false); - } catch (e) { - showError(String(e.message ?? e)); - setStatus("err", "The follow-up query failed."); - } + try { + const result = JSON.parse(await engine.query(cypher)); + lastResult = result; + renderTable(result); + renderJson(result); + setStatus("ok", "Query finished."); + setMeta( + `${plural(result.rows.length, "row")}, ${plural(result.columns.length, "column")}.` + + ` Query took ${result.elapsed_ms.toFixed(2)} ms, after the example put its index and` + + " embeddings in place.", + ); + showPane("table"); + await refreshGraph(false); + } catch (e) { + showError(String(e.message ?? e)); + setStatus("err", "The follow-up query failed."); + } } // --------------------------------------------------------------------------- @@ -990,7 +1163,6 @@ let lastStats = null; // The module's exports, for the one figure only the browser knows: how much WebAssembly heap it has // committed. That number never falls, so it is reported beside the live one rather than instead of // it. -let wasmExports = null; // Live bytes with an empty database, captured after the instance is built and before it is seeded. // Subtracting it is what separates the graph from the engine that holds it. @@ -999,120 +1171,120 @@ let baselineBytes = 0; // KB below a megabyte, because a small graph is a few hundred kilobytes and reporting it as 0.1 MB // says less than 143 KB does. function bytesLabel(bytes) { - if (bytes < 1024) return `${bytes} B`; - if (bytes < 1048576) return `${Math.round(bytes / 1024)} KB`; - return `${(bytes / 1048576).toFixed(1)} MB`; -} - -function renderFooter() { - const note = $("footer-note"); - const counts = lastStats - ? `${plural(lastStats.nodes, "node")} and ${plural(lastStats.edges, "edge")}` - : ""; - - let live = 0; - try { - live = Playground.memoryBytes(); - } catch { - // A build without the counter still reports the counts. - } - const heap = wasmExports?.memory?.buffer?.byteLength ?? 0; - if (live === 0) { - note.textContent = counts; - note.title = ""; - return; - } - - // Two figures rather than three. Splitting the live total into engine and graph was the first - // shape this took, and it reported the same number twice: an empty database allocates a few - // kilobytes, so the graph is very nearly all of it. The baseline is stated in the tooltip - // instead, where it says that rather than implying a division that does not exist. - note.textContent = `${counts} · ${bytesLabel(live)} in use, ${bytesLabel(heap)} heap`; - note.title = - `IssunDB has ${bytesLabel(live)} allocated and not freed. An empty database accounts for` + - ` ${bytesLabel(baselineBytes)} of that, so the rest is graph data and the structures derived` + - ` from it. The WebAssembly heap the browser has committed is ${bytesLabel(heap)}, which only` + - " ever grows, so it stays above the figure in use."; + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1048576) return `${Math.round(bytes / 1024)} KB`; + return `${(bytes / 1048576).toFixed(1)} MB`; +} + +async function renderFooter() { + const note = $("footer-note"); + const counts = lastStats + ? `${plural(lastStats.nodes, "node")} and ${plural(lastStats.edges, "edge")}` + : ""; + + let live = 0; + let heap = 0; + try { + ({live, heap, baseline: baselineBytes} = await engine.memory()); + } catch { + // A build without the counter still reports the counts. + } + if (live === 0) { + note.textContent = counts; + note.title = ""; + return; + } + + // Two figures rather than three. Splitting the live total into engine and graph was the first + // shape this took, and it reported the same number twice: an empty database allocates a few + // kilobytes, so the graph is very nearly all of it. The baseline is stated in the tooltip + // instead, where it says that rather than implying a division that does not exist. + note.textContent = `| ${counts} | ${bytesLabel(live)} in use | ${bytesLabel(heap)} heap |`; + note.title = + `IssunDB has ${bytesLabel(live)} allocated and not freed. An empty database accounts for` + + ` ${bytesLabel(baselineBytes)} of that, so the rest is graph data and the structures derived` + + ` from it. The WebAssembly heap the browser has committed is ${bytesLabel(heap)}, which only` + + " ever grows, so it stays above the figure in use."; } // Hashing the name is what keeps a vertex's color stable across redraws without a table. function hueOf(name) { - let hash = 0; - for (let i = 0; i < name.length; i += 1) { - hash = (hash * 31 + name.charCodeAt(i)) | 0; - } - return Math.abs(hash) % 360; + let hash = 0; + for (let i = 0; i < name.length; i += 1) { + hash = (hash * 31 + name.charCodeAt(i)) | 0; + } + return Math.abs(hash) % 360; } const colorOf = (name) => `hsl(${hueOf(name)} 62% 52%)`; -function refreshSchema() { - let stats; - try { - stats = JSON.parse(db.stats()); - } catch { - return; - } - const rows = []; - for (const [label, n] of Object.entries(stats.label_counts ?? {})) { - rows.push( - `
    ` + - `:${esc(label)}${n}
    `, - ); - } - for (const [type, n] of Object.entries(stats.type_counts ?? {})) { - rows.push( - `
    →` + - `:${esc(type)}${n}
    `, - ); - } - $("schema").innerHTML = rows.length - ? rows.join("") - : '
    Empty database. Run a CREATE.
    '; - lastStats = stats; - renderFooter(); +async function refreshSchema() { + let stats; + try { + stats = JSON.parse(await engine.stats()); + } catch { + return; + } + const rows = []; + for (const [label, n] of Object.entries(stats.label_counts ?? {})) { + rows.push( + `
    ` + + `:${esc(label)}${n}
    `, + ); + } + for (const [type, n] of Object.entries(stats.type_counts ?? {})) { + rows.push( + `
    →` + + `:${esc(type)}${n}
    `, + ); + } + $("schema").innerHTML = rows.length + ? rows.join("") + : '
    Empty database. Run a CREATE.
    '; + lastStats = stats; + renderFooter(); } // --------------------------------------------------------------------------- // Graph view // --------------------------------------------------------------------------- -let snapshot = { nodes: [], edges: [], truncated: false }; +let snapshot = {nodes: [], edges: [], truncated: false}; let snapshotStale = true; -function loadSnapshot() { - let next; - try { - next = JSON.parse(db.graphSnapshot()); - } catch { - next = { nodes: [], edges: [], truncated: false }; - } - // Each surviving node keeps its position, so running a query does not discard a layout the - // user arranged by hand or watched settle. Without this the whole graph re-seeded onto the - // starting circle after every statement. - const previous = new Map(snapshot.nodes.map((node) => [node.id, node])); - for (const node of next.nodes) { - const old = previous.get(node.id); - if (old && old.x !== undefined) { - node.x = old.x; - node.y = old.y; - node.vx = old.vx ?? 0; - node.vy = old.vy ?? 0; - } - } - snapshot = next; - snapshotStale = false; +async function loadSnapshot() { + let next; + try { + next = JSON.parse(await engine.graphSnapshot()); + } catch { + next = {nodes: [], edges: [], truncated: false}; + } + // Each surviving node keeps its position, so running a query does not discard a layout the + // user arranged by hand or watched settle. Without this the whole graph re-seeded onto the + // starting circle after every statement. + const previous = new Map(snapshot.nodes.map((node) => [node.id, node])); + for (const node of next.nodes) { + const old = previous.get(node.id); + if (old && old.x !== undefined) { + node.x = old.x; + node.y = old.y; + node.vx = old.vx ?? 0; + node.vy = old.vy ?? 0; + } + } + snapshot = next; + snapshotStale = false; } // `mayWrite` false means only the highlighting can have changed, so the graph is redrawn // without paying for a fresh scan or another PageRank pass. async function refreshGraph(mayWrite = true) { - // `graphSnapshot` is a full node scan, so paying for it while the graph tab is hidden is - // waste. Switching to the tab loads it instead. - if (mayWrite) snapshotStale = true; - if (!$("pane-graph").classList.contains("on")) return; - if (snapshotStale) loadSnapshot(); - drawGraph(); + // `graphSnapshot` is a full node scan, so paying for it while the graph tab is hidden is + // waste. Switching to the tab loads it instead. + if (mayWrite) snapshotStale = true; + if (!$("pane-graph").classList.contains("on")) return; + if (snapshotStale) await loadSnapshot(); + drawGraph(); } // Velocity-Verlet, with all-pairs repulsion: at the 300-node cap that pass is cheap enough @@ -1120,106 +1292,106 @@ async function refreshGraph(mayWrite = true) { // than read off the element, since the caller needs the same two numbers for the view box and // the two must agree or a fit is computed against a different box than the layout used. function layout(nodes, edges, width, height) { - const index = new Map(nodes.map((n, i) => [n.id, i])); - const links = edges - .map((e) => [index.get(e.source), index.get(e.target)]) - .filter(([a, b]) => a !== undefined && b !== undefined); - - for (const [i, node] of nodes.entries()) { - if (node.x === undefined) { - // Seeded on a circle rather than at random, so a re-layout of the same graph is - // reproducible and the first frame is never a knot at the centre. - const angle = (i / Math.max(nodes.length, 1)) * Math.PI * 2; - const radius = Math.min(width, height) * 0.32; - node.x = width / 2 + Math.cos(angle) * radius; - node.y = height / 2 + Math.sin(angle) * radius; - node.vx = 0; - node.vy = 0; - } - } - - let alpha = 1; - const repulsion = 2600; - const springLength = 78; - const springK = 0.045; - - return function tick() { - alpha *= 0.985; - for (let i = 0; i < nodes.length; i += 1) { - const a = nodes[i]; - for (let j = i + 1; j < nodes.length; j += 1) { - const b = nodes[j]; - let dx = b.x - a.x; - let dy = b.y - a.y; - let d2 = dx * dx + dy * dy; - if (d2 < 0.01) { - // Coincident nodes have no direction to separate along, so nudge them by index - // rather than randomly, which would make a layout unreproducible. - dx = (i - j) * 0.1 + 0.1; - dy = 0.1; - d2 = dx * dx + dy * dy; + const index = new Map(nodes.map((n, i) => [n.id, i])); + const links = edges + .map((e) => [index.get(e.source), index.get(e.target)]) + .filter(([a, b]) => a !== undefined && b !== undefined); + + for (const [i, node] of nodes.entries()) { + if (node.x === undefined) { + // Seeded on a circle rather than at random, so a re-layout of the same graph is + // reproducible and the first frame is never a knot at the centre. + const angle = (i / Math.max(nodes.length, 1)) * Math.PI * 2; + const radius = Math.min(width, height) * 0.32; + node.x = width / 2 + Math.cos(angle) * radius; + node.y = height / 2 + Math.sin(angle) * radius; + node.vx = 0; + node.vy = 0; + } + } + + let alpha = 1; + const repulsion = 2600; + const springLength = 78; + const springK = 0.045; + + return function tick() { + alpha *= 0.985; + for (let i = 0; i < nodes.length; i += 1) { + const a = nodes[i]; + for (let j = i + 1; j < nodes.length; j += 1) { + const b = nodes[j]; + let dx = b.x - a.x; + let dy = b.y - a.y; + let d2 = dx * dx + dy * dy; + if (d2 < 0.01) { + // Coincident nodes have no direction to separate along, so nudge them by index + // rather than randomly, which would make a layout unreproducible. + dx = (i - j) * 0.1 + 0.1; + dy = 0.1; + d2 = dx * dx + dy * dy; + } + const force = repulsion / d2; + const d = Math.sqrt(d2); + const fx = (dx / d) * force; + const fy = (dy / d) * force; + a.vx -= fx; + a.vy -= fy; + b.vx += fx; + b.vy += fy; + } + } + for (const [ai, bi] of links) { + const a = nodes[ai]; + const b = nodes[bi]; + const dx = b.x - a.x; + const dy = b.y - a.y; + const d = Math.hypot(dx, dy) || 0.01; + const force = (d - springLength) * springK; + const fx = (dx / d) * force; + const fy = (dy / d) * force; + a.vx += fx; + a.vy += fy; + b.vx -= fx; + b.vy -= fy; } - const force = repulsion / d2; - const d = Math.sqrt(d2); - const fx = (dx / d) * force; - const fy = (dy / d) * force; - a.vx -= fx; - a.vy -= fy; - b.vx += fx; - b.vy += fy; - } - } - for (const [ai, bi] of links) { - const a = nodes[ai]; - const b = nodes[bi]; - const dx = b.x - a.x; - const dy = b.y - a.y; - const d = Math.hypot(dx, dy) || 0.01; - const force = (d - springLength) * springK; - const fx = (dx / d) * force; - const fy = (dy / d) * force; - a.vx += fx; - a.vy += fy; - b.vx -= fx; - b.vy -= fy; - } - for (const node of nodes) { - node.vx += (width / 2 - node.x) * 0.006; - node.vy += (height / 2 - node.y) * 0.006; - if (node.pinned) { - node.vx = 0; - node.vy = 0; - continue; - } - node.vx *= 0.82; - node.vy *= 0.82; - node.x += node.vx * alpha; - node.y += node.vy * alpha; - const margin = 26; - node.x = Math.max(margin, Math.min(width - margin, node.x)); - node.y = Math.max(margin, Math.min(height - margin, node.y)); - } - return alpha > 0.02; - }; + for (const node of nodes) { + node.vx += (width / 2 - node.x) * 0.006; + node.vy += (height / 2 - node.y) * 0.006; + if (node.pinned) { + node.vx = 0; + node.vy = 0; + continue; + } + node.vx *= 0.82; + node.vy *= 0.82; + node.x += node.vx * alpha; + node.y += node.vy * alpha; + const margin = 26; + node.x = Math.max(margin, Math.min(width - margin, node.x)); + node.y = Math.max(margin, Math.min(height - margin, node.y)); + } + return alpha > 0.02; + }; } const SVG_NS = "http://www.w3.org/2000/svg"; const el = (name, attrs = {}) => { - const node = document.createElementNS(SVG_NS, name); - for (const [k, v] of Object.entries(attrs)) node.setAttribute(k, v); - return node; + const node = document.createElementNS(SVG_NS, name); + for (const [k, v] of Object.entries(attrs)) node.setAttribute(k, v); + return node; }; function labelOf(node) { - return node.labels?.[0] ?? "(none)"; + return node.labels?.[0] ?? "(none)"; } function captionOf(node) { - const props = node.props ?? {}; - for (const key of ["name", "title", "id", "label", "key"]) { - if (typeof props[key] === "string") return props[key]; - } - return `#${node.id}`; + const props = node.props ?? {}; + for (const key of ["name", "title", "id", "label", "key"]) { + if (typeof props[key] === "string") return props[key]; + } + return `#${node.id}`; } // Only these column names are read out of a result, in both cases because guessing from the @@ -1232,206 +1404,213 @@ const GROUP_COLUMNS = new Set(["communityid", "componentid"]); // Fixed rather than hashed off the group id, so two adjacent communities get colors that can be // told apart instead of whatever two hashes happen to land on. const GROUP_PALETTE = [ - "#7e56c2", - "#0b6bcb", - "#1c7c54", - "#c77700", - "#b3261e", - "#00868b", - "#7a5195", - "#556b2f", - "#a0522d", - "#3f51b5", + "#7e56c2", + "#0b6bcb", + "#1c7c54", + "#c77700", + "#b3261e", + "#00868b", + "#7a5195", + "#556b2f", + "#a0522d", + "#3f51b5", ]; function resultOverlay() { - if (!lastResult) return { lit: null, groupIds: null, groupLabel: "" }; - const lower = lastResult.columns.map((c) => c.toLowerCase()); - - const lit = new Set(); - lower.forEach((name, i) => { - if (!NODE_COLUMNS.has(name)) return; - for (const row of lastResult.rows) { - if (Number.isInteger(row[i])) lit.add(row[i]); - } - }); + if (!lastResult) return {lit: null, groupIds: null, groupLabel: ""}; + const lower = lastResult.columns.map((c) => c.toLowerCase()); + + const lit = new Set(); + lower.forEach((name, i) => { + if (!NODE_COLUMNS.has(name)) return; + for (const row of lastResult.rows) { + if (Number.isInteger(row[i])) lit.add(row[i]); + } + }); - const nodeAt = lower.findIndex((name) => NODE_COLUMNS.has(name)); - const groupAt = lower.findIndex((name) => GROUP_COLUMNS.has(name)); - let groupIds = null; - if (nodeAt >= 0 && groupAt >= 0) { - groupIds = new Map(); - for (const row of lastResult.rows) { - if (Number.isInteger(row[nodeAt]) && Number.isInteger(row[groupAt])) { - groupIds.set(row[nodeAt], row[groupAt]); - } + const nodeAt = lower.findIndex((name) => NODE_COLUMNS.has(name)); + const groupAt = lower.findIndex((name) => GROUP_COLUMNS.has(name)); + let groupIds = null; + if (nodeAt >= 0 && groupAt >= 0) { + groupIds = new Map(); + for (const row of lastResult.rows) { + if (Number.isInteger(row[nodeAt]) && Number.isInteger(row[groupAt])) { + groupIds.set(row[nodeAt], row[groupAt]); + } + } + if (groupIds.size === 0) groupIds = null; } - if (groupIds.size === 0) groupIds = null; - } - return { - lit: lit.size ? lit : null, - groupIds, - groupLabel: groupIds ? lastResult.columns[groupAt] : "", - }; + return { + lit: lit.size ? lit : null, + groupIds, + groupLabel: groupIds ? lastResult.columns[groupAt] : "", + }; } // Pointer capture keeps a drag alive once the pointer leaves the element, which improves the gesture // rather than enabling it. It throws when the id is not an active pointer, and letting that escape // abandoned the gesture entirely, since the move listener is attached after this call. function capturePointer(element, pointerId) { - try { - element.setPointerCapture(pointerId); - } catch { - // Without capture the drag still works while the pointer stays over the element. - } + try { + element.setPointerCapture(pointerId); + } catch { + // Without capture the drag still works while the pointer stays over the element. + } } let simGeneration = 0; function drawGraph() { - // A drag whose pointer is released after a redraw calls the previous drawing's `start`, - // which would put a loop over replaced nodes back into the shared handle. - const generation = ++simGeneration; - const svg = $("svg"); - if (sim) { - cancelAnimationFrame(sim); - sim = null; - } - svg.replaceChildren(); - $("inspect").hidden = true; - - // A redraw returns the view to the whole canvas, so a fit is discarded by the next query - // rather than silently framing a graph it was not computed for. - const width = svg.clientWidth || 800; - const height = svg.clientHeight || 500; - setViewBox(svg, { x: 0, y: 0, w: width, h: height }); - - const { nodes, edges, truncated } = snapshot; - $("graph-count").textContent = - `${plural(nodes.length, "node")} and ${plural(edges.length, "edge")}` + - (truncated ? " (capped at 300)" : ""); - - const { lit, groupIds, groupLabel } = resultOverlay(); - - // A node the result did not mention has no group, so it keeps a neutral fill rather than - // borrowing the color of a group it is not in. - const groupOrder = groupIds ? [...new Set(groupIds.values())].sort((a, b) => a - b) : []; - const groupColor = new Map( - groupOrder.map((value, i) => [value, GROUP_PALETTE[i % GROUP_PALETTE.length]]), - ); - const fillOf = (node) => - groupIds - ? (groupColor.get(groupIds.get(node.id)) ?? "var(--md-default-fg-color--lighter)") - : colorOf(labelOf(node)); - - $("legend").innerHTML = groupIds - ? groupOrder - .map( - (value) => - `${esc(groupLabel)} ${value}`, - ) - .join("") - : [...new Set(nodes.map(labelOf))] - .sort() - .map((l) => `${esc(l)}`) - .join(""); - - if (nodes.length === 0) { - svg.append( - el("text", { x: 16, y: 28, fill: "var(--md-default-fg-color--light)", "font-size": "13" }), + // A drag whose pointer is released after a redraw calls the previous drawing's `start`, + // which would put a loop over replaced nodes back into the shared handle. + const generation = ++simGeneration; + const svg = $("svg"); + if (sim) { + cancelAnimationFrame(sim); + sim = null; + } + svg.replaceChildren(); + $("inspect").hidden = true; + + // A redraw returns the view to the whole canvas, so a fit is discarded by the next query + // rather than silently framing a graph it was not computed for. + const width = svg.clientWidth || 800; + const height = svg.clientHeight || 500; + setViewBox(svg, {x: 0, y: 0, w: width, h: height}); + + const {nodes, edges, truncated} = snapshot; + $("graph-count").textContent = + `${plural(nodes.length, "node")} and ${plural(edges.length, "edge")}` + + (truncated ? " (capped at 300)" : ""); + + const {lit, groupIds, groupLabel} = resultOverlay(); + + // A node the result did not mention has no group, so it keeps a neutral fill rather than + // borrowing the color of a group it is not in. + const groupOrder = groupIds ? [...new Set(groupIds.values())].sort((a, b) => a - b) : []; + const groupColor = new Map( + groupOrder.map((value, i) => [value, GROUP_PALETTE[i % GROUP_PALETTE.length]]), ); - svg.lastChild.textContent = "Nothing to draw. Run a CREATE, or click Reset data."; - return; - } - - const linkLayer = el("g"); - const nodeLayer = el("g"); - svg.append(linkLayer, nodeLayer); - - const byId = new Map(nodes.map((n) => [n.id, n])); - const lines = edges.map((e) => { - const line = el("line", { "stroke-width": 1.2 }); - line.dataset.source = e.source; - line.dataset.target = e.target; - linkLayer.append(line); - return line; - }); - - const groups = nodes.map((node) => { - const group = el("g", { class: "node" }); - const circle = el("circle", { - r: NODE_RADIUS, - fill: fillOf(node), + const fillOf = (node) => + groupIds + ? (groupColor.get(groupIds.get(node.id)) ?? "var(--md-default-fg-color--lighter)") + : colorOf(labelOf(node)); + + $("legend").innerHTML = groupIds + ? groupOrder + .map( + (value) => + `${esc(groupLabel)} ${value}`, + ) + .join("") + : [...new Set(nodes.map(labelOf))] + .sort() + .map((l) => `${esc(l)}`) + .join(""); + + if (nodes.length === 0) { + svg.append( + el("text", { + x: 16, + y: 28, + fill: "var(--md-default-fg-color--light)", + "font-size": "13" + }), + ); + svg.lastChild.textContent = "Nothing to draw. Run a CREATE, or click Reset data."; + return; + } + + const linkLayer = el("g"); + const nodeLayer = el("g"); + svg.append(linkLayer, nodeLayer); + + const byId = new Map(nodes.map((n) => [n.id, n])); + const lines = edges.map((e) => { + const line = el("line", {"stroke-width": 1.2}); + line.dataset.source = e.source; + line.dataset.target = e.target; + linkLayer.append(line); + return line; }); - const text = el("text", { "text-anchor": "middle", dy: NODE_RADIUS + 12 }); - text.textContent = captionOf(node); - group.append(circle, text); - if (lit && !lit.has(node.id)) group.classList.add("dim"); - - group.addEventListener("pointerdown", (e) => { - e.stopPropagation(); - node.pinned = true; - capturePointer(group, e.pointerId); - const move = (ev) => { - const point = toSvg(svg, ev); - node.x = point.x; - node.y = point.y; - paint(); - }; - const up = () => { - node.pinned = false; - group.removeEventListener("pointermove", move); - group.removeEventListener("pointerup", up); - group.removeEventListener("pointercancel", up); - group.removeEventListener("lostpointercapture", up); - start(); - }; - group.addEventListener("pointermove", move); - group.addEventListener("pointerup", up); - group.addEventListener("pointercancel", up); - group.addEventListener("lostpointercapture", up); - inspect(node); + + const groups = nodes.map((node) => { + const group = el("g", {class: "node"}); + const circle = el("circle", { + r: NODE_RADIUS, + fill: fillOf(node), + }); + const text = el("text", {"text-anchor": "middle", dy: NODE_RADIUS + 12}); + text.textContent = captionOf(node); + group.append(circle, text); + if (lit && !lit.has(node.id)) group.classList.add("dim"); + + group.addEventListener("pointerdown", (e) => { + e.stopPropagation(); + node.pinned = true; + capturePointer(group, e.pointerId); + const move = (ev) => { + const point = toSvg(svg, ev); + node.x = point.x; + node.y = point.y; + paint(); + }; + const up = () => { + node.pinned = false; + group.removeEventListener("pointermove", move); + group.removeEventListener("pointerup", up); + group.removeEventListener("pointercancel", up); + group.removeEventListener("lostpointercapture", up); + start(); + }; + group.addEventListener("pointermove", move); + group.addEventListener("pointerup", up); + group.addEventListener("pointercancel", up); + group.addEventListener("lostpointercapture", up); + inspect(node); + }); + nodeLayer.append(group); + return group; }); - nodeLayer.append(group); - return group; - }); - - function paint() { - for (const line of lines) { - const a = byId.get(Number(line.dataset.source)); - const b = byId.get(Number(line.dataset.target)); - if (!a || !b) continue; - line.setAttribute("x1", a.x); - line.setAttribute("y1", a.y); - line.setAttribute("x2", b.x); - line.setAttribute("y2", b.y); - } - for (const [i, group] of groups.entries()) { - group.setAttribute("transform", `translate(${nodes[i].x} ${nodes[i].y})`); - } - } - - const tick = layout(nodes, edges, width, height); - function start() { - if (generation !== simGeneration) return; - if (sim) cancelAnimationFrame(sim); - if (REDUCED_MOTION.matches) { - // Settled in one pass and painted once. The bound is where the animated form stops anyway, - // since alpha starts at 1, decays by 0.985 a step, and the loop ends below 0.02. - for (let i = 0; i < 260 && tick(); i += 1); - paint(); - return; - } - const step = () => { - const running = tick(); - paint(); - sim = running ? requestAnimationFrame(step) : null; - }; - sim = requestAnimationFrame(step); - } - paint(); - start(); + + function paint() { + for (const line of lines) { + const a = byId.get(Number(line.dataset.source)); + const b = byId.get(Number(line.dataset.target)); + if (!a || !b) continue; + line.setAttribute("x1", a.x); + line.setAttribute("y1", a.y); + line.setAttribute("x2", b.x); + line.setAttribute("y2", b.y); + } + for (const [i, group] of groups.entries()) { + group.setAttribute("transform", `translate(${nodes[i].x} ${nodes[i].y})`); + } + } + + const tick = layout(nodes, edges, width, height); + + function start() { + if (generation !== simGeneration) return; + if (sim) cancelAnimationFrame(sim); + if (REDUCED_MOTION.matches) { + // Settled in one pass and painted once. The bound is where the animated form stops anyway, + // since alpha starts at 1, decays by 0.985 a step, and the loop ends below 0.02. + for (let i = 0; i < 260 && tick(); i += 1) ; + paint(); + return; + } + const step = () => { + const running = tick(); + paint(); + sim = running ? requestAnimationFrame(step) : null; + }; + sim = requestAnimationFrame(step); + } + + paint(); + start(); } // The view box the graph is currently drawn through. Layout coordinates are canvas pixels, so @@ -1441,159 +1620,159 @@ function drawGraph() { let viewBox = null; function setViewBox(svg, box) { - viewBox = box; - svg.setAttribute("viewBox", `${box.x} ${box.y} ${box.w} ${box.h}`); + viewBox = box; + svg.setAttribute("viewBox", `${box.x} ${box.y} ${box.w} ${box.h}`); } function toSvg(svg, event) { - const rect = svg.getBoundingClientRect(); - const box = viewBox ?? { x: 0, y: 0, w: rect.width, h: rect.height }; - return { - x: box.x + ((event.clientX - rect.left) * box.w) / rect.width, - y: box.y + ((event.clientY - rect.top) * box.h) / rect.height, - }; + const rect = svg.getBoundingClientRect(); + const box = viewBox ?? {x: 0, y: 0, w: rect.width, h: rect.height}; + return { + x: box.x + ((event.clientX - rect.left) * box.w) / rect.width, + y: box.y + ((event.clientY - rect.top) * box.h) / rect.height, + }; } function inspect(node) { - const props = Object.entries(node.props ?? {}); - const rows = props - .map(([k, v]) => `
    ${esc(k)}
    ${esc(JSON.stringify(v))}
    `) - .join(""); - $("inspect").innerHTML = - `
    ${esc(node.labels?.join(":") || "(no label)")} #${node.id}
    ` + - (rows ? `
    ${rows}
    ` : '
    No properties.
    '); - $("inspect").hidden = false; + const props = Object.entries(node.props ?? {}); + const rows = props + .map(([k, v]) => `
    ${esc(k)}
    ${esc(JSON.stringify(v))}
    `) + .join(""); + $("inspect").innerHTML = + `
    ${esc(node.labels?.join(":") || "(no label)")} #${node.id}
    ` + + (rows ? `
    ${rows}
    ` : '
    No properties.
    '); + $("inspect").hidden = false; } // Zoom and pan both move the view box, which is also what Fit sets and what `toSvg` maps a pointer // through, so those three cannot disagree about where the graph is. const ZOOM_STEP = 1.25; -const canvasSize = (svg) => ({ w: svg.clientWidth || 800, h: svg.clientHeight || 500 }); +const canvasSize = (svg) => ({w: svg.clientWidth || 800, h: svg.clientHeight || 500}); // `focal` is the world point to hold still, so a wheel zoom keeps whatever is under the pointer // under the pointer. Without it, zooming in on a corner walks the graph off the canvas. function zoomBy(factor, focal) { - const svg = $("svg"); - if (!viewBox) return; - const { w: canvasW } = canvasSize(svg); - // Bounded, or a few scrolls leave an empty canvas with no way to tell which direction the graph - // went. The clamp is on the width and the same scale is applied to the height, so the box keeps - // the element's aspect ratio and nothing is letterboxed. - const clamped = Math.min(Math.max(viewBox.w * factor, canvasW / 8), canvasW * 4); - const scale = clamped / viewBox.w; - const point = focal ?? { - x: viewBox.x + viewBox.w / 2, - y: viewBox.y + viewBox.h / 2, - }; - setViewBox(svg, { - x: point.x - (point.x - viewBox.x) * scale, - y: point.y - (point.y - viewBox.y) * scale, - w: viewBox.w * scale, - h: viewBox.h * scale, - }); + const svg = $("svg"); + if (!viewBox) return; + const {w: canvasW} = canvasSize(svg); + // Bounded, or a few scrolls leave an empty canvas with no way to tell which direction the graph + // went. The clamp is on the width and the same scale is applied to the height, so the box keeps + // the element's aspect ratio and nothing is letterboxed. + const clamped = Math.min(Math.max(viewBox.w * factor, canvasW / 8), canvasW * 4); + const scale = clamped / viewBox.w; + const point = focal ?? { + x: viewBox.x + viewBox.w / 2, + y: viewBox.y + viewBox.h / 2, + }; + setViewBox(svg, { + x: point.x - (point.x - viewBox.x) * scale, + y: point.y - (point.y - viewBox.y) * scale, + w: viewBox.w * scale, + h: viewBox.h * scale, + }); } $("zoom-in").addEventListener("click", () => zoomBy(1 / ZOOM_STEP)); $("zoom-out").addEventListener("click", () => zoomBy(ZOOM_STEP)); $("svg").addEventListener( - "wheel", - (e) => { - // Claimed rather than shared: the page scrolls as a document, and a wheel over the canvas that - // both zoomed and scrolled the page would be unusable. Hence a non-passive listener. - e.preventDefault(); - zoomBy(e.deltaY > 0 ? ZOOM_STEP : 1 / ZOOM_STEP, toSvg($("svg"), e)); - }, - { passive: false }, + "wheel", + (e) => { + // Claimed rather than shared: the page scrolls as a document, and a wheel over the canvas that + // both zoomed and scrolled the page would be unusable. Hence a non-passive listener. + e.preventDefault(); + zoomBy(e.deltaY > 0 ? ZOOM_STEP : 1 / ZOOM_STEP, toSvg($("svg"), e)); + }, + {passive: false}, ); $("svg").addEventListener("pointerdown", (e) => { - $("inspect").hidden = true; - // A vertex has its own drag handler and stops propagation; this is the guard for anything that - // does not, so a pan cannot start on top of a node. - if (e.target.closest(".node")) return; - - const svg = $("svg"); - const from = viewBox ? { ...viewBox } : null; - if (!from) return; - capturePointer(svg, e.pointerId); - - const move = (ev) => { - // Measured against the box the drag started from rather than the current one, or the pan chases - // itself: each move would be applied to a box the previous move had already shifted. - const rect = svg.getBoundingClientRect(); - const dx = ((ev.clientX - e.clientX) * from.w) / rect.width; - const dy = ((ev.clientY - e.clientY) * from.h) / rect.height; - setViewBox(svg, { x: from.x - dx, y: from.y - dy, w: from.w, h: from.h }); - }; - const up = () => { - svg.removeEventListener("pointermove", move); - svg.removeEventListener("pointerup", up); - svg.removeEventListener("pointercancel", up); - svg.removeEventListener("lostpointercapture", up); - }; - svg.addEventListener("pointermove", move); - svg.addEventListener("pointerup", up); - svg.addEventListener("pointercancel", up); - svg.addEventListener("lostpointercapture", up); + $("inspect").hidden = true; + // A vertex has its own drag handler and stops propagation; this is the guard for anything that + // does not, so a pan cannot start on top of a node. + if (e.target.closest(".node")) return; + + const svg = $("svg"); + const from = viewBox ? {...viewBox} : null; + if (!from) return; + capturePointer(svg, e.pointerId); + + const move = (ev) => { + // Measured against the box the drag started from rather than the current one, or the pan chases + // itself: each move would be applied to a box the previous move had already shifted. + const rect = svg.getBoundingClientRect(); + const dx = ((ev.clientX - e.clientX) * from.w) / rect.width; + const dy = ((ev.clientY - e.clientY) * from.h) / rect.height; + setViewBox(svg, {x: from.x - dx, y: from.y - dy, w: from.w, h: from.h}); + }; + const up = () => { + svg.removeEventListener("pointermove", move); + svg.removeEventListener("pointerup", up); + svg.removeEventListener("pointercancel", up); + svg.removeEventListener("lostpointercapture", up); + }; + svg.addEventListener("pointermove", move); + svg.addEventListener("pointerup", up); + svg.addEventListener("pointercancel", up); + svg.addEventListener("lostpointercapture", up); }); // The layout keeps every vertex inside the canvas, so this only ever zooms in. That is the // direction worth having: a handful of nodes otherwise sit in the middle of a mostly empty // canvas. A redraw resets the view, so there is no "unfit" to provide. $("fit").addEventListener("click", () => { - const svg = $("svg"); - const placed = snapshot.nodes.filter((node) => node.x !== undefined); - if (placed.length === 0) return; - - let minX = Infinity; - let maxX = -Infinity; - let minY = Infinity; - let maxY = -Infinity; - for (const node of placed) { - if (node.x < minX) minX = node.x; - if (node.x > maxX) maxX = node.x; - if (node.y < minY) minY = node.y; - if (node.y > maxY) maxY = node.y; - } - // Clears the largest radius and the caption below it, or fitting would crop the labels it is - // meant to bring into view. - const pad = 42; - minX -= pad; - maxX += pad; - minY -= pad; - maxY += pad; - - // Matched to the element's own aspect ratio. A view box with a different one is letterboxed by - // the default `preserveAspectRatio`, so the fit would come out loose on one axis. - const rect = svg.getBoundingClientRect(); - const aspect = (rect.width || 800) / (rect.height || 500); - let w = maxX - minX; - let h = maxY - minY; - if (w / h > aspect) h = w / aspect; - else w = h * aspect; - - setViewBox(svg, { - x: (minX + maxX) / 2 - w / 2, - y: (minY + maxY) / 2 - h / 2, - w, - h, - }); + const svg = $("svg"); + const placed = snapshot.nodes.filter((node) => node.x !== undefined); + if (placed.length === 0) return; + + let minX = Infinity; + let maxX = -Infinity; + let minY = Infinity; + let maxY = -Infinity; + for (const node of placed) { + if (node.x < minX) minX = node.x; + if (node.x > maxX) maxX = node.x; + if (node.y < minY) minY = node.y; + if (node.y > maxY) maxY = node.y; + } + // Clears the largest radius and the caption below it, or fitting would crop the labels it is + // meant to bring into view. + const pad = 42; + minX -= pad; + maxX += pad; + minY -= pad; + maxY += pad; + + // Matched to the element's own aspect ratio. A view box with a different one is letterboxed by + // the default `preserveAspectRatio`, so the fit would come out loose on one axis. + const rect = svg.getBoundingClientRect(); + const aspect = (rect.width || 800) / (rect.height || 500); + let w = maxX - minX; + let h = maxY - minY; + if (w / h > aspect) h = w / aspect; + else w = h * aspect; + + setViewBox(svg, { + x: (minX + maxX) / 2 - w / 2, + y: (minY + maxY) / 2 - h / 2, + w, + h, + }); }); $("relayout").addEventListener("click", () => { - for (const node of snapshot.nodes) node.x = undefined; - drawGraph(); + for (const node of snapshot.nodes) node.x = undefined; + drawGraph(); }); // The layout reads the viewport size when it starts, so a resize needs a fresh one. Positions // survive, since only an undefined coordinate is re-seeded. let resizeTimer = null; addEventListener("resize", () => { - if (!$("pane-graph").classList.contains("on")) return; - clearTimeout(resizeTimer); - resizeTimer = setTimeout(drawGraph, 150); + if (!$("pane-graph").classList.contains("on")) return; + clearTimeout(resizeTimer); + resizeTimer = setTimeout(drawGraph, 150); }); // --------------------------------------------------------------------------- @@ -1601,55 +1780,55 @@ addEventListener("resize", () => { // --------------------------------------------------------------------------- function download(name, mime, text) { - const url = URL.createObjectURL(new Blob([text], { type: mime })); - const anchor = document.createElement("a"); - anchor.href = url; - anchor.download = name; - anchor.click(); - // Firefox and Safari fetch the object URL asynchronously after the click, so revoking on - // this tick produces an empty download. - setTimeout(() => URL.revokeObjectURL(url), 0); + const url = URL.createObjectURL(new Blob([text], {type: mime})); + const anchor = document.createElement("a"); + anchor.href = url; + anchor.download = name; + anchor.click(); + // Firefox and Safari fetch the object URL asynchronously after the click, so revoking on + // this tick produces an empty download. + setTimeout(() => URL.revokeObjectURL(url), 0); } $("csv").addEventListener("click", () => { - if (!lastResult?.columns.length) return; - const quote = (v) => { - if (v === null || v === undefined) return ""; - const s = typeof v === "object" ? JSON.stringify(v) : String(v); - return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s; - }; - const csv = [ - lastResult.columns.map(quote).join(","), - ...lastResult.rows.map((r) => r.map(quote).join(",")), - ].join("\n"); - download("issundb-result.csv", "text/csv", csv); + if (!lastResult?.columns.length) return; + const quote = (v) => { + if (v === null || v === undefined) return ""; + const s = typeof v === "object" ? JSON.stringify(v) : String(v); + return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s; + }; + const csv = [ + lastResult.columns.map(quote).join(","), + ...lastResult.rows.map((r) => r.map(quote).join(",")), + ].join("\n"); + download("issundb-result.csv", "text/csv", csv); }); $("json-dl").addEventListener("click", () => { - if (!lastResult) return; - const objects = lastResult.rows.map((row) => - Object.fromEntries(lastResult.columns.map((c, i) => [c, row[i]])), - ); - download("issundb-result.json", "application/json", JSON.stringify(objects, null, 2)); + if (!lastResult) return; + const objects = lastResult.rows.map((row) => + Object.fromEntries(lastResult.columns.map((c, i) => [c, row[i]])), + ); + download("issundb-result.json", "application/json", JSON.stringify(objects, null, 2)); }); // The query travels in the fragment, so a shared link never reaches a server even when the // page is hosted on one. const b64url = { - encode: (s) => { - const bytes = new TextEncoder().encode(s); - // Chunked rather than one spread: `String.fromCharCode(...bytes)` raises a RangeError - // past about 130 000 arguments, which a pasted bulk-insert script reaches. - let binary = ""; - for (let i = 0; i < bytes.length; i += 0x8000) { - binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000)); - } - return btoa(binary).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); - }, - decode: (s) => - new TextDecoder().decode( - Uint8Array.from(atob(s.replace(/-/g, "+").replace(/_/g, "/")), (c) => c.charCodeAt(0)), - ), + encode: (s) => { + const bytes = new TextEncoder().encode(s); + // Chunked rather than one spread: `String.fromCharCode(...bytes)` raises a RangeError + // past about 130 000 arguments, which a pasted bulk-insert script reaches. + let binary = ""; + for (let i = 0; i < bytes.length; i += 0x8000) { + binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000)); + } + return btoa(binary).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); + }, + decode: (s) => + new TextDecoder().decode( + Uint8Array.from(atob(s.replace(/-/g, "+").replace(/_/g, "/")), (c) => c.charCodeAt(0)), + ), }; // A fragment reaches no server, but it does have to survive being pasted, and enough clients @@ -1661,48 +1840,48 @@ const MAX_SHARED_SETUP = 24000; let ownHashWrite = false; $("share").addEventListener("click", async () => { - const parts = []; - try { - parts.push(`q=${b64url.encode(editor.value)}`); - } catch { - // Encoding was outside the try before, so a query too large to encode rejected silently. - setStatus("err", "The query is too large to put in a link."); - return; - } - - // Without this the link carried the query alone, so a query over data the sender had created - // returned nothing for whoever opened it. The recipient's boot seeds the sample graph and then - // replays these, which reproduces the state exactly rather than approximating it from a - // snapshot: a snapshot is capped at 300 nodes and carries no relationship properties. - let dropped = 0; - if (setupLog.length > 0) { - let setup = ""; + const parts = []; try { - setup = b64url.encode(setupLog.join(";\n")); + parts.push(`q=${b64url.encode(editor.value)}`); } catch { - setup = ""; - } - if (setup && setup.length <= MAX_SHARED_SETUP) parts.push(`s=${setup}`); - else dropped = setupLog.length; - } - - const note = dropped - ? ` ${plural(dropped, "setup statement")} were too large to include.` - : setupLog.length > 0 - ? ` It carries ${plural(setupLog.length, "setup statement")}.` - : ""; - - const fragment = parts.join("&"); - try { - await navigator.clipboard.writeText( - `${location.origin}${location.pathname}#${fragment}`, - ); - setStatus("ok", `Link copied.${note}`); - } catch { - ownHashWrite = true; - location.hash = fragment; - setStatus("", `The link is in the address bar.${note}`); - } + // Encoding was outside the try before, so a query too large to encode rejected silently. + setStatus("err", "The query is too large to put in a link."); + return; + } + + // Without this the link carried the query alone, so a query over data the sender had created + // returned nothing for whoever opened it. The recipient's boot seeds the sample graph and then + // replays these, which reproduces the state exactly rather than approximating it from a + // snapshot: a snapshot is capped at 300 nodes and carries no relationship properties. + let dropped = 0; + if (setupLog.length > 0) { + let setup = ""; + try { + setup = b64url.encode(setupLog.join(";\n")); + } catch { + setup = ""; + } + if (setup && setup.length <= MAX_SHARED_SETUP) parts.push(`s=${setup}`); + else dropped = setupLog.length; + } + + const note = dropped + ? ` ${plural(dropped, "setup statement")} were too large to include.` + : setupLog.length > 0 + ? ` It carries ${plural(setupLog.length, "setup statement")}.` + : ""; + + const fragment = parts.join("&"); + try { + await navigator.clipboard.writeText( + `${location.origin}${location.pathname}#${fragment}`, + ); + setStatus("ok", `Link copied.${note}`); + } catch { + ownHashWrite = true; + location.hash = fragment; + setStatus("", `The link is in the address bar.${note}`); + } }); // Changing only the fragment is a same-document navigation, so the module is not re-evaluated and @@ -1710,30 +1889,30 @@ $("share").addEventListener("click", async () => { // already in front of the reader, which is how a documentation page's "run this" link behaves on a // second click. addEventListener("hashchange", () => { - // The clipboard fallback above writes the hash itself, and treating that as an incoming link - // would re-run the query as a side effect of copying it. - if (ownHashWrite) { - ownHashWrite = false; - return; - } - const params = new URLSearchParams(location.hash.slice(1)); - // A setup script has to be replayed against a freshly seeded database, or it lands on top of - // whatever is already here and adds a second copy of its data. Reloading is what gives `boot` - // the chance to do it in the right order. - if (params.get("s")) { - location.reload(); - return; - } - let incoming = null; - try { - const encoded = params.get("q"); - incoming = encoded ? b64url.decode(encoded) : params.get("cypher"); - } catch { - incoming = null; - } - if (!incoming) return; - setQuery(incoming); - run(); + // The clipboard fallback above writes the hash itself, and treating that as an incoming link + // would re-run the query as a side effect of copying it. + if (ownHashWrite) { + ownHashWrite = false; + return; + } + const params = new URLSearchParams(location.hash.slice(1)); + // A setup script has to be replayed against a freshly seeded database, or it lands on top of + // whatever is already here and adds a second copy of its data. Reloading is what gives `boot` + // the chance to do it in the right order. + if (params.get("s")) { + location.reload(); + return; + } + let incoming = null; + try { + const encoded = params.get("q"); + incoming = encoded ? b64url.decode(encoded) : params.get("cypher"); + } catch { + incoming = null; + } + if (!incoming) return; + setQuery(incoming); + run(); }); // `default` and `slate` are Material for MkDocs' scheme names, so the playground and the @@ -1743,38 +1922,38 @@ const SUN = "M12 7a5 5 0 100 10 5 5 0 000-10zM12 2v3m0 14v3M2 12h3m14 0h3M4.9 4. const MOON = "M12 3a9 9 0 109 9c0-.5 0-1-.1-1.4A7 7 0 0112 3z"; function currentScheme() { - return document.documentElement.getAttribute("data-md-color-scheme") === "slate" - ? "slate" - : "default"; + return document.documentElement.getAttribute("data-md-color-scheme") === "slate" + ? "slate" + : "default"; } function applyScheme(scheme) { - document.documentElement.setAttribute("data-md-color-scheme", scheme); - // Built through `createElementNS` rather than `innerHTML`, since markup assigned to an SVG - // element is not reliably parsed into the SVG namespace. The icon offers the scheme you - // would switch to, which is the convention Material uses. - const dark = scheme === "slate"; - $("scheme-icon").replaceChildren( - el( - "path", - dark - ? { d: SUN, stroke: "currentColor", "stroke-width": "2", fill: "none" } - : { d: MOON }, - ), - ); + document.documentElement.setAttribute("data-md-color-scheme", scheme); + // Built through `createElementNS` rather than `innerHTML`, since markup assigned to an SVG + // element is not reliably parsed into the SVG namespace. The icon offers the scheme you + // would switch to, which is the convention Material uses. + const dark = scheme === "slate"; + $("scheme-icon").replaceChildren( + el( + "path", + dark + ? {d: SUN, stroke: "currentColor", "stroke-width": "2", fill: "none"} + : {d: MOON}, + ), + ); } $("scheme").addEventListener("click", () => { - const next = currentScheme() === "slate" ? "default" : "slate"; - applyScheme(next); - try { - localStorage.setItem(SCHEME_KEY, next); - } catch { - // Storage being unavailable only costs the choice its persistence. - } - // The graph is drawn with resolved colors rather than custom properties, so it has to be - // repainted for a scheme change to reach it. - if ($("pane-graph").classList.contains("on")) drawGraph(); + const next = currentScheme() === "slate" ? "default" : "slate"; + applyScheme(next); + try { + localStorage.setItem(SCHEME_KEY, next); + } catch { + // Storage being unavailable only costs the choice its persistence. + } + // The graph is drawn with resolved colors rather than custom properties, so it has to be + // repainted for a scheme change to reach it. + if ($("pane-graph").classList.contains("on")) drawGraph(); }); $("toggle-side").addEventListener("click", () => $("side").classList.toggle("hidden")); @@ -1786,14 +1965,11 @@ $("toggle-side").addEventListener("click", () => $("side").classList.toggle("hid // The build stamp comes out of the module rather than a sidecar file the page would have to fetch, // so it is empty for a build made outside a git checkout and the footer then names the version // alone. -function renderPoweredBy() { - const build = Playground.buildRef(); - const engine = build - ? `IssunDB (${Playground.version()}; ${build})` - : `IssunDB (${Playground.version()})`; - $("powered").textContent = - `This playground app is powered by ${engine}; everything` + - " (including the queries) runs safely in your browser."; +function renderPoweredBy({version, build}) { + const named = build ? `IssunDB (${version}; ${build})` : `IssunDB (${version})`; + $("powered").textContent = + `This playground app is powered by ${named}; everything` + + " (including the queries) runs safely in your browser."; } let activeSample = 0; @@ -1801,123 +1977,144 @@ let activeSample = 0; const currentSample = () => SAMPLE_GRAPHS[activeSample] ?? SAMPLE_GRAPHS[0]; function renderSamples() { - const select = $("sample-graph"); - SAMPLE_GRAPHS.forEach((sample, i) => { - const option = document.createElement("option"); - option.value = String(i); - option.textContent = sample.label; - select.append(option); - }); - select.addEventListener("change", () => { - activeSample = Number(select.value); - }); + const select = $("sample-graph"); + SAMPLE_GRAPHS.forEach((sample, i) => { + const option = document.createElement("option"); + option.value = String(i); + option.textContent = sample.label; + select.append(option); + }); + select.addEventListener("change", () => { + activeSample = Number(select.value); + }); } -function seed() { - db.query(currentSample().cypher); - refreshSchema(); +async function seed() { + await engine.query(currentSample().cypher); + await refreshSchema(); } $("reset").addEventListener("click", async () => { - // Freed rather than abandoned. wasm-bindgen registers a finalizer, so an abandoned instance - // is reclaimed eventually, but until then its whole graph is still resident and wasm memory - // never shrinks. The new instance is built first, so a failure leaves the old one usable. - const previous = db; - db = new Playground(); - previous?.free(); - // After the old instance is freed, so the baseline is one empty database rather than two. - baselineBytes = Playground.memoryBytes(); - lastResult = null; - // The discarded writes must not keep travelling in a share link, where replaying them against - // the fresh seed would rebuild the state Reset was clicked to get rid of. - setupLog.length = 0; - // Node ids restart from zero, so a carried-over position would belong to a different node. - snapshot = { nodes: [], edges: [], truncated: false }; - seed(); - await refreshGraph(); - setStatus("ok", `Reset. The ${currentSample().label} sample was re-seeded.`); - setMeta("Run a query to view results."); - showPane("table"); - $("pane-table").innerHTML = - `
    Fresh database, seeded with the ${esc(currentSample().label)} sample.` + - " Pick an example on the left, or write a query.
    "; + ({baseline: baselineBytes} = await engine.reset()); + lastResult = null; + // The discarded writes must not keep travelling in a share link, where replaying them against + // the fresh seed would rebuild the state Reset was clicked to get rid of. + setupLog.length = 0; + // Node ids restart from zero, so a carried-over position would belong to a different node. + snapshot = {nodes: [], edges: [], truncated: false}; + await seed(); + await refreshGraph(); + setStatus("ok", `Reset. The ${currentSample().label} sample was re-seeded.`); + setMeta("Run a query to view results."); + showPane("table"); + $("pane-table").innerHTML = + `
    Fresh database, seeded with the ${esc(currentSample().label)} sample.` + + " Pick an example on the left, or write a query.
    "; }); async function boot() { - applyScheme(currentScheme()); - - wasmExports = await init(); - db = new Playground(); - baselineBytes = Playground.memoryBytes(); - - renderPoweredBy(); - renderSamples(); - renderDemos(); - renderProcedures(); - renderHistory(); - seed(); - - // Three link forms. `q` is the Share button's base64 query and `s` its optional setup script, - // and `cypher` is percent-encoded plain text so a link can be written by hand or generated by a - // docs build. A generator has to encode a plus as %2B, since a fragment read as a query string - // turns a literal one into a space. - const params = new URLSearchParams(location.hash.slice(1)); - - const setup = params.get("s"); - if (setup) { + applyScheme(currentScheme()); + + spawnWorker(); + const info = await engine.boot(); + baselineBytes = info.baseline; + ready = true; + + renderPoweredBy(info); + renderSamples(); + renderDemos(); + renderProcedures(); + renderHistory(); + await seed(); + + // Three link forms. `q` is the Share button's base64 query and `s` its optional setup script, + // and `cypher` is percent-encoded plain text so a link can be written by hand or generated by a + // docs build. A generator has to encode a plus as %2B, since a fragment read as a query string + // turns a literal one into a space. + const params = new URLSearchParams(location.hash.slice(1)); + + const setup = params.get("s"); + if (setup) { + try { + await engine.query(b64url.decode(setup)); + await refreshSchema(); + } catch { + // A setup script that no longer applies leaves the seeded graph in place rather than + // stopping the page from loading. The query it came with still runs, and reports its own + // error if it depended on what failed. + } + } + + let shared = null; try { - db.query(b64url.decode(setup)); - refreshSchema(); + const encoded = params.get("q"); + shared = encoded ? b64url.decode(encoded) : params.get("cypher"); } catch { - // A setup script that no longer applies leaves the seeded graph in place rather than - // stopping the page from loading. The query it came with still runs, and reports its own - // error if it depended on what failed. - } - } - - let shared = null; - try { - const encoded = params.get("q"); - shared = encoded ? b64url.decode(encoded) : params.get("cypher"); - } catch { - shared = null; - } - - const stored = shared ? "" : readStoredEditor().trim(); - if (shared) { - setQuery(shared); - } else if (stored) { - // No caption: the ribbon is for what an example is demonstrating, and the banner below the - // editor already says the query was restored and not run. Three notices for one fact was two - // too many. - setQuery(stored); - } else { - setQuery( - "MATCH (a:Person)-[:KNOWS]->(b:Person)\nRETURN a.name AS from, b.name AS to\nORDER BY from, to", - "A starting query over the seeded sample graph. Press ⌘↵ (or Ctrl↵) to run it.", - ); - } + shared = null; + } + + const stored = shared ? "" : readStoredEditor().trim(); + if (shared) { + setQuery(shared); + } else if (stored) { + // No caption: the ribbon is for what an example is demonstrating, and the banner below the + // editor already says the query was restored and not run. Three notices for one fact was two + // too many. + setQuery(stored); + } else { + setQuery( + "MATCH (a:Person)-[:KNOWS]->(b:Person)\nRETURN a.name AS from, b.name AS to\nORDER BY from, to", + "A starting query over the seeded sample graph. Press ⌘↵ (or Ctrl↵) to run it.", + ); + } - await refreshGraph(); - $("boot").remove(); + await refreshGraph(); + $("boot").remove(); + + // A restored query is deliberately not run. It could be a CREATE, and running it on every + // reload would quietly add another copy of its data. + if (stored) { + showPane("table"); + setStatus("", "Your last query was restored. It has not been run."); + setMeta("Run a query to view results."); + $("pane-table").innerHTML = + '
    Your last query is in the editor. Press ⌘↵ (or Ctrl↵) to run it.
    '; + } else { + await run(); + } +} - // A restored query is deliberately not run. It could be a CREATE, and running it on every - // reload would quietly add another copy of its data. - if (stored) { - showPane("table"); - setStatus("", "Your last query was restored. It has not been run."); - setMeta("Run a query to view results."); - $("pane-table").innerHTML = - '
    Your last query is in the editor. Press ⌘↵ (or Ctrl↵) to run it.
    '; - } else { - await run(); - } +// A boot failure has one likely cause that the browser's own message does not name. The generated +// glue and the wasm binary are written together and reference a snippet directory by a hash of the +// build; hold a cached copy of one against a fresh copy of the other and the pair disagrees about +// that name, which surfaces as an import object field that "is not an Object". Nothing about it +// suggests the real fix, which is to discard the cached half. +function bootAdvice(message) { + if (/snippets\/|is not an Object|WebAssembly\.instantiate/.test(message)) { + return ( + `The cached module does not match the one being served: the JavaScript glue and the ` + + `.wasm file are generated together and disagree about a build hash. ` + + `Reload bypassing the cache (Ctrl+Shift+R, or ` + + `Cmd+Shift+R). If it persists, rerun ` + + `make playground-build, which now clears web/pkg first.` + ); + } + if (/Worker|module worker/i.test(message)) { + return ( + `The engine runs in a module worker, which this browser appears not to support. ` + + `Firefox 114, Safari 15, and Chrome 80 or newer all do.` + ); + } + return ( + `The module is served as web/pkg/; build it with make playground-build ` + + `and serve the directory over HTTP, since a module cannot be loaded from a file:// path.` + ); } boot().catch((e) => { - $("boot").innerHTML = - `
    ` + - `The engine did not load.

    ${esc(String(e))}

    ` + - `The module is served as web/pkg/; build it with make playground-build ` + - `and serve the directory over HTTP, since a module cannot be loaded from a file:// path.
    `; + const message = String(e?.message ?? e); + $("boot").innerHTML = + `
    ` + + `The engine did not load.

    ${esc(message)}

    ` + + `${bootAdvice(message)}
    `; }); diff --git a/web/demos.js b/web/demos.js index f53d734..92cd62e 100644 --- a/web/demos.js +++ b/web/demos.js @@ -31,15 +31,15 @@ const PATH_NOTE = `// Procedure arguments are resolved before planning, so they // The social graph is first because it is what the page seeds on load and what the Examples panel // queries; the other four replace it when Reset Database is pressed with one of them selected. export const SAMPLE_GRAPHS = [ - { - id: "social", - label: "Social network", - cypher: SAMPLE_SOCIAL, - }, - { - id: "articles", - label: "Article corpus", - cypher: `CREATE (a1:Article {title: 'Graph databases', year: 2019, + { + id: "social", + label: "Social network", + cypher: SAMPLE_SOCIAL, + }, + { + id: "articles", + label: "Article corpus", + cypher: `CREATE (a1:Article {title: 'Graph databases', year: 2019, body: 'A graph database stores nodes and relationships instead of tables and joins.'}), (a2:Article {title: 'Vector search', year: 2021, body: 'Approximate nearest neighbour search finds similar embeddings quickly.'}), @@ -61,11 +61,11 @@ export const SAMPLE_GRAPHS = [ (a3)-[:CITES]->(a1), (a5)-[:CITES]->(a2), (a5)-[:CITES]->(a3)`, - }, - { - id: "org", - label: "Org chart", - cypher: `CREATE (rin:Employee {name: 'Rin', title: 'CEO', level: 1}), + }, + { + id: "org", + label: "Org chart", + cypher: `CREATE (rin:Employee {name: 'Rin', title: 'CEO', level: 1}), (sato:Employee {name: 'Sato', title: 'CTO', level: 2}), (mori:Employee {name: 'Mori', title: 'CFO', level: 2}), (kaito:Employee {name: 'Kaito', title: 'Engineering Manager', level: 3}), @@ -78,11 +78,11 @@ export const SAMPLE_GRAPHS = [ (yuki)-[:REPORTS_TO]->(kaito), (hana)-[:REPORTS_TO]->(yuki), (taro)-[:REPORTS_TO]->(mori)`, - }, - { - id: "transport", - label: "Transport network", - cypher: `CREATE (tokyo:City {name: 'Tokyo', country: 'Japan'}), + }, + { + id: "transport", + label: "Transport network", + cypher: `CREATE (tokyo:City {name: 'Tokyo', country: 'Japan'}), (nagoya:City {name: 'Nagoya', country: 'Japan'}), (kyoto:City {name: 'Kyoto', country: 'Japan'}), (osaka:City {name: 'Osaka', country: 'Japan'}), @@ -94,11 +94,11 @@ export const SAMPLE_GRAPHS = [ (tokyo)-[:ROUTE {weight: 500, cost: 14500, capacity: 1100}]->(osaka), (osaka)-[:ROUTE {weight: 480, cost: 15400, capacity: 600}]->(fukuoka), (tokyo)-[:ROUTE {weight: 830, cost: 25000, capacity: 400}]->(sapporo)`, - }, - { - id: "retail", - label: "Retail co-purchase", - cypher: `CREATE (aiko:Customer {name: 'Aiko'}), + }, + { + id: "retail", + label: "Retail co-purchase", + cypher: `CREATE (aiko:Customer {name: 'Aiko'}), (ben:Customer {name: 'Ben'}), (chie:Customer {name: 'Chie'}), (keyboard:Product {name: 'Mechanical keyboard', price: 129, category: 'peripherals'}), @@ -114,11 +114,11 @@ export const SAMPLE_GRAPHS = [ (keyboard)-[:SIMILAR_TO]->(dock), (dock)-[:SIMILAR_TO]->(keyboard), (monitor)-[:SIMILAR_TO]->(lamp)`, - }, - { - id: "knowledge", - label: "Knowledge graph", - cypher: `CREATE (ada:Researcher {name: 'Ada Ito'}), + }, + { + id: "knowledge", + label: "Knowledge graph", + cypher: `CREATE (ada:Researcher {name: 'Ada Ito'}), (bo:Researcher {name: 'Bo Chen'}), (cai:Researcher {name: 'Cai Rossi'}), (lab1:Lab {name: 'Retrieval Group', city: 'Kyoto'}), @@ -140,7 +140,7 @@ export const SAMPLE_GRAPHS = [ (p1)-[:MENTIONS]->(c2), (p3)-[:MENTIONS]->(c1), (p2)-[:MENTIONS]->(c3)`, - }, + }, ]; // The procedure reference the sidebar lists and searches. It lives here rather than in `app.js` @@ -151,445 +151,645 @@ export const SAMPLE_GRAPHS = [ // `requiresVectors` marks the two entries the sample graph cannot satisfy, since it stores no // embeddings. Their snippets are still run, and still have to resolve to a real procedure; only // the empty-index failure is tolerated. +// The `issundb.*` scalar functions. Kept apart from PROCEDURES because they are called in an +// expression rather than through CALL, which is not a presentation detail: a CALL evaluates its +// arguments against no bindings and runs once per statement, so a pairwise score could never see +// the two nodes a MATCH bound. That is why these are functions at all. +// +// Ordinary Cypher functions (toUpper, substring, the temporal family) are deliberately absent. They +// are documented by every Cypher reference there is, while these are documented nowhere else. +export const FUNCTIONS = [ + { + name: "issundb.link.commonNeighbors", + args: "a, b", + yields: "number", + summary: + "How many neighbors two nodes share. The neighborhood is undirected and distinct, so a pair joined by several edges counts once.", + snippet: `MATCH (a:Person {name: 'Ada'}), (b:Person {name: 'Barbara'}) +RETURN issundb.link.commonNeighbors(a, b) AS shared`, + }, + { + name: "issundb.link.jaccard", + args: "a, b", + yields: "number", + summary: + "Shared neighbors over the size of the combined neighborhood, so a pair of quiet nodes is not out-scored by a pair of hubs.", + snippet: `MATCH (a:Person), (b:Person) WHERE id(a) < id(b) +RETURN a.name, b.name, issundb.link.jaccard(a, b) AS score +ORDER BY score DESC, a.name, b.name LIMIT 5`, + }, + { + name: "issundb.link.adamicAdar", + args: "a, b", + yields: "number", + summary: + "Shared neighbors weighted by 1/ln(degree), so a neighbor everybody knows counts for little. A shared neighbor of degree one contributes nothing.", + snippet: `MATCH (a:Person), (b:Person) WHERE id(a) < id(b) +RETURN a.name, b.name, issundb.link.adamicAdar(a, b) AS score +ORDER BY score DESC, a.name, b.name LIMIT 5`, + }, + { + name: "issundb.link.resourceAllocation", + args: "a, b", + yields: "number", + summary: + "Shared neighbors weighted by 1/degree, which penalizes a popular neighbor harder than Adamic-Adar does.", + snippet: `MATCH (a:Person), (b:Person) WHERE id(a) < id(b) +RETURN a.name, b.name, issundb.link.resourceAllocation(a, b) AS score +ORDER BY score DESC, a.name, b.name LIMIT 5`, + }, + { + name: "issundb.link.preferentialAttachment", + args: "a, b", + yields: "number", + summary: + "The product of the two degrees. It ignores shared neighbors entirely, so it scores pairs that have nothing in common.", + snippet: `MATCH (a:Person), (b:Person) WHERE id(a) < id(b) +RETURN a.name, b.name, issundb.link.preferentialAttachment(a, b) AS score +ORDER BY score DESC, a.name, b.name LIMIT 5`, + }, + { + name: "issundb.similarity.jaccard", + args: "listA, listB", + yields: "number", + summary: + "Set similarity over two lists of values, not over the graph. Intersection divided by union.", + snippet: `RETURN issundb.similarity.jaccard([1, 2, 3], [2, 3, 4]) AS score`, + }, + { + name: "issundb.similarity.overlap", + args: "listA, listB", + yields: "number", + summary: + "Intersection divided by the size of the smaller list, so a subset scores 1 however lopsided the pair is.", + snippet: `RETURN issundb.similarity.overlap([1, 2], [1, 2, 3, 4]) AS score`, + }, + { + name: "issundb.distance.cosine", + args: "vectorA, vectorB", + yields: "number", + summary: + "Cosine distance between two embeddings. Either argument may be a node, which resolves to its stored embedding, or a literal vector. Subtract from 1 for cosine similarity; a length mismatch is null.", + snippet: `RETURN issundb.distance.cosine([1.0, 0.0], [1.0, 0.0]) AS d, + 1 - issundb.distance.cosine([1.0, 0.0], [0.0, 1.0]) AS similarity`, + }, + { + name: "issundb.distance.euclidean", + args: "vectorA, vectorB", + yields: "number", + summary: + "Straight-line distance between two embeddings, each either a node or a literal vector.", + snippet: `RETURN issundb.distance.euclidean([0.0, 0.0], [3.0, 4.0]) AS d`, + }, + { + name: "vector_dist", + args: "a, b", + yields: "number", + summary: + "Distance between two embeddings under the graph's configured metric. Either argument may be a node, which resolves to its stored embedding, or a literal vector.", + snippet: `RETURN vector_dist([1.0, 0.0, 0.25], [0.0, 1.0, 0.25]) AS d`, + }, +]; + export const PROCEDURES = [ - { - name: "issundb.pageRank", - args: "[{iterations, damping}]", - yields: "nodeId, score", - summary: - "Ranks nodes by importance. A source spreads its rank across its edges, so parallel edges each carry mass, and dangling mass is not redistributed.", - snippet: `CALL issundb.pageRank({iterations: 20, damping: 0.85}) + { + name: "issundb.pageRank", + args: "[{iterations, damping}]", + yields: "nodeId, score", + summary: + "Ranks nodes by importance. A source spreads its rank across its edges, so parallel edges each carry mass, and dangling mass is not redistributed.", + snippet: `CALL issundb.pageRank({iterations: 20, damping: 0.85}) YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC`, - }, - { - name: "issundb.betweenness", - args: "", - yields: "nodeId, score", - summary: - "How often a node lies on a shortest path between two others, by Brandes' algorithm. Unnormalized, directed, and counted over distinct pairs.", - snippet: `CALL issundb.betweenness() + }, + { + name: "issundb.betweenness", + args: "", + yields: "nodeId, score", + summary: + "How often a node lies on a shortest path between two others, by Brandes' algorithm. Unnormalized, directed, and counted over distinct pairs.", + snippet: `CALL issundb.betweenness() YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC`, - }, - { - name: "issundb.harmonic", - args: "", - yields: "nodeId, score", - summary: - "Sums the reciprocal of the shortest-path distance to every other node, so an unreachable node contributes nothing rather than infinity.", - snippet: `CALL issundb.harmonic() + }, + { + name: "issundb.harmonic", + args: "", + yields: "nodeId, score", + summary: + "Sums the reciprocal of the shortest-path distance to every other node, so an unreachable node contributes nothing rather than infinity.", + snippet: `CALL issundb.harmonic() YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC`, - }, - { - name: "issundb.degree", - args: "[{direction}]", - yields: "nodeId, score", - summary: - "Counts distinct neighbors in one direction, so parallel edges between the same pair count once. Direction is IN, OUT, or BOTH.", - snippet: `CALL issundb.degree({direction: 'OUT'}) + }, + { + name: "issundb.degree", + args: "[{direction}]", + yields: "nodeId, score", + summary: + "Counts distinct neighbors in one direction, so parallel edges between the same pair count once. Direction is IN, OUT, or BOTH.", + snippet: `CALL issundb.degree({direction: 'OUT'}) YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC`, - }, - { - name: "issundb.wcc", - aka: "issundb.connectedComponents", - args: "", - yields: "nodeId, componentId", - summary: - "Weakly connected components by union-find, treating every edge as undirected. The component id is the smallest node id in the component.", - snippet: `CALL issundb.wcc() + }, + { + name: "issundb.wcc", + aka: "issundb.connectedComponents", + args: "", + yields: "nodeId, componentId", + summary: + "Weakly connected components by union-find, treating every edge as undirected. The component id is the smallest node id in the component.", + snippet: `CALL issundb.wcc() YIELD nodeId, componentId RETURN componentId, count(nodeId) AS size ORDER BY componentId`, - }, - { - name: "issundb.scc", - aka: "issundb.stronglyConnectedComponents", - args: "", - yields: "nodeId, componentId", - summary: - "Strongly connected components by Tarjan's algorithm, written iteratively so graph depth cannot reach the call stack. The browser stack is small, so that matters here.", - snippet: `CALL issundb.scc() + }, + { + name: "issundb.scc", + aka: "issundb.stronglyConnectedComponents", + args: "", + yields: "nodeId, componentId", + summary: + "Strongly connected components by Tarjan's algorithm, written iteratively so graph depth cannot reach the call stack. The browser stack is small, so that matters here.", + snippet: `CALL issundb.scc() YIELD nodeId, componentId RETURN componentId, count(nodeId) AS size ORDER BY size DESC, componentId`, - }, - { - name: "issundb.labelPropagation", - args: "[{maxIterations}]", - yields: "nodeId, communityId", - summary: - "Assigns each node the most common community among its neighbors, iterating to a fixed point. Ties break toward the smallest label, so the partition is stable run to run.", - snippet: `CALL issundb.labelPropagation({maxIterations: 10}) + }, + { + name: "issundb.labelPropagation", + args: "[{maxIterations}]", + yields: "nodeId, communityId", + summary: + "Assigns each node the most common community among its neighbors, iterating to a fixed point. Ties break toward the smallest label, so the partition is stable run to run.", + snippet: `CALL issundb.labelPropagation({maxIterations: 10}) +YIELD nodeId, communityId +RETURN nodeId, communityId +ORDER BY communityId, nodeId`, + }, + { + name: "issundb.closeness", + args: "", + yields: "nodeId, score", + summary: + "Reciprocal mean distance to every reachable node, scaled by the fraction of the graph reached, so a node in a small component does not outscore a well-connected one.", + snippet: `CALL issundb.closeness() +YIELD nodeId, score +RETURN nodeId, score +ORDER BY score DESC`, + }, + { + name: "issundb.eigenvector", + args: "[{iterations, tolerance}]", + yields: "nodeId, score", + summary: + "Ranks a node by how important the nodes pointing at it are, by power iteration. Scores are magnitudes scaled to sum to the node count.", + snippet: `CALL issundb.eigenvector({iterations: 100}) +YIELD nodeId, score +RETURN nodeId, score +ORDER BY score DESC`, + }, + { + name: "issundb.katz", + args: "[{alpha, beta, iterations, tolerance}]", + yields: "nodeId, score", + summary: + "Sums the walks reaching a node, attenuating a walk of length k by alpha^k, plus a beta baseline every node receives. Unlike eigenvector centrality it scores a node with no incoming edges.", + snippet: `CALL issundb.katz({alpha: 0.1, beta: 1.0}) +YIELD nodeId, score +RETURN nodeId, score +ORDER BY score DESC`, + }, + { + name: "issundb.clusteringCoefficient", + args: "", + yields: "nodeId, score", + summary: + "The fraction of a node's neighbor pairs that are themselves connected, read as undirected over distinct neighbors so the score stays within 0 and 1.", + snippet: `CALL issundb.clusteringCoefficient() +YIELD nodeId, score +RETURN nodeId, score +ORDER BY score DESC`, + }, + { + name: "issundb.louvain", + args: "", + yields: "nodeId, communityId", + summary: + "Community detection by modularity optimization with coarsening. Separates communities joined by a few edges, which label propagation tends to merge. The community id is the smallest node id it contains.", + snippet: `CALL issundb.louvain() YIELD nodeId, communityId RETURN nodeId, communityId ORDER BY communityId, nodeId`, - }, - { - name: "issundb.communities", - args: "[{maxIterations, topPerCommunity}]", - yields: "communityId, nodeId, rank", - summary: - "Label propagation, with each community's members ranked by PageRank. topPerCommunity keeps only the leading members of each.", - snippet: `CALL issundb.communities({topPerCommunity: 3}) + }, + { + name: "issundb.communities", + args: "[{maxIterations, topPerCommunity, algorithm}]", + yields: "communityId, nodeId, rank", + summary: + "A partition with each community's members ranked by PageRank. algorithm selects labelPropagation (the default) or louvain, and topPerCommunity keeps only the leading members of each.", + snippet: `CALL issundb.communities({topPerCommunity: 3, algorithm: 'louvain'}) YIELD communityId, nodeId, rank RETURN communityId, rank, nodeId ORDER BY communityId, rank`, - }, - { - name: "issundb.shortestPath", - args: "source, target", - yields: "index, nodeId", - summary: - "Breadth-first shortest path by hop count, yielding one row per node along the path in order. An unreachable target yields no rows.", - snippet: `CALL issundb.shortestPath(0, 5) + }, + { + name: "issundb.shortestPath", + args: "source, target", + yields: "index, nodeId", + summary: + "Breadth-first shortest path by hop count, yielding one row per node along the path in order. An unreachable target yields no rows.", + snippet: `CALL issundb.shortestPath(0, 5) YIELD index, nodeId RETURN index, nodeId ORDER BY index`, - }, - { - name: "issundb.dijkstra", - args: "source, target", - yields: "index, nodeId, totalWeight", - summary: - "Least-weight path from a binary heap. The weight is the first present of the weight, cost, capacity, or cap property, defaulting to 1, and totalWeight repeats on every row.", - snippet: `CALL issundb.dijkstra(0, 5) + }, + { + name: "issundb.dijkstra", + args: "source, target", + yields: "index, nodeId, totalWeight", + summary: + "Least-weight path from a binary heap. The weight is the first present of the weight, cost, capacity, or cap property, defaulting to 1, and totalWeight repeats on every row.", + snippet: `CALL issundb.dijkstra(0, 5) YIELD index, nodeId, totalWeight RETURN index, nodeId, totalWeight ORDER BY index`, - }, - { - name: "issundb.triangleCount", - args: "", - yields: "count", - summary: - "Counts assignments of the directed pattern (a)->(b)->(c)->(a), so one cycle of three distinct nodes counts three times, once per rotation, as a Cypher MATCH would return it.", - snippet: `CALL issundb.triangleCount() + }, + { + name: "issundb.triangleCount", + args: "", + yields: "count", + summary: + "Counts assignments of the directed pattern (a)->(b)->(c)->(a), so one cycle of three distinct nodes counts three times, once per rotation, as a Cypher MATCH would return it.", + snippet: `CALL issundb.triangleCount() YIELD count RETURN count AS triangle_rows`, - }, - { - name: "issundb.retrieve.vector", - args: "queryVector [, {k, hops, maxDistance, maxNodes}]", - yields: "nodeId, distance", - summary: - "Vector search seeds expanded by breadth-first traversal. Lower distance is closer, and it is null for a node reached only by expansion.", - snippet: `CALL issundb.retrieve.vector([1.0, 0.0, 0.25], {k: 3, hops: 1}) + }, + { + name: "issundb.retrieve.vector", + args: "queryVector [, {k, hops, maxDistance, maxNodes}]", + yields: "nodeId, distance", + summary: + "Vector search seeds expanded by breadth-first traversal. Lower distance is closer, and it is null for a node reached only by expansion.", + snippet: `CALL issundb.retrieve.vector([1.0, 0.0, 0.25], {k: 3, hops: 1}) YIELD nodeId, distance RETURN nodeId, distance ORDER BY nodeId`, - requiresVectors: true, - }, - { - name: "issundb.retrieve.hybrid", - args: "queryVector, queryText [, config]", - yields: "nodeId, score", - summary: - "Fuses vector and text relevance into one score before expanding. An empty query vector disables vector search, and an empty text query disables text search.", - snippet: `CALL issundb.retrieve.hybrid([1.0, 0.0, 0.25], 'graph', {vectorK: 3, textK: 3, hops: 1}) + requiresVectors: true, + }, + { + name: "issundb.retrieve.hybrid", + args: "queryVector, queryText [, config]", + yields: "nodeId, score", + summary: + "Fuses vector and text relevance into one score before expanding. An empty query vector disables vector search, and an empty text query disables text search.", + snippet: `CALL issundb.retrieve.hybrid([1.0, 0.0, 0.25], 'graph', {vectorK: 3, textK: 3, hops: 1}) YIELD nodeId, score RETURN nodeId, score ORDER BY nodeId`, - requiresVectors: true, - }, + requiresVectors: true, + }, ]; export const DEMO_CATEGORIES = [ - { - label: "Cypher basics", - sample: "social", - requiresLabel: "Person", - docs: "../cypher/", - demos: [ - { - label: "Create nodes", - desc: "Writes nodes and a relationship in one statement, and returns what it made. Every clause of a write statement shares one transaction, so an error anywhere rolls back all of it. Pick a Graph is where a whole dataset comes from.", - cypher: `CREATE (grete:Person {name: 'Grete', city: 'Berlin', age: 34}), + { + label: "Cypher basics", + sample: "social", + requiresLabel: "Person", + docs: "../cypher/", + demos: [ + { + label: "Create nodes", + desc: "Writes nodes and a relationship in one statement, and returns what it made. Every clause of a write statement shares one transaction, so an error anywhere rolls back all of it. Pick a Graph is where a whole dataset comes from.", + cypher: `CREATE (grete:Person {name: 'Grete', city: 'Berlin', age: 34}), (kurt:Person {name: 'Kurt', city: 'Vienna', age: 47}), (grete)-[:KNOWS {since: 1931, weight: 6}]->(kurt) RETURN grete.name AS created, kurt.name AS and_also`, - }, - { - label: "Match and filter", - desc: "Pattern matching with a WHERE predicate. The optimizer splits a top-level AND so each conjunct pushes down to its own lowest binder.", - cypher: `MATCH (p:Person) + }, + { + label: "Match and filter", + desc: "Pattern matching with a WHERE predicate. The optimizer splits a top-level AND so each conjunct pushes down to its own lowest binder.", + cypher: `MATCH (p:Person) WHERE p.city = 'London' AND p.age > 30 RETURN p.name AS name, p.age AS age ORDER BY age DESC`, - }, - { - label: "Traverse", - desc: "Follows a relationship. A typed hop is resolved as a bulk read of the in-memory CSR adjacency rather than a lookup per row.", - cypher: `MATCH (a:Person)-[:KNOWS]->(b:Person) + }, + { + label: "Traverse", + desc: "Follows a relationship. A typed hop is resolved as a bulk read of the in-memory CSR adjacency rather than a lookup per row.", + cypher: `MATCH (a:Person)-[:KNOWS]->(b:Person) RETURN a.name AS from, b.name AS to ORDER BY from, to`, - }, - { - label: "Variable length", - desc: "A path of one to three hops. The relationship variable binds to the whole list of relationships traversed, so size(r) is the hop count.", - cypher: `MATCH (a:Person {name: 'Ada'})-[r:KNOWS*1..3]->(b:Person) + }, + { + label: "Variable length", + desc: "A path of one to three hops. The relationship variable binds to the whole list of relationships traversed, so size(r) is the hop count.", + cypher: `MATCH (a:Person {name: 'Ada'})-[r:KNOWS*1..3]->(b:Person) RETURN b.name AS reached, size(r) AS hops ORDER BY hops, reached`, - }, - { - label: "Aggregate", - desc: "Groups and counts. A count grouped by one endpoint of a single hop lowers to a kernel that emits one entry per group instead of a row per edge.", - cypher: `MATCH (p:Person)-[:KNOWS]->(other) + }, + { + label: "Aggregate", + desc: "Groups and counts. A count grouped by one endpoint of a single hop lowers to a kernel that emits one entry per group instead of a row per edge.", + cypher: `MATCH (p:Person)-[:KNOWS]->(other) RETURN p.city AS city, count(other) AS outgoing ORDER BY outgoing DESC, city`, - }, - { - label: "Update and delete", - desc: "SET assigns a property and a label; a statement's own projection sees its uncommitted writes through a pending-writes overlay.", - cypher: `MATCH (p:Person {name: 'Donald'}) + }, + { + label: "Update and delete", + desc: "SET assigns a property and a label; a statement's own projection sees its uncommitted writes through a pending-writes overlay.", + cypher: `MATCH (p:Person {name: 'Donald'}) SET p.city = 'Palo Alto', p:Retired RETURN p.name AS name, p.city AS city, labels(p) AS labels`, - }, - ], - }, - { - label: "Graph algorithms", - sample: "social", - requiresLabel: "Person", - docs: "../examples/#graph-data-science-in-cypher", - demos: [ - { - label: "PageRank", - desc: "Ranks nodes by importance, and sizes the vertices in the graph view. A source spreads its rank across its edges, so parallel edges each carry mass.", - cypher: `CALL issundb.pageRank({iterations: 20, damping: 0.85}) + }, + ], + }, + { + label: "Graph algorithms", + sample: "social", + requiresLabel: "Person", + docs: "../examples/#graph-data-science-in-cypher", + demos: [ + { + label: "PageRank", + desc: "Ranks nodes by importance, and sizes the vertices in the graph view. A source spreads its rank across its edges, so parallel edges each carry mass.", + cypher: `CALL issundb.pageRank({iterations: 20, damping: 0.85}) YIELD nodeId, score MATCH (p) WHERE id(p) = nodeId RETURN p.name AS name, round(score * 10000) / 10000 AS pagerank ORDER BY pagerank DESC, name`, - }, - { - label: "Betweenness", - desc: "How often each node lies on a shortest path between two others, by Brandes' algorithm. It counts distinct pairs, so two parallel edges are one path.", - cypher: `CALL issundb.betweenness() + }, + { + label: "Betweenness", + desc: "How often each node lies on a shortest path between two others, by Brandes' algorithm. It counts distinct pairs, so two parallel edges are one path.", + cypher: `CALL issundb.betweenness() YIELD nodeId, score MATCH (p) WHERE id(p) = nodeId RETURN p.name AS name, round(score * 100) / 100 AS betweenness ORDER BY betweenness DESC, name`, - }, - { - label: "Degree and harmonic", - desc: "Two more centralities. Degree counts distinct neighbors in the chosen direction; harmonic sums the reciprocal of each shortest-path distance.", - cypher: `CALL issundb.degree({direction: 'OUT'}) + }, + { + label: "Degree and harmonic", + desc: "Two more centralities. Degree counts distinct neighbors in the chosen direction; harmonic sums the reciprocal of each shortest-path distance.", + cypher: `CALL issundb.degree({direction: 'OUT'}) YIELD nodeId, score MATCH (p) WHERE id(p) = nodeId RETURN p.name AS name, score AS out_degree ORDER BY out_degree DESC, name`, - }, - { - label: "Components", - desc: "Weakly connected components by union-find, treating every edge as undirected.", - cypher: `CALL issundb.wcc() + }, + { + label: "Components", + desc: "Weakly connected components by union-find, treating every edge as undirected.", + cypher: `CALL issundb.wcc() YIELD nodeId, componentId MATCH (p) WHERE id(p) = nodeId RETURN componentId, count(p) AS size, collect(p.name) AS members ORDER BY componentId`, - }, - { - label: "Strongly connected", - desc: "Tarjan's algorithm, written iteratively rather than recursively so graph depth cannot reach the call stack. That matters here: the browser stack is small.", - cypher: `CALL issundb.scc() + }, + { + label: "Strongly connected", + desc: "Tarjan's algorithm, written iteratively rather than recursively so graph depth cannot reach the call stack. That matters here: the browser stack is small.", + cypher: `CALL issundb.scc() YIELD nodeId, componentId MATCH (p) WHERE id(p) = nodeId RETURN componentId, count(p) AS size, collect(p.name) AS members ORDER BY size DESC, componentId`, - }, - { - label: "Shortest path", - desc: "Breadth-first shortest path by hop count, traced back through the incoming adjacency. Yields one row per node along the path, in order.", - cypher: `${PATH_NOTE} + }, + { + label: "Shortest path", + desc: "Breadth-first shortest path by hop count, traced back through the incoming adjacency. Yields one row per node along the path, in order.", + cypher: `${PATH_NOTE} CALL issundb.shortestPath(0, 5) YIELD index, nodeId MATCH (p) WHERE id(p) = nodeId RETURN index, p.name AS name ORDER BY index`, - }, - { - label: "Dijkstra", - desc: "Least-weight path from a binary heap. The weight is the first present of the weight, cost, capacity, or cap property, defaulting to 1, and totalWeight repeats on every row.", - cypher: `${PATH_NOTE} + }, + { + label: "Dijkstra", + desc: "Least-weight path from a binary heap. The weight is the first present of the weight, cost, capacity, or cap property, defaulting to 1, and totalWeight repeats on every row.", + cypher: `${PATH_NOTE} CALL issundb.dijkstra(0, 5) YIELD index, nodeId, totalWeight MATCH (p) WHERE id(p) = nodeId RETURN index, p.name AS name, totalWeight ORDER BY index`, - }, - { - label: "Triangles", - desc: "Counts assignments of the directed pattern (a)->(b)->(c)->(a), so one cycle of three distinct nodes counts three times, once per rotation, as a Cypher MATCH would return it. This lowers to a counting kernel that walks the adjacency arrays and tallies integers rather than materializing a row per match.", - cypher: `CALL issundb.triangleCount() + }, + { + label: "Triangles", + desc: "Counts assignments of the directed pattern (a)->(b)->(c)->(a), so one cycle of three distinct nodes counts three times, once per rotation, as a Cypher MATCH would return it. This lowers to a counting kernel that walks the adjacency arrays and tallies integers rather than materializing a row per match.", + cypher: `CALL issundb.triangleCount() YIELD count RETURN count AS triangle_rows`, - }, - { - label: "Communities", - desc: "Label propagation, then each community's members ranked by PageRank. Ties break toward the smallest label, so the partition is stable run to run.", - cypher: `CALL issundb.communities({topPerCommunity: 3}) + }, + { + label: "Communities", + desc: "Label propagation, then each community's members ranked by PageRank. Ties break toward the smallest label, so the partition is stable run to run.", + cypher: `CALL issundb.communities({topPerCommunity: 3}) YIELD communityId, nodeId, rank MATCH (p) WHERE id(p) = nodeId RETURN communityId, rank, p.name AS name ORDER BY communityId, rank`, - }, - ], - }, - { - label: "Query planning", - sample: "social", - requiresLabel: "Person", - docs: "../api-reference/#optimizer-statistics", - demos: [ - { - label: "An index seek", - desc: "A property equality over a labeled scan becomes an index seek, because every scalar node property is auto-indexed. Compare the plan with the range form below.", - cypher: `MATCH (p:Person) WHERE p.name = 'Ada' RETURN p.name, p.city`, - explain: true, - }, - { - label: "A range scan", - desc: "An inequality lowers to a range scan over the same index, bounded rather than seeked.", - cypher: `MATCH (p:Person) WHERE p.age > 40 RETURN p.name, p.age`, - explain: true, - }, - { - label: "A join linearized", - desc: "Two patterns sharing a variable. A join whose one side merely re-scans a variable the other already binds is rewritten into a linear expand-into chain.", - cypher: `MATCH (a:Person)-[:KNOWS]->(b:Person) + }, + ], + }, + { + label: "Link prediction", + sample: "social", + requiresLabel: "Person", + docs: "../api-reference/#link-prediction", + demos: [ + { + label: "Mutual connections", + desc: "How many neighbors two people share, and the same count as a ratio of their combined neighborhood. The neighborhood is undirected and distinct, so who pointed at whom does not matter and a repeated edge counts once.", + cypher: `MATCH (a:Person), (b:Person) +WHERE id(a) < id(b) +RETURN a.name AS a, b.name AS b, + issundb.link.commonNeighbors(a, b) AS mutual, + issundb.link.jaccard(a, b) AS jaccard +ORDER BY mutual DESC, a, b`, + }, + { + label: "Who might know whom", + desc: "The same score, but only for pairs not already connected, which is the question link prediction actually answers. Scoring an existing edge highly proves nothing, so the known pairs are collected first and excluded.", + cypher: `MATCH (x:Person)-[:KNOWS]->(y:Person) +WITH collect(toString(id(x)) + '>' + toString(id(y))) AS links +MATCH (a:Person), (b:Person) +WHERE id(a) < id(b) + AND NOT toString(id(a)) + '>' + toString(id(b)) IN links + AND NOT toString(id(b)) + '>' + toString(id(a)) IN links +RETURN a.name AS a, b.name AS b, + issundb.link.commonNeighbors(a, b) AS mutual, + issundb.link.adamicAdar(a, b) AS score +ORDER BY score DESC, a, b`, + }, + { + label: "The metrics disagree", + desc: "All five on the same pairs. Adamic-Adar and resource allocation discount a neighbor everybody shares, jaccard normalizes by neighborhood size, and preferential attachment ignores shared neighbors entirely: it multiplies the two degrees, so it ranks busy pairs highly even with nothing in common.", + cypher: `MATCH (a:Person), (b:Person) +WHERE id(a) < id(b) +RETURN a.name AS a, b.name AS b, + issundb.link.commonNeighbors(a, b) AS common, + issundb.link.jaccard(a, b) AS jaccard, + issundb.link.adamicAdar(a, b) AS adamicAdar, + issundb.link.resourceAllocation(a, b) AS resourceAlloc, + issundb.link.preferentialAttachment(a, b) AS prefAttach +ORDER BY adamicAdar DESC, prefAttach DESC, a, b`, + }, + ], + }, + { + label: "Query planning", + sample: "social", + requiresLabel: "Person", + docs: "../api-reference/#optimizer-statistics", + demos: [ + { + label: "An index seek", + desc: "A property equality over a labeled scan becomes an index seek, because every scalar node property is auto-indexed. Compare the plan with the range form below.", + cypher: `MATCH (p:Person) WHERE p.name = 'Ada' RETURN p.name, p.city`, + explain: true, + }, + { + label: "A range scan", + desc: "An inequality lowers to a range scan over the same index, bounded rather than seeked.", + cypher: `MATCH (p:Person) WHERE p.age > 40 RETURN p.name, p.age`, + explain: true, + }, + { + label: "A join linearized", + desc: "Two patterns sharing a variable. A join whose one side merely re-scans a variable the other already binds is rewritten into a linear expand-into chain.", + cypher: `MATCH (a:Person)-[:KNOWS]->(b:Person) MATCH (a)-[:KNOWS]->(c:Person) RETURN a.name, b.name, c.name`, - explain: true, - }, - { - label: "An aggregate lowered", - desc: "A grouped count over one hop lowers to the GroupedDegree kernel, which emits one entry per group rather than expanding every edge into a row.", - cypher: `MATCH (p:Person)-[:KNOWS]->(o) RETURN p.name, count(o)`, - explain: true, - }, - ], - }, - { - label: "Full-text search", - sample: "articles", - requiresLabel: "Article", - docs: "../examples/", - demos: [ - { - label: "Index and search", - desc: "Provisions a full-text index over Article.body, then ranks the corpus for a query. The postings are written inside the same transaction as the node, so the index is transactional rather than eventually consistent and a hit is never stale.", - cypher: `MATCH (a:Article) + explain: true, + }, + { + label: "An aggregate lowered", + desc: "A grouped count over one hop lowers to the GroupedDegree kernel, which emits one entry per group rather than expanding every edge into a row.", + cypher: `MATCH (p:Person)-[:KNOWS]->(o) RETURN p.name, count(o)`, + explain: true, + }, + ], + }, + { + label: "Full-text search", + sample: "articles", + requiresLabel: "Article", + docs: "../examples/", + demos: [ + { + label: "Index and search", + desc: "Provisions a full-text index over Article.body, then ranks the corpus for a query. The postings are written inside the same transaction as the node, so the index is transactional rather than eventually consistent and a hit is never stale.", + cypher: `MATCH (a:Article) RETURN a.title AS title, a.year AS year ORDER BY year`, - textIndex: ["Article", "body"], - textSearch: "graph relationships", - }, - ], - }, - { - label: "Vector search", - sample: "social", - requiresLabel: "Person", - docs: "../api-reference/#vector-search-extensions", - demos: [ - { - label: "Nearest neighbours", - desc: "Attaches a three-dimensional embedding to each person, then finds the closest to a query vector. This build uses the exact backend, so these are the true nearest neighbours rather than approximate ones.", - cypher: `MATCH (p:Person) RETURN id(p) AS id, p.name AS name ORDER BY id`, - vectors: { label: "Person", caption: "name" }, - }, - ], - }, - { - label: "GraphRAG", - sample: "articles", - requiresLabel: "Article", - docs: "../hybrid-retrieval/", - demos: [ - { - label: "Retrieve context", - desc: "The shape a retrieval-augmented prompt is assembled from: provision a full-text index over the corpus, then rank it by BM25. The index is written inside the same transaction as the node, so a hit is never stale.", - cypher: `MATCH (a:Article) + textIndex: ["Article", "body"], + textSearch: "graph relationships", + }, + ], + }, + { + label: "Vector search", + sample: "social", + requiresLabel: "Person", + docs: "../api-reference/#vector-search-extensions", + demos: [ + { + label: "Nearest neighbours", + desc: "Attaches a three-dimensional embedding to each person, then finds the closest to a query vector. This build uses the exact backend, so these are the true nearest neighbours rather than approximate ones.", + cypher: `MATCH (p:Person) RETURN id(p) AS id, p.name AS name ORDER BY id`, + vectors: {label: "Person", caption: "name"}, + }, + ], + }, + { + label: "GraphRAG", + sample: "articles", + requiresLabel: "Article", + docs: "../hybrid-retrieval/", + demos: [ + { + label: "Retrieve context", + desc: "The shape a retrieval-augmented prompt is assembled from: provision a full-text index over the corpus, then rank it by BM25. The index is written inside the same transaction as the node, so a hit is never stale.", + cypher: `MATCH (a:Article) RETURN a.title AS title, a.year AS year ORDER BY year`, - textIndex: ["Article", "body"], - textSearch: "graph search relevance", - }, - { - label: "Semantic neighbours", - desc: "The same corpus reached by embedding rather than by wording. This build searches by exact distance, so these are the true nearest neighbours rather than approximate ones.", - cypher: `MATCH (a:Article) + textIndex: ["Article", "body"], + textSearch: "graph search relevance", + }, + { + label: "Semantic neighbours", + desc: "The same corpus reached by embedding rather than by wording. This build searches by exact distance, so these are the true nearest neighbours rather than approximate ones.", + cypher: `MATCH (a:Article) RETURN a.title AS title, a.year AS year ORDER BY year`, - vectors: { label: "Article", caption: "title" }, - }, - { - label: "Fuse text and vectors", - desc: "Hybrid retrieval in one call: vector hits and text hits are scored, fused by reciprocal rank, and expanded over the graph before anything is returned. A node reached only by expansion has a null score, which is why the ordering puts nulls last.", - cypher: `MATCH (a:Article) + vectors: {label: "Article", caption: "title"}, + }, + { + label: "Fuse text and vectors", + desc: "Hybrid retrieval in one call: vector hits and text hits are scored, fused by reciprocal rank, and expanded over the graph before anything is returned. A node reached only by expansion has a null score, which is why the ordering puts nulls last.", + cypher: `MATCH (a:Article) RETURN a.title AS title, a.year AS year ORDER BY year`, - embed: { label: "Article" }, - textIndex: ["Article", "body"], - thenQuery: `CALL issundb.retrieve.hybrid([1.0, 0.0, 0.25], 'graph relevance', + embed: {label: "Article"}, + textIndex: ["Article", "body"], + thenQuery: `CALL issundb.retrieve.hybrid([1.0, 0.0, 0.25], 'graph relevance', {vectorK: 2, textK: 2, hops: 1, textLabel: 'Article', textProperty: 'body'}) YIELD nodeId, score MATCH (a:Article) WHERE id(a) = nodeId RETURN a.title AS title, score ORDER BY score IS NULL, score DESC, title`, - }, - { - label: "Ground an answer", - desc: "What a language model would be handed: the retrieved documents plus the ones they cite, collected into one list per seed. Assembling context is a traversal, which is the argument for keeping it in the database.", - cypher: `MATCH (a:Article)-[:CITES]->(cited:Article) + }, + { + label: "Ground an answer", + desc: "What a language model would be handed: the retrieved documents plus the ones they cite, collected into one list per seed. Assembling context is a traversal, which is the argument for keeping it in the database.", + cypher: `MATCH (a:Article)-[:CITES]->(cited:Article) WHERE a.title IN ['Hybrid retrieval', 'Graph databases'] RETURN a.title AS seed, collect(cited.title) AS also_read ORDER BY seed`, - }, - ], - }, - { - label: "Knowledge graph", - sample: "knowledge", - requiresLabel: "Researcher", - docs: "../cypher/", - demos: [ - { - label: "Entities and relations", - desc: "A small research graph: people, the labs they work in, the papers they wrote, and the concepts those mention. Four labels and three relationship types, which is the shape most knowledge graphs reduce to.", - cypher: `MATCH (r:Researcher)-[:AUTHORED]->(p:Paper)-[:MENTIONS]->(c:Concept) + }, + ], + }, + { + label: "Knowledge graph", + sample: "knowledge", + requiresLabel: "Researcher", + docs: "../cypher/", + demos: [ + { + label: "Entities and relations", + desc: "A small research graph: people, the labs they work in, the papers they wrote, and the concepts those mention. Four labels and three relationship types, which is the shape most knowledge graphs reduce to.", + cypher: `MATCH (r:Researcher)-[:AUTHORED]->(p:Paper)-[:MENTIONS]->(c:Concept) RETURN r.name AS researcher, p.title AS paper, c.name AS concept ORDER BY researcher, paper, concept`, - }, - { - label: "Multi-hop question", - desc: "\"Which cities work on retrieval augmented generation?\" is three hops and a group-by, not a join plan a reader has to write. The optimizer splits the conjunction so each filter pushes down to its own lowest binder.", - cypher: `MATCH (c:Concept {name: 'retrieval augmented generation'})<-[:MENTIONS]-(p:Paper)<-[:AUTHORED]-(r:Researcher)-[:WORKS_IN]->(lab:Lab) + }, + { + label: "Multi-hop question", + desc: "\"Which cities work on retrieval augmented generation?\" is three hops and a group-by, not a join plan a reader has to write. The optimizer splits the conjunction so each filter pushes down to its own lowest binder.", + cypher: `MATCH (c:Concept {name: 'retrieval augmented generation'})<-[:MENTIONS]-(p:Paper)<-[:AUTHORED]-(r:Researcher)-[:WORKS_IN]->(lab:Lab) RETURN lab.city AS city, count(DISTINCT p) AS papers, collect(DISTINCT r.name) AS researchers ORDER BY city`, - }, - { - label: "Shared concepts", - desc: "Two researchers connected by what they write about rather than by an edge between them. The closing hop of a cyclic pattern is fused rather than materialized as a wedge per intermediate node.", - cypher: `MATCH (a:Researcher)-[:AUTHORED]->(:Paper)-[:MENTIONS]->(c:Concept)<-[:MENTIONS]-(:Paper)<-[:AUTHORED]-(b:Researcher) + }, + { + label: "Shared concepts", + desc: "Two researchers connected by what they write about rather than by an edge between them. The closing hop of a cyclic pattern is fused rather than materialized as a wedge per intermediate node.", + cypher: `MATCH (a:Researcher)-[:AUTHORED]->(:Paper)-[:MENTIONS]->(c:Concept)<-[:MENTIONS]-(:Paper)<-[:AUTHORED]-(b:Researcher) WHERE a.name < b.name RETURN a.name AS one, b.name AS other, collect(DISTINCT c.name) AS shared ORDER BY one, other`, - }, - { - label: "Reach by hops", - desc: "How far each concept sits from one researcher, over any of the three relationship types. The relationship variable binds to the whole list traversed, so size(path) is the hop count.", - cypher: `MATCH (r:Researcher {name: 'Ada Ito'})-[path*1..3]->(c:Concept) + }, + { + label: "Reach by hops", + desc: "How far each concept sits from one researcher, over any of the three relationship types. The relationship variable binds to the whole list traversed, so size(path) is the hop count.", + cypher: `MATCH (r:Researcher {name: 'Ada Ito'})-[path*1..3]->(c:Concept) RETURN c.name AS concept, min(size(path)) AS hops ORDER BY hops, concept`, - }, - ], - }, + }, + ], + }, ]; diff --git a/web/format.js b/web/format.js new file mode 100644 index 0000000..554e9e8 --- /dev/null +++ b/web/format.js @@ -0,0 +1,213 @@ +// The Cypher formatter. +// +// Its own module so `scripts/check_playground.mjs` can import it. That check is the whole safety +// argument for running a formatter over a query: it runs every string in `demos.js` before and +// after formatting and compares the rows, so a casing or line-breaking rule that changed what a +// query means fails the build. Reaching the formatter from `app.js` was impossible while it lived +// there, since that module touches the DOM at import time and a Node process has none. + +// Clause phrases that begin a line, longest first so `ON CREATE SET` is recognized before the `SET` +// inside it. +const CLAUSE_PHRASES = [ + ["ON", "CREATE", "SET"], + ["ON", "MATCH", "SET"], + ["OPTIONAL", "MATCH"], + ["DETACH", "DELETE"], + ["ORDER", "BY"], + ["UNION", "ALL"], + ["MATCH"], + ["WHERE"], + ["WITH"], + ["RETURN"], + ["SKIP"], + ["LIMIT"], + ["CREATE"], + ["MERGE"], + ["SET"], + ["REMOVE"], + ["DELETE"], + ["UNWIND"], + ["CALL"], + ["YIELD"], + ["UNION"], + ["FOREACH"], +]; + +// The clauses whose comma-separated items are patterns rather than expressions. Breaking after each +// comma there turns a long line into a readable list of paths; doing it in RETURN would scatter a +// projection over as many lines as it has columns. +const PATTERN_CLAUSES = new Set(["CREATE", "MERGE"]); + +// Deliberately much narrower than the highlighter's keyword set. Uppercasing everything that set +// contains rewrote `issundb.shortestPath` to `issundb.SHORTESTPATH`, and the yield fields `index` +// and `count` to `INDEX` and `COUNT`, all three of which are case-sensitive names rather than +// syntax. So only operators are listed here, and a clause word is uppercased because the phrase +// scan recognized it as one, not because it appears in a list. Function names are left alone: an +// aggregate is conventionally lowercase, and `all(` is not the `ALL` of `UNION ALL`. +const FORMAT_UPPERCASE = new Set([ + "and", + "or", + "xor", + "not", + "in", + "is", + "null", + "true", + "false", + "distinct", + "as", + "asc", + "desc", + "ascending", + "descending", + "starts", + "ends", + "contains", +]); + +const FORMAT_TOKEN = new RegExp( + [ + "(\\/\\/[^\\n]*|\\/\\*[\\s\\S]*?\\*\\/)", + "('(?:[^'\\\\]|\\\\.)*'|\"(?:[^\"\\\\]|\\\\.)*\")", + "([A-Za-z_]\\w*)", + "(\\s+)", + "([^\\s])", + ].join("|"), + "g", +); + +// Line breaking and keyword casing, and nothing else. Spacing within a line is left as written apart +// from collapsing runs of whitespace, because re-spacing would have to know that the `-` in +// `-[:KNOWS]->` and the `*` in `[r*1..3]` are not binary operators. That restraint is what makes the +// pass safe to run on any query: it cannot change what the query means. +export function formatCypher(src) { + const tokens = [...src.matchAll(FORMAT_TOKEN)].map((m) => ({ + comment: m[1], + string: m[2], + word: m[3], + space: m[4], + other: m[5], + text: m[0], + })); + + // A bracket depth per token, so a clause word inside a pattern or a map is not mistaken for the + // start of a line, and the index of every word, so a phrase can be matched by lookahead. + let depth = 0; + const words = []; + tokens.forEach((token, i) => { + token.depth = depth; + if (token.other && "([{".includes(token.other)) depth += 1; + if (token.other && ")]}".includes(token.other)) depth -= 1; + if (token.word) words.push(i); + }); + + const previousWordOf = (index) => { + for (let j = index - 1; j >= 0; j -= 1) { + if (tokens[j].space || tokens[j].comment) continue; + return tokens[j]; + } + return null; + }; + + const nextNonSpaceOf = (index) => { + for (let j = index + 1; j < tokens.length; j += 1) { + if (tokens[j].space) continue; + return tokens[j]; + } + return null; + }; + + // `n.set` and `:Match` are names. Guarding the phrase scan and not only the casing is what stops + // `RETURN n.set` from being broken across two lines at the property. + const isQualifiedName = (index) => { + const previous = previousWordOf(index); + if (previous && (previous.other === "." || previous.other === ":")) return true; + return Boolean(previous && previous.word && previous.word.toLowerCase() === "as"); + }; + + const upperOf = (index) => (index === undefined ? "" : tokens[index].word.toUpperCase()); + const breakAt = new Set(); + const consumed = new Set(); + const phraseWords = new Set(); + words.forEach((i, w) => { + if (consumed.has(i) || tokens[i].depth !== 0 || isQualifiedName(i)) return; + const phrase = CLAUSE_PHRASES.find((candidate) => + candidate.every((word, k) => upperOf(words[w + k]) === word), + ); + if (!phrase) return; + breakAt.add(i); + tokens[i].clause = phrase.join(" "); + phrase.forEach((_, k) => phraseWords.add(words[w + k])); + for (let k = 1; k < phrase.length; k += 1) consumed.add(words[w + k]); + }); + + function shouldUppercase(index) { + if (phraseWords.has(index)) return true; + if (!FORMAT_UPPERCASE.has(tokens[index].word.toLowerCase())) return false; + if (isQualifiedName(index)) return false; + // A word the phrase scan did not claim, followed by an open parenthesis, is a function name + // rather than an operator. A clause keyword is exempt, since `MATCH (` is still a clause. + const next = nextNonSpaceOf(index); + return !(next && next.other === "("); + } + + let out = ""; + let atLineStart = true; + let pendingSpace = false; + let clause = ""; + + const newline = () => { + if (!atLineStart) out += "\n"; + atLineStart = true; + pendingSpace = false; + }; + + tokens.forEach((token, i) => { + if (token.space) { + pendingSpace = out.length > 0; + return; + } + + // A comment runs to the end of its line, so it has to keep one to itself or it would swallow + // whatever the formatter put after it. + if (token.comment) { + newline(); + out += token.text; + out += "\n"; + atLineStart = true; + return; + } + + if (breakAt.has(i)) { + newline(); + clause = token.clause; + } + + if (pendingSpace && !atLineStart) out += " "; + pendingSpace = false; + + if (token.word) { + out += shouldUppercase(i) ? token.word.toUpperCase() : token.word; + atLineStart = false; + return; + } + + if (token.other === ";" && token.depth === 0) { + out += ";\n"; + atLineStart = true; + clause = ""; + return; + } + + if (token.other === "," && token.depth === 0 && PATTERN_CLAUSES.has(clause)) { + out += ",\n" + " ".repeat(clause.length + 1); + atLineStart = true; + return; + } + + out += token.text; + atLineStart = false; + }); + + return out.replace(/[ \t]+$/gm, "").trim(); +} diff --git a/web/index.html b/web/index.html index 31bf232..2c60223 100644 --- a/web/index.html +++ b/web/index.html @@ -1,301 +1,333 @@ - - - + + + IssunDB Playground - - + + - - + + - - -
    -
    -
    Loading IssunDB…
    -
    - It is just an 8 MB WebAssembly module. Hang on. We are almost there. -
    + + +
    +
    +
    Loading IssunDB Playground
    +
    + Hang on. We are almost there.
    +
    -
    - +
    + IssunDB Playground -
    -
    -
    +
    + - - + GitHub + + + -
    -
    +
    +
    -
    -
    -

    Cypher Editor

    - Press Ctrl + Enter to run -
    -
    - - -
    -
    - - - - - - - - -
    -
    +
    +
    +

    Cypher Editor

    + Press Ctrl + Enter to run +
    +
    + + + +
    +
    + + + + + + + + + +
    +
    + + - +
    +
    + + + + +
    + + +
    -
    -
    - - - - -
    - - -
    +
    Run a query to view results.
    -
    Run a query to view results.
    +
    +
    -
    -
    +
    +
    + +
    +
    + + + + +
    + + +
    -
    -
    - -
    -
    - - - - +
    +
    - - -
    - -
    -
    -
    -
    +
    -
    +
    -
    - -
    - -
    +
    + +
    + +
    - - + + diff --git a/web/style.css b/web/style.css index 69b7874..fc13f00 100644 --- a/web/style.css +++ b/web/style.css @@ -16,530 +16,530 @@ :root, [data-md-color-scheme="default"] { - --md-hue: 225deg; + --md-hue: 225deg; - --md-default-fg-color: #000000de; - --md-default-fg-color--light: #0000008a; - --md-default-fg-color--lighter: #00000052; - --md-default-fg-color--lightest: #00000012; - --md-default-bg-color: #fff; + --md-default-fg-color: #000000de; + --md-default-fg-color--light: #0000008a; + --md-default-fg-color--lighter: #00000052; + --md-default-fg-color--lightest: #00000012; + --md-default-bg-color: #fff; - --md-code-fg-color: #36464e; - --md-code-bg-color: #f5f5f5; + --md-code-fg-color: #36464e; + --md-code-bg-color: #f5f5f5; - --md-primary-fg-color: #7e56c2; - --md-primary-fg-color--light: #9574cd; - --md-primary-fg-color--dark: #673ab6; - --md-primary-bg-color: #fff; + --md-primary-fg-color: #7e56c2; + --md-primary-fg-color--light: #9574cd; + --md-primary-fg-color--dark: #673ab6; + --md-primary-bg-color: #fff; - --md-accent-fg-color: #fa0; - --md-typeset-a-color: var(--md-primary-fg-color); + --md-accent-fg-color: #fa0; + --md-typeset-a-color: var(--md-primary-fg-color); - --md-shadow-z1: 0 0.2rem 0.5rem #0000000d, 0 0 0.05rem #0000001a; - --md-shadow-z2: 0 0.2rem 0.5rem #0000001a, 0 0 0.05rem #00000040; + --md-shadow-z1: 0 0.2rem 0.5rem #0000000d, 0 0 0.05rem #0000001a; + --md-shadow-z2: 0 0.2rem 0.5rem #0000001a, 0 0 0.05rem #00000040; - --md-text-font: "Inter", system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, + --md-text-font: "Inter", system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; - --md-code-font: "JetBrains Mono", ui-monospace, "SF Mono", "Cascadia Mono", Menlo, + --md-code-font: "JetBrains Mono", ui-monospace, "SF Mono", "Cascadia Mono", Menlo, Consolas, monospace; - --ok: #1c7c54; - --err: #b3261e; - --panel-bg: var(--md-default-bg-color); - --line: var(--md-default-fg-color--lightest); - - --syn-kw: #7e56c2; - --syn-fn: #0b6bcb; - --syn-str: #0f7b4f; - --syn-num: #b5480c; - --syn-com: var(--md-default-fg-color--light); - --syn-lbl: #b1176b; - --syn-op: var(--md-default-fg-color--light); - - /* The layout is a set of cards on a page, so the page and a card need different - backgrounds. Material's own palette has only one, hence these two. */ - /* Every button, select, and input shares these, so a row of them lines up instead of each one - being sized by its own padding. `min-height` rather than `height`, so a label that wraps grows - the control instead of being clipped by it. */ - --control-h: 34px; - --control-h-sm: 27px; - --control-radius: 6px; - - --page-bg: hsla(var(--md-hue), 16%, 96%, 1); - --card-bg: #fff; - --soft-bg: hsla(var(--md-hue), 16%, 97%, 1); - --accent-fg: #6d4200; - - /* Without this the browser draws its scrollbars, and the form controls it renders natively, - from the light palette regardless of the scheme, which on the dark scheme puts white bars - down the side of every scrolling panel. */ - color-scheme: light; + --ok: #1c7c54; + --err: #b3261e; + --panel-bg: var(--md-default-bg-color); + --line: var(--md-default-fg-color--lightest); + + --syn-kw: #7e56c2; + --syn-fn: #0b6bcb; + --syn-str: #0f7b4f; + --syn-num: #b5480c; + --syn-com: var(--md-default-fg-color--light); + --syn-lbl: #b1176b; + --syn-op: var(--md-default-fg-color--light); + + /* The layout is a set of cards on a page, so the page and a card need different + backgrounds. Material's own palette has only one, hence these two. */ + /* Every button, select, and input shares these, so a row of them lines up instead of each one + being sized by its own padding. `min-height` rather than `height`, so a label that wraps grows + the control instead of being clipped by it. */ + --control-h: 34px; + --control-h-sm: 27px; + --control-radius: 6px; + + --page-bg: hsla(var(--md-hue), 16%, 96%, 1); + --card-bg: #fff; + --soft-bg: hsla(var(--md-hue), 16%, 97%, 1); + --accent-fg: #6d4200; + + /* Without this the browser draws its scrollbars, and the form controls it renders natively, + from the light palette regardless of the scheme, which on the dark scheme puts white bars + down the side of every scrolling panel. */ + color-scheme: light; } [data-md-color-scheme="slate"] { - --md-default-fg-color: hsla(var(--md-hue), 15%, 90%, 0.82); - --md-default-fg-color--light: hsla(var(--md-hue), 15%, 90%, 0.56); - --md-default-fg-color--lighter: hsla(var(--md-hue), 15%, 90%, 0.32); - --md-default-fg-color--lightest: hsla(var(--md-hue), 15%, 90%, 0.12); - --md-default-bg-color: hsla(var(--md-hue), 15%, 14%, 1); + --md-default-fg-color: hsla(var(--md-hue), 15%, 90%, 0.82); + --md-default-fg-color--light: hsla(var(--md-hue), 15%, 90%, 0.56); + --md-default-fg-color--lighter: hsla(var(--md-hue), 15%, 90%, 0.32); + --md-default-fg-color--lightest: hsla(var(--md-hue), 15%, 90%, 0.12); + --md-default-bg-color: hsla(var(--md-hue), 15%, 14%, 1); - --md-code-fg-color: hsla(var(--md-hue), 18%, 86%, 0.82); - --md-code-bg-color: hsla(var(--md-hue), 15%, 18%, 1); + --md-code-fg-color: hsla(var(--md-hue), 18%, 86%, 0.82); + --md-code-bg-color: hsla(var(--md-hue), 15%, 18%, 1); - --md-typeset-a-color: #a47bea; + --md-typeset-a-color: #a47bea; - --ok: #5fd39b; - --err: #f2867d; - --panel-bg: hsla(var(--md-hue), 15%, 18%, 1); + --ok: #5fd39b; + --err: #f2867d; + --panel-bg: hsla(var(--md-hue), 15%, 18%, 1); - --syn-kw: #c4a7ff; - --syn-fn: #7cc4ff; - --syn-str: #7ddba4; - --syn-num: #ffb27a; - --syn-lbl: #ff9ecb; + --syn-kw: #c4a7ff; + --syn-fn: #7cc4ff; + --syn-str: #7ddba4; + --syn-num: #ffb27a; + --syn-lbl: #ff9ecb; - --page-bg: hsla(var(--md-hue), 15%, 11%, 1); - --card-bg: hsla(var(--md-hue), 15%, 15%, 1); - --soft-bg: hsla(var(--md-hue), 15%, 18%, 1); - --accent-fg: #ffc65c; + --page-bg: hsla(var(--md-hue), 15%, 11%, 1); + --card-bg: hsla(var(--md-hue), 15%, 15%, 1); + --soft-bg: hsla(var(--md-hue), 15%, 18%, 1); + --accent-fg: #ffc65c; - color-scheme: dark; + color-scheme: dark; } * { - box-sizing: border-box; + box-sizing: border-box; } /* Every size below is in rem, so this one number scales the whole page. A percentage rather than a pixel count, because it then still follows a reader who has changed their browser's default text size instead of overriding them. 93.75% is 15px against the usual 16. */ html { - font-size: 93.75%; + font-size: 93.75%; } body { - margin: 0; - font-family: var(--md-text-font); - font-size: 1rem; - line-height: 1.5; - color: var(--md-default-fg-color); - background: var(--page-bg); - height: 100vh; - display: flex; - flex-direction: column; - overflow: hidden; + margin: 0; + font-family: var(--md-text-font); + font-size: 1rem; + line-height: 1.5; + color: var(--md-default-fg-color); + background: var(--page-bg); + height: 100vh; + display: flex; + flex-direction: column; + overflow: hidden; } button, select, input { - font: inherit; - color: inherit; + font: inherit; + color: inherit; } button { - cursor: pointer; + cursor: pointer; } a { - color: var(--md-typeset-a-color); + color: var(--md-typeset-a-color); } /* ---------------------------------------------------------------- header ---- */ /* Material's header is a solid primary bar with `--md-primary-bg-color` text. */ header { - display: flex; - align-items: center; - gap: 12px; - padding: 0 14px; - height: 48px; - flex: 0 0 auto; - background: var(--md-primary-fg-color); - color: var(--md-primary-bg-color); - box-shadow: var(--md-shadow-z1); - z-index: 10; + display: flex; + align-items: center; + gap: 12px; + padding: 0 14px; + height: 48px; + flex: 0 0 auto; + background: var(--md-primary-fg-color); + color: var(--md-primary-bg-color); + box-shadow: var(--md-shadow-z1); + z-index: 10; } .brand { - display: flex; - align-items: center; - gap: 9px; - font-weight: 700; - letter-spacing: -0.02em; - font-size: 1.25rem; + display: flex; + align-items: center; + gap: 9px; + font-weight: 700; + letter-spacing: -0.02em; + font-size: 1.25rem; } /* The documentation's logo is drawn with dark strokes, so it needs a light tile to read against the primary-colored bar rather than being recolored. */ .brand-logo { - flex: 0 0 auto; - border-radius: 5px; - background: #fff; - padding: 2px; - box-shadow: 0 1px 4px rgba(0, 0, 0, 0.18); + flex: 0 0 auto; + border-radius: 5px; + background: #fff; + padding: 2px; + box-shadow: 0 1px 4px rgba(0, 0, 0, 0.18); } /* The second word in the accent color, as the reference header does it. */ .brand-sub { - color: var(--md-accent-fg-color); - font-weight: 700; + color: var(--md-accent-fg-color); + font-weight: 700; } .spacer { - flex: 1; + flex: 1; } .site-nav { - display: flex; - align-items: center; - gap: 4px; + display: flex; + align-items: center; + gap: 4px; } header .hdr, .site-nav a { - background: none; - border: 0; - border-radius: 4px; - padding: 6px 9px; - color: var(--md-primary-bg-color); - opacity: 0.88; - font-size: 1rem; - font-weight: 500; - text-decoration: none; - transition: background 0.12s, opacity 0.12s; + background: none; + border: 0; + border-radius: 4px; + padding: 6px 9px; + color: var(--md-primary-bg-color); + opacity: 0.88; + font-size: 1rem; + font-weight: 500; + text-decoration: none; + transition: background 0.12s, opacity 0.12s; } header .hdr:hover, .site-nav a:hover { - background: rgba(255, 255, 255, 0.14); - opacity: 1; + background: rgba(255, 255, 255, 0.14); + opacity: 1; } #toggle-side { - display: none; + display: none; } /* A circle with a hairline ring, which is what distinguishes the icon controls from the text links beside them. */ header .hdr.icon { - display: inline-grid; - place-items: center; - width: 32px; - height: 32px; - padding: 0; - border-radius: 999px; - border: 1px solid rgba(255, 255, 255, 0.28); + display: inline-grid; + place-items: center; + width: 32px; + height: 32px; + padding: 0; + border-radius: 999px; + border: 1px solid rgba(255, 255, 255, 0.28); } /* ----------------------------------------------------------------- shell ---- */ .shell { - flex: 1; - display: flex; - min-height: 0; - overflow-y: auto; - overflow-x: hidden; + flex: 1; + display: flex; + min-height: 0; + overflow-y: auto; + overflow-x: hidden; } /* The whole app is one centered column rather than edge to edge, so on a wide display the editor stays a readable width instead of stretching across it. */ .wrap { - flex: 1; - display: flex; - align-items: flex-start; - gap: 16px; - width: 100%; - max-width: 1280px; - margin: 0 auto; - padding: 16px; + flex: 1; + display: flex; + align-items: flex-start; + gap: 16px; + width: 100%; + max-width: 1280px; + margin: 0 auto; + padding: 16px; } aside { - width: 320px; - flex: 0 0 320px; - display: flex; - flex-direction: column; - gap: 16px; + width: 320px; + flex: 0 0 320px; + display: flex; + flex-direction: column; + gap: 16px; } aside.hidden { - display: none; + display: none; } /* -------------------------------------------------------- cards and panels ---- */ .panel, .card { - background: var(--card-bg); - border: 1px solid var(--line); - border-radius: 8px; - box-shadow: var(--md-shadow-z1); + background: var(--card-bg); + border: 1px solid var(--line); + border-radius: 8px; + box-shadow: var(--md-shadow-z1); } .panel { - flex: 0 0 auto; - padding-bottom: 14px; + flex: 0 0 auto; + padding-bottom: 14px; } .panel > h2, .card-head { - display: flex; - align-items: center; - gap: 8px; - margin: 0; - padding: 11px 14px; - border-bottom: 1px solid var(--line); - font-size: 0.95rem; - font-weight: 600; - letter-spacing: 0.05em; - text-transform: uppercase; - color: var(--md-default-fg-color--light); + display: flex; + align-items: center; + gap: 8px; + margin: 0; + padding: 11px 14px; + border-bottom: 1px solid var(--line); + font-size: 0.95rem; + font-weight: 600; + letter-spacing: 0.05em; + text-transform: uppercase; + color: var(--md-default-fg-color--light); } .panel > h2 svg { - flex: 0 0 auto; - opacity: 0.8; + flex: 0 0 auto; + opacity: 0.8; } .panel-body { - padding: 12px 14px 0; + padding: 12px 14px 0; } .field-label { - font-size: 0.82rem; - font-weight: 600; - color: var(--md-default-fg-color--light); - margin: 12px 0 5px; + font-size: 0.82rem; + font-weight: 600; + color: var(--md-default-fg-color--light); + margin: 12px 0 5px; } /* --------------------------------------------------------------- main area ---- */ main { - flex: 1; - min-width: 0; - display: flex; - flex-direction: column; - gap: 12px; + flex: 1; + min-width: 0; + display: flex; + flex-direction: column; + gap: 12px; } .card { - display: flex; - flex-direction: column; - min-height: 0; + display: flex; + flex-direction: column; + min-height: 0; } .card-head h3 { - margin: 0; - font-size: 1rem; - font-weight: 600; - letter-spacing: 0.05em; - text-transform: uppercase; - color: var(--md-default-fg-color); + margin: 0; + font-size: 1rem; + font-weight: 600; + letter-spacing: 0.05em; + text-transform: uppercase; + color: var(--md-default-fg-color); } .card-head .hint { - margin: 0 0 0 auto; - text-transform: none; - letter-spacing: 0; - font-weight: 400; + margin: 0 0 0 auto; + text-transform: none; + letter-spacing: 0; + font-weight: 400; } .card-foot { - display: flex; - align-items: center; - justify-content: flex-end; - gap: 8px; - padding: 10px 14px; - border-top: 1px solid var(--line); - flex-wrap: wrap; + display: flex; + align-items: center; + justify-content: flex-end; + gap: 8px; + padding: 10px 14px; + border-top: 1px solid var(--line); + flex-wrap: wrap; } .field { - margin-bottom: 10px; + margin-bottom: 10px; } .field label { - display: block; - font-size: 0.82rem; - font-weight: 600; - color: var(--md-default-fg-color--light); - margin-bottom: 4px; + display: block; + font-size: 0.82rem; + font-weight: 600; + color: var(--md-default-fg-color--light); + margin-bottom: 4px; } .control { - width: 100%; - min-height: var(--control-h); - background: var(--md-default-bg-color); - color: var(--md-default-fg-color); - border: 1px solid var(--md-default-fg-color--lighter); - border-radius: var(--control-radius); - padding: 0 10px; - font-size: 0.9rem; + width: 100%; + min-height: var(--control-h); + background: var(--md-default-bg-color); + color: var(--md-default-fg-color); + border: 1px solid var(--md-default-fg-color--lighter); + border-radius: var(--control-radius); + padding: 0 10px; + font-size: 0.9rem; } .control:focus { - outline: none; - border-color: var(--md-primary-fg-color); - box-shadow: 0 0 0 2px color-mix(in srgb, var(--md-primary-fg-color) 22%, transparent); + outline: none; + border-color: var(--md-primary-fg-color); + box-shadow: 0 0 0 2px color-mix(in srgb, var(--md-primary-fg-color) 22%, transparent); } /* The popup inherits neither the select's background nor its color on every platform, so a * dark scheme could render the options as white on white. */ select.control option { - background: var(--md-default-bg-color); - color: var(--md-default-fg-color); + background: var(--md-default-bg-color); + color: var(--md-default-fg-color); } select.control { - appearance: none; - background-image: linear-gradient(45deg, transparent 50%, currentColor 50%), + appearance: none; + background-image: linear-gradient(45deg, transparent 50%, currentColor 50%), linear-gradient(135deg, currentColor 50%, transparent 50%); - background-position: calc(100% - 15px) 52%, calc(100% - 10px) 52%; - background-size: 5px 5px, 5px 5px; - background-repeat: no-repeat; - padding-right: 28px; - cursor: pointer; + background-position: calc(100% - 15px) 52%, calc(100% - 10px) 52%; + background-size: 5px 5px, 5px 5px; + background-repeat: no-repeat; + padding-right: 28px; + cursor: pointer; } /* Two columns of bordered boxes, as the reference sidebar has them: a category's worth of examples stays visible without scrolling past it. */ .demo-buttons { - display: grid; - grid-template-columns: 1fr 1fr; - gap: 6px; - margin-bottom: 10px; + display: grid; + grid-template-columns: 1fr 1fr; + gap: 6px; + margin-bottom: 10px; } .demo { - display: inline-flex; - align-items: center; - justify-content: center; - text-align: center; - min-height: var(--control-h); - background: var(--card-bg); - border: 1px solid var(--md-default-fg-color--lighter); - border-radius: var(--control-radius); - padding: 4px 6px; - font-size: 0.82rem; - line-height: 1.3; - color: var(--md-default-fg-color--light); - transition: border-color 0.12s, color 0.12s, background 0.12s; + display: inline-flex; + align-items: center; + justify-content: center; + text-align: center; + min-height: var(--control-h); + background: var(--card-bg); + border: 1px solid var(--md-default-fg-color--lighter); + border-radius: var(--control-radius); + padding: 4px 6px; + font-size: 0.82rem; + line-height: 1.3; + color: var(--md-default-fg-color--light); + transition: border-color 0.12s, color 0.12s, background 0.12s; } .demo:hover { - border-color: var(--md-primary-fg-color); - color: var(--md-typeset-a-color); + border-color: var(--md-primary-fg-color); + color: var(--md-typeset-a-color); } .demo.active { - background: color-mix(in srgb, var(--md-primary-fg-color) 12%, transparent); - border-color: color-mix(in srgb, var(--md-primary-fg-color) 40%, transparent); - color: var(--md-typeset-a-color); - font-weight: 700; + background: color-mix(in srgb, var(--md-primary-fg-color) 12%, transparent); + border-color: color-mix(in srgb, var(--md-primary-fg-color) 40%, transparent); + color: var(--md-typeset-a-color); + font-weight: 700; } .hint { - margin: 0 0 8px; - font-size: 0.75rem; - line-height: 1.5; - color: var(--md-default-fg-color--light); + margin: 0 0 8px; + font-size: 0.75rem; + line-height: 1.5; + color: var(--md-default-fg-color--light); } .docs-link { - display: inline-flex; - align-items: center; - gap: 4px; - font-size: 0.82rem; - text-decoration: none; + display: inline-flex; + align-items: center; + gap: 4px; + font-size: 0.82rem; + text-decoration: none; } .docs-link:hover { - text-decoration: underline; + text-decoration: underline; } .schema-row { - display: flex; - align-items: center; - gap: 8px; - padding: 3px 0; - font-size: 0.75rem; - font-family: var(--md-code-font); + display: flex; + align-items: center; + gap: 8px; + padding: 3px 0; + font-size: 0.75rem; + font-family: var(--md-code-font); } .schema-row .swatch { - width: 9px; - height: 9px; - border-radius: 2px; - flex: 0 0 auto; + width: 9px; + height: 9px; + border-radius: 2px; + flex: 0 0 auto; } .schema-row .n { - margin-left: auto; - color: var(--md-default-fg-color--light); - font-size: 0.75rem; + margin-left: auto; + color: var(--md-default-fg-color--light); + font-size: 0.75rem; } .empty { - font-size: 0.8rem; - color: var(--md-default-fg-color--light); - font-style: italic; + font-size: 0.8rem; + color: var(--md-default-fg-color--light); + font-style: italic; } .proc-list { - max-height: 200px; - overflow-y: auto; - margin-bottom: 8px; + max-height: 200px; + overflow-y: auto; + margin-bottom: 8px; } .proc { - display: block; - width: 100%; - text-align: left; - background: none; - border: 1px solid transparent; - border-radius: 4px; - padding: 4px 7px; - color: var(--md-default-fg-color--light); + display: block; + width: 100%; + text-align: left; + background: none; + border: 1px solid transparent; + border-radius: 4px; + padding: 4px 7px; + color: var(--md-default-fg-color--light); } .proc:hover { - background: var(--md-default-fg-color--lightest); + background: var(--md-default-fg-color--lightest); } .proc .nm { - display: block; - font-family: var(--md-code-font); - font-size: 0.75rem; - color: var(--md-typeset-a-color); - overflow-wrap: anywhere; + display: block; + font-family: var(--md-code-font); + font-size: 0.75rem; + color: var(--md-typeset-a-color); + overflow-wrap: anywhere; } .proc .yd { - display: block; - font-size: 0.7rem; - margin-top: 1px; + display: block; + font-size: 0.7rem; + margin-top: 1px; } .hist { - display: block; - width: 100%; - text-align: left; - background: none; - border: 0; - border-radius: 4px; - padding: 5px 7px; - font-family: var(--md-code-font); - font-size: 0.75rem; - color: var(--md-default-fg-color--light); - white-space: nowrap; - overflow: hidden; - text-overflow: ellipsis; + display: block; + width: 100%; + text-align: left; + background: none; + border: 0; + border-radius: 4px; + padding: 5px 7px; + font-family: var(--md-code-font); + font-size: 0.75rem; + color: var(--md-default-fg-color--light); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; } .hist:hover { - background: var(--md-default-fg-color--lightest); - color: var(--md-default-fg-color); + background: var(--md-default-fg-color--lightest); + color: var(--md-default-fg-color); } /* ---------------------------------------------------------------- editor ---- */ @@ -547,132 +547,251 @@ select.control { /* The highlighted copy sits under a transparent textarea, so the caret, selection, and * native editing stay real. The two must therefore keep identical metrics. */ .editor { - position: relative; - height: 352px; - min-height: 90px; - resize: vertical; - overflow: hidden; - margin: 14px; - border: 1px solid var(--line); - border-radius: 6px; - background: var(--md-code-bg-color); + position: relative; + height: 352px; + min-height: 90px; + resize: vertical; + overflow: hidden; + margin: 14px; + border: 1px solid var(--line); + border-radius: 6px; + background: var(--md-code-bg-color); } .editor pre, .editor textarea { - position: absolute; - inset: 0; - margin: 0; - padding: 12px 14px; - border: 0; - background: transparent; - font-family: var(--md-code-font); - font-size: 0.92rem; - line-height: 1.5; - tab-size: 2; - white-space: pre-wrap; - overflow-wrap: break-word; - overflow: auto; + position: absolute; + inset: 0; + margin: 0; + padding: 12px 14px; + border: 0; + background: transparent; + font-family: var(--md-code-font); + font-size: 0.92rem; + line-height: 1.5; + tab-size: 2; + white-space: pre-wrap; + overflow-wrap: break-word; + overflow: auto; } .editor pre { - pointer-events: none; - color: var(--md-code-fg-color); + pointer-events: none; + color: var(--md-code-fg-color); } .editor textarea { - color: transparent; - caret-color: var(--md-primary-fg-color); - resize: none; - outline: none; + color: transparent; + caret-color: var(--md-primary-fg-color); + resize: none; + outline: none; } .editor textarea::selection { - background: color-mix(in srgb, var(--md-primary-fg-color) 28%, transparent); + background: color-mix(in srgb, var(--md-primary-fg-color) 28%, transparent); } -.tok-kw { color: var(--syn-kw); font-weight: 600; } -.tok-fn { color: var(--syn-fn); } -.tok-str { color: var(--syn-str); } -.tok-num { color: var(--syn-num); } -.tok-com { color: var(--syn-com); font-style: italic; } -.tok-lbl { color: var(--syn-lbl); } -.tok-op { color: var(--syn-op); } +.tok-kw { + color: var(--syn-kw); + font-weight: 600; +} + +.tok-fn { + color: var(--syn-fn); +} + +.tok-str { + color: var(--syn-str); +} + +.tok-num { + color: var(--syn-num); +} + +.tok-com { + color: var(--syn-com); + font-style: italic; +} + +.tok-lbl { + color: var(--syn-lbl); +} + +.tok-op { + color: var(--syn-op); +} /* --------------------------------------------------------------- buttons ---- */ /* One shape for every button: outlined by default, filled for the primary action and for the one amber action, and a smaller variant for the toolbars. */ .btn { - display: inline-flex; - align-items: center; - justify-content: center; - gap: 7px; - min-height: var(--control-h); - background: var(--card-bg); - border: 1px solid var(--md-default-fg-color--lighter); - border-radius: var(--control-radius); - padding: 0 13px; - color: var(--md-default-fg-color--light); - font-size: 0.9rem; - font-weight: 600; - text-decoration: none; - white-space: nowrap; - transition: border-color 0.12s, color 0.12s, background 0.12s; + display: inline-flex; + align-items: center; + justify-content: center; + gap: 7px; + min-height: var(--control-h); + background: var(--card-bg); + border: 1px solid var(--md-default-fg-color--lighter); + border-radius: var(--control-radius); + padding: 0 13px; + color: var(--md-default-fg-color--light); + font-size: 0.9rem; + font-weight: 600; + text-decoration: none; + white-space: nowrap; + transition: border-color 0.12s, color 0.12s, background 0.12s; } .btn:hover { - border-color: var(--md-primary-fg-color); - color: var(--md-typeset-a-color); + border-color: var(--md-primary-fg-color); + color: var(--md-typeset-a-color); } .btn svg { - flex: 0 0 auto; + flex: 0 0 auto; } .btn.primary { - background: var(--md-primary-fg-color); - border-color: var(--md-primary-fg-color); - color: #fff; - box-shadow: var(--md-shadow-z1); + background: var(--md-primary-fg-color); + border-color: var(--md-primary-fg-color); + color: #fff; + box-shadow: var(--md-shadow-z1); } .btn.primary:hover { - background: var(--md-primary-fg-color--light); - border-color: var(--md-primary-fg-color--light); - color: #fff; + background: var(--md-primary-fg-color--light); + border-color: var(--md-primary-fg-color--light); + color: #fff; } .btn.accent { - background: var(--md-accent-fg-color); - border-color: var(--md-accent-fg-color); - color: rgba(0, 0, 0, 0.82); + background: var(--md-accent-fg-color); + border-color: var(--md-accent-fg-color); + color: rgba(0, 0, 0, 0.82); } .btn.accent:hover { - filter: brightness(1.06); - color: rgba(0, 0, 0, 0.82); + filter: brightness(1.06); + color: rgba(0, 0, 0, 0.82); +} + +/* Inside the editor box, which is the positioning context, so the popup scrolls away with a + resized editor instead of floating over the page. */ +.ac { + position: absolute; + z-index: 20; + min-width: 240px; + max-width: min(420px, 90%); + max-height: 220px; + overflow-y: auto; + border: 1px solid var(--line); + border-radius: 6px; + background: var(--card-bg); + box-shadow: var(--md-shadow-z2, 0 4px 16px rgba(0, 0, 0, 0.18)); +} + +.ac-row { + display: flex; + align-items: center; + gap: 8px; + padding: 5px 10px; + font-family: var(--md-code-font); + font-size: 0.8rem; + cursor: pointer; + white-space: nowrap; +} + +.ac-row.on, +.ac-row:hover { + background: color-mix(in srgb, var(--md-primary-fg-color) 14%, transparent); +} + +.ac-kind { + flex: none; + min-width: 44px; + padding: 1px 5px; + border-radius: 3px; + font-size: 0.66rem; + text-align: center; + text-transform: uppercase; + letter-spacing: 0.04em; + background: var(--md-code-bg-color); + color: var(--md-default-fg-color--light); +} + +.ac-kind.label, +.ac-kind.type { + color: var(--md-primary-fg-color); +} + +.ac-kind.proc, +.ac-kind.fn { + color: var(--md-accent-fg-color); +} + +.ac-text { + overflow: hidden; + text-overflow: ellipsis; +} + +.chips { + display: flex; + gap: 6px; + margin-bottom: 8px; +} + +.chip { + flex: 1; + padding: 4px 8px; + border: 1px solid var(--line); + border-radius: 999px; + background: transparent; + color: var(--md-default-fg-color--light); + font-family: inherit; + font-size: 0.74rem; + cursor: pointer; +} + +.chip:hover { + border-color: var(--md-primary-fg-color); +} + +.chip[aria-pressed="true"] { + background: color-mix(in srgb, var(--md-primary-fg-color) 14%, transparent); + border-color: var(--md-primary-fg-color); + color: var(--md-default-fg-color); +} + +.btn.danger { + background: color-mix(in srgb, var(--err) 12%, var(--card-bg)); + border-color: color-mix(in srgb, var(--err) 42%, var(--line)); + color: var(--err); +} + +.btn.danger:hover { + background: color-mix(in srgb, var(--err) 20%, var(--card-bg)); } .btn:disabled { - opacity: 0.55; - cursor: progress; + opacity: 0.55; + cursor: progress; } .btn.sm { - min-height: var(--control-h-sm); - padding: 0 10px; - font-size: 0.8rem; + min-height: var(--control-h-sm); + padding: 0 10px; + font-size: 0.8rem; } .btn-row { - display: flex; - gap: 8px; + display: flex; + gap: 8px; } .btn-row .btn { - flex: 1; - justify-content: center; + flex: 1; + justify-content: center; } /* ---------------------------------------------------------------- banner ---- */ @@ -682,470 +801,548 @@ select.control { /* No coloured rule down the left edge. The state is carried by a faint tint and by the wording; a dark red or green bar beside the text was the loudest thing on the page for the least it said. */ .banner { - flex: 0 0 auto; - padding: 9px 14px; - border: 1px solid var(--line); - border-radius: var(--control-radius); - background: var(--card-bg); - font-size: 0.9rem; - font-weight: 500; - color: var(--md-default-fg-color--light); - display: flex; - align-items: center; - gap: 9px; + flex: 0 0 auto; + padding: 9px 14px; + border: 1px solid var(--line); + border-radius: var(--control-radius); + background: var(--card-bg); + font-size: 0.9rem; + font-weight: 500; + color: var(--md-default-fg-color--light); + display: flex; + align-items: center; + gap: 9px; } .banner:empty { - display: none; + display: none; } .banner.ok { - background: color-mix(in srgb, var(--ok) 7%, var(--card-bg)); - border-color: color-mix(in srgb, var(--ok) 26%, var(--line)); - color: var(--md-default-fg-color); + background: color-mix(in srgb, var(--ok) 7%, var(--card-bg)); + border-color: color-mix(in srgb, var(--ok) 26%, var(--line)); + color: var(--md-default-fg-color); } .banner.err { - background: color-mix(in srgb, var(--err) 8%, var(--card-bg)); - border-color: color-mix(in srgb, var(--err) 30%, var(--line)); - color: var(--md-default-fg-color); + background: color-mix(in srgb, var(--err) 8%, var(--card-bg)); + border-color: color-mix(in srgb, var(--err) 30%, var(--line)); + color: var(--md-default-fg-color); } .banner.busy { - border-color: color-mix(in srgb, var(--md-primary-fg-color) 26%, var(--line)); + border-color: color-mix(in srgb, var(--md-primary-fg-color) 26%, var(--line)); } .banner .sp { - width: 13px; - height: 13px; - border-radius: 999px; - border: 2px solid var(--md-default-fg-color--lightest); - border-top-color: var(--md-primary-fg-color); - animation: spin 0.7s linear infinite; - flex: 0 0 auto; + width: 13px; + height: 13px; + border-radius: 999px; + border: 2px solid var(--md-default-fg-color--lightest); + border-top-color: var(--md-primary-fg-color); + animation: spin 0.7s linear infinite; + flex: 0 0 auto; } /* --------------------------------------------------------------- results ---- */ .results { - flex: 0 0 auto; + flex: 0 0 auto; } .tabs { - gap: 4px; - padding: 8px 10px; - flex-wrap: wrap; + gap: 4px; + padding: 8px 10px; + flex-wrap: wrap; } .tab { - display: inline-flex; - align-items: center; - gap: 6px; - background: none; - border: 1px solid transparent; - border-radius: 6px; - padding: 5px 10px; - font-size: 0.85rem; - font-weight: 600; - letter-spacing: 0.01em; - text-transform: none; - color: var(--md-default-fg-color--light); + display: inline-flex; + align-items: center; + gap: 6px; + background: none; + border: 1px solid transparent; + border-radius: 6px; + padding: 5px 10px; + font-size: 0.85rem; + font-weight: 600; + letter-spacing: 0.01em; + text-transform: none; + color: var(--md-default-fg-color--light); } .tab:hover { - color: var(--md-default-fg-color); - background: var(--md-default-fg-color--lightest); + color: var(--md-default-fg-color); + background: var(--md-default-fg-color--lightest); } .tab[aria-selected="true"] { - color: var(--md-typeset-a-color); - border-color: color-mix(in srgb, var(--md-primary-fg-color) 40%, transparent); - background: color-mix(in srgb, var(--md-primary-fg-color) 12%, transparent); + color: var(--md-typeset-a-color); + border-color: color-mix(in srgb, var(--md-primary-fg-color) 40%, transparent); + background: color-mix(in srgb, var(--md-primary-fg-color) 12%, transparent); } .result-meta { - flex: 0 0 auto; - padding: 8px 14px; - border-bottom: 1px solid var(--line); - font-size: 0.8rem; - color: var(--md-default-fg-color--light); + flex: 0 0 auto; + padding: 8px 14px; + border-bottom: 1px solid var(--line); + font-size: 0.8rem; + color: var(--md-default-fg-color--light); } /* Taller than the table needs, because the graph view lives in here and a force layout wants room. A fixed height rather than a share of the viewport, since the page scrolls as a document and the layout reads the element's size when it starts. */ .panes { - height: 580px; - display: flex; + height: 580px; + display: flex; } .pane { - flex: 1; - min-width: 0; - min-height: 0; - overflow: auto; - display: none; + flex: 1; + min-width: 0; + min-height: 0; + overflow: auto; + display: none; } .pane.on { - display: block; + display: block; } .pane.graph-pane.on { - display: flex; - flex-direction: column; - overflow: hidden; + display: flex; + flex-direction: column; + overflow: hidden; } table { - border-collapse: collapse; - width: max-content; - min-width: 100%; - font-family: var(--md-code-font); - font-size: 0.82rem; + border-collapse: collapse; + width: max-content; + min-width: 100%; + font-family: var(--md-code-font); + font-size: 0.82rem; } thead th { - position: sticky; - top: 0; - z-index: 1; - background: var(--md-code-bg-color); - border-bottom: 1px solid var(--md-default-fg-color--lighter); - text-align: left; - padding: 8px 12px; - font-weight: 600; - font-size: 0.72rem; - letter-spacing: 0.05em; - text-transform: uppercase; - color: var(--md-default-fg-color--light); - white-space: nowrap; + position: sticky; + top: 0; + z-index: 1; + background: var(--md-code-bg-color); + border-bottom: 1px solid var(--md-default-fg-color--lighter); + text-align: left; + padding: 8px 12px; + font-weight: 600; + font-size: 0.72rem; + letter-spacing: 0.05em; + text-transform: uppercase; + color: var(--md-default-fg-color--light); + white-space: nowrap; } tbody td { - padding: 6px 12px; - border-bottom: 1px solid var(--line); - font-family: var(--md-code-font); - vertical-align: top; - max-width: 460px; - overflow-wrap: anywhere; + padding: 6px 12px; + border-bottom: 1px solid var(--line); + font-family: var(--md-code-font); + vertical-align: top; + max-width: 460px; + overflow-wrap: anywhere; } tbody tr:hover td { - background: var(--md-default-fg-color--lightest); + background: var(--md-default-fg-color--lightest); } td .null { - color: var(--md-default-fg-color--lighter); - font-style: italic; + color: var(--md-default-fg-color--lighter); + font-style: italic; } -td .s { color: var(--syn-str); } -td .n { color: var(--syn-num); } -td .b { color: var(--syn-kw); } +td .s { + color: var(--syn-str); +} + +td .n { + color: var(--syn-num); +} + +td .b { + color: var(--syn-kw); +} .rownum { - color: var(--md-default-fg-color--lighter); - text-align: right; - user-select: none; + color: var(--md-default-fg-color--lighter); + text-align: right; + user-select: none; +} + +/* The plan tree. Depth is carried by a custom property on the row rather than by nesting, because + the engine renders the plan as flat indented lines and rebuilding a DOM hierarchy from them would + add a parser with nothing to show for it. */ +.plan-summary { + padding: 12px 14px; + border-bottom: 1px solid var(--line); + font-size: 0.82rem; + color: var(--md-default-fg-color--light); +} + +.plan-tree { + margin: 0; + padding: 10px 14px; + list-style: none; +} + +.plan-node { + position: relative; + padding: 3px 0 3px calc(var(--depth) * 18px + 16px); + font-family: var(--md-code-font); + font-size: 0.82rem; + line-height: 1.6; +} + +.plan-badge { + position: absolute; + left: calc(var(--depth) * 18px + 2px); + top: 0.72em; + width: 8px; + height: 8px; + border-radius: 50%; + background: var(--md-default-fg-color--lighter); +} + +.plan-badge.kernel { + background: var(--md-accent-fg-color); +} + +.plan-badge.index { + background: var(--md-primary-fg-color); +} + +.plan-badge.pruned { + background: var(--err); +} + +.plan-op { + font-weight: 600; + color: var(--md-default-fg-color); +} + +.plan-op.kernel { + color: var(--md-accent-fg-color); +} + +.plan-detail { + color: var(--md-default-fg-color--light); +} + +.plan-raw { + border-top: 1px solid var(--line); +} + +.plan-raw > summary { + padding: 10px 14px; + font-size: 0.8rem; + color: var(--md-default-fg-color--light); + cursor: pointer; } pre.plan, pre.json { - margin: 0; - padding: 14px; - font-family: var(--md-code-font); - font-size: 0.82rem; - line-height: 1.65; - white-space: pre; - color: var(--md-code-fg-color); - background: var(--md-code-bg-color); - min-height: 100%; + margin: 0; + padding: 14px; + font-family: var(--md-code-font); + font-size: 0.82rem; + line-height: 1.65; + white-space: pre; + color: var(--md-code-fg-color); + background: var(--md-code-bg-color); + min-height: 100%; } /* Flat on every side, as the banner is. The state is carried by a faint tint and a tinted border rather than a coloured rule down one edge. Both variants share this rule, so flattening only the error one would have left the informational one with a bar that read as an accident. */ .notice { - margin: 14px; - padding: 12px 14px; - border: 1px solid var(--line); - border-radius: var(--control-radius); - font-size: 0.85rem; - line-height: 1.55; - box-shadow: var(--md-shadow-z1); + margin: 14px; + padding: 12px 14px; + border: 1px solid var(--line); + border-radius: var(--control-radius); + font-size: 0.85rem; + line-height: 1.55; + box-shadow: var(--md-shadow-z1); } .notice.err { - background: color-mix(in srgb, var(--err) 8%, var(--card-bg)); - border-color: color-mix(in srgb, var(--err) 30%, var(--line)); - color: var(--md-default-fg-color); - font-family: var(--md-code-font); - font-size: 0.82rem; - white-space: pre-wrap; + background: color-mix(in srgb, var(--err) 8%, var(--card-bg)); + border-color: color-mix(in srgb, var(--err) 30%, var(--line)); + color: var(--md-default-fg-color); + font-family: var(--md-code-font); + font-size: 0.82rem; + white-space: pre-wrap; } .notice.info { - background: color-mix(in srgb, var(--md-primary-fg-color) 6%, var(--card-bg)); - border-color: color-mix(in srgb, var(--md-primary-fg-color) 24%, var(--line)); - color: var(--md-default-fg-color--light); + background: color-mix(in srgb, var(--md-primary-fg-color) 6%, var(--card-bg)); + border-color: color-mix(in srgb, var(--md-primary-fg-color) 24%, var(--line)); + color: var(--md-default-fg-color--light); } /* ----------------------------------------------------------- graph view ---- */ .graph-toolbar { - display: flex; - align-items: center; - gap: 10px; - padding: 7px 14px; - border-bottom: 1px solid var(--line); - background: var(--card-bg); - flex: 0 0 auto; - font-size: 0.75rem; - color: var(--md-default-fg-color--light); - flex-wrap: wrap; + display: flex; + align-items: center; + gap: 10px; + padding: 7px 14px; + border-bottom: 1px solid var(--line); + background: var(--card-bg); + flex: 0 0 auto; + font-size: 0.75rem; + color: var(--md-default-fg-color--light); + flex-wrap: wrap; } .legend { - display: flex; - gap: 10px; - flex-wrap: wrap; - align-items: center; + display: flex; + gap: 10px; + flex-wrap: wrap; + align-items: center; } .legend span { - display: inline-flex; - align-items: center; - gap: 5px; - font-family: var(--md-code-font); - font-size: 0.75rem; + display: inline-flex; + align-items: center; + gap: 5px; + font-family: var(--md-code-font); + font-size: 0.75rem; } .legend i { - width: 9px; - height: 9px; - border-radius: 999px; + width: 9px; + height: 9px; + border-radius: 999px; } svg.graph { - flex: 1; - min-height: 0; - width: 100%; - display: block; - cursor: grab; - background: var(--md-code-bg-color); + flex: 1; + min-height: 0; + width: 100%; + display: block; + cursor: grab; + background: var(--md-code-bg-color); } svg.graph:active { - cursor: grabbing; + cursor: grabbing; } svg.graph line { - stroke: var(--md-default-fg-color--lighter); + stroke: var(--md-default-fg-color--lighter); } svg.graph .node circle { - stroke: var(--md-code-bg-color); - stroke-width: 1.5; - cursor: pointer; + stroke: var(--md-code-bg-color); + stroke-width: 1.5; + cursor: pointer; } svg.graph .node text { - font-family: var(--md-text-font); - font-size: 11px; - font-weight: 600; - fill: var(--md-default-fg-color); - paint-order: stroke; - stroke: var(--md-code-bg-color); - stroke-width: 3px; - pointer-events: none; - user-select: none; + font-family: var(--md-text-font); + font-size: 11px; + font-weight: 600; + fill: var(--md-default-fg-color); + paint-order: stroke; + stroke: var(--md-code-bg-color); + stroke-width: 3px; + pointer-events: none; + user-select: none; } svg.graph .node.dim { - opacity: 0.25; + opacity: 0.25; } .inspect { - position: absolute; - right: 12px; - bottom: 12px; - width: 250px; - max-height: 46%; - overflow: auto; - background: var(--panel-bg); - border: 1px solid var(--line); - border-radius: 4px; - box-shadow: var(--md-shadow-z2); - padding: 10px 12px; - font-size: 0.8rem; + position: absolute; + right: 12px; + bottom: 12px; + width: 250px; + max-height: 46%; + overflow: auto; + background: var(--panel-bg); + border: 1px solid var(--line); + border-radius: 4px; + box-shadow: var(--md-shadow-z2); + padding: 10px 12px; + font-size: 0.8rem; } .inspect h5 { - margin: 0 0 6px; - font-size: 0.8rem; - display: flex; - align-items: center; - gap: 6px; + margin: 0 0 6px; + font-size: 0.8rem; + display: flex; + align-items: center; + gap: 6px; } .inspect dl { - margin: 0; - display: grid; - grid-template-columns: auto 1fr; - gap: 2px 9px; - font-family: var(--md-code-font); - font-size: 0.75rem; + margin: 0; + display: grid; + grid-template-columns: auto 1fr; + gap: 2px 9px; + font-family: var(--md-code-font); + font-size: 0.75rem; } .inspect dt { - color: var(--md-default-fg-color--light); + color: var(--md-default-fg-color--light); } .inspect dd { - margin: 0; - overflow-wrap: anywhere; + margin: 0; + overflow-wrap: anywhere; } .graph-pane { - position: relative; + position: relative; } /* ---------------------------------------------------------------- footer ---- */ footer { - flex: 0 0 auto; - padding: 7px 16px; - border-top: 1px solid var(--line); - background: var(--card-bg); - font-size: 0.8rem; - color: var(--md-default-fg-color--light); - display: flex; - gap: 12px; - align-items: center; + flex: 0 0 auto; + padding: 7px 16px; + border-top: 1px solid var(--line); + background: var(--card-bg); + font-size: 0.8rem; + color: var(--md-default-fg-color--light); + display: flex; + gap: 12px; + align-items: center; } .boot { - position: fixed; - inset: 0; - display: grid; - place-items: center; - align-content: center; - background: var(--md-default-bg-color); - z-index: 50; - gap: 12px; - text-align: center; - color: var(--md-default-fg-color--light); + position: fixed; + inset: 0; + display: grid; + place-items: center; + align-content: center; + background: var(--md-default-bg-color); + z-index: 50; + gap: 12px; + text-align: center; + color: var(--md-default-fg-color--light); } .boot .sp { - width: 26px; - height: 26px; - border-radius: 999px; - border: 2.5px solid var(--md-default-fg-color--lightest); - border-top-color: var(--md-primary-fg-color); - animation: spin 0.7s linear infinite; - margin: 0 auto; + width: 26px; + height: 26px; + border-radius: 999px; + border: 2.5px solid var(--md-default-fg-color--lightest); + border-top-color: var(--md-primary-fg-color); + animation: spin 0.7s linear infinite; + margin: 0 auto; } @keyframes spin { - to { - transform: rotate(360deg); - } + to { + transform: rotate(360deg); + } } /* The force layout settles over an animation loop and the boot indicator spins, so both are motion this has to answer for. `drawGraph` reads the same preference and settles the layout in one pass instead of animating it. */ @media (prefers-reduced-motion: reduce) { - *, - *::before, - *::after { - animation-duration: 0s !important; - animation-iteration-count: 1 !important; - transition-duration: 0s !important; - scroll-behavior: auto !important; - } + *, + *::before, + *::after { + animation-duration: 0s !important; + animation-iteration-count: 1 !important; + transition-duration: 0s !important; + scroll-behavior: auto !important; + } } @media (max-width: 1080px) { - .wrap { - gap: 12px; - padding: 12px; - } + .wrap { + gap: 12px; + padding: 12px; + } - aside { - width: 296px; - flex: 0 0 296px; - } + aside { + width: 296px; + flex: 0 0 296px; + } } @media (max-width: 820px) { - #toggle-side { - display: inline-grid; - } - - /* Overlaid rather than beside the editor. It carries the page background because the panels - inside it are cards with gaps between them, so without one the editor shows through. */ - aside { - position: absolute; - z-index: 20; - top: 48px; - height: calc(100% - 48px); - width: 300px; - flex: 0 0 300px; - padding: 12px; - background: var(--page-bg); - box-shadow: var(--md-shadow-z2); - } + #toggle-side { + display: inline-grid; + } + + /* Overlaid rather than beside the editor. It carries the page background because the panels + inside it are cards with gaps between them, so without one the editor shows through. */ + aside { + position: absolute; + z-index: 20; + top: 48px; + height: calc(100% - 48px); + width: 300px; + flex: 0 0 300px; + padding: 12px; + background: var(--page-bg); + box-shadow: var(--md-shadow-z2); + } } @media (max-width: 600px) { - header { - gap: 6px; - padding: 0 8px; - } - - /* The brand text is the one header item a query never needs, and the footer names the build - anyway. Dropping it keeps the nav links and the scheme toggle reachable without a horizontal - scroll; the mark stays, so the header is still identifiable. */ - .brand-name { - display: none; - } - - header .hdr { - padding: 6px 6px; - font-size: 0.85rem; - } - - .wrap { - padding: 8px; - gap: 8px; - } - - /* Every action fits on one row at a wider width; here they wrap instead of being clipped. */ - .card-foot { - justify-content: stretch; - } - - .card-foot .btn { - flex: 1; - justify-content: center; - } - - .tab { - padding: 7px 9px; - font-size: 0.8rem; - } - - .inspect { - width: auto; - left: 12px; - } - - /* The footer sentence needs more than one line at this width, and without wrapping the graph - counts beside it would be squeezed to nothing. */ - footer { - flex-wrap: wrap; - gap: 2px 12px; - } + header { + gap: 6px; + padding: 0 8px; + } + + /* The brand text is the one header item a query never needs, and the footer names the build + anyway. Dropping it keeps the nav links and the scheme toggle reachable without a horizontal + scroll; the mark stays, so the header is still identifiable. */ + .brand-name { + display: none; + } + + header .hdr { + padding: 6px 6px; + font-size: 0.85rem; + } + + .wrap { + padding: 8px; + gap: 8px; + } + + /* Every action fits on one row at a wider width; here they wrap instead of being clipped. */ + .card-foot { + justify-content: stretch; + } + + .card-foot .btn { + flex: 1; + justify-content: center; + } + + .tab { + padding: 7px 9px; + font-size: 0.8rem; + } + + .inspect { + width: auto; + left: 12px; + } + + /* The footer sentence needs more than one line at this width, and without wrapping the graph + counts beside it would be squeezed to nothing. */ + footer { + flex-wrap: wrap; + gap: 2px 12px; + } } diff --git a/web/worker.js b/web/worker.js new file mode 100644 index 0000000..a0e6663 --- /dev/null +++ b/web/worker.js @@ -0,0 +1,80 @@ +// The engine, off the main thread. +// +// A wasm call cannot yield, so running the module on the page's own thread froze the tab for the +// whole of a query: no repaint, no scrolling, no way to give up. That was tolerable while the +// sample graphs were the only data, and stopped being so once the analytics procedures landed, +// since an all-pairs pass over a graph loaded from a share link runs for as long as it runs. +// +// The engine therefore owns this worker and the page owns nothing but a promise per call. The +// graph lives here too, which is what makes termination the only possible cancel: see the note on +// the page side. + +import init, {Playground} from "./pkg/issundb_wasm.js"; + +let db = null; +let wasmMemory = null; +let baseline = 0; + +/// A build without the allocation counter still answers everything else, so a missing figure is +/// reported as zero rather than failing the call that asked. +function liveBytes() { + try { + return Playground.memoryBytes(); + } catch { + return 0; + } +} + +function memory() { + return {live: liveBytes(), heap: wasmMemory?.buffer?.byteLength ?? 0, baseline}; +} + +const OPS = { + async boot() { + const exports = await init(); + // The heap figure is the browser's, and only this side holds the module's memory object. + wasmMemory = exports.memory; + db = new Playground(); + baseline = liveBytes(); + let build = ""; + try { + build = Playground.buildRef(); + } catch { + // A module built outside a git checkout carries no stamp. + } + return {version: Playground.version(), build, ...memory()}; + }, + + reset() { + // Freed rather than abandoned. wasm-bindgen registers a finalizer, so an abandoned instance is + // reclaimed eventually, but until then its whole graph is still resident and wasm memory never + // shrinks. The new instance is built first, so a failure leaves the old one usable. + const previous = db; + db = new Playground(); + previous?.free(); + // After the old instance is freed, so the baseline is one empty database rather than two. + baseline = liveBytes(); + return memory(); + }, + + query: (cypher) => db.query(cypher), + explain: (cypher) => db.explain(cypher), + stats: () => db.stats(), + graphSnapshot: () => db.graphSnapshot(), + createTextIndex: (label, property) => void db.createTextIndex(label, property), + textSearch: (query, k) => db.textSearch(query, k), + upsertVector: (id, vector) => void db.upsertVector(id, vector), + vectorSearch: (vector, k) => db.vectorSearch(vector, k), + memory: () => memory(), +}; + +self.onmessage = async ({data: {id, op, args}}) => { + try { + const handler = OPS[op]; + if (!handler) throw new Error(`unknown engine operation: ${op}`); + self.postMessage({id, ok: true, value: await handler(...(args ?? []))}); + } catch (e) { + // Only the message survives structured cloning, and it is the whole of what the page shows. + self.postMessage({id, ok: false, error: String(e?.message ?? e)}); + } +};