Deterministic Canonical Embedding System

A TypeScript library for building symbolic-first retrieval systems.

This project turns inputs such as dates, quantities, places, concepts, events, claims, document spans, code symbols, and graph edges into deterministic canonical forms, stable hashes, and hierarchical path indexes. Embeddings are supported, but they are deliberately downstream: vectors rank candidates after symbolic filters have already narrowed the result set.

Why This Exists

Embedding search is useful, but similarity is not identity. Two strings can be close in vector space while meaning different things, and two different surface forms can mean exactly the same thing. Systems that rely on vectors as the primary truth layer are hard to audit, hard to migrate, and hard to debug when retrieval needs exact semantics.

This repository separates identity, meaning, and navigation:

Field	Purpose
exactRef	Precise occurrence identity, including source context when present.
canonicalHash	Deterministic denotational identity for normalized meaning.
canonicalPaths	Hierarchical lookup routes for exact, subtree, prefix, and intersection queries.

That split lets an application answer questions like:

• "Find objects in June 2026."
• "Find events in Antwerp involving this participant."
• "Find this exact source occurrence, not just the same normalized value."
• "Rank the already-matched symbolic candidates with embeddings."

The design goal is simple: use deterministic symbolic indexes for correctness, then use embeddings for ranking and approximation.

What It Provides

• Deterministic canonicalizers for the built-in v1 type catalog.
• Stable JSON serialization and SHA-256 identity helpers.
• Validated canonical path grammar and reversible path encoding.
• In-memory canonical path trie and row-backed path index implementations.
• Synchronous and asynchronous ingestion pipelines.
• Query planners that intersect symbolic criteria in selective order.
• Optional access logging and composite-path reindex recommendations.
• Optional embedding record generation, vector storage, and symbolic-first ranking.
• PostgreSQL object, path, access-log, and vector-store backends.
• Optional PostgreSQL ltree path index support.
• Merkle trie roots and proof verification helpers.
• Release, benchmark, migration, and live PostgreSQL validation scripts.

Install And Test

Requires Node.js 22 or newer.

npm install
npm test

Quick Start: In-Memory System

import { createCanonicalPathTrieSystem } from "deterministic-canonical-embedding-system";

const system = createCanonicalPathTrieSystem();

const object = system.ingest({
  value: "2026-06-25",
  typeHint: "time",
  metadata: { source: "example", sourceId: "message_1" },
});

const results = system.query({
  criteria: [
    {
      path: ["time", "month", "2026", "06"],
      required: true,
      matchMode: "subtree",
    },
  ],
  mode: "canonical",
});

console.log(object.canonicalHash, results.objectIds);

The system stores the original object, computes stable identity fields, indexes all generated canonical paths, and queries those paths without using vector similarity as the source of truth.

Ingestion Model

Single-object ingestion validates input, classifies it, resolves a canonicalizer, generates canonical paths, writes the stored object, and inserts path postings.

Batch ingestion uses the same flow, with duplicate accounting based on object IDs derived from exactRef. Duplicate exact occurrences return the existing object and reinsert its stored paths, which makes retry and repair flows idempotent.

Direct single-object helpers have narrow dependency contracts:

• ingestObject needs store get/put and trie insert.
• asyncIngestObject needs async store get/put and path-index insert.

Batch and system facades require broader dependencies because they also support paged duplicate scans, query planning, and deletion.

Query Planning

QueryPlanner and AsyncQueryPlanner use symbolic indexes first. They estimate candidate counts, start with the most selective required criterion, intersect the remaining criteria, and report diagnostics for the chosen plan.

Planner options can be wired through the system facade:

import {
  createCanonicalPathTrieSystem,
  InMemoryAccessLog,
  InMemoryVectorStore,
} from "deterministic-canonical-embedding-system";

const accessLog = new InMemoryAccessLog();
const vectorStore = new InMemoryVectorStore();
const system = createCanonicalPathTrieSystem({
  accessLog,
  vectorStore,
  broadSubtreeWarningThreshold: 500,
});

const object = system.ingest({
  value: "2026-06-25",
  typeHint: "time",
});

system.delete(object.id);

If rank: "embedding" is requested, embedding ranking is applied only after the symbolic candidate set has been selected.

PostgreSQL Usage

The PostgreSQL factory wires the async system facade with PostgreSQL-backed object storage, path indexes, access logs, and optional vector storage.

import {
  applyPostgresMigrations,
  createPostgresCanonicalPathTrieSystem,
  POSTGRES_BASE_MIGRATIONS,
} from "deterministic-canonical-embedding-system";

await applyPostgresMigrations(client, POSTGRES_BASE_MIGRATIONS);

const system = createPostgresCanonicalPathTrieSystem(client, {
  accessLog: "postgres",
  transaction: "postgres",
});

const batch = await system.ingestBatch([
  {
    value: "2026-06-25",
    typeHint: "time",
    metadata: { sourceId: "message_1" },
  },
]);

await system.delete(batch.objectIds[0]);

With transaction: "postgres", single-object ingestion, batch ingestion, and system-level deletion run inside PostgreSQL transactions.

Optional PostgreSQL ltree Index

Use the optional ltree migration and select the ltree path backend:

import {
  applyPostgresMigrations,
  createPostgresCanonicalPathTrieSystem,
  POSTGRES_ALL_MIGRATIONS,
} from "deterministic-canonical-embedding-system";

await applyPostgresMigrations(client, POSTGRES_ALL_MIGRATIONS);

const system = createPostgresCanonicalPathTrieSystem(client, {
  pathIndex: "ltree",
  accessLog: "postgres",
  transaction: "postgres",
});

The ltree backend uses reversible s_ plus UTF-8 hex labels, so arbitrary canonical path segments remain deterministic while satisfying ltree label restrictions.

Reindexing And Access Logs

Access logs record query criteria, candidate counts, selected plans, result counts, latency, and timestamps. The advisor can scan bounded access-log pages and recommend materialized compound paths for repeated expensive query patterns.

import {
  materializeAsyncCompositePathIndexes,
  recommendCompositePathIndexesFromAsyncAccessLog,
} from "deterministic-canonical-embedding-system";

const recommendations = await recommendCompositePathIndexesFromAsyncAccessLog(accessLog);
await materializeAsyncCompositePathIndexes(pathIndex, recommendations, {
  lookupMode: "subtree",
  protocolVersion: "v1",
});

Materialization and revocation have method-minimal dependency contracts, so maintenance jobs can pass small adapters instead of full storage implementations.

Embeddings

Embedding records can be generated and stored after ingestion. The default helper can store exact and canonical embeddings for each object.

import {
  storeObjectEmbeddings,
  AsyncVectorStoreAdapter,
  InMemoryVectorStore,
} from "deterministic-canonical-embedding-system";

const vectorStore = new AsyncVectorStoreAdapter(new InMemoryVectorStore());

await storeObjectEmbeddings(object, provider, vectorStore, {
  embeddingTypes: ["canonical"],
});

Pass the same vector store to a system or planner to enable embedding ranking. The vector layer never defines object identity.

Documentation Map

Topic	Document
Canonical path grammar	docs/canonical_path_grammar.md
Canonicalizer behavior and fixtures	docs/canonicalizer_catalog_semantics.md
Path validation and encoding	docs/path_validation_encoding.md
Trie and path-index semantics	docs/canonical_path_trie_semantics.md
Stable identity hashing	docs/stable_identity_hashing.md
Ingestion pipeline	docs/ingestion_pipeline_semantics.md
Query planner semantics	docs/query_planner_semantics.md
Access logs	docs/access_log_semantics.md
Reindex advisor lifecycle	docs/reindex_advisor_lifecycle.md
Embedding layer	docs/embedding_layer_semantics.md
Vector store behavior	docs/vector_store_semantics.md
Object store behavior	docs/object_store_semantics.md
System facade lifecycle	docs/system_lifecycle_semantics.md
PostgreSQL production notes	docs/postgres_production_readiness.md
Live PostgreSQL validation	docs/postgres_live_validation.md
Merkle trie proofs	docs/merkle_trie_proofs.md
Performance validation	docs/performance_scale_validation.md
Release checks	docs/release_readiness.md
Operational commands	docs/operational_tooling.md

Development

npm run build
npm test
npm run release:check

Useful optional commands:

npm run benchmark
npm run postgres:migrate
npm run test:postgres