You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGELOG.md
+31-1Lines changed: 31 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,6 +6,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
8
8
9
+
## [1.10.2]
10
+
11
+
### Added
12
+
13
+
-**Identity funnel** — `Vertex.identity_funnel`: ordered fallback branches digested into the synthetic `id`, so sources that identify the same entity by different keys (email here, phone + country there) share one authored policy. The first branch whose `when_all_present` fields are all present and non-empty wins; `include_branch_id` (default true) puts the winning branch id in the digest payload so two branches over equal values cannot collide. `hash_identity_properties` is the single-branch case and remains authored as-is — neither form is rewritten into the other, so legacy digests are byte-identical. `digest: sha256` only; `uuid5` is rejected pending a namespace policy. New models in `architecture/schema/identity_funnel.py`, digest helpers in `architecture/schema/identity_digest.py`. Runnable walkthrough: `examples/17-identity-funnel/`, docs in `docs/concepts/schema/vertex_identity.md`.
14
+
-**`mode: funnel` on `ReplaceIdentityOp`** — `FunnelIdentityTarget` joins the `to:` discriminated union, so a funnel is authorable as an evolution op and diffable like any other identity change. The differ compares the funnel itself (branch order, ids and field sets all feed the key), and a change emits `REKEY_VERTEX` at CRITICAL risk. `Vertex.has_identity_funnel` / `digest_source_fields` and `VertexConfig.identity_funnel_vertices` expose the policy for introspection; `identity_mode` still resolves to `hash`, since flat and funnel share one write path.
15
+
-**Cross-resource identity discovery** — `graflo/db/cross_resource_identity.py`: given two or more sampled resources that may describe the same vertex, propose a shared identity policy (natural / composite / funnel / hash) plus per-resource field maps, suggested rename steps, and the evidence behind the choice. Deterministic inference over samples — no LLM, no live database. Input is `SourceSample.samples_by_resource` verbatim, so nothing sits between sampling and inference; declared `primary_key` / `foreign_keys` are ground truth and short-circuit the heuristic. `apply_proposal_to_vertex` is a separate, explicit call that rebuilds the vertex through validation and refuses a `no_viable_identity` proposal. Runnable walkthrough: `examples/18-cross-resource-identity/`; concept page `docs/concepts/schema/cross_resource_identity.md`.
16
+
-**Proposal only, by construction.** Fuzzy signals (name similarity, value overlap) align *columns*; whether a field-set is a key is settled by exact equality after normalization plus bootstrap resampling. Nothing probabilistic reaches the write path.
17
+
-**Uniqueness is evaluated within each resource, never over the pooled rows.** Resources describing the same entities hold the same key values, so a pooled-uniqueness test would reject exactly the keys worth finding. The proposal reports `uniqueness_by_resource` and `shared_key_values`.
18
+
-**Value overlap is a mandatory floor; name similarity cannot veto it.**`email_address` vs `customer_email` scores 0.37 on names while sharing every value — requiring both thresholds independently would let the weak signal override the strong one.
19
+
-`CrossResourceStrategy` is separate from the shipped `IdentityStrategy` (which uses `unary`), mapped at the proposal boundary rather than renaming a published literal.
20
+
-**Graph operation revision chains** — ordered, content-hashed, replayable change sets over a `GraphManifest`, in `architecture/evolution/`. Concept page: `docs/concepts/schema/manifest_evolution.md`. Deliberately **not** Alembic-shaped: Alembic's core is a reversible `upgrade`/`downgrade` pair, and several ops are lossy, so the model is a git log — forward chain plus inverses on the reversible subset, with rollback by replay from the base.
21
+
-**`codec`** — `TypeAdapter(list[RevisionOp])` plus YAML (de)serialization. No `TypeAdapter` over the op union existed, so a heterogeneous op list could not be loaded from YAML at all and change sets had no substrate. `RevisionOp` is `ManifestOp` minus the binary `compose_manifests`. Serialization **verifies its own output**: the compact form is re-validated and the full form used when it would not load back — `IdentityTarget.mode` has a default, so dropping defaults strips the discriminator a nested union needs. All 29 revision ops now have a round-trip test; there was none before.
22
+
-**`autogenerate`** — `diff_manifests(base, target, hints=...) -> (ops, warnings)`, the first producer of `ManifestOp` values in the codebase (every op was previously hand-built; `SchemaDiff` emits description records on a disjoint plane and never sees `ingestion_model` or `bindings`). Its contract is the **replay invariant** — `manifest_hash(apply_evolution(base, ops)) == manifest_hash(target)` — which `diff_manifests_verified` checks, reporting the residual rather than passing an incomplete change set as complete. Renames are ambiguous by construction and come from explicit `RenameHints`; the differ never guesses a rename from a drop plus an add.
23
+
-**`inverse`** — `invert_op(op, manifest=pre_state)` for the reversible subset, `None` plus a stated reason for `merge_vertices`, `merge_edges`, `change_field_types`, `sanitize` and `project_manifest`. Round-trip tested by content hash for every reversible op.
24
+
-**`revision`** — `Revision` (ops, `down_revision`, hashes before/after, content-derived id), `RevisionChain` (validates linear links *and* that each step's start hash matches its predecessor's end hash), `apply_revisions` (re-verifies both hashes at every step), `downgrade_to` (prefers exact replay from the base; inversion refuses rather than approximating), and `FileRevisionStore` at `.graflo/revisions/`. This is the replay guarantee `MigrationRecord` cannot give — it stores bare op-type names, dropping the targets and values a replay needs.
25
+
-**`revision` CLI** — `new`, `apply`, `history`, `verify`, `downgrade`. Distinct from `migrate_schema`, which targets a database; applying a chain to a live database is not supported yet.
26
+
-**`manifest_hash` and friends moved** from `migrate/io.py` (L6) to `architecture/evolution/hashing.py` (L4), since the revision layer cannot import `migrate`. Re-exported from `migrate.io`, where they have always been imported from.
27
+
-**Empty vertex-index entries are pruned in `Schema.finish_init`.**`{"party": []}` and an absent key describe the same profile but hash differently, so an index or secondary-identity removal left a manifest that compared unequal to an identically-authored one — invisible until content hashes started deciding whether a replay was correct.
28
+
-**Shared inference helpers are now importable** — `column_values`, `eligible_columns`, `minimize_key_fields`, `greedy_unique_key` and `prepare_inference_samples` lost their underscore prefix in `db/identity_inference.py` so cross-resource inference reuses them instead of copying. Behaviour unchanged.
29
+
-**`SourceSample` rejects duplicate resource names.**`samples_by_resource` keys by name, so a duplicate silently discarded every sample but the last.
30
+
-**Meta-ontology `1.2.0`** — `gf:IdentityFunnel` / `gf:IdentityBranch` classes, `gf:hasIdentityFunnel`, `gf:hasIdentityBranch`, `gf:hasBranchCondition`, `gf:funnelDigest`, `gf:funnelIncludeBranchId`, `gf:branchId`. Branch order is carried by `gf:artifactIndex` rather than triple order, because branch order decides the key. Mirrored in `graflo-context.jsonld` and `rdf/namespace.py`; `docs/assets/graflo-ontology-viz/` regenerated.
31
+
32
+
### Fixed
33
+
34
+
- **Hash-identity vertices are no longer collapsed and dropped during casting.** A hash vertex resolves to `identity: ["id"]`, but `id` was computed only at write time in `DBWriter`. At assemble time it was still empty, so `merge_doc_basis(docs, ("id",))` folded **every document in the batch into one** ("if no documents have index keys, all documents are merged into a single document"), and `drop_empty_identity_docs` — on by default — then deleted the survivor together with its edge tuples. Edge endpoints were hit independently: `_push_edges` matches on `identity_fields(source)`, and the writer's hook never touched the endpoint doc copies in `gc.edges`. Digest computation now runs at **assemble time** via `ensure_digest_identities_in_acc_vertex`, alongside `ensure_assigned_uuids_in_acc_vertex` — before edge assembly and before dedup — and the writer hook remains as an idempotent safety net. Present since hash mode shipped in `[1.8.11]`; missed because all existing coverage was unit-level on the writer, with no example or e2e test casting a hash vertex.
35
+
-**Behaviour change:** manifests using `hash_identity_properties` now write the correct N documents where they previously wrote one (or zero, with edges missing). The digests themselves are unchanged.
36
+
- A document whose flat hash sources are **all** empty now yields no identity instead of digesting `{field: null}` — otherwise every empty document shared one key and merged into a single vertex.
37
+
-**Identity funnels survive renames and merges.**`RenameVertexPropertiesOp` rewrites funnel branch fields and conditions, and `merge_vertex_models` carries a funnel through — refusing, rather than silently unioning, when sources declare *different* funnels or mix a funnel with flat hash properties. A rewriter that forgets an identity field-set silently rekeys the graph.
38
+
9
39
## [1.10.1]
10
40
11
41
### Added
@@ -20,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
20
50
21
51
### Fixed
22
52
23
-
-**RDF round-trip no longer silently degrades a vertex's identity mode** (`CORE-RDF-001`). The serializer wrote only `gf:blank` and `gf:identityName`, so `assigned`, `hash_identity_properties` and `secondary_identities` were dropped and every vertex read back as `natural` — a wrong-but-valid schema, which is the worst failure shape for a documented round-trip format. All four identity modes (`natural`, `hash`, `blank`, `assigned`) now survive, and `examples/16-secondary-identities` round-trips to canonical equality.
53
+
-**RDF round-trip no longer silently degrades a vertex's identity mode.** The serializer wrote only `gf:blank` and `gf:identityName`, so `assigned`, `hash_identity_properties` and `secondary_identities` were dropped and every vertex read back as `natural` — a wrong-but-valid schema, which is the worst failure shape for a documented round-trip format. All four identity modes (`natural`, `hash`, `blank`, `assigned`) now survive, and `examples/16-secondary-identities` round-trips to canonical equality.
24
54
-**Identity field order survives.** Identity nodes are `BNode`s and RDF triples are unordered, so a multi-field `identity` could come back permuted. They now carry `gf:artifactIndex` and are read through `_ordered_nodes`; graphs written by the previous serializer still parse (a missing index degrades to arbitrary order rather than failing).
25
55
-**`strict_references=True` now rejects a pipeline `vertex:` step naming an undeclared vertex.**`filter_vertex_config_for_resource` intersects a resource's vertex names with the schema's and silently drops unknowns, so a resource that ingested nothing validated clean — a name mismatch between the vertex definition and the step was invisible. **Behaviour change:** manifests that previously passed under `strict_references=True` may now fail, which is the point; lenient validation is unchanged.
26
56
-`ChunkerFactory._guess_chunker_type` raises the documented `ValueError` for a file with no extension instead of `IndexError`, so callers scanning a directory can skip it like any other unknown type.
"comment": "Ordered fallback branches deriving a deterministic synthetic id. The first branch whose fields are all present wins. Generalizes hash_identity_properties, which is the single-branch case.",
"comment": "Ordered fallback branches deriving a deterministic synthetic id. The first branch whose fields are all present wins. Generalizes hash_identity_properties, which is the single-branch case.",
0 commit comments