334 add data samples

CHANGELOG.md

@@ -6,6 +6,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [1.10.1]
+
+### Added
+
+- **Meta-ontology `1.1.0`** (`graflo/rdf/ontology/graflo.ttl`) — `gf:assigned` (datatype, boolean), `gf:hasHashIdentity` (object property reusing `gf:Identity`), `gf:SecondaryIdentity` (class) and `gf:hasSecondaryIdentity`. `gf:hasIdentity`'s domain widens to include `gf:SecondaryIdentity`, so a secondary identity's field list reuses the existing identity-chain vocabulary rather than adding a parallel one. Mirrored in `graflo-context.jsonld` and `rdf/namespace.py`; `docs/assets/graflo-ontology-viz/` regenerated.
+- **Resource sampling** — a connector-driven sampling primitive, the shared input stage for schema inference (algorithmic or agentic). Previously `infer_manifest` performed this privately for PostgreSQL and nothing else could reach it.
+- **`architecture.onto_sample`** (new, L2 leaf models) — `ResourceSample` holds documents **verbatim as JSON**: flat rows for tables, arbitrarily nested for APIs, nothing flattened at the boundary. `connector` records where the documents came from, so the relation that later becomes a resource plus its `resource_connector` binding survives the round trip. `SourceSample.samples_by_resource` returns `dict[str, list[dict]]` — the shape cross-resource identity inference consumes, so no adapter is needed. `ForeignKeyHint` carries *declared* references, which are ground truth for edge inference rather than a name-suffix guess.
+- **Sampling and profiling are separate operations.** `ResourceSampler` (`hq/sampler.py`) pulls documents; `profile_sample` derives the description. `ResourceProfile` is **path-keyed** (`address.city`, `items[].sku`) with a `FieldType`, null rate, cardinality and nesting depth, so hierarchical and tabular sources take one code path — a flat column list cannot represent a nested response at all. `ResourceProfile.flat_docs` projects nested documents into flat records, which is what makes `IdentityInferencer` (flat-records only) usable on an API source.
+- **`GraphEngine.sample_resources(...)`** — accepts a `PostgresConfig`, a `Bindings` block, or a file/directory path. Bindings-driven sampling reuses `RegistryBuilder.discover_files` and the `resource_connector` mapping, so provenance comes for free instead of being reconstructed. PostgreSQL sampling carries `primary_key` and `foreign_keys` through from introspection. Documents are normalized to JSON-safe values (`datetime`, `Decimal`, `memoryview`, numpy scalars), capped by `max_docs` and `max_cell_chars`, with `truncated` set when anything was dropped or clipped.
+- **`PostgresConfig.from_dsn(dsn)`** — a bare `uri=` derives only host and port, leaving `username`/`password`/`database` unset, so `to_sqlalchemy_connection_string()` raised. Parses credentials, database and a `search_path` from either `?schema=` or libpq `?options=-csearch_path=`.
+- Concept page: [Sampling and profiling](docs/concepts/schema/sampling_and_profiling.md), cross-linked from the identity inference guide.
+
+### Fixed
+
+- **RDF round-trip no longer silently degrades a vertex's identity mode** (`CORE-RDF-001`). The serializer wrote only `gf:blank` and `gf:identityName`, so `assigned`, `hash_identity_properties` and `secondary_identities` were dropped and every vertex read back as `natural` — a wrong-but-valid schema, which is the worst failure shape for a documented round-trip format. All four identity modes (`natural`, `hash`, `blank`, `assigned`) now survive, and `examples/16-secondary-identities` round-trips to canonical equality.
+- **Identity field order survives.** Identity nodes are `BNode`s and RDF triples are unordered, so a multi-field `identity` could come back permuted. They now carry `gf:artifactIndex` and are read through `_ordered_nodes`; graphs written by the previous serializer still parse (a missing index degrades to arbitrary order rather than failing).
+- **`strict_references=True` now rejects a pipeline `vertex:` step naming an undeclared vertex.** `filter_vertex_config_for_resource` intersects a resource's vertex names with the schema's and silently drops unknowns, so a resource that ingested nothing validated clean — a name mismatch between the vertex definition and the step was invisible. **Behaviour change:** manifests that previously passed under `strict_references=True` may now fail, which is the point; lenient validation is unchanged.
+- `ChunkerFactory._guess_chunker_type` raises the documented `ValueError` for a file with no extension instead of `IndexError`, so callers scanning a directory can skip it like any other unknown type.
+- The CSV/TSV chunker no longer raises `RuntimeError: generator raised StopIteration` (PEP 479) on an empty file; it yields no rows.
+
 ## [1.10.0]
 
 

docs/assets/graflo-ontology-viz/embed.html

@@ -2,7 +2,7 @@
 <html lang="en">
 <head>
   <meta charset="utf-8" />
-  <title>GraFlo Ontology (v1.0.0)</title>
+  <title>GraFlo Ontology (v1.1.0)</title>
   <link rel="stylesheet" href="graph-view.css" />
 </head>
 <body class="embed">
@@ -213,6 +213,13 @@
       "source": "https://ontology.growgraph.dev/graflo/Schema",
       "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
     },
+    {
+      "id": "sub:https://ontology.growgraph.dev/graflo/SecondaryIdentity->https://ontology.growgraph.dev/graflo/GrafloArtifact",
+      "kind": "subClassOf",
+      "label": "subClassOf",
+      "source": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+      "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
+    },
     {
       "id": "sub:https://ontology.growgraph.dev/graflo/SparqlConnector->https://ontology.growgraph.dev/graflo/BoundConnector",
       "kind": "subClassOf",
@@ -367,6 +374,20 @@
       "source": "https://ontology.growgraph.dev/graflo/Vertex",
       "target": "https://ontology.growgraph.dev/graflo/Identity"
     },
+    {
+      "id": "prop:https://ontology.growgraph.dev/graflo/hasHashIdentity",
+      "kind": "objectProperty",
+      "label": "hasHashIdentity",
+      "source": "https://ontology.growgraph.dev/graflo/Vertex",
+      "target": "https://ontology.growgraph.dev/graflo/Identity"
+    },
+    {
+      "id": "prop:https://ontology.growgraph.dev/graflo/hasSecondaryIdentity",
+      "kind": "objectProperty",
+      "label": "hasSecondaryIdentity",
+      "source": "https://ontology.growgraph.dev/graflo/Vertex",
+      "target": "https://ontology.growgraph.dev/graflo/SecondaryIdentity"
+    },
     {
       "id": "prop:https://ontology.growgraph.dev/graflo/edgeSource",
       "kind": "objectProperty",
@@ -791,6 +812,13 @@
       "label": "Schema",
       "local": "Schema"
     },
+    {
+      "comment": "Alternate named field-set that identifies a vertex for lookup only. Edge endpoints may match on it; upserts always use the primary identity.",
+      "id": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+      "kind": "gf",
+      "label": "SecondaryIdentity",
+      "local": "SecondaryIdentity"
+    },
     {
       "comment": null,
       "id": "https://ontology.growgraph.dev/graflo/SparqlConnector",
@@ -877,7 +905,7 @@
     }
   ],
   "ontology": "https://ontology.growgraph.dev/graflo",
-  "version": "1.0.0"
+  "version": "1.1.0"
 };</script>
   <script src="graph-view.js"></script>
 </body>

docs/assets/graflo-ontology-viz/graph-data.json

@@ -189,6 +189,13 @@
             "source": "https://ontology.growgraph.dev/graflo/Schema",
             "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
         },
+        {
+            "id": "sub:https://ontology.growgraph.dev/graflo/SecondaryIdentity->https://ontology.growgraph.dev/graflo/GrafloArtifact",
+            "kind": "subClassOf",
+            "label": "subClassOf",
+            "source": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+            "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
+        },
         {
             "id": "sub:https://ontology.growgraph.dev/graflo/SparqlConnector->https://ontology.growgraph.dev/graflo/BoundConnector",
             "kind": "subClassOf",
@@ -343,6 +350,20 @@
             "source": "https://ontology.growgraph.dev/graflo/Vertex",
             "target": "https://ontology.growgraph.dev/graflo/Identity"
         },
+        {
+            "id": "prop:https://ontology.growgraph.dev/graflo/hasHashIdentity",
+            "kind": "objectProperty",
+            "label": "hasHashIdentity",
+            "source": "https://ontology.growgraph.dev/graflo/Vertex",
+            "target": "https://ontology.growgraph.dev/graflo/Identity"
+        },
+        {
+            "id": "prop:https://ontology.growgraph.dev/graflo/hasSecondaryIdentity",
+            "kind": "objectProperty",
+            "label": "hasSecondaryIdentity",
+            "source": "https://ontology.growgraph.dev/graflo/Vertex",
+            "target": "https://ontology.growgraph.dev/graflo/SecondaryIdentity"
+        },
         {
             "id": "prop:https://ontology.growgraph.dev/graflo/edgeSource",
             "kind": "objectProperty",
@@ -767,6 +788,13 @@
             "label": "Schema",
             "local": "Schema"
         },
+        {
+            "comment": "Alternate named field-set that identifies a vertex for lookup only. Edge endpoints may match on it; upserts always use the primary identity.",
+            "id": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+            "kind": "gf",
+            "label": "SecondaryIdentity",
+            "local": "SecondaryIdentity"
+        },
         {
             "comment": null,
             "id": "https://ontology.growgraph.dev/graflo/SparqlConnector",
@@ -853,5 +881,5 @@
         }
     ],
     "ontology": "https://ontology.growgraph.dev/graflo",
-    "version": "1.0.0"
+    "version": "1.1.0"
 }

docs/assets/graflo-ontology-viz/index.html

@@ -2,13 +2,13 @@
 <html lang="en">
 <head>
   <meta charset="utf-8" />
-  <title>GraFlo Ontology (v1.0.0)</title>
+  <title>GraFlo Ontology (v1.1.0)</title>
   <link rel="stylesheet" href="graph-view.css" />
 </head>
 <body>
   <div class="layout">
     <aside class="sidebar">
-      <h1>GraFlo Ontology (v1.0.0)</h1>
+      <h1>GraFlo Ontology (v1.1.0)</h1>
       <p>Class graph from <code>graflo.ttl</code>: hierarchy laid out by <code>subClassOf</code> (force-assisted, stable), standalone classes in the top-left. All property edges are shown; thick arrows are <code>subClassOf</code>.</p>
       <div class="control">
         <label for="search">Search classes</label>
@@ -236,6 +236,13 @@ <h1>GraFlo Ontology (v1.0.0)</h1>
       "source": "https://ontology.growgraph.dev/graflo/Schema",
       "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
     },
+    {
+      "id": "sub:https://ontology.growgraph.dev/graflo/SecondaryIdentity->https://ontology.growgraph.dev/graflo/GrafloArtifact",
+      "kind": "subClassOf",
+      "label": "subClassOf",
+      "source": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+      "target": "https://ontology.growgraph.dev/graflo/GrafloArtifact"
+    },
     {
       "id": "sub:https://ontology.growgraph.dev/graflo/SparqlConnector->https://ontology.growgraph.dev/graflo/BoundConnector",
       "kind": "subClassOf",
@@ -390,6 +397,20 @@ <h1>GraFlo Ontology (v1.0.0)</h1>
       "source": "https://ontology.growgraph.dev/graflo/Vertex",
       "target": "https://ontology.growgraph.dev/graflo/Identity"
     },
+    {
+      "id": "prop:https://ontology.growgraph.dev/graflo/hasHashIdentity",
+      "kind": "objectProperty",
+      "label": "hasHashIdentity",
+      "source": "https://ontology.growgraph.dev/graflo/Vertex",
+      "target": "https://ontology.growgraph.dev/graflo/Identity"
+    },
+    {
+      "id": "prop:https://ontology.growgraph.dev/graflo/hasSecondaryIdentity",
+      "kind": "objectProperty",
+      "label": "hasSecondaryIdentity",
+      "source": "https://ontology.growgraph.dev/graflo/Vertex",
+      "target": "https://ontology.growgraph.dev/graflo/SecondaryIdentity"
+    },
     {
       "id": "prop:https://ontology.growgraph.dev/graflo/edgeSource",
       "kind": "objectProperty",
@@ -814,6 +835,13 @@ <h1>GraFlo Ontology (v1.0.0)</h1>
       "label": "Schema",
       "local": "Schema"
     },
+    {
+      "comment": "Alternate named field-set that identifies a vertex for lookup only. Edge endpoints may match on it; upserts always use the primary identity.",
+      "id": "https://ontology.growgraph.dev/graflo/SecondaryIdentity",
+      "kind": "gf",
+      "label": "SecondaryIdentity",
+      "local": "SecondaryIdentity"
+    },
     {
       "comment": null,
       "id": "https://ontology.growgraph.dev/graflo/SparqlConnector",
@@ -900,7 +928,7 @@ <h1>GraFlo Ontology (v1.0.0)</h1>
     }
   ],
   "ontology": "https://ontology.growgraph.dev/graflo",
-  "version": "1.0.0"
+  "version": "1.1.0"
 };</script>
   <script src="graph-view.js"></script>
 </body>

docs/concepts/index.md

@@ -67,6 +67,10 @@ flowchart LR
 
 Entry points: **`GraphEngine.migrate_graph()`** (schema + data), **`infer_schema_from_graph()`** (schema only), **`export_graph()`** (in-memory `GraFloOutput`). See [Graph export and migration](operations/graph_export_migration.md) and the [Graph DB migration guide](../guides/graph_db_migration.md).
 
+- **SourceSample** — a bounded, verbatim-JSON sample of one or more resources, retaining the
+  connector each document came from. The shared input stage for schema inference, algorithmic or
+  agentic; produced by `GraphEngine.sample_resources()`. See
+  [Sampling and profiling](schema/sampling_and_profiling.md).
 - **Source Instance** — a concrete data artifact (a file, a table, a SPARQL endpoint, an API, a Kafka topic), wrapped by an `AbstractDataSource` with a `DataSourceType` (`FILE`, `SQL`, `SPARQL`, `API`, `KAFKA`, `IN_MEMORY`).
 - **Resource** — a reusable transformation pipeline (actor steps: descend, transform, vertex, edge) that maps raw records to graph elements. Data sources bind to Resources by name via the `DataSourceRegistry`.
 - **GraphManifest** — the canonical top-level contract that composes `schema`, `ingestion_model`, and `bindings`. High-level **contract evolution** (remove/merge vertex types and keep ingestion aligned) is described in [Manifest evolution](schema/manifest_evolution.md).