From 7ec2da40d3621aa2adc09d54d49b590fb36f79c9 Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 23 Apr 2026 11:57:55 -0700 Subject: [PATCH 01/27] Add predictive skills (rai-predictive-modeling + rai-predictive-training) Two-skill workflow mirroring the prescriptive pattern: - rai-predictive-modeling: concepts, Snowflake loading, task relationships, graph edges, and PropertyTransformer features. Includes node-classification, link-prediction, and regression examples with generic concept names (no domain lock-in). - rai-predictive-training: GNN constructor, fit, predictions, evaluation, register/load. Includes regression-specific sanity checks guidance (epoch count, R^2 < 0 interpretation, target-profiling). Discovery integration (minimal additions on top of latest main): - rai-discovery/SKILL.md: add rai-predictive-modeling to the Formulation skill list. - rai-discovery/references/predictive.md: replace "Future" platform-status language with a two-modes description (pre-computed vs GNN training); cross-reference the new skills. - rai-graph-analysis/SKILL.md: one-line cross-reference to rai-predictive-modeling for GNN graph construction. Rebased from branch tip onto origin/main; prior 19 commits of pre-split history dropped in favor of a single clean commit against latest main. --- .../rai-discovery/references/predictive.md | 10 +- .../rai/skills/rai-graph-analysis/SKILL.md | 1 + skills/rai-predictive-modeling/SKILL.md | 305 +++++++++++ .../examples/link_prediction_snowflake.py | 80 +++ .../examples/node_classification_snowflake.py | 86 ++++ .../examples/regression_snowflake.py | 85 +++ .../references/auto-discovery.md | 105 ++++ .../references/property-transformer-types.md | 66 +++ .../references/task-relationships.md | 126 +++++ skills/rai-predictive-training/SKILL.md | 482 ++++++++++++++++++ .../examples/register_and_load.py | 53 ++ .../examples/train_link_prediction.py | 46 ++ .../examples/train_node_classification.py | 45 ++ .../examples/train_regression.py | 46 ++ .../references/evaluation-debugging.md | 138 +++++ .../references/hyperparameters.md | 91 ++++ .../references/prediction-attributes.md | 67 +++ .../references/task-types-and-metrics.md | 68 +++ 18 files changed, 1895 insertions(+), 5 deletions(-) create mode 100644 skills/rai-predictive-modeling/SKILL.md create mode 100644 skills/rai-predictive-modeling/examples/link_prediction_snowflake.py create mode 100644 skills/rai-predictive-modeling/examples/node_classification_snowflake.py create mode 100644 skills/rai-predictive-modeling/examples/regression_snowflake.py create mode 100644 skills/rai-predictive-modeling/references/auto-discovery.md create mode 100644 skills/rai-predictive-modeling/references/property-transformer-types.md create mode 100644 skills/rai-predictive-modeling/references/task-relationships.md create mode 100644 skills/rai-predictive-training/SKILL.md create mode 100644 skills/rai-predictive-training/examples/register_and_load.py create mode 100644 skills/rai-predictive-training/examples/train_link_prediction.py create mode 100644 skills/rai-predictive-training/examples/train_node_classification.py create mode 100644 skills/rai-predictive-training/examples/train_regression.py create mode 100644 skills/rai-predictive-training/references/evaluation-debugging.md create mode 100644 skills/rai-predictive-training/references/hyperparameters.md create mode 100644 skills/rai-predictive-training/references/prediction-attributes.md create mode 100644 skills/rai-predictive-training/references/task-types-and-metrics.md diff --git a/plugins/rai/skills/rai-discovery/references/predictive.md b/plugins/rai/skills/rai-discovery/references/predictive.md index a8721cd..d1ca739 100644 --- a/plugins/rai/skills/rai-discovery/references/predictive.md +++ b/plugins/rai/skills/rai-discovery/references/predictive.md @@ -11,7 +11,7 @@ Predictive reasoning uses historical data patterns to forecast outcomes, classify entities, or detect anomalies. -**Current platform status:** The RAI predictive reasoner is not yet integrated into the platform. Today, predictive capabilities are delivered via **pre-computed prediction tables** — external ML outputs loaded into Snowflake and mapped as ontology concepts. Discovery should identify both pre-computed predictions already in the data and predictive questions the data could support. +**Two modes:** Predictive capabilities can be delivered via **pre-computed prediction tables** (external ML outputs loaded into Snowflake) or via the **RAI predictive pipeline** (GNN-based models trained directly on the knowledge graph — see `rai-predictive-modeling` and `rai-predictive-training`). Discovery should identify both pre-computed predictions already in the data and predictive questions the data could support via GNN training. | Type | Question Pattern | Ontology Signal | |------|-----------------|-----------------| @@ -39,7 +39,7 @@ Problem type: `classification`, `regression`, `forecasting`, `anomaly_detection` ### mode How prediction is delivered: - **`pre_computed`**: A prediction/forecast table already exists in the schema. Discovery identifies it and suggests downstream use by other reasoners. -- **`rai_predictive`**: Future — when the RAI predictive reasoner is platform-integrated. +- **`rai_predictive`**: Build and train a graph neural network (GNN) using the RAI predictive pipeline (**early access** — APIs and behavior may change). See `rai-predictive-modeling` for data modeling and `rai-predictive-training` for training and evaluation. ### target_concept / target_property What to predict. E.g., `Entity` / `risk_value`, or `Customer` / `churn_flag`. @@ -106,7 +106,7 @@ A `RiskPrediction` table with `predicted_risk_prob` and `risk_tier` per entity p ## Output Concepts -Predictive reasoning (whether pre-computed or future RAI-native) adds concepts to the ontology that downstream reasoners consume: +Predictive reasoning (whether pre-computed or via the RAI predictive pipeline) adds concepts to the ontology that downstream reasoners consume: | Prediction Type | Output Concept | Downstream Use | |----------------|----------------|----------------| @@ -128,11 +128,11 @@ What ontology patterns indicate prediction potential: - Look for columns named `predicted_*`, `probability`, `risk_*`, `forecast_*`, `confidence` - Check if the prediction table links to other ontology concepts via FK (e.g., entity_id linking predictions to Entity concept) -### For rai_predictive mode (future) +### For rai_predictive mode (GNN training) - **Feature availability**: Target property with sufficient non-null values; 3+ candidate features with variance - **Temporal span**: For forecasting, at least 2 full cycles of the target period (period-level prediction needs 2+ periods of history) - **Label quality**: For classification, labels exist and are reasonably balanced (flag extreme imbalance like 99%/1%) - **Row count**: Rough minimums (regression 50+, classification 30+ per class, forecasting 2+ full periods) - **Feature-target relationship**: At least some features plausibly related to target (domain signal) -**Minimum viable ontology for prediction:** For pre-computed: a prediction table exists and links to other concepts. For future rai_predictive: at least one concept with a target property (what to predict) and 2+ feature properties (what to predict from), backed by sufficient historical data. +**Minimum viable ontology for prediction:** For pre-computed: a prediction table exists and links to other concepts. For rai_predictive (GNN): at least one concept with a target property (what to predict) and 2+ feature properties (what to predict from), backed by sufficient historical data. See `rai-predictive-modeling` for the full data modeling workflow. diff --git a/plugins/rai/skills/rai-graph-analysis/SKILL.md b/plugins/rai/skills/rai-graph-analysis/SKILL.md index 8c21d67..57b51be 100644 --- a/plugins/rai/skills/rai-graph-analysis/SKILL.md +++ b/plugins/rai/skills/rai-graph-analysis/SKILL.md @@ -38,6 +38,7 @@ description: Graph algorithm selection and execution on PyRel v1 models. Covers - Ontology design decisions (concept modeling, data mapping) — see `rai-ontology-design` - Optimization formulation (variables, constraints, objectives) — see `rai-prescriptive-problem-formulation` - Business rule authoring (validation, classification, alerting) — see `rai-rules-authoring` +- GNN graph construction for predictive pipelines — see `rai-predictive-modeling` **Overview (process steps):** 1. Study the existing model — understand base definitions, coding conventions, and what's already wired diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md new file mode 100644 index 0000000..b6bbcb2 --- /dev/null +++ b/skills/rai-predictive-modeling/SKILL.md @@ -0,0 +1,305 @@ +--- +name: rai-predictive-modeling +description: Build GNN data models -- concepts, Snowflake data loading, task relationships, graph edges, and PropertyTransformer features. Use when defining entity types, loading data, or configuring graph structure for a predictive GNN pipeline. +--- + +# Predictive Modeling + + +> **Early access.** The RAI predictive reasoner (GNN) is in early access — APIs, engine requirements, and behavior may change. Confirm the latest surface with the RelationalAI team before production use. + +## Summary + +**What:** Data modeling workflow for GNN pipelines -- from imports through graph construction and feature configuration. + +**When to use:** +- Defining concepts and loading data from Snowflake +- Building graph structure (edges, self-references) +- Configuring task relationships (train/val/test splits) +- Setting up PropertyTransformer features + +**When NOT to use:** +- Training, predictions, evaluation, model management -- see `rai-predictive-training` +- Graph algorithms (centrality, community detection) -- see `rai-graph-analysis` + +**Overview:** 6 steps: imports -> concepts -> populate -> task relationships -> graph -> features + +--- + +## Quick Reference + +```python +# Imports +from relationalai.semantics import Model, select, define, Integer, String, Any +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +model = Model("") +Concept, Table, Relationship = model.Concept, model.Table, model.Relationship +``` + +| Pattern | Code | +|---------|------| +| Single PK | `User = Concept("User", identify_by={"user_id": Integer})` | +| Composite PK | `Class = Concept("Class", identify_by={"courseid": Integer, "year": Integer})` | +| No PK (task table) | `TrainTable = Concept("TrainTable")` | + +```python +# Graph init +gnn_graph = Graph(model, directed=True, weighted=False) +Edge = gnn_graph.Edge + +# PropertyTransformer +pt = PropertyTransformer( + category=[User.locale, User.gender], + continuous=[User.birthyear], + datetime=[User.joinedAt, Event.start_time], + time_col=[Event.start_time], +) +``` + +--- + +## Imports and Model Setup + +```python +from relationalai.semantics import Model, select, define, Integer, String, Any +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +model = Model("") +Concept, Table, Relationship = model.Concept, model.Table, model.Relationship +``` + +Additional type imports as needed: `Date`, `DateTime`, `Float`. + +--- + +## Define and Populate Concepts + +Three concept categories show up in a GNN pipeline, distinguished by whether they declare a primary key and how they participate in the graph: + +| Category | `identify_by`? | Role | Constraints | +|----------|---------------|------|-------------| +| **Graph (node)** | yes | Source, target, or other node entities the GNN reasons over | Can carry features and `time_col` | +| **Edge-intermediary** | no | Used only as `src=`/`dst=` in `Edge.new(...)` to express many-to-many or attributed relationships | **Cannot carry `time_col`** -- `time_col` only propagates for node concepts (with `identify_by`); see `rai-predictive-training` § Known Limitations | +| **Task table** | no | Holds train/val/test split rows, joined to a graph concept by FK | Not used in edges; not a feature source | + +> If you have an existing ontology from `rai-build-starter-ontology`, create a new `Model` for the GNN pipeline -- concepts need `identify_by` for GNN to resolve primary keys. + +### Graph (node) Concepts + +The `identify_by` key names must exist as columns in the Snowflake table. Column-name matching is **case-insensitive** in both `identify_by` keys and property accesses -- a Snowflake column `FOO_BAR` can be referenced as `Concept.foo_bar`, `Concept.FOO_BAR`, or any other casing. Spelling still has to match exactly. Check `INFORMATION_SCHEMA.COLUMNS` or run `DESCRIBE TABLE` to confirm the columns before writing `identify_by` or property accesses. + +```python +User = Concept("User", identify_by={"user_id": Integer}) +Event = Concept("Event", identify_by={"event_id": Integer}) +``` + +### Edge-intermediary Concepts + +When a many-to-many or attributed relationship is best modeled as its own concept (e.g. `Interaction` between `User` and `Item`), and that concept's row identity isn't needed downstream, you can omit `identify_by`: + +```python +EventAttendee = Concept("EventAttendee") # used only in Edge.new(src=..., dst=...) +``` + +If the intermediary needs to carry the temporal column for `has_time_column=True`, give it an `identify_by` (promoting it to a graph node concept). `time_col` does not propagate from edge-intermediary concepts. + +### Task Table Concepts + +Task table concepts have no `identify_by`: + +```python +train_table_concept = Concept("TrainTable") +val_table_concept = Concept("ValidationTable") +test_table_concept = Concept("TestTable") +``` + +### Populate from Snowflake + +```python +define(Customer.new(Table("DB.SCHEMA.CUSTOMERS").to_schema())) +define(train_table_concept.new(Table("DB.TASKS.TRAIN").to_schema())) +``` + +The GNN pipeline expects pre-existing train/val/test split tables in Snowflake. Each split table must contain: a join key column matching a source concept PK, a label/target column (train/val only), and optionally a timestamp column. + +`PropertyTransformer` and the task-table pattern also work with concepts populated from local data via `model.data(df)` -- not just `Table(...).to_schema()`. Useful when some concept data lives in local CSVs (e.g. optimizer parameters) while the graph comes from Snowflake. + +**Timestamp column type matters for the GNN datetime pipeline.** Columns intended for `time_col` / `datetime` features need a type the trainer accepts; native Snowflake `TIMESTAMP_NTZ` has been observed to be silently incompatible (loads cleanly, but the trainer doesn't pick the column up as temporal). VARCHAR ISO-8601 is the safer default for time-bearing columns, though large-scale loads can still trip a server-side `ValidationError` (see `rai-predictive-training` § Known Limitations). Confirm the trainer's currently-accepted timestamp formats with the RelationalAI team if you're hitting datetime errors at scale. + +--- + +## Task Relationships + +Relationships encode the task structure using a template string with three parts: +- **Head** = source concept (the concept being predicted on) +- **"at" clause** = optional timestamp field +- **"has" clause** = label (classification/regression) or target concept (link prediction) + +### Relationship Arity Rules + +| Task Type | Train/Val template | Test template | +|-----------|-------------------|---------------| +| classification (no time) | `f"{Source} has {Any:label}"` | `f"{Source}"` | +| classification (with time) | `f"{Source} at {Any:ts} has {Any:label}"` | `f"{Source} at {Any:ts}"` | +| regression (no time) | `f"{Source} has {Any:value}"` | `f"{Source}"` | +| regression (with time) | `f"{Source} at {Any:ts} has {Any:value}"` | `f"{Source} at {Any:ts}"` | +| link_prediction | `f"{Source} has {Target}"` | `f"{Source}"` | +| repeated_link_prediction | `f"{Source} at {Any:ts} has {Target}"` | `f"{Source} at {Any:ts}"` | + +For full code examples of all task type patterns, see [references/task-relationships.md](references/task-relationships.md). + +--- + +## Graph and Edges + +```python +gnn_graph = Graph(model, directed=True, weighted=False) +Edge = gnn_graph.Edge +``` + +### Standard Edges (FK field equality) + +```python +define(Edge.new(src=Interaction, dst=User)).where( + Interaction.user_id == User.user_id) +``` + +### Self-Referential Edges (use `.ref()`) + +```python +PostRef = Post.ref() +define(Edge.new(src=Post, dst=PostRef)).where( + PostRef.parent_id == Post.id) +``` + +### Mediated Self-Reference + +```python +PeopleRef = People.ref() +define(Edge.new(src=People, dst=PeopleRef)).where( + People.Id == Related.person1, + PeopleRef.Id == Related.person2, +) +``` + +### Multiple Typed Edges Between Same Pair + +```python +BB1Edge = Concept("BB1Edge", extends=[Edge]) +BB2Edge = Concept("BB2Edge", extends=[Edge]) + +Bref = B.ref() +define(BB1Edge.new(src=B, dst=Bref)).where(B.field1 == Bref.id) +define(BB2Edge.new(src=B, dst=Bref)).where(B.field2 == Bref.id) +``` + +--- + +## Feature Configuration + +The `PropertyTransformer` annotates concept fields with their semantic types for the GNN. + +```python +pt = PropertyTransformer( + category=[User.locale, User.gender, Event.city, Event.state, Event.country], + datetime=[User.joinedAt, Event.start_time], + continuous=[User.birthyear], + time_col=[Event.start_time], +) +``` + +### Feature Type Guidelines + +| Data type | Annotation | +|-----------|-----------| +| Boolean flags, enum/status codes | `category` | +| Ages, prices, ratings | `continuous` | +| Free-form text, names, descriptions | `text` | +| Dates, timestamps | `datetime` | +| Explicit integer values (not IDs) | `integer` | + +The `integer` parameter is a distinct type from `continuous` -- use it for whole-number counts or ordinal values where float precision is not meaningful (e.g. review counts, position ranks): + +```python +pt = PropertyTransformer( + integer=[Review.num_votes, Standing.position], + continuous=[Review.rating, Result.points], + ... +) +``` + +### Feature Selection Strategy + +- **Drop all PKs and FKs.** Graph structure already captures relationships; IDs add noise. Example: `drop=[Study.nct_id, Outcome.id, Outcome.nct_id, ...]` +- **Start with minimal `text` fields.** Text embedding is expensive and too many text fields dilute signal. Begin with 3-5 key text fields, add more only if metrics improve. +- **Use `category` for discrete location/status fields.** Fields like city, state, country have limited cardinality. +- **Use `continuous` for numeric measurements.** Counts, scores, percentages. +- **Lean feature sets beat everything-in.** In practice, reducing ~30 text fields to 5 improved AUROC from 57% to 68%. + +### Graph metrics as features + +Centrality, community labels, and other graph-algorithm outputs from `rai-graph-analysis` can feed the GNN as features once they're materialized as concept properties. Compute the metric on a separate Graph instance (the algorithm graph -- often a different topology from the GNN graph), bind the result, then include in the PropertyTransformer: + +```python +# Algorithm graph (e.g. node-to-node, distinct from the GNN's bipartite/edge-intermediary graph) +algo_graph = Graph(model, directed=False) +define(algo_graph.Edge.new(src=Source, dst=SourceRef)).where(...) + +# Bind metric output as a Concept property +Source.pagerank = model.Property(f"{Source} has {Float:pagerank}") +model.define(Source.pagerank(graph_algo_result)) + +# Include as a continuous (or category) feature +pt = PropertyTransformer( + continuous=[Source.pagerank, ...], + ... +) +``` + +Two-graph setups are common (the GNN graph and the algorithm graph have different shapes); name them distinctly to avoid confusion. + +PropertyTransformer is optional -- omitting it auto-infers all field types. For production, explicit annotation is recommended. Use `drop` to exclude fields or entire concepts: `drop=[Interaction, Item.internal_code]`. + +For the full feature type reference including drop patterns, see [references/property-transformer-types.md](references/property-transformer-types.md). + +--- + +## Common Pitfalls + +| Mistake | Cause | Fix | +|---------|-------|-----| +| Concept name is plural (e.g. "Customers") | Naming convention | Use singular names: `Concept("Customer")` | +| Task table concept has `identify_by` | Task tables don't need primary keys | Use plain `Concept("TrainTable")` with no `identify_by` | +| Snowflake table name not fully qualified | Missing database or schema prefix | Use `"DATABASE.SCHEMA.TABLE"` format | +| Test Relationship includes label/target | Test data should not contain the answer | Omit the "has" clause: `f"{Source}"` or `f"{Source} at {Any:ts}"` | +| Positional args in `define(Train(...))` don't match template | Template and population call must align | Match the order: source, [timestamp], [label/target] | +| Self-referential edge without `.ref()` | Same concept on both sides creates ambiguity | Use `PostRef = Post.ref()` for the destination | +| `time_col` fields not in `datetime` list | Both lists must include the field | Add time columns to both `datetime=[...]` and `time_col=[...]` | +| Task table concept used in edge definition | Only graph concepts participate in edges | Edges connect domain entities, not task tables | +| Missing type import | e.g. using `Date` without importing it | Add missing types to the import line | +| Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | +| `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | +| Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | + +--- + +## Examples + +| Pattern | Description | File | +|---------|-------------|------| +| Node classification | Binary classification data model | [examples/node_classification_snowflake.py](examples/node_classification_snowflake.py) | +| Link prediction | Repeated link prediction data model | [examples/link_prediction_snowflake.py](examples/link_prediction_snowflake.py) | +| Regression | Regression-with-time data model | [examples/regression_snowflake.py](examples/regression_snowflake.py) | + +--- + +## Reference files + +| Reference | Description | File | +|-----------|-------------|------| +| Task relationships | Relationship template patterns for all task types with code examples | [references/task-relationships.md](references/task-relationships.md) | +| PropertyTransformer types | Full feature type reference, drop patterns, and guidelines | [references/property-transformer-types.md](references/property-transformer-types.md) | +| Auto-discovery | SQL templates for discovering PKs, FKs, edges, and task structure | [references/auto-discovery.md](references/auto-discovery.md) | diff --git a/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py b/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py new file mode 100644 index 0000000..d591cb2 --- /dev/null +++ b/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py @@ -0,0 +1,80 @@ +""" +GNN Link Prediction -- Data Modeling (Phases 1-6) +================================================= +Repeated link prediction on a bipartite User-Item graph with an Interaction +edge-intermediary concept carrying timestamps. + +Demonstrates: concepts, population, task relationships (link prediction with +time), graph edges via an intermediary concept, and PropertyTransformer. + +For training and prediction, see `rai-predictive-training`. +""" + +# -- Phase 1: Imports & Model Setup -- +from relationalai.semantics import Model, select, define, Integer, Any +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +model = Model("gnn_link_prediction_example") +Concept, Table, Relationship = model.Concept, model.Table, model.Relationship + +# -- Phase 2: Define Concepts -- +# graph (node) concepts -- User is source (predicting from), Item is target (predicting to). +# Interaction has its own identify_by because it carries `time_col` (timestamp); time_col +# only propagates for node concepts, so an edge-intermediary version (no identify_by) would +# fail validation. See `rai-predictive-training` § Known Limitations. +User = Concept("User", identify_by={"user_id": Integer}) +Item = Concept("Item", identify_by={"item_id": Integer}) +Interaction = Concept("Interaction", identify_by={"interaction_id": Integer}) + +# task table concepts +train_table_concept = Concept("TrainTable") +val_table_concept = Concept("ValidationTable") +test_table_concept = Concept("TestTable") + +# -- Phase 3: Populate Concepts (from Snowflake) -- +define(User.new(Table("DB.SCHEMA.USERS").to_schema())) +define(Item.new(Table("DB.SCHEMA.ITEMS").to_schema())) +define(Interaction.new(Table("DB.SCHEMA.INTERACTIONS").to_schema())) + +define(train_table_concept.new(Table("DB.SCHEMA.TRAIN_LINK").to_schema())) +define(val_table_concept.new(Table("DB.SCHEMA.VAL_LINK").to_schema())) +define(test_table_concept.new(Table("DB.SCHEMA.TEST_LINK").to_schema())) + +# -- Phase 4: Setup Task Relationships -- repeated_link_prediction (with time) +# Train/Val carry the Target concept in the "has" clause (no {Any:label}). +# Test omits the target: the GNN predicts which Item each User links to. +Train = Relationship(f"{User} at {Any:timestamp} has {Item}") +define(Train(User, train_table_concept.timestamp, Item)).where( + User.user_id == train_table_concept.user_id, + Item.item_id == train_table_concept.item_id, +) + +Val = Relationship(f"{User} at {Any:timestamp} has {Item}") +define(Val(User, val_table_concept.timestamp, Item)).where( + User.user_id == val_table_concept.user_id, + Item.item_id == val_table_concept.item_id, +) + +Test = Relationship(f"{User} at {Any:timestamp}") +define(Test(User, test_table_concept.timestamp)).where( + User.user_id == test_table_concept.user_id, +) + +# -- Phase 5: Build Graph & Edges -- +gnn_graph = Graph(model, directed=True, weighted=False) +Edge = gnn_graph.Edge + +define(Edge.new(src=Interaction, dst=User)).where( + Interaction.user_id == User.user_id) +define(Edge.new(src=Interaction, dst=Item)).where( + Interaction.item_id == Item.item_id) + +# -- Phase 6: Configure PropertyTransformer -- +pt = PropertyTransformer( + category=[User.region, User.status, Item.category, Interaction.channel], + continuous=[User.age, Interaction.value], + text=[Item.name], + datetime=[Interaction.timestamp], + time_col=[Interaction.timestamp], +) diff --git a/skills/rai-predictive-modeling/examples/node_classification_snowflake.py b/skills/rai-predictive-modeling/examples/node_classification_snowflake.py new file mode 100644 index 0000000..1b405e8 --- /dev/null +++ b/skills/rai-predictive-modeling/examples/node_classification_snowflake.py @@ -0,0 +1,86 @@ +""" +GNN Node Classification -- Data Modeling (Phases 1-6) +===================================================== +Binary classification on user data from Snowflake with temporal features. +Demonstrates: concepts, population, task relationships, graph, and features. + +For training and prediction, see `rai-predictive-training`. +""" + +# -- Phase 1: Imports & Model Setup -- +from relationalai.semantics import Model, select, define, Integer, String, Any +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +model = Model("gnn_node_classification_example") +Concept, Table, Relationship = model.Concept, model.Table, model.Relationship + +# -- Phase 2: Define Concepts -- +# graph (node) concepts +User = Concept("User", identify_by={"user_id": Integer}) +Event = Concept("Event", identify_by={"event_id": Integer}) +# edge-intermediary concept (no identify_by, used only as Edge src/dst) +EventAttendee = Concept("EventAttendee") + +# task table concepts +train_table_concept = Concept("TrainTable") +val_table_concept = Concept("ValidationTable") +test_table_concept = Concept("TestTable") + +# -- Phase 3: Populate Concepts (from Snowflake) -- +define(User.new(Table("DB.SCHEMA.USERS").to_schema())) +define(Event.new(Table("DB.SCHEMA.EVENTS").to_schema())) +define(EventAttendee.new(Table("DB.SCHEMA.EVENT_ATTENDEES").to_schema())) + +define(train_table_concept.new(Table("DB.SCHEMA.TRAIN").to_schema())) +define(val_table_concept.new(Table("DB.SCHEMA.VAL").to_schema())) +define(test_table_concept.new(Table("DB.SCHEMA.TEST").to_schema())) + +# -- Phase 4: Setup Task Relationships -- +Train = Relationship(f"{User} at {Any:timestamp} has {Any:target}") +define(Train(User, train_table_concept.timestamp, train_table_concept.target)).where( + User.user_id == train_table_concept.user_id +) + +Val = Relationship(f"{User} at {Any:timestamp} has {Any:target}") +define(Val(User, val_table_concept.timestamp, val_table_concept.target)).where( + User.user_id == val_table_concept.user_id +) + +Test = Relationship(f"{User} at {Any:timestamp}") +define(Test(User, test_table_concept.timestamp)).where( + User.user_id == test_table_concept.user_id +) + +# -- Phase 5: Build Graph & Edges -- +gnn_graph = Graph(model, directed=True, weighted=False) +Edge = gnn_graph.Edge + +define(Edge.new(src=Event, dst=User)).where( + Event.user_id == User.user_id) +define(Edge.new(src=EventAttendee, dst=Event)).where( + EventAttendee.event == Event.event_id) +define(Edge.new(src=EventAttendee, dst=User)).where( + EventAttendee.user_id == User.user_id) + +# -- Phase 6: Configure PropertyTransformer -- +category_user = [User.locale, User.gender] +datetime_user = [User.joinedAt] +continuous_user = [User.birthyear] + +category_event = [Event.city, Event.state, Event.zip, Event.country] +datetime_event = [Event.start_time] +continuous_event = [Event.lat, Event.lng] + +category_event_attendee = [EventAttendee.status] +datetime_event_attendee = [EventAttendee.start_time] + +# time_col only propagates for node concepts -- list it on Event (a node), not +# on EventAttendee (edge-intermediary, no identify_by). See +# `rai-predictive-training` § Known Limitations for the failure mode this avoids. +pt = PropertyTransformer( + category=[*category_user, *category_event, *category_event_attendee], + datetime=[*datetime_user, *datetime_event, *datetime_event_attendee], + continuous=[*continuous_user, *continuous_event], + time_col=[Event.start_time], +) diff --git a/skills/rai-predictive-modeling/examples/regression_snowflake.py b/skills/rai-predictive-modeling/examples/regression_snowflake.py new file mode 100644 index 0000000..0550531 --- /dev/null +++ b/skills/rai-predictive-modeling/examples/regression_snowflake.py @@ -0,0 +1,85 @@ +""" +GNN Regression -- Data Modeling (Phases 1-6) +============================================= +Regression with temporal features on a bipartite User-Item graph. +The source concept (Interaction) carries the numeric target to predict. + +Demonstrates: concepts, population, regression task relationships with +`{Any:value}`, graph edges, and PropertyTransformer with time_col. + +For training and prediction, see `rai-predictive-training`. +""" + +# -- Phase 1: Imports & Model Setup -- +from relationalai.semantics import Model, select, define, Integer, Any +from relationalai.semantics.reasoners.graph import Graph +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +model = Model("gnn_regression_example") +Concept, Table, Relationship = model.Concept, model.Table, model.Relationship + +# -- Phase 2: Define Concepts -- +# graph concepts -- the source concept (the one being predicted on) needs its +# own primary key. If the source table lacks one, add a row_number column in +# Snowflake first (e.g. via a view or derived table). +User = Concept("User", identify_by={"user_id": Integer}) +Item = Concept("Item", identify_by={"item_id": Integer}) +Interaction = Concept("Interaction", identify_by={"interaction_id": Integer}) + +# task table concepts +train_table_concept = Concept("TrainTable") +val_table_concept = Concept("ValidationTable") +test_table_concept = Concept("TestTable") + +# -- Phase 3: Populate Concepts (from Snowflake) -- +define(User.new(Table("DB.SCHEMA.USERS").to_schema())) +define(Item.new(Table("DB.SCHEMA.ITEMS").to_schema())) +define(Interaction.new(Table("DB.SCHEMA.INTERACTIONS").to_schema())) + +define(train_table_concept.new(Table("DB.SCHEMA.TRAIN").to_schema())) +define(val_table_concept.new(Table("DB.SCHEMA.VAL").to_schema())) +define(test_table_concept.new(Table("DB.SCHEMA.TEST").to_schema())) + +# -- Phase 4: Setup Task Relationships -- regression (with time) +# Train/Val carry the numeric target in the "has" clause as {Any:value}. +# Test omits the target: the GNN predicts it. +Train = Relationship(f"{Interaction} at {Any:timestamp} has {Any:value}") +define(Train(Interaction, train_table_concept.timestamp, train_table_concept.value)).where( + Interaction.interaction_id == train_table_concept.interaction_id, +) + +Val = Relationship(f"{Interaction} at {Any:timestamp} has {Any:value}") +define(Val(Interaction, val_table_concept.timestamp, val_table_concept.value)).where( + Interaction.interaction_id == val_table_concept.interaction_id, +) + +Test = Relationship(f"{Interaction} at {Any:timestamp}") +define(Test(Interaction, test_table_concept.timestamp)).where( + Interaction.interaction_id == test_table_concept.interaction_id, +) + +# -- Phase 5: Build Graph & Edges -- +gnn_graph = Graph(model, directed=True, weighted=False) +Edge = gnn_graph.Edge + +define(Edge.new(src=Interaction, dst=User)).where( + Interaction.user_id == User.user_id, +) +define(Edge.new(src=Interaction, dst=Item)).where( + Interaction.item_id == Item.item_id, +) + +# -- Phase 6: Configure PropertyTransformer -- +# Drop PKs/FKs explicitly -- fields not listed in any category get auto-inferred +# as features, so PKs/FKs must be in `drop=[...]` to actually be excluded. +pt = PropertyTransformer( + category=[User.region, User.status, Item.category, Interaction.channel], + continuous=[User.age], + text=[Item.name], + datetime=[Interaction.timestamp], + time_col=[Interaction.timestamp], + drop=[ + User.user_id, Item.item_id, Interaction.interaction_id, + Interaction.user_id, Interaction.item_id, + ], +) diff --git a/skills/rai-predictive-modeling/references/auto-discovery.md b/skills/rai-predictive-modeling/references/auto-discovery.md new file mode 100644 index 0000000..6ff8931 --- /dev/null +++ b/skills/rai-predictive-modeling/references/auto-discovery.md @@ -0,0 +1,105 @@ +# Auto-Discovery + +After the user provides table names, the agent automatically discovers schema details by querying Snowflake. This reference documents the conversation templates and discovery process. + +## Conversation Templates + +### Phase 1a -- Source Tables + +Ask exactly this: + +``` +Phase 1a: Source Tables + +What are your **source table** fully qualified names? +(e.g., `MY_DB.MY_SCHEMA.CUSTOMERS`, `MY_DB.MY_SCHEMA.TRANSACTIONS`) + +If you have a schema diagram or image, feel free to share it and I'll extract the details. +``` + +### Phase 1b -- Task Tables + +Ask exactly this (after user responds to 1a): + +``` +Phase 1b: Task Tables + +What are your **task table** fully qualified names for train/val/test? +(e.g., `MY_DB.TASKS.TRAIN`, `MY_DB.TASKS.VAL`, `MY_DB.TASKS.TEST`) +``` + +### Phase 1c -- Experiment Artifacts + +Ask exactly this (after user responds to 1b): + +``` +Phase 1c: Experiment Artifacts + +What Snowflake database and schema should we use for **experiment artifacts**? +(e.g., `MY_DB.EXPERIMENTS`) +``` + +## What to Auto-Discover + +Once the user provides the table names, the agent must automatically discover the following by querying Snowflake (`DESCRIBE TABLE` or `INFORMATION_SCHEMA`). Use the snowflake-schema tool to get the schema of each table. + +1. **Column names and types** for all source and task tables +2. **Primary keys** -- identify PK columns +3. **Foreign key relationships** -- detect FK columns by matching column names across tables (e.g., `customer_id` in `TRANSACTIONS` matches `customer_id` PK in `CUSTOMERS`) +4. **Graph concepts** -- each source table becomes a concept (use singular form of table name) +5. **Edges** -- derived from FK relationships found above +6. **Task structure** -- from task table columns, infer: + - Join key (column matching a source concept PK) + - Label/target column (non-key, non-timestamp column) or target concept (for link prediction) + - Time column (columns with DATE/TIMESTAMP type) +7. **Task type** -- infer from the label column: + - Binary/boolean or 2-value categorical -> `binary_classification` + - Multi-value categorical -> `multiclass_classification` + - Numeric/float -> `regression` + - Column matching another concept's PK -> `link_prediction` (ask user to confirm) + +## Link Prediction Detection + +If link prediction is detected, after presenting the discovery summary, ask the user: + +``` +I detected a **link prediction** task. One more question: + +Are you predicting **new** links (connections that don't exist yet) or **repeated** interactions (e.g., a customer re-purchasing an item they've bought before)? + +- **New links** -> `link_prediction` +- **Repeated interactions** -> `repeated_link_prediction` +``` + +## Summary Table Template + +Present the discovery results to the user as a summary table for confirmation before proceeding: + +``` +Here's what I discovered from your tables: + +**Source Tables & Concepts:** +| Table | Concept | PK | Other Columns | +|-------|---------|-----|---------------| +| ... | ... | ... | ... | + +**Edges (FK relationships):** +| From | To | Join Condition | +|------|-----|---------------| +| ... | ... | ... | + +**Task Tables:** +| Split | Table | Join Key -> Concept | Label/Target | Time Column | +|-------|-------|-------------------|--------------|-------------| +| Train | ... | ... | ... | ... | +| Val | ... | ... | ... | ... | +| Test | ... | ... | ... (none) | ... | + +**Inferred task type:** `` + +Does this look correct? I'll proceed with this structure. +``` + +## Fallback + +If the agent cannot connect to Snowflake or auto-discovery fails, fall back to asking the user for column details directly. diff --git a/skills/rai-predictive-modeling/references/property-transformer-types.md b/skills/rai-predictive-modeling/references/property-transformer-types.md new file mode 100644 index 0000000..a778552 --- /dev/null +++ b/skills/rai-predictive-modeling/references/property-transformer-types.md @@ -0,0 +1,66 @@ +# PropertyTransformer Feature Types + +The `PropertyTransformer` class specifies how concept fields are transformed into GNN-compatible features. + +## Feature Types + +| Type | PropertyTransformer kwarg | Description | Example fields | +|------|--------------------------|-------------|----------------| +| Category | `category=[...]` | Discrete categorical values (int or string) | Gender, product code, membership status | +| Continuous | `continuous=[...]` | Numeric continuous values (float) | Age, price, rating | +| Text | `text=[...]` | Text strings (embedded via language model) | Product name, description, comment | +| Datetime | `datetime=[...]` | Timestamps or dates | Transaction date, creation date | +| Integer | `integer=[...]` | Whole-number counts or ordinal values (not IDs) | Review counts, position ranks | +| Drop | `drop=[...]` | Exclude field from model entirely | Foreign keys, sensitive data, redundant IDs | + +## Special Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `time_col` | list or single | Time column(s) for temporal models. Must NOT appear in `drop`. When multiple concepts have date fields, list all of them: `time_col=[Interaction.timestamp, Session.started_at, Order.placed_at, ...]`. Fields in `time_col` must also appear in `datetime`. | + +## Usage + +```python +from relationalai.semantics.reasoners.predictive import PropertyTransformer + +pt = PropertyTransformer( + category=[User.status, Item.category, Interaction.channel], + continuous=[User.age, Interaction.value], + text=[Item.name], + datetime=[Interaction.timestamp], + drop=[User.internal_code, Item.legacy_sku], + time_col=[Interaction.timestamp], +) +``` + +## Drop Patterns + +### Drop specific fields +```python +drop=[Item.legacy_sku, Item.internal_code] +``` + +### Drop all fields of a concept (identifier columns) +```python +drop=[User] # drops all User fields (including primary key) +``` + +### Mixed: drop entire concept + specific fields from another +```python +drop=[Interaction, Item.legacy_sku, Item.internal_code] +``` + +## Default Behavior + +Fields not mentioned in any category are auto-inferred by the GNN engine (equivalent to the `Infer` embedding type). This is usually fine for most fields, but explicitly annotating them improves reproducibility. + +## Guidelines + +- **Primary key / identifier fields**: Usually `drop` (they don't carry predictive signal) +- **Foreign key join columns**: Usually `drop` (the graph structure captures the relationship) +- **Numeric IDs that encode meaning** (e.g. product_code): Use `category` +- **Free-form text**: Use `text` +- **Dates/timestamps**: Use `datetime`. If it's the temporal ordering column, also add to `time_col` +- **Boolean flags**: Use `category` +- **Continuous measurements**: Use `continuous` diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md new file mode 100644 index 0000000..c871e0d --- /dev/null +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -0,0 +1,126 @@ +# Task Relationships + +Relationships encode the task structure using a template string with three parts: +- **Head** = source concept (the concept being predicted on) +- **"at" clause** = optional timestamp field +- **"has" clause** = label (classification/regression) or target concept (link prediction) + +## Relationship Arity Rules + +| Task Type | Train/Val template | Test template | +|-----------|-------------------|---------------| +| classification (no time) | `f"{Source} has {Any:label}"` | `f"{Source}"` | +| classification (with time) | `f"{Source} at {Any:ts} has {Any:label}"` | `f"{Source} at {Any:ts}"` | +| regression (no time) | `f"{Source} has {Any:value}"` | `f"{Source}"` | +| regression (with time) | `f"{Source} at {Any:ts} has {Any:value}"` | `f"{Source} at {Any:ts}"` | +| link_prediction | `f"{Source} has {Target}"` | `f"{Source}"` | +| repeated_link_prediction | `f"{Source} at {Any:ts} has {Target}"` | `f"{Source} at {Any:ts}"` | + +## Node Classification (with time) + +```python +Train = Relationship(f"{User} at {Any:timestamp} has {Any:target}") +define(Train(User, train_table_concept.timestamp, train_table_concept.target)).where( + User.user_id == train_table_concept.user_id +) + +Val = Relationship(f"{User} at {Any:timestamp} has {Any:target}") +define(Val(User, val_table_concept.timestamp, val_table_concept.target)).where( + User.user_id == val_table_concept.user_id +) + +Test = Relationship(f"{User} at {Any:timestamp}") +define(Test(User, test_table_concept.timestamp)).where( + User.user_id == test_table_concept.user_id +) +``` + +## Node Classification (no time) + +```python +Train = Relationship(f"{User} has {Any:target}") +Val = Relationship(f"{User} has {Any:target}") +Test = Relationship(f"{User}") +``` + +## Regression (with time) + +Numeric target on the source concept (e.g. a per-row value). + +```python +Train = Relationship(f"{Interaction} at {Any:timestamp} has {Any:value}") +define(Train(Interaction, train_table_concept.timestamp, train_table_concept.value)).where( + Interaction.interaction_id == train_table_concept.interaction_id, +) + +Val = Relationship(f"{Interaction} at {Any:timestamp} has {Any:value}") +define(Val(Interaction, val_table_concept.timestamp, val_table_concept.value)).where( + Interaction.interaction_id == val_table_concept.interaction_id, +) + +Test = Relationship(f"{Interaction} at {Any:timestamp}") +define(Test(Interaction, test_table_concept.timestamp)).where( + Interaction.interaction_id == test_table_concept.interaction_id, +) +``` + +## Regression (no time) + +```python +Train = Relationship(f"{Interaction} has {Any:value}") +Val = Relationship(f"{Interaction} has {Any:value}") +Test = Relationship(f"{Interaction}") +``` + +## Link Prediction (with time / repeated_link_prediction) + +```python +Train = Relationship(f"{User} at {Any:timestamp} has {Item}") +define(Train(User, train_table_concept.timestamp, Item)).where( + User.user_id == train_table_concept.user_id, + Item.item_id == train_table_concept.item_id, +) + +Val = Relationship(f"{User} at {Any:timestamp} has {Item}") +define(Val(User, val_table_concept.timestamp, Item)).where( + User.user_id == val_table_concept.user_id, + Item.item_id == val_table_concept.item_id, +) + +Test = Relationship(f"{User} at {Any:timestamp}") +define(Test(User, test_table_concept.timestamp)).where( + User.user_id == test_table_concept.user_id, +) +``` + +## Link Prediction (no time) + +```python +Train = Relationship(f"{User} has {Item}") +Val = Relationship(f"{User} has {Item}") +Test = Relationship(f"{User}") +``` + +## Alternative: select() fragments + +Instead of `Relationship` + `define()`, you can use `select()` directly. Both forms are accepted by the GNN constructor: + +```python +Train = select(User, train_table_concept.timestamp, Item).where( + User.user_id == train_table_concept.user_id, + Item.item_id == train_table_concept.item_id, +) + +Val = select(User, val_table_concept.timestamp, Item).where( + User.user_id == val_table_concept.user_id, + Item.item_id == val_table_concept.item_id, +) + +Test = select(User, test_table_concept.timestamp).where( + User.user_id == test_table_concept.user_id, +) +``` + +## Post-training aggregation (rollup shape) + +A common real-world shape is: train the GNN on fine-grained events (e.g. a `Transaction` source), then aggregate predictions up to a coarser entity (e.g. `Article`) for downstream rules or optimization. This lives on the **consumption side**, not in the Relationship template -- see `rai-predictive-training` § Aggregation and bridge concepts for the `aggregates.(Source.predictions.).per(Target).where(...)` pattern. diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md new file mode 100644 index 0000000..f5d4c62 --- /dev/null +++ b/skills/rai-predictive-training/SKILL.md @@ -0,0 +1,482 @@ +--- +name: rai-predictive-training +description: Configure and train GNN models, generate predictions, evaluate results, and manage trained models. Use after building the data model with rai-predictive-modeling, when ready to run training, evaluate, or manage GNN models. +--- + +# Predictive Training + + +> **Early access.** The RAI predictive reasoner (GNN) is in early access — APIs, engine requirements, and behavior may change. Confirm the latest surface with the RelationalAI team before production use. + +## Summary + +**What:** Training, evaluation, and model management workflow for GNN pipelines. + +**When to use:** +- Configuring the GNN estimator and hyperparameters +- Training models with `fit()` +- Generating predictions on test data +- Evaluating and debugging results +- Registering or loading saved models + +**When NOT to use:** +- Defining concepts, loading data, building graphs -- see `rai-predictive-modeling` + +**Overview:** 4 steps: configure GNN -> train -> predict/evaluate -> optional: register/load. + + +## Quick Reference + +### Node Classification (minimal) + +```python +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + train=Train, validation=Val, + task_type="binary_classification", eval_metric="roc_auc", + has_time_column=True, device="cuda", n_epochs=5, +) +gnn.fit() +User.predictions = gnn.predictions(domain=Test) +``` + +### Default Metrics + +| Task Type | Suggested Metric | +|-----------|-----------------| +| binary_classification | `roc_auc` | +| multiclass_classification | `accuracy` | +| multilabel_classification | `multilabel_auprc_macro` | +| regression | `rmse` | +| link_prediction | `link_prediction_precision@5` | +| repeated_link_prediction | `link_prediction_precision@5` | + +### Prediction Attributes + +| Task Type | Attributes | +|-----------|-----------| +| classification | `.probs`, `.predicted_labels` | +| regression | `.predicted_value` | +| link prediction | `.rank`, `.scores`, `.predicted_` | + +--- + +## GNN Constructor + +### Required Parameters + +| Parameter | Description | +|-----------|-------------| +| `exp_database`, `exp_schema` | Snowflake location for experiment artifacts | +| `graph` | Graph object with edges defined | +| `train`, `validation` | Relationship objects | +| `task_type` | Task type string | +| `eval_metric` | Evaluation metric string | + +### Optional Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `property_transformer` | None | PropertyTransformer instance (omit for auto-inference) | +| `has_time_column` | False | Set `True` when Relationships use the "at" keyword | +| `dataset_alias` | None | Custom alias for the dataset | +| `stream_logs` | True | Stream training logs to console. Set `False` if log streaming is slow or unreliable — training continues server-side regardless | +| `parallel_reasoners_init` | True | Initialize reasoners in parallel at construction time | + +### Node Classification Example + +```python +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + train=Train, validation=Val, + task_type="binary_classification", + eval_metric="roc_auc", + has_time_column=True, + device="cuda", n_epochs=5, lr=0.005, +) +gnn.fit() +``` + +### Link Prediction Example (temporal) + +```python +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + train=Train, validation=Val, + task_type="repeated_link_prediction", + eval_metric="link_prediction_precision@5", + has_time_column=True, + device="cuda", n_epochs=5, lr=0.005, + head_layers=2, num_negative=20, label_smoothing=True, +) +gnn.fit() +``` + +**Note:** `gnn.fit()` trains at most once per GNN instance. If training has already completed (or is in progress), subsequent calls to `fit()` are silent no-ops. To retrain -- e.g. with different hyperparameters -- construct a new `GNN` instance. + +**Multi-GNN pipelines on the same model.** Train multiple GNNs over the same entity set (e.g. regression + classification + link-prediction on the same graph) by reusing one `Graph` and one `PropertyTransformer` across all `GNN` instances; vary `task_type`, `eval_metric`, `train`/`validation`, and the source/target concepts. Bind each task's predictions to a **distinct attribute name** -- the convention `Source.predictions` collides if one source concept hosts more than one task. + +```python +shared = dict(graph=gnn_graph, property_transformer=pt) +gnn_a = GNN(**shared, train=TrainA, validation=ValA, task_type="regression", eval_metric="rmse", ...) +gnn_b = GNN(**shared, train=TrainB, validation=ValB, task_type="binary_classification", eval_metric="roc_auc", ...) +gnn_c = GNN(**shared, train=TrainC, validation=ValC, task_type="repeated_link_prediction", eval_metric="link_prediction_precision@5", ...) +for g in (gnn_a, gnn_b, gnn_c): g.fit() + +# Distinct attributes when a source concept hosts multiple predictions: +Item.value_predictions = gnn_a.predictions(domain=TestA) +User.label_predictions = gnn_b.predictions(domain=TestB) +User.link_predictions = gnn_c.predictions(domain=TestC) +``` + +Hyperparameters can also be passed as a dictionary: + +```python +train_config = {"device": "cuda", "n_epochs": 10, "lr": 0.001, "train_batch_size": 512} +gnn = GNN(exp_database="DB", exp_schema="EXPERIMENTS", ..., **train_config) +gnn.fit() +``` + +--- + +## Common Hyperparameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `device` | `"cuda"` | `"cuda"` (GPU) or `"cpu"` | +| `n_epochs` | 5 | Number of training epochs | +| `lr` | 0.005 | Learning rate | +| `train_batch_size` | 256 | Training batch size | + +For link prediction, also consider: `head_layers=2`, `num_negative=20`, `label_smoothing=True`. + +**`device="cuda"` is a paired requirement.** The client-side flag alone is not enough — the predictive reasoner engine must also be GPU-sized in `raiconfig.yaml`. Configure both or neither; mismatched settings silently fall back or fail. Heuristic: CPU HIGHMEM tiers trade training speed for more RAM; GPU is faster per epoch when the dataset fits in the GPU VM's CPU memory, HIGHMEM otherwise. + +**A GNN workflow touches multiple reasoner engines — size each per its role.** At minimum, the **predictive** engine runs `fit()` and prediction jobs, and the **logic** engine runs model definitions, queries, and downstream rule evaluation; predict-then-optimize pipelines also need the **prescriptive** engine. Each is configured independently in `raiconfig.yaml` under `reasoners:` with its own `name` and `size` — appropriate sizing differs per role (GPU for predictive training, HIGHMEM CPU for logic query and rule workloads, and per-problem sizing for prescriptive). Mis-sizing one engine doesn't error loudly; the workflow still runs and silently under-performs or hits memory limits on that engine's step. + +**Auto-suspend during iteration.** Set a low `auto_suspend_mins` on every engine you're using — idle pool cost can dominate total spend on small workloads. Warm pools make sense only for scheduled/production cadence. Specific tier names and per-cloud memory-vs-compute tradeoffs change over time — ask the RelationalAI team for current sizing. Full `raiconfig.yaml` structure (including the `reasoners:` block for all engine types) lives in the RAI configuration/setup skill. + +**Resume suspended GPU pools before runs -- required, not optional.** A suspended GPU compute pool does not auto-resume when `gnn.fit()` or `gnn.predictions()` submits; the client has been observed polling indefinitely (over 90 minutes idle) with no error surfaced. Before any training or prediction run -- especially after a period of idle when auto-suspend has fired -- explicitly resume the pool so it is `READY` when the job submits. Run `ALTER COMPUTE POOL RESUME` in Snowflake (the `` is the predictive reasoner entry in `raiconfig.yaml`), or use the RAI CLI's reasoner resume. See the RAI configuration/setup skill for current CLI syntax. + +For all hyperparameters and tuning guidance, see [references/hyperparameters.md](references/hyperparameters.md). + +--- + +## Training + +### fit() Stages + +`gnn.fit()` runs three stages internally: +1. Data preparation and feature extraction +2. Model training over `n_epochs` +3. Evaluation on the validation set + +### Known Limitations + +`has_time_column=True` has two known failure modes; both share the same workaround (turn temporal off — switch to non-temporal Relationships and `has_time_column=False`): + +1. **Edge-intermediary `time_col`.** When the concept carrying `time_col` is used only as an edge intermediary (no `identify_by`), validation fails with "no time column defined in data tables". `time_col` only propagates for node concepts. +2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. + +--- + +## Predictions + +After training, generate predictions on the test set: + +```python +Source.predictions = gnn.predictions(domain=Test) +``` + +### Classification (binary, multiclass, multilabel) + +```python +User.predictions = gnn.predictions(domain=Test) + +select( + User.user_id, + User.predictions.probs, + User.predictions.predicted_labels, +).where(User.predictions).inspect() +``` + +### Regression + +```python +Unit.predictions = gnn.predictions(domain=Test) + +select( + Unit.unit_id, + Unit.predictions.predicted_value, +).where(Unit.predictions).inspect() +``` + +### Link Prediction + +```python +User.predictions = gnn.predictions(domain=Test) + +select( + User.user_id, + Item.item_id, + User.predictions.rank, + User.predictions.scores, +).where( + User.predictions.predicted_item == Item, +).inspect() +``` + +The `predicted_` attribute name is always lowercase: Target `Item` -> `.predicted_item`. + +### As DataFrame + +Replace `.inspect()` with `.to_df()` to get a pandas DataFrame: + +```python +df = select( + User.user_id, + User.predictions.probs, + User.predictions.predicted_labels, +).where(User.predictions).to_df() +``` + +### Dictionary-Style Field Indexing + +The prediction relation also supports dictionary-style field indexing, useful when the source concept name conflicts with an existing attribute: + +```python +PredRelation = gnn.predictions(domain=Test) +select( + PredRelation["beer"].name, + PredRelation["timestamp"], + PredRelation["prediction"].predicted_labels, + PredRelation["prediction"].probs, +).inspect() +``` + +For the full prediction attributes reference (per-task attribute types, code shapes), see [references/prediction-attributes.md](references/prediction-attributes.md). The summary table is in Quick Reference above. + +--- + +## Using Predictions Downstream + +Once `Source.predictions = gnn.predictions(...)` runs, predictions are bound to the source concept and accessible via `Source.predictions.` throughout the **same `Model`**. Other reasoners (rules, prescriptive, graph) consume them by deriving new properties from those attributes. + +### Same-model pattern (default) + +Keep training, prediction, and downstream reasoning in one `Model`. This is the idiomatic RAI flow for predict-then-optimize and predict-then-rules chains: + +```python +# 1. Train and bind predictions +Item.predictions = gnn.predictions(domain=Test) + +# 2. Derive a regular property from the prediction +Item.predicted_value = model.Property(f"{Item} has {Float:predicted_value}") +model.define(Item.predicted_value(Item.predictions.predicted_value)) + +# 3a. Predictive -> Rules: boolean flag +Item.is_high = model.Relationship(f"{Item} is high") +model.where(Item.predicted_value > threshold).define(Item.is_high()) + +# 3b. Predictive -> Prescriptive: Item.predicted_value can appear in +# Problem(model, Float) constraint / objective expressions. +``` + +### Cross-session pattern (explicit persistence) + +If training and downstream reasoning run in separate processes, persist predictions to Snowflake and reload them as a fresh `Concept`: + +```python +# Training session: save predictions DataFrame +df = select(Source.source_id, Source.predictions.predicted_value) \ + .where(Source.predictions).to_df() +# Then write_pandas(conn, df, "MY_PREDICTIONS", auto_create_table=True, overwrite=True) +# Grant SELECT on MY_PREDICTIONS to APPLICATION RELATIONALAI. + +# Downstream session: load as a Concept in a new Model +Prediction = Concept("Prediction", identify_by={"source_id": Integer}) +model.define(Prediction.new(Table("DB.SCHEMA.MY_PREDICTIONS").to_schema())) +# Derive properties from Prediction, apply rules, run a solver, etc. +``` + +`database=` and `schema=` on `GNN(...)` are optional and omitted throughout this skill. For durable persistence, use the explicit `write_pandas` path above. + +### Aggregation and bridge concepts + +When the downstream reasoner's scope differs from the GNN source -- e.g. per-source predictions feeding a per-target optimizer -- aggregate predictions via `aggregates.(...).per(Target).where(join)` and attach the result to a **bridge concept** representing the downstream scope: + +```python +# GNN source predicts a value per Source (e.g. per-event regression); +# downstream scope is OptTarget, one row per coarser entity. +OptTarget = Concept("OptTarget", identify_by={"opt_target_id": Integer}) +OptTarget.total_predicted_value = model.Property(f"{OptTarget} has {Float:total_predicted_value}") + +agg = aggregates.sum(Source.predictions.predicted_value).per(OptTarget).where( + Interaction.target_id == OptTarget.opt_target_id, + Interaction.source_id == Source.source_id, +) +model.define(OptTarget.total_predicted_value(agg)) +``` + +The bridge concept (`OptTarget`) separates *what the GNN predicted at Source scope* from *what the downstream reasoner consumes at Target scope*. Skipping the bridge and trying to use `Source.predictions.predicted_value` directly in a Target-scoped constraint forces ad-hoc joins inside each rule or objective expression. For classification or link-prediction predictions, swap `sum`/`predicted_value` for `avg`/`probs` or `count`/`scores` per the rule below. + +**Choose the aggregation function by target shape.** Use `sum` for additive or count-like predictions (per-event regression values rolled up to an entity total), `avg` for proportional or probability-like predictions (mean predicted score across related source entities), `count` for link-prediction hits. Mixing them produces values that look numerically fine but don't mean what downstream expects. + +**Non-additive blending of multiple signals** (e.g. combining several GNN outputs, or a GNN probability with a rule-derived flag) is also a derived-property step, not a built-in `aggregates.`. Express it as ordinary arithmetic in the property definition: a multiplicative composite (`predicted_a * (1 - w * avg_b) * (1 + w * avg_c)`) for risk-uplift-style logic, or a weighted interpolation (`alpha * rule_signal + (1 - alpha) * gnn_probs`) for hybrid scoring. Keep the bridge concept distinct from the GNN source so the blend is a regular Property the downstream reasoner can consume. + +**Denormalize if the target was pre-scaled at training.** If the training target was normalized (e.g. to `[0, 1]`, or z-scored), raw predictions carry that scale too. Record the denormalization factor alongside the derived property and apply it before feeding into constraints or objectives that expect real-world units -- otherwise the downstream reasoner sees tiny numbers where it expected the real-world quantity. + +For a full predict-then-optimize example chaining multiple GNNs into optimizers with bridge + aggregation, see the `retail_planning` template in the templates repo. + +--- + +## Evaluation & Debugging + +After `gnn.fit()`, inspect what data the engine received: + +```python +# Visual graph of the dataset schema +graph_viz = gnn.visualize_dataset(show_dtypes=True) +graph_viz.write_png("dataset_schema.png") + +# Print the data config to console +gnn.dataset.print_data_config() +``` + +If results are poor, see [references/evaluation-debugging.md](references/evaluation-debugging.md) § Tuning Poor Results for the ordered checklist (dataset inspection → text-feature reduction → hyperparameter tuning) plus regression-specific sanity checks, multi-metric framing, and leakage diagnostics. + +--- + +## Model Management + +### Register a Model + +After `gnn.fit()` completes: + +```python +gnn.register_model( + model_database="DB", + model_schema="MODEL_REGISTRY", + model_name="my_predictor", + version_name="v1", + comment="Initial training run", # optional +) +``` + +### Load by Registry Key + +```python +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + source_concept=User, + task_type="binary_classification", + has_time_column=True, + model_database="DB", model_schema="MODEL_REGISTRY", + model_name="my_predictor", version_name="v1", +) +gnn.load() +User.predictions = gnn.predictions(domain=Test) +``` + +### Load by Run ID + +Same as above, replacing the registry key params with `model_run_id=""`. + +### What to Include vs. Omit When Loading + +| Include | Omit | +|---------|------| +| `exp_database`, `exp_schema` | `database`, `schema` (now optional) | +| `graph`, `property_transformer` | `train`, `validation` | +| `source_concept` (required) | `eval_metric` | +| `task_type` (required) | hyperparameters (`device`, `n_epochs`, etc.) | +| `has_time_column=True` (if model was trained with time column) | | +| `target_concept` (required for link prediction only) | | +| model identifier (registry key or run ID) | | + +### Train-Register-Load Workflow + +**Session 1: Train and Register** + +```python +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + train=Train, validation=Val, + task_type="binary_classification", eval_metric="roc_auc", + has_time_column=True, device="cuda", n_epochs=5, +) +gnn.fit() +gnn.register_model( + model_database="DB", model_schema="MODEL_REGISTRY", + model_name="my_predictor", version_name="v1", +) +``` + +**Session 2: Load and Predict** + +```python +# Rebuild graph and property_transformer (same structure as training) +gnn_graph = Graph(model, directed=True, weighted=False) +# ... define edges ... +pt = PropertyTransformer(...) + +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + source_concept=User, + task_type="binary_classification", + has_time_column=True, + model_database="DB", model_schema="MODEL_REGISTRY", + model_name="my_predictor", version_name="v1", +) +gnn.load() +User.predictions = gnn.predictions(domain=Test) +``` + +--- + +## Common Pitfalls + +| Mistake | Cause | Fix | +|---------|-------|-----| +| Missing `has_time_column=True` | Templates with the "at" keyword require the flag so the trainer finds the time column | Set `has_time_column=True` when templates contain "at" | +| Using `.predicted_Item` (uppercase) | Target-attribute names are always lowercased from the Target concept name | Use `.predicted_item` | +| Invalid `task_type`/`eval_metric` combination | Not every metric applies to every task type | Check [references/task-types-and-metrics.md](references/task-types-and-metrics.md) for valid pairs | +| `register_model()` before `fit()` | Registration requires a trained model | Always call `gnn.fit()` before `gnn.register_model()` | +| Omitting `graph`/`property_transformer` when loading | Load reconstructs against the same schema used during training | Provide the same `graph` and `property_transformer` used during training | +| Passing training-only params when loading | Load ignores training-time params | Omit `train`, `validation`, and hyperparameters when loading | +| Omitting `source_concept` when loading | Required to bind the loaded model to the source concept for prediction | Add `source_concept=` to the load constructor | +| Omitting `task_type` when loading | Not persisted in the registry | Add `task_type=""` to the load constructor | +| Omitting `target_concept` for link-prediction load | Required to resolve the prediction target concept | Add `target_concept=` for link prediction | +| Omitting `has_time_column` when loading a temporal model | Not persisted in the registry | Re-supply `has_time_column=True` at load time | +| `has_time_column=True` fails with "no time column defined in data tables" | The concept carrying `time_col` is an edge, not a node — `time_col` only propagates for node concepts | Use `has_time_column=False` with non-temporal Relationships as workaround | +| `has_time_column=True` fails with `ValidationError: Error processing datetime column ''` at scale | Server-side datetime processing rejects the column despite clean data, node-level concept, and correct `datetime`/`time_col` config — second known limitation | Verify the timestamp column type matches the GNN datetime pipeline's expected format (see `rai-predictive-modeling`); fall back to non-temporal Relationships if it persists | +| Experiment schema not accessible by the RAI native app | RAI app needs explicit grants to read from the experiment schema | `GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; GRANT ALL ON SCHEMA . TO APPLICATION RELATIONALAI` | +| `gnn.fit()` or `gnn.predictions()` hangs with no error output | GPU compute pool is suspended; client polls indefinitely instead of auto-resuming or failing fast (observed >90 min idle) | Run `ALTER COMPUTE POOL RESUME` (or the RAI CLI equivalent) before the run — pool name comes from the predictive reasoner entry in `raiconfig.yaml` | + +--- + +## Examples + +| Pattern | Description | File | +|---------|-------------|------| +| Node classification | Binary classification training + prediction | [examples/train_node_classification.py](examples/train_node_classification.py) | +| Link prediction | Repeated link prediction training + prediction | [examples/train_link_prediction.py](examples/train_link_prediction.py) | +| Regression | Regression training + prediction | [examples/train_regression.py](examples/train_regression.py) | +| Register and load | Complete train-register-load workflow across sessions | [examples/register_and_load.py](examples/register_and_load.py) | + +--- + +## Reference Files + +| Reference | Description | File | +|-----------|-------------|------| +| Task types and metrics | All valid (task_type, eval_metric) combinations | [references/task-types-and-metrics.md](references/task-types-and-metrics.md) | +| Hyperparameters | Full hyperparameter table with types, defaults, and tuning guidance | [references/hyperparameters.md](references/hyperparameters.md) | +| Prediction attributes | Prediction attributes by task type with usage examples | [references/prediction-attributes.md](references/prediction-attributes.md) | +| Evaluation & debugging | Dataset inspection, result checking, and tuning steps | [references/evaluation-debugging.md](references/evaluation-debugging.md) | diff --git a/skills/rai-predictive-training/examples/register_and_load.py b/skills/rai-predictive-training/examples/register_and_load.py new file mode 100644 index 0000000..76745bd --- /dev/null +++ b/skills/rai-predictive-training/examples/register_and_load.py @@ -0,0 +1,53 @@ +""" +GNN Model Management -- Register and Load Workflow +==================================================== +Demonstrates the train-register-load pattern across sessions. + +Session 1: Train a model and register it to Snowflake Model Registry. +Session 2: Load the registered model and generate predictions. +""" +from relationalai.semantics.reasoners.predictive import GNN + +# -- Session 1: Train and Register ------------------------------------------- +# Assumes data model from `rai-predictive-modeling`: +# gnn_graph, pt, Train, Val, Test, User + +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + train=Train, validation=Val, + task_type="binary_classification", eval_metric="roc_auc", + has_time_column=True, + device="cuda", n_epochs=5, +) +gnn.fit() + +gnn.register_model( + model_database="DB", + model_schema="MODEL_REGISTRY", + model_name="my_predictor", + version_name="v1", + comment="Initial training run", +) + + +# -- Session 2: Load and Predict --------------------------------------------- +# Rebuild graph and PropertyTransformer (same structure as training session) +# gnn_graph = Graph(model, directed=True, weighted=False) +# ... define edges ... +# pt = PropertyTransformer(...) + +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, property_transformer=pt, + source_concept=User, + task_type="binary_classification", + has_time_column=True, + model_database="DB", + model_schema="MODEL_REGISTRY", + model_name="my_predictor", + version_name="v1", +) +gnn.load() + +User.predictions = gnn.predictions(domain=Test) diff --git a/skills/rai-predictive-training/examples/train_link_prediction.py b/skills/rai-predictive-training/examples/train_link_prediction.py new file mode 100644 index 0000000..14ebacb --- /dev/null +++ b/skills/rai-predictive-training/examples/train_link_prediction.py @@ -0,0 +1,46 @@ +""" +GNN Link Prediction -- Training & Prediction +============================================== +Repeated link prediction training and prediction. + +Assumes data model from `rai-predictive-modeling`: + - gnn_graph: Graph with edges defined + - pt: PropertyTransformer instance (passed as `property_transformer=pt`) + - Train, Val, Test: Relationship objects + - User: source concept, Item: target concept +""" +from relationalai.semantics import select +from relationalai.semantics.reasoners.predictive import GNN + +# -- Train GNN --------------------------------------------------------------- +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, + property_transformer=pt, + train=Train, + validation=Val, + task_type="repeated_link_prediction", + eval_metric="link_prediction_precision@5", + has_time_column=True, + device="cuda", + n_epochs=5, + train_batch_size=256, + lr=0.005, + head_layers=2, + num_negative=20, + label_smoothing=True, +) +gnn.fit() + +# -- Predict & Inspect ------------------------------------------------------- +# .predicted_ attribute name is always lowercase: Target `Item` -> .predicted_item +User.predictions = gnn.predictions(domain=Test) + +select( + User.user_id, + Item.item_id, + User.predictions.rank, + User.predictions.scores, +).where( + User.predictions.predicted_item == Item, +).inspect() diff --git a/skills/rai-predictive-training/examples/train_node_classification.py b/skills/rai-predictive-training/examples/train_node_classification.py new file mode 100644 index 0000000..405ac32 --- /dev/null +++ b/skills/rai-predictive-training/examples/train_node_classification.py @@ -0,0 +1,45 @@ +""" +GNN Node Classification -- Training & Prediction +================================================== +Binary classification training and prediction on user data. + +Assumes data model from `rai-predictive-modeling`: + - gnn_graph: Graph with edges defined + - pt: PropertyTransformer instance (passed as `property_transformer=pt`) + - Train, Val, Test: Relationship objects + - User: source concept (head of Relationship template) +""" +from relationalai.semantics import select +from relationalai.semantics.reasoners.predictive import GNN + +# -- Train GNN --------------------------------------------------------------- +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, + property_transformer=pt, + train=Train, + validation=Val, + task_type="binary_classification", + eval_metric="roc_auc", + has_time_column=True, + device="cuda", + n_epochs=5, +) +gnn.fit() + +# -- Predict & Inspect ------------------------------------------------------- +User.predictions = gnn.predictions(domain=Test) + +select( + User.user_id, + User.predictions.probs, + User.predictions.predicted_labels, +).where(User.predictions).inspect() + +df = select( + User.user_id, + User.predictions.probs, + User.predictions.predicted_labels, +).where(User.predictions).to_df() + +print(f"Predictions: {len(df)} rows, {len(df.dropna())} after dropping NaNs") diff --git a/skills/rai-predictive-training/examples/train_regression.py b/skills/rai-predictive-training/examples/train_regression.py new file mode 100644 index 0000000..2ca1d42 --- /dev/null +++ b/skills/rai-predictive-training/examples/train_regression.py @@ -0,0 +1,46 @@ +""" +GNN Regression -- Training & Prediction +======================================== +Regression training and prediction with temporal features. + +Assumes data model from `rai-predictive-modeling`: + - gnn_graph: Graph with edges defined + - pt: PropertyTransformer instance (passed as `property_transformer=pt`) + - Train, Val, Test: Relationship objects + - Interaction: source concept (head of Relationship template) +""" +from relationalai.semantics import select +from relationalai.semantics.reasoners.predictive import GNN + +# -- Train GNN --------------------------------------------------------------- +# Regression typically needs more epochs than classification. Start with 20-50; +# 5 (the classification default) is a smoke test and usually plateaus at the mean. +gnn = GNN( + exp_database="DB", exp_schema="EXPERIMENTS", + graph=gnn_graph, + property_transformer=pt, + train=Train, + validation=Val, + task_type="regression", + eval_metric="rmse", + has_time_column=True, + device="cuda", + n_epochs=20, + lr=0.005, +) +gnn.fit() + +# -- Predict & Inspect ------------------------------------------------------- +Interaction.predictions = gnn.predictions(domain=Test) + +select( + Interaction.interaction_id, + Interaction.predictions.predicted_value, +).where(Interaction.predictions).inspect() + +df = select( + Interaction.interaction_id, + Interaction.predictions.predicted_value, +).where(Interaction.predictions).to_df() + +print(f"Predictions: {len(df)} rows") diff --git a/skills/rai-predictive-training/references/evaluation-debugging.md b/skills/rai-predictive-training/references/evaluation-debugging.md new file mode 100644 index 0000000..7c42ebd --- /dev/null +++ b/skills/rai-predictive-training/references/evaluation-debugging.md @@ -0,0 +1,138 @@ +# Evaluation & Debugging + +Detailed patterns for inspecting datasets, checking prediction results, and tuning model performance. + +## Inspecting the Dataset + +After `gnn.fit()`, inspect what data the engine received: + +```python +# Visual graph of the dataset schema (requires pydot) +graph_viz = gnn.visualize_dataset() +graph_viz.write_png("dataset_schema.png") + +# With data types shown +graph_viz = gnn.visualize_dataset(show_dtypes=True) +graph_viz.write_png("dataset_schema.png") +``` + +### Metadata and Data Config + +```python +# Export full metadata as a dictionary (useful for debugging feature types) +config = gnn.dataset.metadata_dict + +# Print the data config to console +gnn.dataset.print_data_config() +``` + +### Prediction-step timing expectations + +`gnn.predictions(...)` runs a 4-step sequence (prepare test table -> load model -> submit prediction job -> load results into the logic engine). This sequence carries fixed overhead independent of test-set size, so small test sets still incur meaningful wall-clock time. Subsequent predictions in the same session are faster due to caching; fresh `GNN` instances re-pay the full cost. Don't optimize feature choices based on a first-run prediction time. + +## Accessing Prediction Results + +### Via Source Concept Attribute + +```python +Source.predictions = gnn.predictions(domain=Test) + +select( + Source.id, + Source.predictions.probs, + Source.predictions.predicted_labels, +).where(Source.predictions).inspect() +``` + +### Via prediction_concept + +Access the underlying prediction concept directly -- useful when you need to reference it without binding it to a source concept attribute: + +```python +PredResult = gnn.prediction_concept +select(Source.source_id, PredResult.predicted_labels, PredResult.probs).where( + Source.predictions(DateTime, PredResult) +).inspect() +``` + +### Dictionary-Style Field Indexing + +Useful when the source concept name conflicts with an existing attribute: + +```python +PredRelation = gnn.predictions(domain=Test) +select( + PredRelation["beer"].name, + PredRelation["timestamp"], + PredRelation["prediction"].predicted_labels, + PredRelation["prediction"].probs, +).inspect() +``` + +## Evaluating Results + +### What "good" means + +Ultimately a prediction is good if it supports the business question. That's the ground truth. Business-utility is hard to measure upfront, though, so training-time evaluation relies on intrinsic metrics as proxies. Pick the proxy that most resembles downstream use -- RMSE if the answer is a numeric value, Spearman rho if the answer is a rank-ordering, recall-at-precision if the answer is a gated decision -- and triangulate with the others. + +### Reading the training loss + +`gnn.fit()` prints per-epoch train and validation loss. The trajectory diagnoses training health before any test-set metric: + +| Loss pattern | Likely cause | Action | +|--------------|-------------|--------| +| Both losses still decreasing at the last epoch | Not converged | Train longer (bump `n_epochs`) | +| Train loss decreasing, val loss plateau or rising | Overfitting | Stop earlier, reduce capacity, or add regularization | +| Both losses flat at a high value | Under-capacity, weak features, or LR too small | Check features; try a larger `lr` | +| Long plateau then step-change improvement | Model just learned a structural pattern | Keep training past the plateau -- best epoch may come late | + +### Use multiple metrics + +No single number describes quality. Before trusting predictions, look at: + +- **Task metric vs a predict-baseline.** Predict-mean for regression, majority-class for classification. Compute the lift: `(baseline - model) / baseline`. Near-zero lift means the model hasn't learned anything useful. +- **Correlation** (Pearson + Spearman). Can be high even when absolute error is poor -- the model may have learned ranking but not magnitudes. +- **Prediction-range vs target-range.** `stddev(predicted) / stddev(target)`. A tight prediction band means the model is hedging toward the mean. +- **Error distribution.** Look at the residual histogram, not just aggregates -- a few huge errors can dominate RMSE while most predictions are fine. +- **Sanity-check the prediction DataFrame before using it downstream.** After `.to_df()`, verify the predicted column is free of NaN and stays in the expected range (`predicted_value >= 0` for non-negative targets, `probs` in `[0, 1]` for classification, `scores` non-null for link prediction). Silent NaN/garbage can propagate through a derived property or optimizer constraint and surface as a cryptic solver failure later. + + Pattern (warn-not-block — keeps the pipeline running while flagging suspicious output): + + ```python + df = select(Source.id, Source.predictions.).where(Source.predictions).to_df() + col = df[""] + if col.isna().any() or (col < 0).any(): # adjust bounds per task type + print(f"WARNING: {Source.__name__} predictions contain NaN or out-of-range values") + ``` + + Run a check per GNN in a multi-GNN pipeline; cheap and catches silent failures before they reach derived properties or solver constraints. + +## Tuning Poor Results + +If results are significantly worse than expected, check these in order: + +1. **Inspect the dataset** -- run `gnn.visualize_dataset(show_dtypes=True)` and `gnn.dataset.print_data_config()` to verify feature types and edges match expectations. +2. **Reduce text features** -- too many text fields dilute signal. Start with 3-5 key text fields, add more only if metrics improve. In practice, reducing ~30 text fields to 5 improved AUROC from 57% to 68%. +3. **Adjust hyperparameters** -- see [hyperparameters.md](hyperparameters.md) "Tuning When Results Are Poor" section for symptom-based guidance. + +### Regression-specific sanity checks + +Regression typically needs **more epochs than classification** -- `n_epochs=5` (the quickstart default) is a smoke-test, not a training run. For a first real attempt, bump well above the default and let the loss trajectory (see "Reading the training loss" above) tell you when to stop -- if val-loss is still decreasing at the last epoch, you need more. + +**Under-fitting checklist** (cheapest diagnostic first): + +- **Profile the target distribution before training** -- `SELECT MIN, MAX, AVG, STDDEV FROM ` anchors what RMSE values mean. The same RMSE that's tight on a [0,1]-normalized target is meaningless on an unnormalized one. +- **Val-RMSE vs `stddev(target)`** -- if val-RMSE plateaus at or above the target's stddev, the model has collapsed to the mean. +- **Prediction-band vs target-band** -- if `stddev(predicted)` is noticeably narrower than `stddev(target)`, the model is hedging toward the mean. Under-trained regardless of RMSE. +- **Ranking vs magnitudes** -- if Pearson/Spearman correlation is moderate (>0.3) but RMSE doesn't beat the predict-mean baseline, the model has learned *ranking* but not *magnitudes*. This is under-fitting, not a feature problem -- train longer. +- **R² < 0 early in training is normal** -- it clears as the model learns the target's scale. If it persists past the early training phase, revisit features or learning rate. + +### Suspiciously-good results + +If a first-pass GNN returns R² > 0.95 (regression), AUROC > 0.98, or accuracy > 0.95 (classification), pause and check for leakage before trusting the model: + +- Is the target/label column also listed in the `PropertyTransformer` (category/continuous/...) by accident? +- Is a feature a near-duplicate of the label (a derived property that encodes the target)? +- Does the train/val/test split share entities in ways that let the model memorize — e.g., the same source entity appears in all three splits with the label tied to that entity? Especially common in `repeated_link_prediction`, where the same (source, target) pair can recur across splits. + +Strong features can legitimately produce high scores, but a cheap verification pass prevents shipping a leaky model. diff --git a/skills/rai-predictive-training/references/hyperparameters.md b/skills/rai-predictive-training/references/hyperparameters.md new file mode 100644 index 0000000..3058603 --- /dev/null +++ b/skills/rai-predictive-training/references/hyperparameters.md @@ -0,0 +1,91 @@ +# GNN Hyperparameters + +Hyperparameters are passed as `**train_params` kwargs to the `GNN(...)` constructor. + +## Common Hyperparameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `device` | str | `"cuda"` | Compute device: `"cuda"` (GPU) or `"cpu"` | +| `n_epochs` | int | 5 | Number of training epochs | +| `lr` | float | 0.005 | Learning rate | +| `train_batch_size` | int | 256 | Training batch size | +| `head_layers` | int | 2 | Number of prediction head layers | +| `seed` | int | - | Random seed for reproducibility | +| `channels` | int | 64 | Hidden channel dimension | + +## Link Prediction Hyperparameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `num_negative` | int | 20 | Number of negative samples per positive | +| `label_smoothing` | bool | True | Apply label smoothing during training | + +## Advanced Hyperparameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `temporal_strategy` | str | - | Temporal modeling strategy (e.g. `"last"`) | +| `text_embedder` | str | - | Text embedding model (e.g. `"model2vec-potion-base-4M"`) | +| `max_iters` | int | - | Maximum training iterations | + +## GNN Constructor Operational Flags + +These are named parameters on `GNN(...)`, not train_params: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `test_batch_size` | int | None | Batch size for prediction/inference | +| `stream_logs` | bool | True | Stream training logs to console | +| `use_current_time` | bool | True | Use current time for temporal models | + +## Example Configurations + +### Node Classification (small dataset) +```python +train_params = {"device": "cpu", "n_epochs": 10, "seed": 42} +``` + +### Node Classification (large dataset) +```python +train_params = {"device": "cuda", "n_epochs": 5, "lr": 0.005, "train_batch_size": 256} +``` + +### Link Prediction +```python +train_params = { + "device": "cuda", + "n_epochs": 5, + "train_batch_size": 256, + "lr": 0.005, + "head_layers": 2, + "num_negative": 20, + "label_smoothing": True, +} +``` + +### Regression with Temporal Data +```python +train_params = { + "device": "cuda", + "n_epochs": 5, + "channels": 64, + "head_layers": 2, + "temporal_strategy": "last", +} +``` + +## Tuning When Results Are Poor + +Two heuristics before the symptom table: + +- **`lr` is usually the first knob to sweep.** The default is a starting point, not a recommendation. If training isn't producing learning (flat losses, no convergence), try `lr` above and below the default before concluding features are the problem. +- **Message-passing depth vs graph diameter.** A GNN propagates signal one hop per layer (or one level per neighbor-sampling step). If the source concept sits far from the concepts carrying predictive signal in the schema, the model's depth must reach them -- otherwise distant nodes never contribute. The depth parameter is passed through `train_params` to the trainer; inspect the trainer's accepted kwargs (e.g. on `gnn.trainer` after construction) to find the exact name. + +| Symptom | Likely cause | Action | +|---------|-------------|--------| +| Validation metric still improving at last epoch | Not enough training | Increase `n_epochs` | +| Training loss oscillates or diverges | Learning rate too high | Lower `lr` | +| Good training metric, poor validation metric | Overfitting | Reduce `n_epochs`, reduce text features, or increase `train_batch_size` | +| Very slow convergence on large dataset | Batch too small or lr too high | Increase `train_batch_size`, decrease `lr` | +| Poor results despite hyperparameter sweeps | Signal can't reach the source concept | Check the graph depth matches the schema's diameter; otherwise reduce noisy features (drop PKs/FKs, trim text fields) | diff --git a/skills/rai-predictive-training/references/prediction-attributes.md b/skills/rai-predictive-training/references/prediction-attributes.md new file mode 100644 index 0000000..a32a305 --- /dev/null +++ b/skills/rai-predictive-training/references/prediction-attributes.md @@ -0,0 +1,67 @@ +# Prediction Attributes by Task Type + +After calling `gnn.predictions(domain=Test)`, the prediction results are attached to the source concept (head of the Relationship) and accessed via `select(...)`. + +## Classification (binary, multiclass, multilabel) + +| Attribute | Type | Description | +|-----------|------|-------------| +| `Source.predictions.probs` | float/array | Probability distribution over classes | +| `Source.predictions.predicted_labels` | int/str | Predicted class label (argmax of probs) | + +```python +Source.predictions = gnn.predictions(domain=Test) +select( + Source.id, + Source.predictions.probs, + Source.predictions.predicted_labels, +).where(Source.predictions).inspect() +``` + +## Regression + +| Attribute | Type | Description | +|-----------|------|-------------| +| `Source.predictions.predicted_value` | float | Predicted continuous value | + +```python +Source.predictions = gnn.predictions(domain=Test) +select( + Source.id, + Source.predictions.predicted_value, +).where(Source.predictions).inspect() +``` + +## Link Prediction (link_prediction, repeated_link_prediction) + +| Attribute | Type | Description | +|-----------|------|-------------| +| `Source.predictions.rank` | int | Ranking position (1, 2, 3, ...) | +| `Source.predictions.scores` | float | Relevance/similarity score | +| `Source.predictions.predicted_` | reference | Predicted target concept instance | + +The `predicted_` attribute name is derived from the target concept in the Relationship template. For example, if the Relationship tail is `Item`, the attribute is `predicted_item`. + +```python +Source.predictions = gnn.predictions(domain=Test) +select( + Source.source_id, + Target.target_id, + Source.predictions.rank, + Source.predictions.scores, +).where( + Source.predictions.predicted_target == Target, +).inspect() +``` + +## Using `.to_df()` Instead of `.inspect()` + +Replace `.inspect()` with `.to_df()` to get a pandas DataFrame: + +```python +df = select( + Source.id, + Source.predictions.probs, + Source.predictions.predicted_labels, +).where(Source.predictions).to_df() +``` diff --git a/skills/rai-predictive-training/references/task-types-and-metrics.md b/skills/rai-predictive-training/references/task-types-and-metrics.md new file mode 100644 index 0000000..ab3ed05 --- /dev/null +++ b/skills/rai-predictive-training/references/task-types-and-metrics.md @@ -0,0 +1,68 @@ +# Task Types and Evaluation Metrics + +Valid `(task_type, eval_metric)` combinations for the GNN constructor. + +## Binary Classification + +| task_type | eval_metric | +|-----------|-------------| +| `"binary_classification"` | `"accuracy"` | +| `"binary_classification"` | `"f1"` | +| `"binary_classification"` | `"roc_auc"` | +| `"binary_classification"` | `"average_precision"` | + +## Multiclass Classification + +| task_type | eval_metric | +|-----------|-------------| +| `"multiclass_classification"` | `"accuracy"` | +| `"multiclass_classification"` | `"macro_f1"` | +| `"multiclass_classification"` | `"micro_f1"` | + +## Multilabel Classification + +| task_type | eval_metric | +|-----------|-------------| +| `"multilabel_classification"` | `"multilabel_auprc_micro"` | +| `"multilabel_classification"` | `"multilabel_auroc_micro"` | +| `"multilabel_classification"` | `"multilabel_precision_micro"` | +| `"multilabel_classification"` | `"multilabel_auprc_macro"` | +| `"multilabel_classification"` | `"multilabel_auroc_macro"` | +| `"multilabel_classification"` | `"multilabel_precision_macro"` | + +## Regression + +| task_type | eval_metric | +|-----------|-------------| +| `"regression"` | `"r2"` | +| `"regression"` | `"mae"` | +| `"regression"` | `"rmse"` | + +## Link Prediction + +| task_type | eval_metric | +|-----------|-------------| +| `"link_prediction"` | `"link_prediction_precision@k"` | +| `"link_prediction"` | `"link_prediction_recall@k"` | +| `"link_prediction"` | `"link_prediction_map@k"` | + +## Repeated Link Prediction (temporal) + +| task_type | eval_metric | +|-----------|-------------| +| `"repeated_link_prediction"` | `"link_prediction_precision@k"` | +| `"repeated_link_prediction"` | `"link_prediction_recall@k"` | +| `"repeated_link_prediction"` | `"link_prediction_map@k"` | + +Replace `@k` with the desired top-k value, e.g. `"link_prediction_precision@5"`. + +## Task Type Summary + +| Task Type | has_time_column | Train Relationship template | Test Relationship template | +|-----------|-----------------|---------------------------|--------------------------| +| binary_classification | optional | `f"{Source} has {Any:label}"` | `f"{Source}"` | +| multiclass_classification | optional | `f"{Source} has {Any:label}"` | `f"{Source}"` | +| multilabel_classification | optional | `f"{Source} has {Any:label}"` | `f"{Source}"` | +| regression | optional | `f"{Source} has {Any:value}"` | `f"{Source}"` | +| link_prediction | False | `f"{Source} has {Target}"` | `f"{Source}"` | +| repeated_link_prediction | True | `f"{Source} at {Any:ts} has {Target}"` | `f"{Source} at {Any:ts}"` | From f6af8c7a942621b2ce0e3d382172ab691e53f4d2 Mon Sep 17 00:00:00 2001 From: pkouki Date: Mon, 27 Apr 2026 13:27:27 +0300 Subject: [PATCH 02/27] docs(rai-predictive-training): sync gap fixes from rai-predictive source skill Pulls in content present in the rai-predictive source skill (PyRel repo) that hadn't reached the rai-predictive-training target. SKILL.md: - Predictions section: add Pattern 1 (attribute bind) / Pattern 2 (Python variable bind) explanation with [Duplicate relationship] warning - Predictions section: add gnn.prediction_concept direct-access subsection - Evaluation & Debugging: add gnn.dataset.metadata_dict, basic visualize_dataset() variant, and pydot note - Common Pitfalls: add 7 missing rows (select fragments to train/validation, reassigning Source.predictions, model_name/version_name with spaces, duplicate (model_name, version_name), register_model on load-mode, fit on load-mode, load on fit-mode) references/task-types-and-metrics.md: - Restore note that @k is optional (target had reduced this to a single example, dropping the "omit for no top-k cutoff" guidance) Co-Authored-By: Claude Sonnet 4.6 --- skills/rai-predictive-training/SKILL.md | 39 ++++++++++++++++++- .../references/task-types-and-metrics.md | 2 +- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index f5d4c62..2eb6524 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -185,11 +185,21 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] ## Predictions -After training, generate predictions on the test set: +After training, generate predictions on the test set. There are two valid patterns: +**Pattern 1 — bind to a concept attribute (query with `select()`):** ```python Source.predictions = gnn.predictions(domain=Test) ``` +This registers the predictions as a named relationship on the concept. Each attribute name can only be assigned **once per session** — assigning `Source.predictions` a second time raises `[Duplicate relationship]`. Use a different attribute name if you need multiple prediction sets on the same concept (e.g. `Source.predictions_v2`). + +**Pattern 2 — assign to a plain Python variable (call as many times as needed):** +```python +predictions = gnn.predictions(domain=Test) +# can be called again without error +predictions = gnn.predictions(domain=Test) +``` +Use this pattern when calling `predictions()` multiple times in the same session. ### Classification (binary, multiclass, multilabel) @@ -257,6 +267,17 @@ select( ).inspect() ``` +### Accessing `prediction_concept` Directly + +`gnn.predictions()` returns a Relationship. You can also access the underlying prediction concept directly via `gnn.prediction_concept` — useful when you need to reference it without binding it to a source concept attribute: + +```python +PredResult = gnn.prediction_concept +select(Beer.name, PredResult.predicted_labels, PredResult.probs).where( + Beer.churn(DateTime, PredResult) +).inspect() +``` + For the full prediction attributes reference (per-task attribute types, code shapes), see [references/prediction-attributes.md](references/prediction-attributes.md). The summary table is in Quick Reference above. --- @@ -338,10 +359,17 @@ For a full predict-then-optimize example chaining multiple GNNs into optimizers After `gnn.fit()`, inspect what data the engine received: ```python -# Visual graph of the dataset schema +# Visual graph of the dataset schema (requires pydot) +graph_viz = gnn.visualize_dataset() +graph_viz.write_png("dataset_schema.png") + +# With data types shown graph_viz = gnn.visualize_dataset(show_dtypes=True) graph_viz.write_png("dataset_schema.png") +# Export full metadata as a dictionary (useful for debugging feature types) +config = gnn.dataset.metadata_dict + # Print the data config to console gnn.dataset.print_data_config() ``` @@ -446,14 +474,21 @@ User.predictions = gnn.predictions(domain=Test) |---------|-------|-----| | Missing `has_time_column=True` | Templates with the "at" keyword require the flag so the trainer finds the time column | Set `has_time_column=True` when templates contain "at" | | Using `.predicted_Item` (uppercase) | Target-attribute names are always lowercased from the Target concept name | Use `.predicted_item` | +| Assigning `Source.predictions = gnn.predictions(...)` twice in the same session | Concept attributes are immutable once registered — reassignment raises `[Duplicate relationship]` | Either use a different attribute name (e.g. `Source.predictions_v2`) or assign to a plain Python variable instead: `predictions = gnn.predictions(...)` | | Invalid `task_type`/`eval_metric` combination | Not every metric applies to every task type | Check [references/task-types-and-metrics.md](references/task-types-and-metrics.md) for valid pairs | +| Passing `select(...)` fragments to `train=` or `validation=` | GNN expects Relationship objects | Use `train=Train` with the Relationship object directly | | `register_model()` before `fit()` | Registration requires a trained model | Always call `gnn.fit()` before `gnn.register_model()` | +| `model_name` or `version_name` with spaces or special characters (e.g. `"my model"`, `"v1.0!"`) | Snowflake rejects non-identifier strings as model names, but validation only happens after full training completes | Use plain alphanumeric names with underscores only (e.g. `"my_model"`, `"V1"`) | +| Calling `register_model()` with a `(model_name, version_name)` pair that already exists in the registry | The registry enforces uniqueness — duplicate versions raise `ModelManagerError` | Use a new `version_name` (e.g. `"V2"`) or delete the existing version first | +| Calling `register_model()` on a GNN instance created in load mode | Load-mode GNN instances cannot re-register — only fit-mode instances can register models | Call `register_model()` on the `fit_gnn` instance after `fit()`, not on the `gnn` instance after `load()` | | Omitting `graph`/`property_transformer` when loading | Load reconstructs against the same schema used during training | Provide the same `graph` and `property_transformer` used during training | | Passing training-only params when loading | Load ignores training-time params | Omit `train`, `validation`, and hyperparameters when loading | | Omitting `source_concept` when loading | Required to bind the loaded model to the source concept for prediction | Add `source_concept=` to the load constructor | | Omitting `task_type` when loading | Not persisted in the registry | Add `task_type=""` to the load constructor | | Omitting `target_concept` for link-prediction load | Required to resolve the prediction target concept | Add `target_concept=` for link prediction | | Omitting `has_time_column` when loading a temporal model | Not persisted in the registry | Re-supply `has_time_column=True` at load time | +| Calling `fit()` on a GNN instance created in load mode | Load-mode GNN instances do not support training | Create a separate fit-mode GNN instance (with `train=`, `validation=`) and call `fit()` on that | +| Calling `load()` on a GNN instance created in fit mode (with `train=`, `validation=`) | Fit-mode GNN instances do not support `load()` | Create a separate load-mode GNN instance (with `source_concept=`, `model_name=`, `version_name=`) and call `load()` on that | | `has_time_column=True` fails with "no time column defined in data tables" | The concept carrying `time_col` is an edge, not a node — `time_col` only propagates for node concepts | Use `has_time_column=False` with non-temporal Relationships as workaround | | `has_time_column=True` fails with `ValidationError: Error processing datetime column ''` at scale | Server-side datetime processing rejects the column despite clean data, node-level concept, and correct `datetime`/`time_col` config — second known limitation | Verify the timestamp column type matches the GNN datetime pipeline's expected format (see `rai-predictive-modeling`); fall back to non-temporal Relationships if it persists | | Experiment schema not accessible by the RAI native app | RAI app needs explicit grants to read from the experiment schema | `GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; GRANT ALL ON SCHEMA . TO APPLICATION RELATIONALAI` | diff --git a/skills/rai-predictive-training/references/task-types-and-metrics.md b/skills/rai-predictive-training/references/task-types-and-metrics.md index ab3ed05..b1b0751 100644 --- a/skills/rai-predictive-training/references/task-types-and-metrics.md +++ b/skills/rai-predictive-training/references/task-types-and-metrics.md @@ -54,7 +54,7 @@ Valid `(task_type, eval_metric)` combinations for the GNN constructor. | `"repeated_link_prediction"` | `"link_prediction_recall@k"` | | `"repeated_link_prediction"` | `"link_prediction_map@k"` | -Replace `@k` with the desired top-k value, e.g. `"link_prediction_precision@5"`. +`@k` is optional. Omit it to evaluate without a top-k cutoff (e.g. `"link_prediction_precision"`), or append a value to restrict to the top k results (e.g. `"link_prediction_precision@5"`). ## Task Type Summary From 1aad66fb4c95f9486931f3b6f4383d14e054745d Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 27 Apr 2026 10:09:27 -0700 Subject: [PATCH 03/27] rai-predictive-training: post-#28 cleanups - Replace Beer.churn placeholder with generic Source/predictions in the new prediction_concept Directly snippet (matches Source/Target convention used elsewhere in the skill). - Drop the wrong "Passing select(...) fragments to train=/validation=" Pitfalls row: PyRel's GNN constructor accepts both Relationship and Fragment (verified in source: train: Optional[Relationship | Fragment | Chain] with explicit isinstance check). The "Alternative: select() fragments" section in task-relationships.md correctly advertises this. - Drop duplicate "Source.predictions twice" Pitfalls row -- already covered by the Pattern 1 / Pattern 2 prose in the Predictions section. - Compress the "Accessing prediction_concept Directly" subsection from a full H3 with code block to an inline note (one-line code). - Compress the inspect-the-dataset code block: keep one visualize_dataset variant (with show_dtypes=True) instead of two. SKILL.md: 517 -> 496 lines, back under the 500 cap. --- skills/rai-predictive-training/SKILL.md | 37 ++++++------------------- 1 file changed, 8 insertions(+), 29 deletions(-) diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 2eb6524..4e60df1 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -185,21 +185,17 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] ## Predictions -After training, generate predictions on the test set. There are two valid patterns: +After training, generate predictions on the test set. Two valid binding patterns: -**Pattern 1 — bind to a concept attribute (query with `select()`):** ```python +# Pattern 1 — bind to a concept attribute (queryable via select()): Source.predictions = gnn.predictions(domain=Test) -``` -This registers the predictions as a named relationship on the concept. Each attribute name can only be assigned **once per session** — assigning `Source.predictions` a second time raises `[Duplicate relationship]`. Use a different attribute name if you need multiple prediction sets on the same concept (e.g. `Source.predictions_v2`). -**Pattern 2 — assign to a plain Python variable (call as many times as needed):** -```python -predictions = gnn.predictions(domain=Test) -# can be called again without error +# Pattern 2 — assign to a plain Python variable (re-callable): predictions = gnn.predictions(domain=Test) ``` -Use this pattern when calling `predictions()` multiple times in the same session. + +Each concept-attribute name can be assigned **once per session** — re-binding `Source.predictions` raises `[Duplicate relationship]`. To call `predictions()` multiple times in one session, use Pattern 2 or a fresh attribute name (e.g. `Source.predictions_v2`). ### Classification (binary, multiclass, multilabel) @@ -267,16 +263,7 @@ select( ).inspect() ``` -### Accessing `prediction_concept` Directly - -`gnn.predictions()` returns a Relationship. You can also access the underlying prediction concept directly via `gnn.prediction_concept` — useful when you need to reference it without binding it to a source concept attribute: - -```python -PredResult = gnn.prediction_concept -select(Beer.name, PredResult.predicted_labels, PredResult.probs).where( - Beer.churn(DateTime, PredResult) -).inspect() -``` +**Direct access via `gnn.prediction_concept`.** Exposes the underlying prediction concept without binding to a source attribute — useful when the source concept name conflicts with an existing attribute. Use it as the head in `select(...)`: `select(Source.source_id, gnn.prediction_concept.predicted_labels).where(Source.predictions(DateTime, gnn.prediction_concept)).inspect()`. For the full prediction attributes reference (per-task attribute types, code shapes), see [references/prediction-attributes.md](references/prediction-attributes.md). The summary table is in Quick Reference above. @@ -359,18 +346,12 @@ For a full predict-then-optimize example chaining multiple GNNs into optimizers After `gnn.fit()`, inspect what data the engine received: ```python -# Visual graph of the dataset schema (requires pydot) -graph_viz = gnn.visualize_dataset() -graph_viz.write_png("dataset_schema.png") - -# With data types shown +# Visual schema with data types (requires pydot; omit show_dtypes for the simple variant) graph_viz = gnn.visualize_dataset(show_dtypes=True) graph_viz.write_png("dataset_schema.png") -# Export full metadata as a dictionary (useful for debugging feature types) +# Full metadata dict (debugging feature types) and data-config printout config = gnn.dataset.metadata_dict - -# Print the data config to console gnn.dataset.print_data_config() ``` @@ -474,9 +455,7 @@ User.predictions = gnn.predictions(domain=Test) |---------|-------|-----| | Missing `has_time_column=True` | Templates with the "at" keyword require the flag so the trainer finds the time column | Set `has_time_column=True` when templates contain "at" | | Using `.predicted_Item` (uppercase) | Target-attribute names are always lowercased from the Target concept name | Use `.predicted_item` | -| Assigning `Source.predictions = gnn.predictions(...)` twice in the same session | Concept attributes are immutable once registered — reassignment raises `[Duplicate relationship]` | Either use a different attribute name (e.g. `Source.predictions_v2`) or assign to a plain Python variable instead: `predictions = gnn.predictions(...)` | | Invalid `task_type`/`eval_metric` combination | Not every metric applies to every task type | Check [references/task-types-and-metrics.md](references/task-types-and-metrics.md) for valid pairs | -| Passing `select(...)` fragments to `train=` or `validation=` | GNN expects Relationship objects | Use `train=Train` with the Relationship object directly | | `register_model()` before `fit()` | Registration requires a trained model | Always call `gnn.fit()` before `gnn.register_model()` | | `model_name` or `version_name` with spaces or special characters (e.g. `"my model"`, `"v1.0!"`) | Snowflake rejects non-identifier strings as model names, but validation only happens after full training completes | Use plain alphanumeric names with underscores only (e.g. `"my_model"`, `"V1"`) | | Calling `register_model()` with a `(model_name, version_name)` pair that already exists in the registry | The registry enforces uniqueness — duplicate versions raise `ModelManagerError` | Use a new `version_name` (e.g. `"V2"`) or delete the existing version first | From 6c04f78b8bf4edb0b08bb6fa8a696de7d35e6de5 Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 27 Apr 2026 10:09:51 -0700 Subject: [PATCH 04/27] rai-predictive-training: remove engine-resume guidance The "Resume suspended GPU pools before runs" paragraph and the matching "gnn.fit() hangs with no error" Pitfalls row were added based on a single test report. The predictive team can't reproduce the behavior in their own runs, so the guidance is removed pending root-cause clarity to avoid steering users toward a workaround for an issue that may not be universal. Auto-suspend, multi-engine sizing, and GPU pairing notes (separate concerns) are unchanged. --- skills/rai-predictive-training/SKILL.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 4e60df1..8d16444 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -159,8 +159,6 @@ For link prediction, also consider: `head_layers=2`, `num_negative=20`, `label_s **Auto-suspend during iteration.** Set a low `auto_suspend_mins` on every engine you're using — idle pool cost can dominate total spend on small workloads. Warm pools make sense only for scheduled/production cadence. Specific tier names and per-cloud memory-vs-compute tradeoffs change over time — ask the RelationalAI team for current sizing. Full `raiconfig.yaml` structure (including the `reasoners:` block for all engine types) lives in the RAI configuration/setup skill. -**Resume suspended GPU pools before runs -- required, not optional.** A suspended GPU compute pool does not auto-resume when `gnn.fit()` or `gnn.predictions()` submits; the client has been observed polling indefinitely (over 90 minutes idle) with no error surfaced. Before any training or prediction run -- especially after a period of idle when auto-suspend has fired -- explicitly resume the pool so it is `READY` when the job submits. Run `ALTER COMPUTE POOL RESUME` in Snowflake (the `` is the predictive reasoner entry in `raiconfig.yaml`), or use the RAI CLI's reasoner resume. See the RAI configuration/setup skill for current CLI syntax. - For all hyperparameters and tuning guidance, see [references/hyperparameters.md](references/hyperparameters.md). --- @@ -471,7 +469,6 @@ User.predictions = gnn.predictions(domain=Test) | `has_time_column=True` fails with "no time column defined in data tables" | The concept carrying `time_col` is an edge, not a node — `time_col` only propagates for node concepts | Use `has_time_column=False` with non-temporal Relationships as workaround | | `has_time_column=True` fails with `ValidationError: Error processing datetime column ''` at scale | Server-side datetime processing rejects the column despite clean data, node-level concept, and correct `datetime`/`time_col` config — second known limitation | Verify the timestamp column type matches the GNN datetime pipeline's expected format (see `rai-predictive-modeling`); fall back to non-temporal Relationships if it persists | | Experiment schema not accessible by the RAI native app | RAI app needs explicit grants to read from the experiment schema | `GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; GRANT ALL ON SCHEMA . TO APPLICATION RELATIONALAI` | -| `gnn.fit()` or `gnn.predictions()` hangs with no error output | GPU compute pool is suspended; client polls indefinitely instead of auto-resuming or failing fast (observed >90 min idle) | Run `ALTER COMPUTE POOL RESUME` (or the RAI CLI equivalent) before the run — pool name comes from the predictive reasoner entry in `raiconfig.yaml` | --- From 42f1d7b1392145904bd4a3cf6696e2c89c43be0c Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 27 Apr 2026 10:12:21 -0700 Subject: [PATCH 05/27] predictive skills: intent-routing + auto-discovery user-input boundary - rai-predictive-training/SKILL.md: add a "By user intent" bullet list in the Summary so an agent reading the skill can quickly route to the right sections based on goal (train+val, train+predict+downstream, or train+register+reload). Same Model carries through all three; only the gnn.* call sequence and which sections you exercise differ. - rai-predictive-modeling/SKILL.md: add a "User-input boundary" callout at the top of Define and Populate Concepts -- the user-input boundary is the 3 prompts in auto-discovery.md (source FQNs, task FQNs, experiment artifact location); auto-derive everything else from INFORMATION_SCHEMA / DESCRIBE TABLE. - rai-predictive-modeling/references/auto-discovery.md: rename the "What to Auto-Discover" section to "What to Auto-Discover (and what NOT to ask)" with explicit "Do not ask the user" framing -- column names, PKs, FKs, label/target columns, timestamp columns, task type, feature types are all auto-derivable; asking the user creates friction (often they don't know without checking the schema). Both SKILL.md files stay under the 500-line cap (modeling 307, training 497). --- skills/rai-predictive-modeling/SKILL.md | 2 ++ skills/rai-predictive-modeling/references/auto-discovery.md | 4 ++-- skills/rai-predictive-training/SKILL.md | 4 ++++ 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index b6bbcb2..8806835 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -77,6 +77,8 @@ Additional type imports as needed: `Date`, `DateTime`, `Float`. ## Define and Populate Concepts +> **User-input boundary:** the only things you need from the user are the 3 inputs in [`references/auto-discovery.md`](references/auto-discovery.md) -- source table FQNs, task table FQNs, and the experiment-artifact location. Auto-derive PKs, FKs, columns, types, edges, task type, and timestamp candidates from `INFORMATION_SCHEMA` / `DESCRIBE TABLE`. Don't ask the user for column-level details. + Three concept categories show up in a GNN pipeline, distinguished by whether they declare a primary key and how they participate in the graph: | Category | `identify_by`? | Role | Constraints | diff --git a/skills/rai-predictive-modeling/references/auto-discovery.md b/skills/rai-predictive-modeling/references/auto-discovery.md index 6ff8931..5417f07 100644 --- a/skills/rai-predictive-modeling/references/auto-discovery.md +++ b/skills/rai-predictive-modeling/references/auto-discovery.md @@ -39,9 +39,9 @@ What Snowflake database and schema should we use for **experiment artifacts**? (e.g., `MY_DB.EXPERIMENTS`) ``` -## What to Auto-Discover +## What to Auto-Discover (and what NOT to ask) -Once the user provides the table names, the agent must automatically discover the following by querying Snowflake (`DESCRIBE TABLE` or `INFORMATION_SCHEMA`). Use the snowflake-schema tool to get the schema of each table. +The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experiment location). **Do not ask the user** for column names, PKs, FKs, label/target columns, timestamp columns, task type, or feature types — those are friction the user often can't answer without checking the schema themselves. Query Snowflake (`DESCRIBE TABLE` or `INFORMATION_SCHEMA.COLUMNS`; use the snowflake-schema tool) and infer: 1. **Column names and types** for all source and task tables 2. **Primary keys** -- identify PK columns diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 8d16444..24617e7 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -24,6 +24,10 @@ description: Configure and train GNN models, generate predictions, evaluate resu **Overview:** 4 steps: configure GNN -> train -> predict/evaluate -> optional: register/load. +**By user intent — sections to focus on:** +- Train + read validation metric → Quick Reference + GNN Constructor + `gnn.fit()` +- + predict + downstream rule / optimization → also Predictions + Using Predictions Downstream +- + register + reload across sessions → also Model Management ## Quick Reference From 29329da14a0c155042739f89b1b3315345d4f7a2 Mon Sep 17 00:00:00 2001 From: pkouki Date: Tue, 28 Apr 2026 16:10:23 +0300 Subject: [PATCH 06/27] docs(rai-predictive-modeling): add link prediction task table format validation Adds a warning section before the link prediction relationship patterns explaining that task tables must be flat (one row per src/tgt pair). Includes detection guidance for VARIANT/array columns and a Snowflake LATERAL FLATTEN fix for unnesting arrays. Co-Authored-By: Claude Sonnet 4.6 --- .../references/task-relationships.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index c871e0d..ec05f22 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -72,6 +72,25 @@ Val = Relationship(f"{Interaction} has {Any:value}") Test = Relationship(f"{Interaction}") ``` +## Link Prediction — Task Table Format Requirements + +The GNN framework requires link prediction task tables in **flat format**: one row per `(src, timestamp, tgt)` pair. + +| Split | Required columns | Notes | +|-------|-----------------|-------| +| Train | `src_id`, `timestamp`, `tgt_id` | One target per row | +| Val | `src_id`, `timestamp`, `tgt_id` | One target per row | +| Test | `src_id`, `timestamp` | No target column | + +> ⚠️ **If your target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will silently fail or error. Before proceeding, choose one of: +> 1. Use a flat version of the table if one exists. +> 2. Create a Snowflake view that unnests the array into individual rows: +> ```sql +> SELECT src_id, timestamp, f.value::INT AS tgt_id +> FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f +> ``` +> 3. Proceed to observe the error. + ## Link Prediction (with time / repeated_link_prediction) ```python From e77198c87b1c7fba162ef2c3cd485790d6bb3f6b Mon Sep 17 00:00:00 2001 From: pkouki Date: Tue, 28 Apr 2026 19:05:12 +0300 Subject: [PATCH 07/27] docs(rai-predictive-modeling): add VARIANT check and approval gate for link prediction task tables Co-Authored-By: Claude Sonnet 4.6 --- skills/rai-predictive-modeling/SKILL.md | 1 + .../references/task-relationships.md | 15 +++++++-------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 8806835..5c30de9 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -285,6 +285,7 @@ For the full feature type reference including drop patterns, see [references/pro | Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | | `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | | Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | +| Link prediction target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair | Run `DESCRIBE TABLE` on each split table before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN view and wait for user approval before creating anything in Snowflake | --- diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index ec05f22..5b86cad 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -82,14 +82,13 @@ The GNN framework requires link prediction task tables in **flat format**: one r | Val | `src_id`, `timestamp`, `tgt_id` | One target per row | | Test | `src_id`, `timestamp` | No target column | -> ⚠️ **If your target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will silently fail or error. Before proceeding, choose one of: -> 1. Use a flat version of the table if one exists. -> 2. Create a Snowflake view that unnests the array into individual rows: -> ```sql -> SELECT src_id, timestamp, f.value::INT AS tgt_id -> FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f -> ``` -> 3. Proceed to observe the error. +> **Before writing link prediction task table definitions, run `DESCRIBE TABLE` on each split table and check column types.** +> +> ⚠️ **If the target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN view to the user and wait for explicit approval before creating anything in Snowflake: +> ```sql +> SELECT src_id, timestamp, f.value::INT AS tgt_id +> FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f +> ``` ## Link Prediction (with time / repeated_link_prediction) From a5999114b58f2bf16a4bd48c681466e8bb13a7ae Mon Sep 17 00:00:00 2001 From: pkouki Date: Tue, 28 Apr 2026 19:14:14 +0300 Subject: [PATCH 08/27] docs(rai-predictive-modeling): extend VARIANT check to all three split tables including test Co-Authored-By: Claude Sonnet 4.6 --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../rai-predictive-modeling/references/task-relationships.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 5c30de9..c0e0f41 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -285,7 +285,7 @@ For the full feature type reference including drop patterns, see [references/pro | Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | | `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | | Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | -| Link prediction target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair | Run `DESCRIBE TABLE` on each split table before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN view and wait for user approval before creating anything in Snowflake | +| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN view and wait for user approval before creating anything in Snowflake | --- diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index 5b86cad..d0cd7ce 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -82,9 +82,9 @@ The GNN framework requires link prediction task tables in **flat format**: one r | Val | `src_id`, `timestamp`, `tgt_id` | One target per row | | Test | `src_id`, `timestamp` | No target column | -> **Before writing link prediction task table definitions, run `DESCRIBE TABLE` on each split table and check column types.** +> **Before writing link prediction task table definitions, run `DESCRIBE TABLE` on all three split tables (train, val, and test) and check column types. This includes the test table — some users provide labels there for evaluation purposes.** > -> ⚠️ **If the target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN view to the user and wait for explicit approval before creating anything in Snowflake: +> ⚠️ **If any join key or target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN view to the user and wait for explicit approval before creating anything in Snowflake: > ```sql > SELECT src_id, timestamp, f.value::INT AS tgt_id > FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f From 5a29775f733ff36b8224614ed2c602972a44eb2c Mon Sep 17 00:00:00 2001 From: pkouki Date: Tue, 28 Apr 2026 19:36:18 +0300 Subject: [PATCH 09/27] docs(rai-predictive-modeling): use CTAS instead of VIEW for VARIANT flattening Snowflake does not support change tracking on LATERAL views, so the recommended fix is CREATE TABLE AS SELECT + ALTER TABLE SET CHANGE_TRACKING. Co-Authored-By: Claude Sonnet 4.6 --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../references/task-relationships.md | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index c0e0f41..3b3b194 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -285,7 +285,7 @@ For the full feature type reference including drop patterns, see [references/pro | Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | | `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | | Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | -| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN view and wait for user approval before creating anything in Snowflake | +| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN table (not a view — Snowflake does not support change tracking on LATERAL views) and wait for user approval before creating anything in Snowflake | --- diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index d0cd7ce..04e4e78 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -86,8 +86,12 @@ The GNN framework requires link prediction task tables in **flat format**: one r > > ⚠️ **If any join key or target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN view to the user and wait for explicit approval before creating anything in Snowflake: > ```sql +> -- Use CREATE TABLE (not VIEW) — Snowflake does not support change tracking on LATERAL views +> CREATE OR REPLACE TABLE my_db.my_schema.my_task_table_flat AS > SELECT src_id, timestamp, f.value::INT AS tgt_id -> FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f +> FROM my_task_table, LATERAL FLATTEN(input => tgt_array) f; +> +> ALTER TABLE my_db.my_schema.my_task_table_flat SET CHANGE_TRACKING = TRUE; > ``` ## Link Prediction (with time / repeated_link_prediction) From c7a08e330cdefbc5aea950cfc75ef49ddf7c8812 Mon Sep 17 00:00:00 2001 From: pkouki Date: Wed, 29 Apr 2026 10:29:01 +0300 Subject: [PATCH 10/27] docs(rai-predictive-modeling): distinguish VARIANT on joined vs non-joined columns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VARIANT on a column not used in any relationship join is a non-blocking warning — no flattening needed. Only columns used in joins require a LATERAL FLATTEN table fix. Co-Authored-By: Claude Sonnet 4.6 --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../rai-predictive-modeling/references/task-relationships.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 3b3b194..7851716 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -285,7 +285,7 @@ For the full feature type reference including drop patterns, see [references/pro | Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | | `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | | Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | -| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found, propose a LATERAL FLATTEN table (not a view — Snowflake does not support change tracking on LATERAL views) and wait for user approval before creating anything in Snowflake | +| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found on a column used in a join, propose a LATERAL FLATTEN table (not a view — Snowflake does not support change tracking on LATERAL views) and wait for user approval. If `VARIANT` is on a column not used in any join, it is a non-blocking warning — no action needed, keep the original table. | --- diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index 04e4e78..44deff1 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -84,7 +84,7 @@ The GNN framework requires link prediction task tables in **flat format**: one r > **Before writing link prediction task table definitions, run `DESCRIBE TABLE` on all three split tables (train, val, and test) and check column types. This includes the test table — some users provide labels there for evaluation purposes.** > -> ⚠️ **If any join key or target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN view to the user and wait for explicit approval before creating anything in Snowflake: +> ⚠️ **If any join key or target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN table to the user and wait for explicit approval before creating anything in Snowflake: > ```sql > -- Use CREATE TABLE (not VIEW) — Snowflake does not support change tracking on LATERAL views > CREATE OR REPLACE TABLE my_db.my_schema.my_task_table_flat AS @@ -93,6 +93,8 @@ The GNN framework requires link prediction task tables in **flat format**: one r > > ALTER TABLE my_db.my_schema.my_task_table_flat SET CHANGE_TRACKING = TRUE; > ``` +> +> **If the `VARIANT` column is not used in any relationship join** (e.g. an extra column in the test table), it produces a non-blocking warning — no action needed, keep the original table as-is. ## Link Prediction (with time / repeated_link_prediction) From ec205987c94dba07ad59071b3eea975dcd529896 Mon Sep 17 00:00:00 2001 From: cafzal Date: Wed, 29 Apr 2026 10:08:22 -0700 Subject: [PATCH 11/27] docs(rai-predictive-modeling): tighten link-prediction VARIANT guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review-driven fixes applied on top of pkouki's content: - Use `tgt_id` consistently in the format-requirements section (table, warning prose, and SQL recipe) — was previously `tgt_id` in the table but `target_id` in the warning prose. - Replace the ⚠️ emoji in the warning blockquote with bold `**Warning:**` to match the no-emoji convention used elsewhere in this skill. - Compress the SKILL.md Common Pitfalls row to a 1-line trigger that points to the reference section for the joined-vs-non-joined branch and the LATERAL FLATTEN recipe — keeps a single source of truth. - Append `(VARIANT check)` to the reference section heading so an agent grepping for "VARIANT" hits the section title directly. No semantic changes; pure cleanup of an already-validated finding. --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../rai-predictive-modeling/references/task-relationships.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 7851716..119d084 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -285,7 +285,7 @@ For the full feature type reference including drop patterns, see [references/pro | Column name has spaces or special characters | Python identifier rules prevent `Concept.weight(kg)` | Use `getattr(People, "weight(kg)")` to reference the field | | `identify_by` key or property access doesn't match Snowflake column name | Typo or wrong column — matching is case-insensitive, but the column name must exist | Check `INFORMATION_SCHEMA.COLUMNS` / run `DESCRIBE TABLE` for the exact spelling | | Train/Val/Test Relationships have different schemas | Test omits the label but also changes concept or timestamp structure | Train, Val, and Test must share the same concept and timestamp structure — only the label/target is omitted in Test | -| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair — applies to train, val, and test (some users include labels in test for evaluation) | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; if `VARIANT` is found on a column used in a join, propose a LATERAL FLATTEN table (not a view — Snowflake does not support change tracking on LATERAL views) and wait for user approval. If `VARIANT` is on a column not used in any join, it is a non-blocking warning — no action needed, keep the original table. | +| Link prediction join key or target column is `VARIANT` in task table | Task table stores target IDs as an array instead of one row per pair | Run `DESCRIBE TABLE` on all three split tables before writing task relationships; see `references/task-relationships.md` § Link Prediction — Task Table Format Requirements (VARIANT check) for the joined-vs-non-joined branch and the `LATERAL FLATTEN` recipe | --- diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/skills/rai-predictive-modeling/references/task-relationships.md index 44deff1..b681c35 100644 --- a/skills/rai-predictive-modeling/references/task-relationships.md +++ b/skills/rai-predictive-modeling/references/task-relationships.md @@ -72,7 +72,7 @@ Val = Relationship(f"{Interaction} has {Any:value}") Test = Relationship(f"{Interaction}") ``` -## Link Prediction — Task Table Format Requirements +## Link Prediction — Task Table Format Requirements (VARIANT check) The GNN framework requires link prediction task tables in **flat format**: one row per `(src, timestamp, tgt)` pair. @@ -84,7 +84,7 @@ The GNN framework requires link prediction task tables in **flat format**: one r > **Before writing link prediction task table definitions, run `DESCRIBE TABLE` on all three split tables (train, val, and test) and check column types. This includes the test table — some users provide labels there for evaluation purposes.** > -> ⚠️ **If any join key or target column is `VARIANT` type** (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.target_id == train_table_concept.target_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN table to the user and wait for explicit approval before creating anything in Snowflake: +> **Warning:** if any join key or target column is `VARIANT` type (e.g. a JSON array of target IDs per row), the table is in the wrong format and the join `Target.tgt_id == train_table_concept.tgt_id` will fail with `[UnresolvedType]`. Do not flatten automatically — propose creating a LATERAL FLATTEN table to the user and wait for explicit approval before creating anything in Snowflake: > ```sql > -- Use CREATE TABLE (not VIEW) — Snowflake does not support change tracking on LATERAL views > CREATE OR REPLACE TABLE my_db.my_schema.my_task_table_flat AS From 9669763a290930e99d6243ea4f52a3471bcc8193 Mon Sep 17 00:00:00 2001 From: cafzal Date: Tue, 28 Apr 2026 11:11:46 -0700 Subject: [PATCH 12/27] predictive skills: add gap-fixes from full-PaySim run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three issues hit during the v1/fraud-detection full-PaySim run that the skills didn't anticipate. Adding them so the next adapter doesn't rediscover them: rai-predictive-modeling: - Promote the TIMESTAMP_NTZ -> VARCHAR ISO-8601 guidance out of body prose into a Common Pitfalls row with the exact ALTER TABLE / TO_CHAR fix. - Add a new pitfall row (and § Populate from Snowflake callout) for pandas timestamp[ns] parquet -> Snowflake TIMESTAMP_NTZ silently multiplying values by 1000 on COPY INTO. rai-predictive-training: - § GNN Constructor: pre-resume SYSTEM_COMPUTE_POOL_GPU before long fit() runs (otherwise the SDK hangs forever with no progress signal — we lost 91 idle minutes to a suspended pool). - § Known Limitations #2: spell out the four mechanical steps of the has_time_column=False fallback (skill said "fall back" but adapters still had to figure out which Relationship template + PropertyTransformer + temporal_strategy edits go together). - Same section: add the engine-side cache-invalidation footgun. After an ALTER TABLE column-type change, the engine's compiled-relation artifact retains the old type even after stream delete + recreate. The real error lives in problems.json via GET_TRANSACTION_ARTIFACTS, not in the "transaction was aborted (runtime error)" client wrapper. Workaround: rename Model(...) to force a fresh RAI relation namespace. --- skills/rai-predictive-modeling/SKILL.md | 6 +++++- skills/rai-predictive-training/SKILL.md | 10 ++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 119d084..1533903 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -129,7 +129,11 @@ The GNN pipeline expects pre-existing train/val/test split tables in Snowflake. `PropertyTransformer` and the task-table pattern also work with concepts populated from local data via `model.data(df)` -- not just `Table(...).to_schema()`. Useful when some concept data lives in local CSVs (e.g. optimizer parameters) while the graph comes from Snowflake. -**Timestamp column type matters for the GNN datetime pipeline.** Columns intended for `time_col` / `datetime` features need a type the trainer accepts; native Snowflake `TIMESTAMP_NTZ` has been observed to be silently incompatible (loads cleanly, but the trainer doesn't pick the column up as temporal). VARCHAR ISO-8601 is the safer default for time-bearing columns, though large-scale loads can still trip a server-side `ValidationError` (see `rai-predictive-training` § Known Limitations). Confirm the trainer's currently-accepted timestamp formats with the RelationalAI team if you're hitting datetime errors at scale. +**Timestamp column types for the GNN datetime pipeline.** Columns intended for `time_col` / `datetime` features should match a format the trainer accepts; if you're not sure what's currently supported, ask the RelationalAI team. + +> **Do schema changes (any column type, not just timestamps) before the first `Model(...)` bind**, not after — `ALTER`-ing a column type on an already-bound table can leave a stale compiled-relation signature on the engine that survives stream delete + recreate. See `rai-predictive-training` § Known Limitations for the symptom, the diagnostic path, and the workaround. + +**Avoid `timestamp[ns]` parquet payloads when bulk-loading via `COPY INTO TIMESTAMP_NTZ`.** Snowflake interprets the integer payload as `timestamp[us]`, multiplying every value by 1000 — pandas' default `datetime64[ns]` -> parquet round-trip silently lands timestamps tens of millions of years in the future. Two safe options: write the timestamp column as ISO-8601 strings into parquet, or load the underlying integer time index (e.g. an hour offset) and rebuild server-side via `DATEADD(HOUR, , ''::TIMESTAMP_NTZ)` after `COPY INTO`. --- diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 24617e7..2e6876e 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -163,6 +163,13 @@ For link prediction, also consider: `head_layers=2`, `num_negative=20`, `label_s **Auto-suspend during iteration.** Set a low `auto_suspend_mins` on every engine you're using — idle pool cost can dominate total spend on small workloads. Warm pools make sense only for scheduled/production cadence. Specific tier names and per-cloud memory-vs-compute tradeoffs change over time — ask the RelationalAI team for current sizing. Full `raiconfig.yaml` structure (including the `reasoners:` block for all engine types) lives in the RAI configuration/setup skill. +**Pre-flight check the GPU compute pool before long `fit()` runs.** The predictive reasoner provisions onto a Snowpark Container Services GPU compute pool. Confirming the pool is active up front is cheap hygiene; if it's auto-suspended, jobs may queue without surfacing a clear progress signal. Substitute the pool name your account uses — `SYSTEM_COMPUTE_POOL_GPU` is the Snowflake-provided default; some accounts have a custom pool referenced from `raiconfig.yaml`: + +```sql +SHOW COMPUTE POOLS; -- list all + check state +ALTER COMPUTE POOL RESUME; -- if state=SUSPENDED +``` + For all hyperparameters and tuning guidance, see [references/hyperparameters.md](references/hyperparameters.md). --- @@ -183,6 +190,8 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] 1. **Edge-intermediary `time_col`.** When the concept carrying `time_col` is used only as an edge intermediary (no `identify_by`), validation fails with "no time column defined in data tables". `time_col` only propagates for node concepts. 2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. +**Engine-side compiled-relation cache footgun (not datetime-specific — surfaces after any column-type change on a bound table):** the engine caches the compiled relation type per RAI relation name and doesn't invalidate it on `ALTER TABLE`, even after stream delete + recreate. Symptom: `Encountered reference to a base relation with a mismatched signature` — but the client only shows the opaque `Failed to pull data into index: transaction was aborted (runtime error)` wrapper. Pull the real error via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` (presigned URL) and look at the `report` field. Workaround: rename `Model(...)` to force a fresh RAI relation namespace (downstream queries that depend on the old name need updating). Better: do schema changes before the first bind — see `rai-predictive-modeling` § Populate from Snowflake. + --- ## Predictions @@ -472,6 +481,7 @@ User.predictions = gnn.predictions(domain=Test) | Calling `load()` on a GNN instance created in fit mode (with `train=`, `validation=`) | Fit-mode GNN instances do not support `load()` | Create a separate load-mode GNN instance (with `source_concept=`, `model_name=`, `version_name=`) and call `load()` on that | | `has_time_column=True` fails with "no time column defined in data tables" | The concept carrying `time_col` is an edge, not a node — `time_col` only propagates for node concepts | Use `has_time_column=False` with non-temporal Relationships as workaround | | `has_time_column=True` fails with `ValidationError: Error processing datetime column ''` at scale | Server-side datetime processing rejects the column despite clean data, node-level concept, and correct `datetime`/`time_col` config — second known limitation | Verify the timestamp column type matches the GNN datetime pipeline's expected format (see `rai-predictive-modeling`); fall back to non-temporal Relationships if it persists | +| `SnowflakeTableObjectsException: Failed to pull data into index: transaction was aborted (runtime error)` | Opaque client wrapper that hides the actual server-side error (commonly a stale compiled-relation signature after schema drift, but other causes possible) | Pull `problems.json` via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` (presigned URL) and read the `report` field for the real error. For the schema-drift case specifically, see § Known Limitations | | Experiment schema not accessible by the RAI native app | RAI app needs explicit grants to read from the experiment schema | `GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; GRANT ALL ON SCHEMA . TO APPLICATION RELATIONALAI` | --- From cdc1d55dd68e656344190b0ef396926865324d2e Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 30 Apr 2026 19:28:41 -0700 Subject: [PATCH 13/27] predictive skills: add config learnings from PR #49 template runs Adds runbook entries derived from a real end-to-end run of the subscriber_retention and demand_forecasting templates: experiment- schema setup DDL, worker-not-ready recovery, train-job-matches-stale- experiment behavior, has_time_column workaround, JOBS history rollover, and de-recommendation of CREATE_GNN_SERVICE for QUEUED job recovery. --- plugins/rai/skills/rai-health/SKILL.md | 28 ++++++++++ plugins/rai/skills/rai-setup/SKILL.md | 2 + skills/rai-predictive-modeling/SKILL.md | 33 ++++++++++++ skills/rai-predictive-training/SKILL.md | 72 ++++++++++++++++++++++++- 4 files changed, 134 insertions(+), 1 deletion(-) diff --git a/plugins/rai/skills/rai-health/SKILL.md b/plugins/rai/skills/rai-health/SKILL.md index 3328fa8..e0ca264 100644 --- a/plugins/rai/skills/rai-health/SKILL.md +++ b/plugins/rai/skills/rai-health/SKILL.md @@ -315,6 +315,34 @@ for the full step-by-step recovery checklist, schema reference, and official doc --- +## Predictive train jobs stuck QUEUED + +A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The pod-level status does not reflect in-pod worker state — the worker can be desynced and still report ready. + +**Recovery:** suspend then resume the predictive reasoner to force a worker recycle. Do **not** call `CREATE_GNN_SERVICE()`. + +```sql +-- 1. Confirm a stuck train job +SELECT ID, STATE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN +FROM RELATIONALAI.API.JOBS +WHERE STATE IN ('QUEUED','RUNNING') + AND PAYLOAD LIKE '%"job_type": "train"%' +ORDER BY CREATED_ON ASC; + +-- 2. Recycle the worker +CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', ''); +CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); + +-- 3. Wait for STATUS=READY, kill the stuck client, then resubmit with a bumped Model name +CALL RELATIONALAI.API.GET_REASONER('predictive', ''); +``` + +> **Do not use `CALL RELATIONALAI.EXPERIMENTAL.CREATE_GNN_SERVICE();` to recover stuck predictive train jobs.** It targets a legacy GNN service path that is orthogonal to the predictive reasoner — in PyRel 1.0.x the predictive reasoner serves train jobs in-pod and does not depend on it. The call typically fails with an image-mismatch error (`Invalid image specified in service spec: image 'rai-gnn-app:' does not exist in current application version`); that error does **not** mean GNN training is broken, just that this code path is retired. The right escalation is `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` on the predictive reasoner itself. + +See `rai-predictive-training` § Worker not ready to accept jobs for the matching client-side symptom and post-recycle resubmit pattern, and § Train-job ID disappears from `RELATIONALAI.API.JOBS` for stalled-job forensics. + +--- + ## Step 6 — CDC Engine Management > **CDC engine ≠ reasoner engine.** The CDC pipeline runs on a dedicated managed engine diff --git a/plugins/rai/skills/rai-setup/SKILL.md b/plugins/rai/skills/rai-setup/SKILL.md index d7f6ec7..896eb62 100644 --- a/plugins/rai/skills/rai-setup/SKILL.md +++ b/plugins/rai/skills/rai-setup/SKILL.md @@ -37,6 +37,8 @@ The RelationalAI Native App for Snowflake must be installed in your account by a The `rai_developer` role is the standard role for running PyRel programs. Custom Snowflake roles also work if granted the `rai_user` application role — see [User Access](https://docs.relational.ai/manage/user-access). +**Predictive (GNN) workflows need additional schema setup beyond the base install** — a customer-owned database + schema with `USAGE` and `CREATE EXPERIMENT`/`CREATE MODEL` granted to `APPLICATION RELATIONALAI`. Without it the very first `gnn.fit()` fails. See `rai-predictive-modeling` § Prerequisites for the DDL. The predictive submodule (`relationalai.semantics.reasoners.predictive`) also requires a `relationalai` version that ships it — confirm the minimum with the RelationalAI team before pinning. + Support / docs: support@relational.ai · sales@relational.ai · [docs.relational.ai](https://docs.relational.ai/) --- diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 1533903..4044c99 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -26,6 +26,39 @@ description: Build GNN data models -- concepts, Snowflake data loading, task rel --- +## Prerequisites + +### Experiment schema setup (one-time, ACCOUNTADMIN) + +GNN training writes experiment artifacts to a Snowflake schema. The RELATIONALAI native app must own write access on it. Without this, the very first `gnn.fit()` fails with a message like *"Schema does not exist or the GNN RelationalAI Native App lacks permissions"* (the wording rotates between "Schema does not exist" and "Database does not exist" depending on which grant is missing first — both grants below are required). + +```sql +-- Use a database you own (NOT a Snowflake-shared/marketplace database). +-- Shared DBs reject schema creation: "Creating schema on shared database +-- '' is not allowed." +CREATE DATABASE IF NOT EXISTS ; +CREATE SCHEMA IF NOT EXISTS .EXPERIMENTS; + +GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; +GRANT ALL PRIVILEGES ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; +``` + +Then in the script: + +```python +gnn = GNN( + exp_database="", + exp_schema="EXPERIMENTS", + ... +) +``` + +### `relationalai` package version + +The predictive submodule (`relationalai.semantics.reasoners.predictive`) is not in every published `relationalai` release — `from relationalai.semantics.reasoners.predictive import GNN` will raise `ModuleNotFoundError` on releases that pre-date it. Confirm the current minimum version with the RelationalAI team before pinning in `pyproject.toml`. + +--- + ## Quick Reference ```python diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 2e6876e..973b56f 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -188,10 +188,80 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] `has_time_column=True` has two known failure modes; both share the same workaround (turn temporal off — switch to non-temporal Relationships and `has_time_column=False`): 1. **Edge-intermediary `time_col`.** When the concept carrying `time_col` is used only as an edge intermediary (no `identify_by`), validation fails with "no time column defined in data tables". `time_col` only propagates for node concepts. -2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. +2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Reproduced on a daily date column at ~27K-row scale, so the threshold for "scale" is low. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. Concrete fallback shape: + + ```python + # Before (fails) + pt = PropertyTransformer( + datetime=[Sale.date], + time_col=[Sale.date], # <-- remove + ... + ) + gnn = GNN(has_time_column=True, ..., temporal_strategy="last") # <-- both go + + # After (works) — keep date as a plain datetime feature; do the temporal split + # in pandas before building the task tables. + pt = PropertyTransformer( + datetime=[Sale.date], + # time_col disabled — see Known Limitation + ... + ) + gnn = GNN(has_time_column=False, ...) + # Train relationship loses the date arg: + Train = Relationship(f"{Sale} has {Any:value}") + model.define(Train(Sale, TrainTable.unit_sales)).where(...) + ``` **Engine-side compiled-relation cache footgun (not datetime-specific — surfaces after any column-type change on a bound table):** the engine caches the compiled relation type per RAI relation name and doesn't invalidate it on `ALTER TABLE`, even after stream delete + recreate. Symptom: `Encountered reference to a base relation with a mismatched signature` — but the client only shows the opaque `Failed to pull data into index: transaction was aborted (runtime error)` wrapper. Pull the real error via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` (presigned URL) and look at the `report` field. Workaround: rename `Model(...)` to force a fresh RAI relation namespace (downstream queries that depend on the old name need updating). Better: do schema changes before the first bind — see `rai-predictive-modeling` § Populate from Snowflake. +### Worker not ready to accept jobs + +`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` returns `STATUS='READY'`. The reasoner status reflects pod state, not in-pod worker state — the worker can be desynced and still report READY. The real error surfaces only when the SDK polls and the worker reports back: + +> Request failed for external function CREATE_JOB with remote service error: 400 `{"status":"Not Found","message":"worker is not ready to accept jobs - please retry the job later"}` + +**Recovery:** when train jobs are stuck QUEUED for >5 minutes despite the reasoner being READY, force the worker to recycle by suspending and resuming the predictive reasoner. + +```sql +-- 1. Check non-terminal train jobs +SELECT ID, STATE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN +FROM RELATIONALAI.API.JOBS +WHERE STATE IN ('QUEUED','RUNNING') + AND PAYLOAD LIKE '%"job_type": "train"%' +ORDER BY CREATED_ON ASC; + +-- 2. Force the worker to recycle +CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', ''); +CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); + +-- 3. Wait for STATUS=READY again, then resubmit (kill the stuck client first) +CALL RELATIONALAI.API.GET_REASONER('predictive', ''); +``` + +After resume, jobs that were QUEUED on the old worker error out cleanly with the worker-not-ready message. Submit a fresh run with a bumped `Model("...")` name (see § Stale-experiment matching) to start a new experiment. `CREATE_GNN_SERVICE()` is **not** the right escalation here — the predictive reasoner serves train jobs in-pod in PyRel 1.0.x and does not depend on the legacy GNN service. See `rai-health` § Predictive train jobs stuck QUEUED. + +### Stale-experiment matching: bump `Model("...")` name on retry + +`_wait_obtain_model_run_id` in `relationalai.semantics.reasoners.predictive.estimator` matches a just-submitted training job to the most recent experiment with the same `Model("...")` name + concept name. If a previous run with the same model name was killed mid-flight or failed at the prediction step, a new train submission can match the **stale** `model_run_id` and then hang at "Step 2/4: Preparing model for prediction" because the stale model artifacts don't fit the current dataset. + +**Symptom:** Step 1/3, 2/3, 3/3 (training submit) all succeed in seconds. Step 1/4 (test-table prep) succeeds. Step 2/4 ("Preparing model for prediction…") polls forever (>30 min) with no progress, no JOBS row, no traceback. + +**Workaround:** bump the `Model("...")` name on every re-run after a killed/failed run. + +```python +model = Model("my_template_local_v2") # bump on re-run +``` + +--- + +## Troubleshooting + +### Train-job ID disappears from `RELATIONALAI.API.JOBS` + +`RELATIONALAI.API.JOBS` history rolls — train jobs that were `STATE='QUEUED'` for ~30 min can be canceled or roll out of retention and disappear from the table entirely. Client-side `_wait_obtain_model_run_id` polling does not time-bound itself and will keep going indefinitely against an ID that no longer exists. + +If a `gnn.fit()` client has been polling for >30 min with no progress and `SELECT * FROM RELATIONALAI.API.JOBS WHERE ID = ''` returns no row, the client will not self-terminate — kill it manually. Do **not** rely on `RELATIONALAI.API.JOBS` for forensics on stalled jobs older than the retention window; capture state earlier (or rely on `GET_TRANSACTION_ARTIFACTS` / engine logs) when investigating. + --- ## Predictions From 426c72f85f5ad62f9a713455ab37f35b58480872 Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 30 Apr 2026 20:12:33 -0700 Subject: [PATCH 14/27] predictive skills: tighten config-learnings additions against SDK source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After grounding each addition against the PyRel gnn3 source and the relationalai_gnns runtime, three claims needed correction or sharpening: - L-1 (rai-predictive-modeling Prerequisites): replace the over-broad GRANT ALL PRIVILEGES with the four minimum grants prescribed by relationalai_gnns.core.diagnostics.PermissionDiagnostic. Keep ALL as a working superset for non-least-privilege deployments. Note the error surfaces as PermissionError specifically (not RuntimeError) so it can be caught explicitly. - L-3 (rai-predictive-training): the original "SDK matches train jobs to experiments by Model name" claim was wrong — _wait_obtain_model_run_id reads train_job.model_run_id directly, no name matching. Replaced with the actual mechanism: gnn.fit() is idempotent (estimator.py:483-490); reusing a GNN instance after a partial failure silently no-ops and the next predictions() resolves the previous job's id. Workaround is to re-instantiate GNN(...), not to bump the Model("...") name. - L-6 (Stalled train job forensics): tightened to cite the actual job_manager.py:332-340 polling loop semantics. Removed the speculative "after ~30 min" timing — JOBS retention varies by Snowflake/native-app version and isn't observable from the SDK. - rai-health: same SDK-grounding for the SUSPEND/RESUME runbook; framed the recovery as empirical (not source-documented) since the SDK has no worker-readiness probe beyond GET_REASONER status. Reinforced that the SDK never invokes CREATE_GNN_SERVICE, so its image-mismatch error has no bearing on predictive train submission. Net-new additions: - gnn.fit() idempotency as its own subsection in rai-predictive-training (the actual root cause customers will hit when retrying after partial failures in notebooks). - Hyperparameter validation note: unknown keys raise ValueError with difflib-suggested corrections via validate_train_params. --- plugins/rai/skills/rai-health/SKILL.md | 10 +++---- skills/rai-predictive-modeling/SKILL.md | 16 +++++++--- skills/rai-predictive-training/SKILL.md | 40 ++++++++++++++----------- 3 files changed, 39 insertions(+), 27 deletions(-) diff --git a/plugins/rai/skills/rai-health/SKILL.md b/plugins/rai/skills/rai-health/SKILL.md index e0ca264..094ea5e 100644 --- a/plugins/rai/skills/rai-health/SKILL.md +++ b/plugins/rai/skills/rai-health/SKILL.md @@ -317,9 +317,9 @@ for the full step-by-step recovery checklist, schema reference, and official doc ## Predictive train jobs stuck QUEUED -A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The pod-level status does not reflect in-pod worker state — the worker can be desynced and still report ready. +A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The SDK only checks reasoner-pod status before submitting — the in-pod worker queue can be out of sync with that status, and the SDK has no way to detect it (`relationalai_gnns/core/connector.py::_check_engine_availability`). -**Recovery:** suspend then resume the predictive reasoner to force a worker recycle. Do **not** call `CREATE_GNN_SERVICE()`. +**Recovery (empirical):** suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Do **not** call `CREATE_GNN_SERVICE()`. ```sql -- 1. Confirm a stuck train job @@ -333,13 +333,13 @@ ORDER BY CREATED_ON ASC; CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', ''); CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); --- 3. Wait for STATUS=READY, kill the stuck client, then resubmit with a bumped Model name +-- 3. Wait for STATUS=READY, kill the stuck client, then resubmit (re-instantiate GNN(...)) CALL RELATIONALAI.API.GET_REASONER('predictive', ''); ``` -> **Do not use `CALL RELATIONALAI.EXPERIMENTAL.CREATE_GNN_SERVICE();` to recover stuck predictive train jobs.** It targets a legacy GNN service path that is orthogonal to the predictive reasoner — in PyRel 1.0.x the predictive reasoner serves train jobs in-pod and does not depend on it. The call typically fails with an image-mismatch error (`Invalid image specified in service spec: image 'rai-gnn-app:' does not exist in current application version`); that error does **not** mean GNN training is broken, just that this code path is retired. The right escalation is `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` on the predictive reasoner itself. +> **Do not use `CALL RELATIONALAI.EXPERIMENTAL.CREATE_GNN_SERVICE();` to recover stuck predictive train jobs.** The SDK never invokes it — train submission goes through `.api.exec_job_async('GNN', , ...)` against the predictive reasoner directly (`relationalai_gnns/core/connector.py::exec_job`). `CREATE_GNN_SERVICE` targets a separate code path; if it fails with an image-mismatch error like `Invalid image specified in service spec: image 'rai-gnn-app:' does not exist in current application version`, that does **not** mean GNN training is broken — it just means that orthogonal path can't be brought up. The right escalation is `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` on the predictive reasoner itself. -See `rai-predictive-training` § Worker not ready to accept jobs for the matching client-side symptom and post-recycle resubmit pattern, and § Train-job ID disappears from `RELATIONALAI.API.JOBS` for stalled-job forensics. +See `rai-predictive-training` § Worker not ready to accept jobs for the matching client-side symptom and § Stalled train job: SDK polls without a timeout for stalled-job forensics. --- diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index 4044c99..a5fcaec 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -30,7 +30,9 @@ description: Build GNN data models -- concepts, Snowflake data loading, task rel ### Experiment schema setup (one-time, ACCOUNTADMIN) -GNN training writes experiment artifacts to a Snowflake schema. The RELATIONALAI native app must own write access on it. Without this, the very first `gnn.fit()` fails with a message like *"Schema does not exist or the GNN RelationalAI Native App lacks permissions"* (the wording rotates between "Schema does not exist" and "Database does not exist" depending on which grant is missing first — both grants below are required). +GNN training writes experiment artifacts to a Snowflake schema. The RELATIONALAI native app must have write access on it. Without this the first `gnn.fit()` raises `PermissionError` (from `relationalai_gnns.core.diagnostics.PermissionDiagnostic`) whose message names the missing grant — typically *"Database does not exist or the GNN RelationalAI Native App lacks permissions"* or *"Schema does not exist or ..."*. + +The diagnostic prescribes exactly four grants on top of the database+schema: ```sql -- Use a database you own (NOT a Snowflake-shared/marketplace database). @@ -39,10 +41,14 @@ GNN training writes experiment artifacts to a Snowflake schema. The RELATIONALAI CREATE DATABASE IF NOT EXISTS ; CREATE SCHEMA IF NOT EXISTS .EXPERIMENTS; -GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; -GRANT ALL PRIVILEGES ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; +GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; +GRANT USAGE ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; +GRANT CREATE EXPERIMENT ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; +GRANT CREATE MODEL ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; ``` +`GRANT ALL PRIVILEGES ON SCHEMA .EXPERIMENTS` is a working superset if you don't need least-privilege. + Then in the script: ```python @@ -53,9 +59,11 @@ gnn = GNN( ) ``` +The error is a `PermissionError`, not a generic `RuntimeError` — code that wraps `gnn.fit()` can catch it specifically. + ### `relationalai` package version -The predictive submodule (`relationalai.semantics.reasoners.predictive`) is not in every published `relationalai` release — `from relationalai.semantics.reasoners.predictive import GNN` will raise `ModuleNotFoundError` on releases that pre-date it. Confirm the current minimum version with the RelationalAI team before pinning in `pyproject.toml`. +The predictive submodule (`relationalai.semantics.reasoners.predictive`) is not in every published `relationalai` release — `from relationalai.semantics.reasoners.predictive import GNN` raises `ModuleNotFoundError` on releases that pre-date it. Pin a release that ships the submodule (or install from the development branch when iterating against unreleased changes). --- diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index 973b56f..ea4fd64 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -144,6 +144,8 @@ gnn = GNN(exp_database="DB", exp_schema="EXPERIMENTS", ..., **train_config) gnn.fit() ``` +**Unknown keys raise actionable errors.** `validate_train_params` (`relationalai.semantics.reasoners.predictive.preparation`) rejects unknown `**train_params` keys with a `ValueError` and uses `difflib` to suggest near-matches; if you accidentally pass a `GNN(...)` constructor parameter (e.g. `task_type=`) inside `train_params`, the message tells you it belongs on the constructor. Read the suggestion before second-guessing the typo. + --- ## Common Hyperparameters @@ -188,39 +190,40 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] `has_time_column=True` has two known failure modes; both share the same workaround (turn temporal off — switch to non-temporal Relationships and `has_time_column=False`): 1. **Edge-intermediary `time_col`.** When the concept carrying `time_col` is used only as an edge intermediary (no `identify_by`), validation fails with "no time column defined in data tables". `time_col` only propagates for node concepts. -2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Reproduced on a daily date column at ~27K-row scale, so the threshold for "scale" is low. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. Concrete fallback shape: +2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Reproduced on a daily date column at ~27K-row scale, so the threshold for "scale" is low. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. The full fallback is to make four coordinated changes: drop `time_col=` from `PropertyTransformer`, set `has_time_column=False`, drop `temporal_strategy=`, and rewrite the `Train`/`Val`/`Test` `Relationship`s to drop the date argument. Then preserve the temporal split in pandas before building the task tables: ```python - # Before (fails) + # Before (fails at scale) pt = PropertyTransformer( datetime=[Sale.date], time_col=[Sale.date], # <-- remove ... ) gnn = GNN(has_time_column=True, ..., temporal_strategy="last") # <-- both go + Train = Relationship(f"{Sale} at {Any:date} has {Any:value}") # <-- drop date arg - # After (works) — keep date as a plain datetime feature; do the temporal split - # in pandas before building the task tables. + # After (works) — keep date as a plain datetime feature; the + # temporal split lives in pandas (train_mask/val_mask/test_mask). pt = PropertyTransformer( - datetime=[Sale.date], - # time_col disabled — see Known Limitation + datetime=[Sale.date], # convention: don't pass time_col= when has_time_column=False ... ) gnn = GNN(has_time_column=False, ...) - # Train relationship loses the date arg: Train = Relationship(f"{Sale} has {Any:value}") model.define(Train(Sale, TrainTable.unit_sales)).where(...) ``` + The `PropertyTransformer` API itself accepts `time_col=` independent of `GNN(has_time_column=...)`; the rule "don't pass `time_col=` when `has_time_column=False`" is a convention to avoid dead annotations, not a code-enforced check. + **Engine-side compiled-relation cache footgun (not datetime-specific — surfaces after any column-type change on a bound table):** the engine caches the compiled relation type per RAI relation name and doesn't invalidate it on `ALTER TABLE`, even after stream delete + recreate. Symptom: `Encountered reference to a base relation with a mismatched signature` — but the client only shows the opaque `Failed to pull data into index: transaction was aborted (runtime error)` wrapper. Pull the real error via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` (presigned URL) and look at the `report` field. Workaround: rename `Model(...)` to force a fresh RAI relation namespace (downstream queries that depend on the old name need updating). Better: do schema changes before the first bind — see `rai-predictive-modeling` § Populate from Snowflake. ### Worker not ready to accept jobs -`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` returns `STATUS='READY'`. The reasoner status reflects pod state, not in-pod worker state — the worker can be desynced and still report READY. The real error surfaces only when the SDK polls and the worker reports back: +`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` returns `STATUS='READY'`. The SDK only checks reasoner-pod readiness via `api.get_reasoner` before submitting (`relationalai_gnns/core/connector.py::_check_engine_availability`); it has no notion of an in-pod worker queue, so a desynced worker on a READY pod is invisible to the client. The server-side error surfaces only when the SDK polls and the `CREATE_JOB` external function reports back: > Request failed for external function CREATE_JOB with remote service error: 400 `{"status":"Not Found","message":"worker is not ready to accept jobs - please retry the job later"}` -**Recovery:** when train jobs are stuck QUEUED for >5 minutes despite the reasoner being READY, force the worker to recycle by suspending and resuming the predictive reasoner. +**Recovery (empirical, not source-documented):** when train jobs are stuck QUEUED for >5 minutes despite the reasoner being READY, suspending and resuming the predictive reasoner has been observed to clear the desync. ```sql -- 1. Check non-terminal train jobs @@ -238,29 +241,30 @@ CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); CALL RELATIONALAI.API.GET_REASONER('predictive', ''); ``` -After resume, jobs that were QUEUED on the old worker error out cleanly with the worker-not-ready message. Submit a fresh run with a bumped `Model("...")` name (see § Stale-experiment matching) to start a new experiment. `CREATE_GNN_SERVICE()` is **not** the right escalation here — the predictive reasoner serves train jobs in-pod in PyRel 1.0.x and does not depend on the legacy GNN service. See `rai-health` § Predictive train jobs stuck QUEUED. +After resume, jobs that were QUEUED on the old worker error out cleanly with the worker-not-ready message; resubmit (re-instantiate `GNN(...)` — see § `gnn.fit()` is idempotent below). `CREATE_GNN_SERVICE()` is **not** the right escalation: in PyRel 1.0.x the SDK submits training via `CALL .api.exec_job_async('GNN', , ...)` against the predictive reasoner directly (`relationalai_gnns/core/connector.py::exec_job`), with no reference to `CREATE_GNN_SERVICE`. See `rai-health` § Predictive train jobs stuck QUEUED. -### Stale-experiment matching: bump `Model("...")` name on retry +### `gnn.fit()` is idempotent — re-instantiate `GNN(...)` on retry -`_wait_obtain_model_run_id` in `relationalai.semantics.reasoners.predictive.estimator` matches a just-submitted training job to the most recent experiment with the same `Model("...")` name + concept name. If a previous run with the same model name was killed mid-flight or failed at the prediction step, a new train submission can match the **stale** `model_run_id` and then hang at "Step 2/4: Preparing model for prediction" because the stale model artifacts don't fit the current dataset. +`gnn.fit()` is a silent no-op if `self.train_job` already exists and isn't `FAILED` (`relationalai/semantics/reasoners/predictive/estimator.py:483-490`). Calling `gnn.fit()` a second time on the same `GNN` Python object will **not** submit a new training job — it just logs `Training job already running/completed` and returns. The next `gnn.predictions(...)` call then resolves the *previous* `train_job.model_run_id` (which is the previous job's `job_id` per `relationalai_gnns/core/job_manager.py:132-146`). -**Symptom:** Step 1/3, 2/3, 3/3 (training submit) all succeed in seconds. Step 1/4 (test-table prep) succeeds. Step 2/4 ("Preparing model for prediction…") polls forever (>30 min) with no progress, no JOBS row, no traceback. +**Symptom this explains**: a re-run of a notebook cell or a retry after a killed mid-flight run reports `model_run_id` from a much-earlier job, not the freshly-submitted one. Subsequent prediction calls operate on stale model artifacts and can hang at "Step 2/4: Preparing model for prediction". -**Workaround:** bump the `Model("...")` name on every re-run after a killed/failed run. +**Workaround:** re-instantiate `GNN(...)` on every retry. Bumping `Model("...")` name is **not** the right fix for this specific bug (the SDK has no name-based experiment matching at the `_wait_obtain_model_run_id` layer) — that bump is the workaround for the engine-side compiled-relation cache footgun above, a separate issue. To force a fresh training run after a partial failure: ```python -model = Model("my_template_local_v2") # bump on re-run +gnn = GNN(...) # build a new instance — required +gnn.fit() # submits a fresh job ``` --- ## Troubleshooting -### Train-job ID disappears from `RELATIONALAI.API.JOBS` +### Stalled train job: SDK polls without a timeout -`RELATIONALAI.API.JOBS` history rolls — train jobs that were `STATE='QUEUED'` for ~30 min can be canceled or roll out of retention and disappear from the table entirely. Client-side `_wait_obtain_model_run_id` polling does not time-bound itself and will keep going indefinitely against an ID that no longer exists. +`relationalai_gnns/core/job_manager.py::JobMonitor._wait_for_completion` (line 332-340) polls `get_status()` every 5 seconds with no timeout, no retry cap, and no max-poll-count. While the underlying job stays in `QUEUED` or `RUNNING`, the loop is unbounded. If the row is removed from `RELATIONALAI.API.JOBS` (history retention varies by Snowflake/native-app version), `get_status()` raises rather than self-terminating cleanly — but a row that *stays* QUEUED indefinitely will never trigger that exit path. -If a `gnn.fit()` client has been polling for >30 min with no progress and `SELECT * FROM RELATIONALAI.API.JOBS WHERE ID = ''` returns no row, the client will not self-terminate — kill it manually. Do **not** rely on `RELATIONALAI.API.JOBS` for forensics on stalled jobs older than the retention window; capture state earlier (or rely on `GET_TRANSACTION_ARTIFACTS` / engine logs) when investigating. +If a `gnn.fit()` client has been polling for an unreasonable amount of time and `SELECT * FROM RELATIONALAI.API.JOBS WHERE ID = ''` shows the row is still QUEUED (or returns no row), kill the client manually. Use the SUSPEND/RESUME runbook in § Worker not ready to accept jobs to recover, then re-instantiate `GNN(...)` and resubmit. Do **not** rely on `RELATIONALAI.API.JOBS` for forensics on long-stalled jobs; capture state earlier with `GET_TRANSACTION_ARTIFACTS` or engine logs when investigating. --- From f8c3f088d915550a0a2a9ef7a963d8e4d5023297 Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 30 Apr 2026 20:15:41 -0700 Subject: [PATCH 15/27] predictive-training: dedupe SUSPEND/RESUME runbook with rai-health MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'Worker not ready to accept jobs' section originally repeated the same 3-statement SQL recovery block that lives in rai-health § Predictive train jobs stuck QUEUED. Drop the duplicate SQL and replace with a pointer; keep the SDK-level explanation (what gnn.fit submits, why the SDK can't see worker desync) since that's training-skill territory. Result: one canonical runbook home (rai-health, the operational SQL skill), one canonical SDK explanation (rai-predictive-training). --- skills/rai-predictive-training/SKILL.md | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/skills/rai-predictive-training/SKILL.md b/skills/rai-predictive-training/SKILL.md index ea4fd64..fc91a1b 100644 --- a/skills/rai-predictive-training/SKILL.md +++ b/skills/rai-predictive-training/SKILL.md @@ -219,29 +219,11 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] ### Worker not ready to accept jobs -`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` returns `STATUS='READY'`. The SDK only checks reasoner-pod readiness via `api.get_reasoner` before submitting (`relationalai_gnns/core/connector.py::_check_engine_availability`); it has no notion of an in-pod worker queue, so a desynced worker on a READY pod is invisible to the client. The server-side error surfaces only when the SDK polls and the `CREATE_JOB` external function reports back: +`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `GET_REASONER` returns `STATUS='READY'`. The SDK only checks reasoner-pod readiness via `api.get_reasoner` before submitting (`relationalai_gnns/core/connector.py::_check_engine_availability`); it has no notion of an in-pod worker queue, so a desynced worker on a READY pod is invisible to the client. The server-side error surfaces only when the SDK polls and the `CREATE_JOB` external function reports back: > Request failed for external function CREATE_JOB with remote service error: 400 `{"status":"Not Found","message":"worker is not ready to accept jobs - please retry the job later"}` -**Recovery (empirical, not source-documented):** when train jobs are stuck QUEUED for >5 minutes despite the reasoner being READY, suspending and resuming the predictive reasoner has been observed to clear the desync. - -```sql --- 1. Check non-terminal train jobs -SELECT ID, STATE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN -FROM RELATIONALAI.API.JOBS -WHERE STATE IN ('QUEUED','RUNNING') - AND PAYLOAD LIKE '%"job_type": "train"%' -ORDER BY CREATED_ON ASC; - --- 2. Force the worker to recycle -CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', ''); -CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); - --- 3. Wait for STATUS=READY again, then resubmit (kill the stuck client first) -CALL RELATIONALAI.API.GET_REASONER('predictive', ''); -``` - -After resume, jobs that were QUEUED on the old worker error out cleanly with the worker-not-ready message; resubmit (re-instantiate `GNN(...)` — see § `gnn.fit()` is idempotent below). `CREATE_GNN_SERVICE()` is **not** the right escalation: in PyRel 1.0.x the SDK submits training via `CALL .api.exec_job_async('GNN', , ...)` against the predictive reasoner directly (`relationalai_gnns/core/connector.py::exec_job`), with no reference to `CREATE_GNN_SERVICE`. See `rai-health` § Predictive train jobs stuck QUEUED. +For the SQL recovery runbook (suspend + resume + re-check), see `rai-health` § Predictive train jobs stuck QUEUED. After recovery, resubmit by re-instantiating `GNN(...)` — see § `gnn.fit()` is idempotent below for why bumping `Model("...")` name alone is not enough. ### `gnn.fit()` is idempotent — re-instantiate `GNN(...)` on retry From a195281f4b93d5a6fc04e69d29b53ce07699a207 Mon Sep 17 00:00:00 2001 From: Foula Date: Tue, 28 Apr 2026 15:06:38 +0000 Subject: [PATCH 16/27] added some notes --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../references/auto-discovery.md | 75 ++++++++++++++++++- 2 files changed, 74 insertions(+), 3 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index a5fcaec..b4854b8 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -118,7 +118,7 @@ Additional type imports as needed: `Date`, `DateTime`, `Float`. ## Define and Populate Concepts -> **User-input boundary:** the only things you need from the user are the 3 inputs in [`references/auto-discovery.md`](references/auto-discovery.md) -- source table FQNs, task table FQNs, and the experiment-artifact location. Auto-derive PKs, FKs, columns, types, edges, task type, and timestamp candidates from `INFORMATION_SCHEMA` / `DESCRIBE TABLE`. Don't ask the user for column-level details. +> **User-input boundary:** the only things you need from the user are the 3 inputs in [`references/auto-discovery.md`](references/auto-discovery.md) -- source table FQNs, task table FQNs, and the experiment-artifact location. Auto-derive PKs, FKs, columns, types, edges, task type, and timestamp candidates from Snowflake schema introspection. Use the in-skill `get_table_schema(table_name, database, schema)` helper in `references/auto-discovery.md` as the default schema source before any manual SQL fallback. Don't ask the user for column-level details. Three concept categories show up in a GNN pipeline, distinguished by whether they declare a primary key and how they participate in the graph: diff --git a/skills/rai-predictive-modeling/references/auto-discovery.md b/skills/rai-predictive-modeling/references/auto-discovery.md index 5417f07..978ea4e 100644 --- a/skills/rai-predictive-modeling/references/auto-discovery.md +++ b/skills/rai-predictive-modeling/references/auto-discovery.md @@ -2,8 +2,14 @@ After the user provides table names, the agent automatically discovers schema details by querying Snowflake. This reference documents the conversation templates and discovery process. +## How to use this workflow + +Walk through each phase **sequentially**. For each phase, use the **exact question template** below -- do not rephrase, reorder, or add extra questions. Wait for the user's answers before proceeding to the next phase. If the user provides information that covers multiple phases, acknowledge it and skip to the next uncovered phase. + ## Conversation Templates +**Phase 1 is split into three sub-steps. Ask each one separately and wait for the user's response before moving to the next.** + ### Phase 1a -- Source Tables Ask exactly this: @@ -41,7 +47,7 @@ What Snowflake database and schema should we use for **experiment artifacts**? ## What to Auto-Discover (and what NOT to ask) -The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experiment location). **Do not ask the user** for column names, PKs, FKs, label/target columns, timestamp columns, task type, or feature types — those are friction the user often can't answer without checking the schema themselves. Query Snowflake (`DESCRIBE TABLE` or `INFORMATION_SCHEMA.COLUMNS`; use the snowflake-schema tool) and infer: +The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experiment location). **Do not ask the user** for column names, PKs, FKs, label/target columns, timestamp columns, task type, or feature types — those are friction the user often can't answer without checking the schema themselves. Use the in-skill helper below first (`get_table_schema(table_name, database, schema)`), then infer: 1. **Column names and types** for all source and task tables 2. **Primary keys** -- identify PK columns @@ -55,9 +61,74 @@ The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experime 7. **Task type** -- infer from the label column: - Binary/boolean or 2-value categorical -> `binary_classification` - Multi-value categorical -> `multiclass_classification` + - Multiple label columns for the same row, or array/list-of-labels target -> `multilabel_classification` - Numeric/float -> `regression` - Column matching another concept's PK -> `link_prediction` (ask user to confirm) +**Note: multiclass vs multilabel** +- **Multiclass**: each row has exactly one label chosen from many classes (e.g., `sports` *or* `news` *or* `finance`). +- **Multilabel**: each row can have multiple labels at the same time (e.g., `sports` *and* `news`). + +### Required execution pattern (Snowpark first, then helper) + +Set up a Snowpark session once (follow `rai-setup`), then reuse it for schema discovery. + +Use this implementation directly: + +```python +import re +from relationalai.config import SnowflakeConnection, create_config +from snowflake import snowpark + +_IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9_]+$") +session: snowpark.Session = create_config().get_session(SnowflakeConnection) + + +def get_table_schema(table_name: str, database: str, schema: str) -> list[dict]: + """Return Snowflake table columns as [{'column_name': ..., 'data_type': ...}].""" + table_name = table_name.strip() + database = database.strip() + schema = schema.strip() + + if not table_name or not database or not schema: + return [{"error": "table_name, database, and schema are required and cannot be empty."}] + + for field_name, value in [("database", database), ("schema", schema), ("table_name", table_name)]: + if not _IDENTIFIER_RE.fullmatch(value): + return [{"error": f"Invalid {field_name}: '{value}'. Use only letters, numbers, and underscores."}] + + query = """ + SELECT COLUMN_NAME, DATA_TYPE + FROM {database}.INFORMATION_SCHEMA.COLUMNS + WHERE TABLE_SCHEMA = '{schema}' + AND TABLE_NAME = '{table}' + ORDER BY ORDINAL_POSITION + """.format( + database=database.upper(), + schema=schema.upper(), + table=table_name.upper(), + ) + + try: + rows = session.sql(query).collect() + except Exception as exc: + return [{"error": f"Snowflake query failed: {exc}"}] + + if not rows: + return [{"error": f"No columns found for {database}.{schema}.{table_name}. Check the name and permissions."}] + + return [{"column_name": row["COLUMN_NAME"], "data_type": row["DATA_TYPE"]} for row in rows] +``` + +For each user-provided fully qualified table name `DB.SCHEMA.TABLE`: + +1. Parse into `database=DB`, `schema=SCHEMA`, `table_name=TABLE`. +2. Call the in-skill helper `get_table_schema(table_name=TABLE, database=DB, schema=SCHEMA)`. +3. Treat the returned `column_name` values as canonical Snowflake column names (case-insensitive matching allowed for concept/property references, but spelling must match). +4. If the helper returns an `error`, retry once after uppercasing parts; if still failing, ask the user for `DESCRIBE TABLE` output for only the failing table. + +Use `DESCRIBE TABLE` / manual SQL only as fallback when the Snowpark helper cannot return schema. + ## Link Prediction Detection If link prediction is detected, after presenting the discovery summary, ask the user: @@ -102,4 +173,4 @@ Does this look correct? I'll proceed with this structure. ## Fallback -If the agent cannot connect to Snowflake or auto-discovery fails, fall back to asking the user for column details directly. +If the helper cannot connect to Snowflake or auto-discovery fails, fall back to asking the user for `DESCRIBE TABLE` output (or column lists with types) for only the affected tables. From 7a4be581c2c1bcb14c15728783d25ab945541f17 Mon Sep 17 00:00:00 2001 From: Foula Date: Tue, 28 Apr 2026 17:42:54 +0000 Subject: [PATCH 17/27] small fix --- skills/rai-predictive-modeling/SKILL.md | 2 +- .../rai-predictive-modeling/evals/evals.json | 67 +++++++++++++++++ .../references/auto-discovery.md | 8 +- .../rai-predictive-training/evals/evals.json | 75 +++++++++++++++++++ 4 files changed, 147 insertions(+), 5 deletions(-) create mode 100644 skills/rai-predictive-modeling/evals/evals.json create mode 100644 skills/rai-predictive-training/evals/evals.json diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index b4854b8..f5925a1 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -125,7 +125,7 @@ Three concept categories show up in a GNN pipeline, distinguished by whether the | Category | `identify_by`? | Role | Constraints | |----------|---------------|------|-------------| | **Graph (node)** | yes | Source, target, or other node entities the GNN reasons over | Can carry features and `time_col` | -| **Edge-intermediary** | no | Used only as `src=`/`dst=` in `Edge.new(...)` to express many-to-many or attributed relationships | **Cannot carry `time_col`** -- `time_col` only propagates for node concepts (with `identify_by`); see `rai-predictive-training` § Known Limitations | +| **Edge-intermediary** | no | Used only as `src=`/`dst=` in `Edge.new(...)` to express many-to-many or attributed relationships | **Cannot carry `time_col`** -- `time_col` only propagates for node concepts| | **Task table** | no | Holds train/val/test split rows, joined to a graph concept by FK | Not used in edges; not a feature source | > If you have an existing ontology from `rai-build-starter-ontology`, create a new `Model` for the GNN pipeline -- concepts need `identify_by` for GNN to resolve primary keys. diff --git a/skills/rai-predictive-modeling/evals/evals.json b/skills/rai-predictive-modeling/evals/evals.json new file mode 100644 index 0000000..8ab81ef --- /dev/null +++ b/skills/rai-predictive-modeling/evals/evals.json @@ -0,0 +1,67 @@ +{ + "skill_name": "rai-predictive-modeling", + "evals": [ + { + "id": 1, + "prompt": "I want to set up a GNN data model. My source tables are MYDB.MYSCHEMA.USERS, MYDB.MYSCHEMA.EVENTS, MYDB.MYSCHEMA.RSVPS. Task tables are MYDB.TASKS.TRAIN, MYDB.TASKS.VAL, MYDB.TASKS.TEST. Experiment artifacts: MYDB.EXPERIMENTS. What do you need from me? Reply briefly.", + "expected_output": "The agent should NOT ask for column names, primary keys, foreign keys, label/target columns, timestamps, task type, or feature types. It should indicate it will auto-discover via Snowflake schema introspection (using INFORMATION_SCHEMA or the get_table_schema helper).", + "expectations": [ + "The agent does not ask the user for column names, primary keys, or foreign keys", + "The agent does not ask the user for the label/target column or task type", + "The agent indicates it will introspect Snowflake schema (mentions INFORMATION_SCHEMA, DESCRIBE TABLE, or auto-discovery)" + ] + }, + { + "id": 2, + "prompt": "In a GNN node-classification model with two source concepts (User, Event) and a many-to-many EventAttendee join table that has only event_id, user_id, status columns, how should I declare the three concepts? Also declare the train/val/test task table concepts. Show the PyRel code only.", + "expected_output": "User and Event use identify_by; EventAttendee uses no identify_by (edge-intermediary); task table concepts also have no identify_by.", + "expectations": [ + "User and Event are declared with an identify_by argument", + "EventAttendee is declared without identify_by (edge-intermediary)", + "Train/Val/Test task table concepts are declared without identify_by" + ] + }, + { + "id": 3, + "prompt": "Write the train/validation/test Relationships for a temporal binary classification GNN where the source is Driver, the timestamp column is named 'date', and the label is 'did_not_finish'. Show the PyRel code with template strings.", + "expected_output": "Train/Val use 'at' and 'has' clauses; Test omits the label/target.", + "expectations": [ + "Train Relationship template includes both 'at {Any:date}' and 'has {Any:did_not_finish}'", + "Val Relationship template includes both 'at {Any:date}' and 'has {Any:did_not_finish}'", + "Test Relationship template includes 'at {Any:date}' but does NOT include 'has {Any:did_not_finish}'" + ] + }, + { + "id": 4, + "prompt": "I have a Post concept with a parent_id column referencing Post.id (replies). Define a self-referential GNN edge from Post to its parent Post. Show the PyRel code only.", + "expected_output": "Use Post.ref() to create a separate reference for the destination side of the self-edge.", + "expectations": [ + "The code uses Post.ref() (or equivalent .ref() pattern) to disambiguate the two sides of the self-edge", + "The Edge.new(...) call uses src=Post and dst set to the .ref() reference", + "The where clause joins on parent_id and id" + ] + }, + { + "id": 5, + "prompt": "I have an Event concept (a node, with identify_by) that carries a start_time timestamp, and an EventAttendee edge-intermediary concept (no identify_by) that also carries a copy of start_time. Where should I put time_col in the PropertyTransformer, and why?", + "expected_output": "time_col should be on Event (a node concept). Putting time_col on EventAttendee (edge-intermediary, no identify_by) fails because time_col only propagates for node concepts.", + "expectations": [ + "The agent recommends time_col=[Event.start_time] on the node concept, not on EventAttendee", + "The agent explains time_col only propagates for node concepts (concepts with identify_by)", + "The agent does NOT recommend putting time_col on EventAttendee" + ] + }, + { + "id": 6, + "prompt": "What feature types should I use in PropertyTransformer for: (a) a country code column, (b) a price column, (c) a timestamp column, (d) a free-form review text column, (e) a primary key column 'user_id'? Briefly justify each.", + "expected_output": "(a) category, (b) continuous, (c) datetime, (d) text, (e) drop / exclude — PKs and FKs should be dropped because graph structure already captures relationships.", + "expectations": [ + "Country code is mapped to category", + "Price is mapped to continuous", + "Timestamp is mapped to datetime", + "Free-form review text is mapped to text", + "Primary key user_id is dropped or excluded with the justification that PKs add noise / graph structure already captures relationships" + ] + } + ] +} diff --git a/skills/rai-predictive-modeling/references/auto-discovery.md b/skills/rai-predictive-modeling/references/auto-discovery.md index 978ea4e..aa56104 100644 --- a/skills/rai-predictive-modeling/references/auto-discovery.md +++ b/skills/rai-predictive-modeling/references/auto-discovery.md @@ -34,20 +34,20 @@ What are your **task table** fully qualified names for train/val/test? (e.g., `MY_DB.TASKS.TRAIN`, `MY_DB.TASKS.VAL`, `MY_DB.TASKS.TEST`) ``` -### Phase 1c -- Experiment Artifacts +### Phase 1c -- Experiment Tracking Ask exactly this (after user responds to 1b): ``` -Phase 1c: Experiment Artifacts +Phase 1c: Experiment Tracking -What Snowflake database and schema should we use for **experiment artifacts**? +What Snowflake database and schema should we use for **experiment tracking**? (e.g., `MY_DB.EXPERIMENTS`) ``` ## What to Auto-Discover (and what NOT to ask) -The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experiment location). **Do not ask the user** for column names, PKs, FKs, label/target columns, timestamp columns, task type, or feature types — those are friction the user often can't answer without checking the schema themselves. Use the in-skill helper below first (`get_table_schema(table_name, database, schema)`), then infer: +The user-input boundary is the 3 prompts above (source FQNs, task FQNs, experiment db and schema). **Do not ask the user** for column names, PKs, FKs, label/target columns, timestamp columns, task type, or feature types — those are friction the user often can't answer without checking the schema themselves. Use the in-skill helper below first (`get_table_schema(table_name, database, schema)`), then infer: 1. **Column names and types** for all source and task tables 2. **Primary keys** -- identify PK columns diff --git a/skills/rai-predictive-training/evals/evals.json b/skills/rai-predictive-training/evals/evals.json new file mode 100644 index 0000000..058a46f --- /dev/null +++ b/skills/rai-predictive-training/evals/evals.json @@ -0,0 +1,75 @@ +{ + "skill_name": "rai-predictive-training", + "evals": [ + { + "id": 1, + "prompt": "Construct a GNN for binary node classification with these inputs already defined: gnn_graph, pt (PropertyTransformer), Train and Val Relationships (both use 'at' for timestamp), and source concept User. I want to use GPU and 5 epochs. Pick a reasonable eval_metric. Set exp_database='DB', exp_schema='EXPERIMENTS'. Show only the GNN(...) constructor call and the gnn.fit() call.", + "expected_output": "Constructor uses task_type='binary_classification', a valid binary metric (roc_auc is the suggested default), has_time_column=True, device='cuda', n_epochs=5, exp_database/exp_schema, and passes graph, property_transformer, train, validation. Then gnn.fit() is called.", + "expectations": [ + "task_type is set to 'binary_classification'", + "eval_metric is a valid binary classification metric (roc_auc is the suggested default)", + "has_time_column=True is set", + "device='cuda' and n_epochs=5 are set", + "graph, property_transformer, train, validation are all passed as constructor arguments", + "gnn.fit() is called after construction" + ] + }, + { + "id": 2, + "prompt": "After training a binary classification GNN with `gnn = GNN(...)` and `gnn.fit()`, I want to bind predictions to the User concept and select user_id, probabilities, and predicted labels into a pandas DataFrame. Show only the prediction + select code.", + "expected_output": "Predictions are bound via User.predictions = gnn.predictions(domain=Test). Then a select reads User.user_id, User.predictions.probs, User.predictions.predicted_labels with .where(User.predictions) and .to_df().", + "expectations": [ + "Predictions are bound via User.predictions = gnn.predictions(domain=Test)", + "The select reads User.predictions.probs and User.predictions.predicted_labels", + "A where clause filters on User.predictions", + ".to_df() is used to materialize the DataFrame" + ] + }, + { + "id": 3, + "prompt": "I want to train two GNNs over the same model: one regression (RMSE) on Item.price and one binary classification (ROC AUC) on User.is_active. Both share the same gnn_graph and PropertyTransformer pt. Show the GNN constructor calls, the .fit() calls, and how to bind both sets of predictions on Test data without an attribute-name conflict.", + "expected_output": "Two GNN instances sharing graph + pt, with different task_type/eval_metric. Predictions bound to distinct attribute names (e.g. Item.price_predictions, User.activity_predictions) — not both called 'predictions'.", + "expectations": [ + "Two separate GNN instances are created sharing the same graph and property_transformer", + "First GNN uses task_type='regression' with eval_metric='rmse'", + "Second GNN uses task_type='binary_classification' with eval_metric='roc_auc'", + "Predictions are bound to distinct attribute names per task (e.g. .price_predictions, .activity_predictions) rather than both as .predictions" + ] + }, + { + "id": 4, + "prompt": "I trained and registered a binary-classification GNN in session 1 as model_name='dnf_predictor', version_name='v1' in DB.MODEL_REGISTRY. Source concept is Driver, has_time_column=True. In session 2, I have already rebuilt gnn_graph and pt with the same structure. Show only the load-mode GNN(...) constructor call and the load + predict-on-Test code.", + "expected_output": "Load-mode constructor passes graph, property_transformer, source_concept=Driver, task_type='binary_classification', has_time_column=True, model_database/model_schema/model_name/version_name. Omits train/validation/eval_metric/hyperparameters. Calls gnn.load() (not gnn.fit()).", + "expectations": [ + "source_concept=Driver is provided", + "task_type='binary_classification' is provided (not persisted in the registry)", + "has_time_column=True is provided (not persisted in the registry)", + "model_database, model_schema, model_name, and version_name are all provided", + "train and validation are NOT passed", + "device, n_epochs, lr (or other hyperparameters) are NOT passed", + "gnn.load() is called, not gnn.fit()" + ] + }, + { + "id": 5, + "prompt": "After training a regression GNN that produces Item.predictions.predicted_value, I want to use the prediction in a downstream optimization solver scoped at the Order level (one row per Order). Items relate to Orders via Interaction.item_id and Interaction.order_id. Sketch the bridge concept and the aggregation pattern as PyRel code.", + "expected_output": "Aggregate Item.predictions.predicted_value with aggregates.sum(...).per(Order).where(Interaction joins). Bind result to a derived Property on Order (e.g. Order.total_predicted_value) so the solver consumes it cleanly at the Order scope.", + "expectations": [ + "A bridge concept at the Order scope is used (predictions are not consumed directly at Item scope)", + "aggregates.sum (or another appropriate aggregation) per Order is used", + "The aggregation joins Item with Order via the Interaction relationship", + "The result is bound as a regular Property on the bridge concept (e.g. Order.total_predicted_value)" + ] + }, + { + "id": 6, + "prompt": "I set device='cuda' on the GNN constructor but training is unexpectedly slow or fails to use the GPU. What is the most likely cause and how do I fix it?", + "expected_output": "device='cuda' on GNN(...) is paired with engine sizing in raiconfig.yaml — the predictive reasoner engine must also be GPU-sized. Setting only the client flag silently falls back or fails.", + "expectations": [ + "The agent identifies that device='cuda' alone is not enough — the predictive reasoner engine must also be GPU-sized", + "The agent points to raiconfig.yaml (or the reasoners section) as the place to set engine size", + "The agent describes this as a paired requirement (both must be configured, or neither)" + ] + } + ] +} From 0ca1f58c1c61a952e99b790541d646d49169ba44 Mon Sep 17 00:00:00 2001 From: Foula Date: Wed, 29 Apr 2026 09:08:46 +0000 Subject: [PATCH 18/27] remove predictive eval artifacts from tracking Made-with: Cursor --- .../rai-predictive-modeling/evals/evals.json | 67 ----------------- .../rai-predictive-training/evals/evals.json | 75 ------------------- 2 files changed, 142 deletions(-) delete mode 100644 skills/rai-predictive-modeling/evals/evals.json delete mode 100644 skills/rai-predictive-training/evals/evals.json diff --git a/skills/rai-predictive-modeling/evals/evals.json b/skills/rai-predictive-modeling/evals/evals.json deleted file mode 100644 index 8ab81ef..0000000 --- a/skills/rai-predictive-modeling/evals/evals.json +++ /dev/null @@ -1,67 +0,0 @@ -{ - "skill_name": "rai-predictive-modeling", - "evals": [ - { - "id": 1, - "prompt": "I want to set up a GNN data model. My source tables are MYDB.MYSCHEMA.USERS, MYDB.MYSCHEMA.EVENTS, MYDB.MYSCHEMA.RSVPS. Task tables are MYDB.TASKS.TRAIN, MYDB.TASKS.VAL, MYDB.TASKS.TEST. Experiment artifacts: MYDB.EXPERIMENTS. What do you need from me? Reply briefly.", - "expected_output": "The agent should NOT ask for column names, primary keys, foreign keys, label/target columns, timestamps, task type, or feature types. It should indicate it will auto-discover via Snowflake schema introspection (using INFORMATION_SCHEMA or the get_table_schema helper).", - "expectations": [ - "The agent does not ask the user for column names, primary keys, or foreign keys", - "The agent does not ask the user for the label/target column or task type", - "The agent indicates it will introspect Snowflake schema (mentions INFORMATION_SCHEMA, DESCRIBE TABLE, or auto-discovery)" - ] - }, - { - "id": 2, - "prompt": "In a GNN node-classification model with two source concepts (User, Event) and a many-to-many EventAttendee join table that has only event_id, user_id, status columns, how should I declare the three concepts? Also declare the train/val/test task table concepts. Show the PyRel code only.", - "expected_output": "User and Event use identify_by; EventAttendee uses no identify_by (edge-intermediary); task table concepts also have no identify_by.", - "expectations": [ - "User and Event are declared with an identify_by argument", - "EventAttendee is declared without identify_by (edge-intermediary)", - "Train/Val/Test task table concepts are declared without identify_by" - ] - }, - { - "id": 3, - "prompt": "Write the train/validation/test Relationships for a temporal binary classification GNN where the source is Driver, the timestamp column is named 'date', and the label is 'did_not_finish'. Show the PyRel code with template strings.", - "expected_output": "Train/Val use 'at' and 'has' clauses; Test omits the label/target.", - "expectations": [ - "Train Relationship template includes both 'at {Any:date}' and 'has {Any:did_not_finish}'", - "Val Relationship template includes both 'at {Any:date}' and 'has {Any:did_not_finish}'", - "Test Relationship template includes 'at {Any:date}' but does NOT include 'has {Any:did_not_finish}'" - ] - }, - { - "id": 4, - "prompt": "I have a Post concept with a parent_id column referencing Post.id (replies). Define a self-referential GNN edge from Post to its parent Post. Show the PyRel code only.", - "expected_output": "Use Post.ref() to create a separate reference for the destination side of the self-edge.", - "expectations": [ - "The code uses Post.ref() (or equivalent .ref() pattern) to disambiguate the two sides of the self-edge", - "The Edge.new(...) call uses src=Post and dst set to the .ref() reference", - "The where clause joins on parent_id and id" - ] - }, - { - "id": 5, - "prompt": "I have an Event concept (a node, with identify_by) that carries a start_time timestamp, and an EventAttendee edge-intermediary concept (no identify_by) that also carries a copy of start_time. Where should I put time_col in the PropertyTransformer, and why?", - "expected_output": "time_col should be on Event (a node concept). Putting time_col on EventAttendee (edge-intermediary, no identify_by) fails because time_col only propagates for node concepts.", - "expectations": [ - "The agent recommends time_col=[Event.start_time] on the node concept, not on EventAttendee", - "The agent explains time_col only propagates for node concepts (concepts with identify_by)", - "The agent does NOT recommend putting time_col on EventAttendee" - ] - }, - { - "id": 6, - "prompt": "What feature types should I use in PropertyTransformer for: (a) a country code column, (b) a price column, (c) a timestamp column, (d) a free-form review text column, (e) a primary key column 'user_id'? Briefly justify each.", - "expected_output": "(a) category, (b) continuous, (c) datetime, (d) text, (e) drop / exclude — PKs and FKs should be dropped because graph structure already captures relationships.", - "expectations": [ - "Country code is mapped to category", - "Price is mapped to continuous", - "Timestamp is mapped to datetime", - "Free-form review text is mapped to text", - "Primary key user_id is dropped or excluded with the justification that PKs add noise / graph structure already captures relationships" - ] - } - ] -} diff --git a/skills/rai-predictive-training/evals/evals.json b/skills/rai-predictive-training/evals/evals.json deleted file mode 100644 index 058a46f..0000000 --- a/skills/rai-predictive-training/evals/evals.json +++ /dev/null @@ -1,75 +0,0 @@ -{ - "skill_name": "rai-predictive-training", - "evals": [ - { - "id": 1, - "prompt": "Construct a GNN for binary node classification with these inputs already defined: gnn_graph, pt (PropertyTransformer), Train and Val Relationships (both use 'at' for timestamp), and source concept User. I want to use GPU and 5 epochs. Pick a reasonable eval_metric. Set exp_database='DB', exp_schema='EXPERIMENTS'. Show only the GNN(...) constructor call and the gnn.fit() call.", - "expected_output": "Constructor uses task_type='binary_classification', a valid binary metric (roc_auc is the suggested default), has_time_column=True, device='cuda', n_epochs=5, exp_database/exp_schema, and passes graph, property_transformer, train, validation. Then gnn.fit() is called.", - "expectations": [ - "task_type is set to 'binary_classification'", - "eval_metric is a valid binary classification metric (roc_auc is the suggested default)", - "has_time_column=True is set", - "device='cuda' and n_epochs=5 are set", - "graph, property_transformer, train, validation are all passed as constructor arguments", - "gnn.fit() is called after construction" - ] - }, - { - "id": 2, - "prompt": "After training a binary classification GNN with `gnn = GNN(...)` and `gnn.fit()`, I want to bind predictions to the User concept and select user_id, probabilities, and predicted labels into a pandas DataFrame. Show only the prediction + select code.", - "expected_output": "Predictions are bound via User.predictions = gnn.predictions(domain=Test). Then a select reads User.user_id, User.predictions.probs, User.predictions.predicted_labels with .where(User.predictions) and .to_df().", - "expectations": [ - "Predictions are bound via User.predictions = gnn.predictions(domain=Test)", - "The select reads User.predictions.probs and User.predictions.predicted_labels", - "A where clause filters on User.predictions", - ".to_df() is used to materialize the DataFrame" - ] - }, - { - "id": 3, - "prompt": "I want to train two GNNs over the same model: one regression (RMSE) on Item.price and one binary classification (ROC AUC) on User.is_active. Both share the same gnn_graph and PropertyTransformer pt. Show the GNN constructor calls, the .fit() calls, and how to bind both sets of predictions on Test data without an attribute-name conflict.", - "expected_output": "Two GNN instances sharing graph + pt, with different task_type/eval_metric. Predictions bound to distinct attribute names (e.g. Item.price_predictions, User.activity_predictions) — not both called 'predictions'.", - "expectations": [ - "Two separate GNN instances are created sharing the same graph and property_transformer", - "First GNN uses task_type='regression' with eval_metric='rmse'", - "Second GNN uses task_type='binary_classification' with eval_metric='roc_auc'", - "Predictions are bound to distinct attribute names per task (e.g. .price_predictions, .activity_predictions) rather than both as .predictions" - ] - }, - { - "id": 4, - "prompt": "I trained and registered a binary-classification GNN in session 1 as model_name='dnf_predictor', version_name='v1' in DB.MODEL_REGISTRY. Source concept is Driver, has_time_column=True. In session 2, I have already rebuilt gnn_graph and pt with the same structure. Show only the load-mode GNN(...) constructor call and the load + predict-on-Test code.", - "expected_output": "Load-mode constructor passes graph, property_transformer, source_concept=Driver, task_type='binary_classification', has_time_column=True, model_database/model_schema/model_name/version_name. Omits train/validation/eval_metric/hyperparameters. Calls gnn.load() (not gnn.fit()).", - "expectations": [ - "source_concept=Driver is provided", - "task_type='binary_classification' is provided (not persisted in the registry)", - "has_time_column=True is provided (not persisted in the registry)", - "model_database, model_schema, model_name, and version_name are all provided", - "train and validation are NOT passed", - "device, n_epochs, lr (or other hyperparameters) are NOT passed", - "gnn.load() is called, not gnn.fit()" - ] - }, - { - "id": 5, - "prompt": "After training a regression GNN that produces Item.predictions.predicted_value, I want to use the prediction in a downstream optimization solver scoped at the Order level (one row per Order). Items relate to Orders via Interaction.item_id and Interaction.order_id. Sketch the bridge concept and the aggregation pattern as PyRel code.", - "expected_output": "Aggregate Item.predictions.predicted_value with aggregates.sum(...).per(Order).where(Interaction joins). Bind result to a derived Property on Order (e.g. Order.total_predicted_value) so the solver consumes it cleanly at the Order scope.", - "expectations": [ - "A bridge concept at the Order scope is used (predictions are not consumed directly at Item scope)", - "aggregates.sum (or another appropriate aggregation) per Order is used", - "The aggregation joins Item with Order via the Interaction relationship", - "The result is bound as a regular Property on the bridge concept (e.g. Order.total_predicted_value)" - ] - }, - { - "id": 6, - "prompt": "I set device='cuda' on the GNN constructor but training is unexpectedly slow or fails to use the GPU. What is the most likely cause and how do I fix it?", - "expected_output": "device='cuda' on GNN(...) is paired with engine sizing in raiconfig.yaml — the predictive reasoner engine must also be GPU-sized. Setting only the client flag silently falls back or fails.", - "expectations": [ - "The agent identifies that device='cuda' alone is not enough — the predictive reasoner engine must also be GPU-sized", - "The agent points to raiconfig.yaml (or the reasoners section) as the place to set engine size", - "The agent describes this as a paired requirement (both must be configured, or neither)" - ] - } - ] -} From 843e0fd6e40186cd52ad3026ddbfc8b9e2120d00 Mon Sep 17 00:00:00 2001 From: Foula Date: Wed, 29 Apr 2026 17:37:37 +0000 Subject: [PATCH 19/27] fixed skills --- skills/rai-predictive-modeling/SKILL.md | 31 +++++++------------ .../examples/link_prediction_snowflake.py | 9 ++---- .../examples/node_classification_snowflake.py | 4 --- 3 files changed, 14 insertions(+), 30 deletions(-) diff --git a/skills/rai-predictive-modeling/SKILL.md b/skills/rai-predictive-modeling/SKILL.md index f5925a1..ff2ec32 100644 --- a/skills/rai-predictive-modeling/SKILL.md +++ b/skills/rai-predictive-modeling/SKILL.md @@ -83,7 +83,7 @@ Concept, Table, Relationship = model.Concept, model.Table, model.Relationship |---------|------| | Single PK | `User = Concept("User", identify_by={"user_id": Integer})` | | Composite PK | `Class = Concept("Class", identify_by={"courseid": Integer, "year": Integer})` | -| No PK (task table) | `TrainTable = Concept("TrainTable")` | +| No PK (e.g. task table) | `TrainTable = Concept("TrainTable")` | ```python # Graph init @@ -118,17 +118,18 @@ Additional type imports as needed: `Date`, `DateTime`, `Float`. ## Define and Populate Concepts -> **User-input boundary:** the only things you need from the user are the 3 inputs in [`references/auto-discovery.md`](references/auto-discovery.md) -- source table FQNs, task table FQNs, and the experiment-artifact location. Auto-derive PKs, FKs, columns, types, edges, task type, and timestamp candidates from Snowflake schema introspection. Use the in-skill `get_table_schema(table_name, database, schema)` helper in `references/auto-discovery.md` as the default schema source before any manual SQL fallback. Don't ask the user for column-level details. +> **User-input boundary:** the only things you need from the user are the 3 inputs in [`references/auto-discovery.md`](references/auto-discovery.md) -- source table FQNs, task table FQNs, and the experiment tracking database and schema. Auto-derive PKs, FKs, columns, types, edges, task type, and timestamp candidates from Snowflake schema introspection. Use the in-skill `get_table_schema(table_name, database, schema)` helper in `references/auto-discovery.md` as the default schema source before any manual SQL fallback. Don't ask the user for column-level details. -Three concept categories show up in a GNN pipeline, distinguished by whether they declare a primary key and how they participate in the graph: +Two concept categories show up in a GNN pipeline, distinguished by their role in the graph: -| Category | `identify_by`? | Role | Constraints | -|----------|---------------|------|-------------| -| **Graph (node)** | yes | Source, target, or other node entities the GNN reasons over | Can carry features and `time_col` | -| **Edge-intermediary** | no | Used only as `src=`/`dst=` in `Edge.new(...)` to express many-to-many or attributed relationships | **Cannot carry `time_col`** -- `time_col` only propagates for node concepts| -| **Task table** | no | Holds train/val/test split rows, joined to a graph concept by FK | Not used in edges; not a feature source | +| Category | Role | +|----------|------| +| **Graph (node)** | Source, target, or other node entities the GNN reasons over -- can carry features and `time_col` | +| **Task table** | Holds train/val/test split rows, joined to a graph concept by FK -- not used in edges; not a feature source | -> If you have an existing ontology from `rai-build-starter-ontology`, create a new `Model` for the GNN pipeline -- concepts need `identify_by` for GNN to resolve primary keys. +`identify_by` is not required by the GNN pipeline. Pass it when you want to declare an explicit primary key for a graph concept (matches a Snowflake column); omit it for task tables and for graph concepts where you don't need an explicit PK. + +> If you have an existing ontology from `rai-build-starter-ontology`, create a new `Model` for the GNN pipeline. ### Graph (node) Concepts @@ -139,16 +140,6 @@ User = Concept("User", identify_by={"user_id": Integer}) Event = Concept("Event", identify_by={"event_id": Integer}) ``` -### Edge-intermediary Concepts - -When a many-to-many or attributed relationship is best modeled as its own concept (e.g. `Interaction` between `User` and `Item`), and that concept's row identity isn't needed downstream, you can omit `identify_by`: - -```python -EventAttendee = Concept("EventAttendee") # used only in Edge.new(src=..., dst=...) -``` - -If the intermediary needs to carry the temporal column for `has_time_column=True`, give it an `identify_by` (promoting it to a graph node concept). `time_col` does not propagate from edge-intermediary concepts. - ### Task Table Concepts Task table concepts have no `identify_by`: @@ -291,7 +282,7 @@ pt = PropertyTransformer( Centrality, community labels, and other graph-algorithm outputs from `rai-graph-analysis` can feed the GNN as features once they're materialized as concept properties. Compute the metric on a separate Graph instance (the algorithm graph -- often a different topology from the GNN graph), bind the result, then include in the PropertyTransformer: ```python -# Algorithm graph (e.g. node-to-node, distinct from the GNN's bipartite/edge-intermediary graph) +# Algorithm graph (often a different topology from the GNN graph) algo_graph = Graph(model, directed=False) define(algo_graph.Edge.new(src=Source, dst=SourceRef)).where(...) diff --git a/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py b/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py index d591cb2..c35a1d6 100644 --- a/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py +++ b/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py @@ -2,10 +2,10 @@ GNN Link Prediction -- Data Modeling (Phases 1-6) ================================================= Repeated link prediction on a bipartite User-Item graph with an Interaction -edge-intermediary concept carrying timestamps. +concept carrying timestamps. Demonstrates: concepts, population, task relationships (link prediction with -time), graph edges via an intermediary concept, and PropertyTransformer. +time), graph edges, and PropertyTransformer. For training and prediction, see `rai-predictive-training`. """ @@ -19,10 +19,7 @@ Concept, Table, Relationship = model.Concept, model.Table, model.Relationship # -- Phase 2: Define Concepts -- -# graph (node) concepts -- User is source (predicting from), Item is target (predicting to). -# Interaction has its own identify_by because it carries `time_col` (timestamp); time_col -# only propagates for node concepts, so an edge-intermediary version (no identify_by) would -# fail validation. See `rai-predictive-training` § Known Limitations. +# graph (node) concepts User = Concept("User", identify_by={"user_id": Integer}) Item = Concept("Item", identify_by={"item_id": Integer}) Interaction = Concept("Interaction", identify_by={"interaction_id": Integer}) diff --git a/skills/rai-predictive-modeling/examples/node_classification_snowflake.py b/skills/rai-predictive-modeling/examples/node_classification_snowflake.py index 1b405e8..39224fe 100644 --- a/skills/rai-predictive-modeling/examples/node_classification_snowflake.py +++ b/skills/rai-predictive-modeling/examples/node_classification_snowflake.py @@ -19,7 +19,6 @@ # graph (node) concepts User = Concept("User", identify_by={"user_id": Integer}) Event = Concept("Event", identify_by={"event_id": Integer}) -# edge-intermediary concept (no identify_by, used only as Edge src/dst) EventAttendee = Concept("EventAttendee") # task table concepts @@ -75,9 +74,6 @@ category_event_attendee = [EventAttendee.status] datetime_event_attendee = [EventAttendee.start_time] -# time_col only propagates for node concepts -- list it on Event (a node), not -# on EventAttendee (edge-intermediary, no identify_by). See -# `rai-predictive-training` § Known Limitations for the failure mode this avoids. pt = PropertyTransformer( category=[*category_user, *category_event, *category_event_attendee], datetime=[*datetime_user, *datetime_event, *datetime_event_attendee], From 3f04b703d0a2de80564147d6926e68bbee623892 Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 30 Apr 2026 20:36:58 -0700 Subject: [PATCH 20/27] structural: migrate predictive skills to plugins/rai/skills/ layout Moves rai-predictive-modeling and rai-predictive-training from skills// to plugins/rai/skills// to align with main's plugin layout. The other PR #21 file edits (rai-health, rai-setup, rai-discovery, rai-graph-analysis) were auto-relocated by git's rename detection during the merge from main. Removes the empty skills/ top-level directory. PR #21 is now mergeable to main without further structural work. --- {skills => plugins/rai/skills}/rai-predictive-modeling/SKILL.md | 0 .../rai-predictive-modeling/examples/link_prediction_snowflake.py | 0 .../examples/node_classification_snowflake.py | 0 .../rai-predictive-modeling/examples/regression_snowflake.py | 0 .../skills}/rai-predictive-modeling/references/auto-discovery.md | 0 .../references/property-transformer-types.md | 0 .../rai-predictive-modeling/references/task-relationships.md | 0 {skills => plugins/rai/skills}/rai-predictive-training/SKILL.md | 0 .../skills}/rai-predictive-training/examples/register_and_load.py | 0 .../rai-predictive-training/examples/train_link_prediction.py | 0 .../rai-predictive-training/examples/train_node_classification.py | 0 .../skills}/rai-predictive-training/examples/train_regression.py | 0 .../rai-predictive-training/references/evaluation-debugging.md | 0 .../skills}/rai-predictive-training/references/hyperparameters.md | 0 .../rai-predictive-training/references/prediction-attributes.md | 0 .../rai-predictive-training/references/task-types-and-metrics.md | 0 16 files changed, 0 insertions(+), 0 deletions(-) rename {skills => plugins/rai/skills}/rai-predictive-modeling/SKILL.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/examples/link_prediction_snowflake.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/examples/node_classification_snowflake.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/examples/regression_snowflake.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/references/auto-discovery.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/references/property-transformer-types.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-modeling/references/task-relationships.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/SKILL.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/examples/register_and_load.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/examples/train_link_prediction.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/examples/train_node_classification.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/examples/train_regression.py (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/references/evaluation-debugging.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/references/hyperparameters.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/references/prediction-attributes.md (100%) rename {skills => plugins/rai/skills}/rai-predictive-training/references/task-types-and-metrics.md (100%) diff --git a/skills/rai-predictive-modeling/SKILL.md b/plugins/rai/skills/rai-predictive-modeling/SKILL.md similarity index 100% rename from skills/rai-predictive-modeling/SKILL.md rename to plugins/rai/skills/rai-predictive-modeling/SKILL.md diff --git a/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py b/plugins/rai/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py similarity index 100% rename from skills/rai-predictive-modeling/examples/link_prediction_snowflake.py rename to plugins/rai/skills/rai-predictive-modeling/examples/link_prediction_snowflake.py diff --git a/skills/rai-predictive-modeling/examples/node_classification_snowflake.py b/plugins/rai/skills/rai-predictive-modeling/examples/node_classification_snowflake.py similarity index 100% rename from skills/rai-predictive-modeling/examples/node_classification_snowflake.py rename to plugins/rai/skills/rai-predictive-modeling/examples/node_classification_snowflake.py diff --git a/skills/rai-predictive-modeling/examples/regression_snowflake.py b/plugins/rai/skills/rai-predictive-modeling/examples/regression_snowflake.py similarity index 100% rename from skills/rai-predictive-modeling/examples/regression_snowflake.py rename to plugins/rai/skills/rai-predictive-modeling/examples/regression_snowflake.py diff --git a/skills/rai-predictive-modeling/references/auto-discovery.md b/plugins/rai/skills/rai-predictive-modeling/references/auto-discovery.md similarity index 100% rename from skills/rai-predictive-modeling/references/auto-discovery.md rename to plugins/rai/skills/rai-predictive-modeling/references/auto-discovery.md diff --git a/skills/rai-predictive-modeling/references/property-transformer-types.md b/plugins/rai/skills/rai-predictive-modeling/references/property-transformer-types.md similarity index 100% rename from skills/rai-predictive-modeling/references/property-transformer-types.md rename to plugins/rai/skills/rai-predictive-modeling/references/property-transformer-types.md diff --git a/skills/rai-predictive-modeling/references/task-relationships.md b/plugins/rai/skills/rai-predictive-modeling/references/task-relationships.md similarity index 100% rename from skills/rai-predictive-modeling/references/task-relationships.md rename to plugins/rai/skills/rai-predictive-modeling/references/task-relationships.md diff --git a/skills/rai-predictive-training/SKILL.md b/plugins/rai/skills/rai-predictive-training/SKILL.md similarity index 100% rename from skills/rai-predictive-training/SKILL.md rename to plugins/rai/skills/rai-predictive-training/SKILL.md diff --git a/skills/rai-predictive-training/examples/register_and_load.py b/plugins/rai/skills/rai-predictive-training/examples/register_and_load.py similarity index 100% rename from skills/rai-predictive-training/examples/register_and_load.py rename to plugins/rai/skills/rai-predictive-training/examples/register_and_load.py diff --git a/skills/rai-predictive-training/examples/train_link_prediction.py b/plugins/rai/skills/rai-predictive-training/examples/train_link_prediction.py similarity index 100% rename from skills/rai-predictive-training/examples/train_link_prediction.py rename to plugins/rai/skills/rai-predictive-training/examples/train_link_prediction.py diff --git a/skills/rai-predictive-training/examples/train_node_classification.py b/plugins/rai/skills/rai-predictive-training/examples/train_node_classification.py similarity index 100% rename from skills/rai-predictive-training/examples/train_node_classification.py rename to plugins/rai/skills/rai-predictive-training/examples/train_node_classification.py diff --git a/skills/rai-predictive-training/examples/train_regression.py b/plugins/rai/skills/rai-predictive-training/examples/train_regression.py similarity index 100% rename from skills/rai-predictive-training/examples/train_regression.py rename to plugins/rai/skills/rai-predictive-training/examples/train_regression.py diff --git a/skills/rai-predictive-training/references/evaluation-debugging.md b/plugins/rai/skills/rai-predictive-training/references/evaluation-debugging.md similarity index 100% rename from skills/rai-predictive-training/references/evaluation-debugging.md rename to plugins/rai/skills/rai-predictive-training/references/evaluation-debugging.md diff --git a/skills/rai-predictive-training/references/hyperparameters.md b/plugins/rai/skills/rai-predictive-training/references/hyperparameters.md similarity index 100% rename from skills/rai-predictive-training/references/hyperparameters.md rename to plugins/rai/skills/rai-predictive-training/references/hyperparameters.md diff --git a/skills/rai-predictive-training/references/prediction-attributes.md b/plugins/rai/skills/rai-predictive-training/references/prediction-attributes.md similarity index 100% rename from skills/rai-predictive-training/references/prediction-attributes.md rename to plugins/rai/skills/rai-predictive-training/references/prediction-attributes.md diff --git a/skills/rai-predictive-training/references/task-types-and-metrics.md b/plugins/rai/skills/rai-predictive-training/references/task-types-and-metrics.md similarity index 100% rename from skills/rai-predictive-training/references/task-types-and-metrics.md rename to plugins/rai/skills/rai-predictive-training/references/task-types-and-metrics.md From cbcc7e64673d5e0926e69be1d4885c87fe61d65a Mon Sep 17 00:00:00 2001 From: cafzal Date: Thu, 30 Apr 2026 20:44:47 -0700 Subject: [PATCH 21/27] predictive skills: dev-skills-review fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - rai-predictive-training/SKILL.md: 563→514 lines via consolidation - Replaced fragmented 'Known Limitations' / 'Worker not ready' / 'gnn.fit is idempotent' / 'Stalled train job' subsections (split between Training and Troubleshooting) with one tight symptom→fix table under a single 'Known Limitations & Runtime Troubleshooting' heading - Removed three Common Pitfalls rows that duplicated the new inline table - Extracted full has_time_column=True before/after fallback code, SDK source citations, and full troubleshoot prose to references/known- limitations.md (rewritten as a quick lookup, not essay prose) - Reference Files table updated with new entry + load trigger - Both new skills' descriptions: added negative-boundary clauses ('Not for X — see other-skill') so routing failures from over-broad WHEN clauses are caught at description match time --- .../skills/rai-predictive-modeling/SKILL.md | 2 +- .../skills/rai-predictive-training/SKILL.md | 73 ++++--------------- .../references/known-limitations.md | 59 +++++++++++++++ 3 files changed, 73 insertions(+), 61 deletions(-) create mode 100644 plugins/rai/skills/rai-predictive-training/references/known-limitations.md diff --git a/plugins/rai/skills/rai-predictive-modeling/SKILL.md b/plugins/rai/skills/rai-predictive-modeling/SKILL.md index ff2ec32..f6aeec5 100644 --- a/plugins/rai/skills/rai-predictive-modeling/SKILL.md +++ b/plugins/rai/skills/rai-predictive-modeling/SKILL.md @@ -1,6 +1,6 @@ --- name: rai-predictive-modeling -description: Build GNN data models -- concepts, Snowflake data loading, task relationships, graph edges, and PropertyTransformer features. Use when defining entity types, loading data, or configuring graph structure for a predictive GNN pipeline. +description: Build GNN data models -- concepts, Snowflake data loading, task relationships, graph edges, and PropertyTransformer features. Use when defining entity types, loading data, or configuring graph structure for a predictive GNN pipeline. Not for training, predictions, evaluation, or model management (see `rai-predictive-training`). --- # Predictive Modeling diff --git a/plugins/rai/skills/rai-predictive-training/SKILL.md b/plugins/rai/skills/rai-predictive-training/SKILL.md index fc91a1b..c3b37fc 100644 --- a/plugins/rai/skills/rai-predictive-training/SKILL.md +++ b/plugins/rai/skills/rai-predictive-training/SKILL.md @@ -1,6 +1,6 @@ --- name: rai-predictive-training -description: Configure and train GNN models, generate predictions, evaluate results, and manage trained models. Use after building the data model with rai-predictive-modeling, when ready to run training, evaluate, or manage GNN models. +description: Configure and train GNN models, generate predictions, evaluate results, and manage trained models. Use after building the data model with `rai-predictive-modeling`, when ready to run training, evaluate, or manage GNN models. Not for concepts, data loading, edges, or feature configuration (see `rai-predictive-modeling`). --- # Predictive Training @@ -185,68 +185,20 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] 2. Model training over `n_epochs` 3. Evaluation on the validation set -### Known Limitations +### Known Limitations & Runtime Troubleshooting -`has_time_column=True` has two known failure modes; both share the same workaround (turn temporal off — switch to non-temporal Relationships and `has_time_column=False`): +GNN training has runtime gotchas that surface as opaque or no-error symptoms in the client. Use this table to recognize each one; load `references/known-limitations.md` for full causes (with SDK source citations), the before/after fallback code for `has_time_column=True` at scale, and the `GET_TRANSACTION_ARTIFACTS` recipe. -1. **Edge-intermediary `time_col`.** When the concept carrying `time_col` is used only as an edge intermediary (no `identify_by`), validation fails with "no time column defined in data tables". `time_col` only propagates for node concepts. -2. **Datetime column processing at scale.** On larger Snowflake-loaded datasets the trainer can fail server-side with `ValidationError: Error processing datetime column ''` even with the time-bearing concept as a node, clean data, and the column properly listed in both `datetime=[...]` and `time_col=[...]`. The failure is loud at submit time, not silent. Reproduced on a daily date column at ~27K-row scale, so the threshold for "scale" is low. Confirm the timestamp column type matches what the GNN datetime pipeline accepts (see `rai-predictive-modeling` § Define and Populate Concepts) and fall back to non-temporal Relationships if the issue persists. The full fallback is to make four coordinated changes: drop `time_col=` from `PropertyTransformer`, set `has_time_column=False`, drop `temporal_strategy=`, and rewrite the `Train`/`Val`/`Test` `Relationship`s to drop the date argument. Then preserve the temporal split in pandas before building the task tables: +| Symptom | Recover via | +|---|---| +| `has_time_column=True` fails with `no time column defined in data tables` | `time_col` only propagates for node concepts — switch to non-temporal Relationships + `has_time_column=False` | +| `has_time_column=True` fails with `ValidationError: Error processing datetime column` at scale (~27K rows is enough) | Same fallback; full code shape in `references/known-limitations.md` | +| Train job stays `QUEUED` indefinitely while reasoner reports `READY` | `rai-health` § Predictive train jobs stuck QUEUED (`SUSPEND_REASONER` + `RESUME_REASONER_ASYNC`) | +| `gnn.fit()` returns a `model_run_id` from an earlier job after a partial failure or notebook re-run | `gnn.fit()` is idempotent if `self.train_job` exists and isn't FAILED — re-instantiate `GNN(...)` on every retry, not bump `Model("...")` | +| Client polls forever with no progress | `JobMonitor._wait_for_completion` has no timeout — kill the client manually + recover via the QUEUED runbook | +| `Failed to pull data into index: transaction was aborted (runtime error)` | Opaque wrapper — pull `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` for the real error. For the schema-drift / compiled-relation-cache cause: rename `Model(...)` | - ```python - # Before (fails at scale) - pt = PropertyTransformer( - datetime=[Sale.date], - time_col=[Sale.date], # <-- remove - ... - ) - gnn = GNN(has_time_column=True, ..., temporal_strategy="last") # <-- both go - Train = Relationship(f"{Sale} at {Any:date} has {Any:value}") # <-- drop date arg - - # After (works) — keep date as a plain datetime feature; the - # temporal split lives in pandas (train_mask/val_mask/test_mask). - pt = PropertyTransformer( - datetime=[Sale.date], # convention: don't pass time_col= when has_time_column=False - ... - ) - gnn = GNN(has_time_column=False, ...) - Train = Relationship(f"{Sale} has {Any:value}") - model.define(Train(Sale, TrainTable.unit_sales)).where(...) - ``` - - The `PropertyTransformer` API itself accepts `time_col=` independent of `GNN(has_time_column=...)`; the rule "don't pass `time_col=` when `has_time_column=False`" is a convention to avoid dead annotations, not a code-enforced check. - -**Engine-side compiled-relation cache footgun (not datetime-specific — surfaces after any column-type change on a bound table):** the engine caches the compiled relation type per RAI relation name and doesn't invalidate it on `ALTER TABLE`, even after stream delete + recreate. Symptom: `Encountered reference to a base relation with a mismatched signature` — but the client only shows the opaque `Failed to pull data into index: transaction was aborted (runtime error)` wrapper. Pull the real error via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` (presigned URL) and look at the `report` field. Workaround: rename `Model(...)` to force a fresh RAI relation namespace (downstream queries that depend on the old name need updating). Better: do schema changes before the first bind — see `rai-predictive-modeling` § Populate from Snowflake. - -### Worker not ready to accept jobs - -`gnn.fit()` submits successfully (Step 3/3 logs `Training job submitted`) but the train job sits in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely, even though `GET_REASONER` returns `STATUS='READY'`. The SDK only checks reasoner-pod readiness via `api.get_reasoner` before submitting (`relationalai_gnns/core/connector.py::_check_engine_availability`); it has no notion of an in-pod worker queue, so a desynced worker on a READY pod is invisible to the client. The server-side error surfaces only when the SDK polls and the `CREATE_JOB` external function reports back: - -> Request failed for external function CREATE_JOB with remote service error: 400 `{"status":"Not Found","message":"worker is not ready to accept jobs - please retry the job later"}` - -For the SQL recovery runbook (suspend + resume + re-check), see `rai-health` § Predictive train jobs stuck QUEUED. After recovery, resubmit by re-instantiating `GNN(...)` — see § `gnn.fit()` is idempotent below for why bumping `Model("...")` name alone is not enough. - -### `gnn.fit()` is idempotent — re-instantiate `GNN(...)` on retry - -`gnn.fit()` is a silent no-op if `self.train_job` already exists and isn't `FAILED` (`relationalai/semantics/reasoners/predictive/estimator.py:483-490`). Calling `gnn.fit()` a second time on the same `GNN` Python object will **not** submit a new training job — it just logs `Training job already running/completed` and returns. The next `gnn.predictions(...)` call then resolves the *previous* `train_job.model_run_id` (which is the previous job's `job_id` per `relationalai_gnns/core/job_manager.py:132-146`). - -**Symptom this explains**: a re-run of a notebook cell or a retry after a killed mid-flight run reports `model_run_id` from a much-earlier job, not the freshly-submitted one. Subsequent prediction calls operate on stale model artifacts and can hang at "Step 2/4: Preparing model for prediction". - -**Workaround:** re-instantiate `GNN(...)` on every retry. Bumping `Model("...")` name is **not** the right fix for this specific bug (the SDK has no name-based experiment matching at the `_wait_obtain_model_run_id` layer) — that bump is the workaround for the engine-side compiled-relation cache footgun above, a separate issue. To force a fresh training run after a partial failure: - -```python -gnn = GNN(...) # build a new instance — required -gnn.fit() # submits a fresh job -``` - ---- - -## Troubleshooting - -### Stalled train job: SDK polls without a timeout - -`relationalai_gnns/core/job_manager.py::JobMonitor._wait_for_completion` (line 332-340) polls `get_status()` every 5 seconds with no timeout, no retry cap, and no max-poll-count. While the underlying job stays in `QUEUED` or `RUNNING`, the loop is unbounded. If the row is removed from `RELATIONALAI.API.JOBS` (history retention varies by Snowflake/native-app version), `get_status()` raises rather than self-terminating cleanly — but a row that *stays* QUEUED indefinitely will never trigger that exit path. - -If a `gnn.fit()` client has been polling for an unreasonable amount of time and `SELECT * FROM RELATIONALAI.API.JOBS WHERE ID = ''` shows the row is still QUEUED (or returns no row), kill the client manually. Use the SUSPEND/RESUME runbook in § Worker not ready to accept jobs to recover, then re-instantiate `GNN(...)` and resubmit. Do **not** rely on `RELATIONALAI.API.JOBS` for forensics on long-stalled jobs; capture state earlier with `GET_TRANSACTION_ARTIFACTS` or engine logs when investigating. +`CREATE_GNN_SERVICE()` is **not** the right escalation for any predictive train issue — the SDK submits training in-pod against the predictive reasoner, not via that legacy path (`relationalai_gnns/core/connector.py::exec_job`). See `rai-health` § Predictive train jobs stuck QUEUED. --- @@ -561,3 +513,4 @@ User.predictions = gnn.predictions(domain=Test) | Hyperparameters | Full hyperparameter table with types, defaults, and tuning guidance | [references/hyperparameters.md](references/hyperparameters.md) | | Prediction attributes | Prediction attributes by task type with usage examples | [references/prediction-attributes.md](references/prediction-attributes.md) | | Evaluation & debugging | Dataset inspection, result checking, and tuning steps | [references/evaluation-debugging.md](references/evaluation-debugging.md) | +| Known limitations & runtime troubleshooting | `has_time_column=True` failure-mode fallback code; SUSPEND/RESUME runbook; `gnn.fit()` idempotency; `JobMonitor._wait_for_completion` polling; `GET_TRANSACTION_ARTIFACTS` recipe — load when the symptom→fix table in SKILL.md is too compact | [references/known-limitations.md](references/known-limitations.md) | diff --git a/plugins/rai/skills/rai-predictive-training/references/known-limitations.md b/plugins/rai/skills/rai-predictive-training/references/known-limitations.md new file mode 100644 index 0000000..62fa854 --- /dev/null +++ b/plugins/rai/skills/rai-predictive-training/references/known-limitations.md @@ -0,0 +1,59 @@ +# GNN runtime troubleshooting (lookup) + +Quick symptom→fix lookup. Load when SKILL.md § Known Limitations & Runtime Troubleshooting needs the full code shape or SDK source citation. + +--- + +## `has_time_column=True` fails + +| Symptom | Cause | Fix | +|---|---|---| +| `no time column defined in data tables` | `time_col` propagates only for node concepts (with `identify_by`); your time-bearing concept is an edge intermediary | Below — full fallback | +| `ValidationError: Error processing datetime column ''` (often at scale, ~27K rows is enough) | Server-side datetime processor rejects the column despite clean data + correct `datetime`/`time_col` config | Below — full fallback | + +**Fallback (works for both):** drop `time_col=`, set `has_time_column=False`, drop `temporal_strategy=`, drop the date arg from `Train`/`Val`/`Test` Relationships. Keep the temporal split in pandas: + +```python +# Before +pt = PropertyTransformer(datetime=[Sale.date], time_col=[Sale.date], ...) +gnn = GNN(has_time_column=True, ..., temporal_strategy="last") +Train = Relationship(f"{Sale} at {Any:date} has {Any:value}") + +# After +pt = PropertyTransformer(datetime=[Sale.date], ...) +gnn = GNN(has_time_column=False, ...) +Train = Relationship(f"{Sale} has {Any:value}") +model.define(Train(Sale, TrainTable.unit_sales)).where(...) +``` + +--- + +## `transaction was aborted (runtime error)` (opaque wrapper) + +| Symptom | Cause | Fix | +|---|---|---| +| `Failed to pull data into index: transaction was aborted (runtime error)` after a column type change | Engine cached the compiled relation per name; `ALTER TABLE` + stream recreate doesn't invalidate it. Real error: `Encountered reference to a base relation with a mismatched signature` | `CALL RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` → presigned URL → `problems.json` → `report` field. Then rename `Model(...)` to force a fresh namespace, or do schema changes before the first bind | + +--- + +## Train job `QUEUED` while reasoner `READY` + +| Symptom | Cause | Fix | +|---|---|---| +| `worker is not ready to accept jobs - please retry the job later` (server-side, surfaces only on poll) | SDK only probes pod via `api.get_reasoner` (`relationalai_gnns/core/connector.py::_check_engine_availability`); in-pod worker can be desynced | `rai-health` § Predictive train jobs stuck QUEUED (suspend + resume the predictive reasoner). After recovery, re-instantiate `GNN(...)` and resubmit | + +--- + +## `gnn.fit()` returns a stale `model_run_id` + +| Symptom | Cause | Fix | +|---|---|---| +| Re-run reports `model_run_id` from a much-earlier job; prediction hangs at "Step 2/4: Preparing model for prediction" | `gnn.fit()` is a silent no-op when `self.train_job` exists and isn't `FAILED` (`estimator.py:483-490`); `predictions()` then resolves the previous `job_id` (`job_manager.py:132-146`) | Re-instantiate `GNN(...)` on every retry. Bumping `Model("...")` is **not** the right fix — that's the workaround for the cache footgun above | + +--- + +## Client polls forever + +| Symptom | Cause | Fix | +|---|---|---| +| `gnn.fit()` polling for an unreasonable amount of time | `JobMonitor._wait_for_completion` (`job_manager.py:332-340`) polls every 5s with no timeout/retry-cap | Kill the client manually. Recover via the QUEUED runbook, re-instantiate `GNN(...)`, resubmit | From 99b0263d9f1d1fe29678b311bbdcfb2b91713cbf Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 10:02:58 -0700 Subject: [PATCH 22/27] rai-discovery: route GNN use cases (link prediction, node classification, node regression) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Predictive-coupled discovery edits — apply the translation/routing table to the three product-supported task families and surface them in the SKILL.md tables that already enumerate predictive routing. Reasoner-agnostic discovery improvements (description/summary rewrite, per-reasoner skill load table, reference-row translation framing for prescriptive/graph/rules) ship separately on PR #39 since they help existing workflows without depending on the predictive skills. - references/predictive.md: replace the classification/regression/anomaly/ clustering question-type list with the three product-supported families (node_classification, node_regression, link_prediction); add the User-Type → GNN Task Type translation table mapping each user-facing type to the granular `task_type` / `eval_metric` / `has_time_column`; add `link_target_concept` field for link prediction; per-period forecasting routes to `regression` with `has_time_column=True` (not a separate task type). Anomaly/clustering are not supported natively — only emit as `pre_computed` if an external table exists. - examples/predictive_routing.md: add three GNN-mode walkthroughs (churn → node classification, unit output → node regression, recommendation → link prediction) carrying both the user-facing type and the technical GNN fields; update the existing pre-computed example to use the new `node_classification` value. - SKILL.md predictive rows in the Quick Reference, Reasoner Classification, Cumulative Discovery, Reference Files, Examples, and routing-fields tables — refreshed to the three-family vocabulary and naming rai-predictive-modeling / rai-predictive-training as handoff targets. --- plugins/rai/skills/rai-discovery/SKILL.md | 18 +-- .../examples/predictive_routing.md | 109 +++++++++++++++++- .../rai-discovery/references/predictive.md | 87 ++++++++++---- 3 files changed, 184 insertions(+), 30 deletions(-) diff --git a/plugins/rai/skills/rai-discovery/SKILL.md b/plugins/rai/skills/rai-discovery/SKILL.md index b66ca1a..48d50fc 100644 --- a/plugins/rai/skills/rai-discovery/SKILL.md +++ b/plugins/rai/skills/rai-discovery/SKILL.md @@ -41,7 +41,7 @@ description: Translation, ideation, and routing layer between an ontology and th |--------------------|----------|-----------------| | Constrained resources, costs, capacities | **Prescriptive** | "What should we do?" — allocate, schedule, route | | Network topology, graph structure | **Graph** | "What patterns exist?" — centrality, clusters, paths | -| Temporal data, features, historical outcomes | **Predictive** | "What will happen?" — forecast, classify | +| Labels/values per entity, historical pair data, graph topology | **Predictive** | "What will happen?" / "Which Y for each X?" — node classification, node regression, link prediction | | Threshold/status fields, business rules | **Rules** | "Is this valid?" — compliance, classification | | Feasibility | Meaning | Next Step | @@ -113,7 +113,7 @@ Each suggestion must be tagged with one or more reasoner types. Use these signal |--------|-----------------|------------------| | Optimizing decisions over constrained resources | **Prescriptive** | "What should we do?" — allocate, schedule, route, price | | Understanding structure, connectivity, influence | **Graph** | "What patterns exist?" — who is central, what clusters exist, shortest path | -| Predicting outcomes from features | **Predictive** | "What will happen?" — forecast, classify, detect anomalies | +| Predicting node labels/values or future links from features and graph topology | **Predictive** | "What will happen?" / "Which Y for each X?" — node classification, node regression, link prediction | | Enforcing business rules and logical constraints | **Rules** | "Is this valid?" — compliance, classification, derivation | **Disambiguation rules:** @@ -185,8 +185,9 @@ Each reasoner adds new concepts and properties to the ontology. Discovery should | Graph centrality | `node.centrality_score` | Predictive: centrality as feature. Prescriptive: weight allocation by node importance. | | Graph reachability | impact_count, affected flags | Prescriptive: minimize disruption to high-impact nodes. Rules: alert on critical dependencies. | | Graph WCC / community | WCC: `(node, component_id_node)` membership (access `.id` to get its identifying value; cast to `int` only for integer-identified nodes); community: `node.community_label` (int) | Prescriptive: optimize within-cluster vs cross-cluster. Rules: flag isolated components. | -| Predictive forecasting | `Forecast.predicted_value` | Prescriptive: optimize against predicted demand/delays. | -| Predictive classification | `Entity.risk_probability` | Rules: flag above threshold. Prescriptive: incorporate risk as constraint. | +| Predictive node classification | `Entity.predictions` with `.probs`, `.predicted_labels` | Rules: flag above threshold. Prescriptive: incorporate risk/class as constraint. | +| Predictive node regression | `Entity.predictions.predicted_value` (incl. per-period forecasts) | Prescriptive: optimize against predicted values, often via aggregation/bridge concept. | +| Predictive link prediction | `User.predictions` with `.rank`, `.scores`, `.predicted_` | Prescriptive: top-K predicted pairs as candidate edges in assignment/matching. Rules: flag pairs above score threshold. | ### How to suggest cumulative questions @@ -400,7 +401,7 @@ Each suggestion includes a `reasoners` field — an ordered list specifying the | **prescriptive** | `decision_scope`, `forcing_requirement`, `objective_property`, `decision_variable`, `scenario_parameter`, `competing_objectives` | | **graph** | `algorithm`, `graph_construction` (`node_concept`, `directed`, `weighted`, `edge_definition`), `target_filter`, `output_binding` | | **rules** | `rule_type`, `source_concept`, `condition_properties`, `join_path`, `threshold`, `output_type`, `output_property`, `downstream_use` | -| **predictive** | `type`, `mode` (`pre_computed` or `rai_predictive`), `target_concept`, `target_property`, `feature_properties`, `output_concept`, `pre_computed_table` | +| **predictive** | User-facing: `type` (`node_classification` \| `node_regression` \| `link_prediction`), `mode` (`pre_computed` \| `rai_predictive`). Concept routing: `target_concept`, `target_property` (classification/regression), `link_target_concept` (link prediction only), `feature_properties`, `output_concept`, `pre_computed_table`. GNN task routing (for `rai_predictive` mode): `task_type` (`binary_classification` \| `multiclass_classification` \| `multilabel_classification` \| `regression` \| `link_prediction` \| `repeated_link_prediction`), `eval_metric`, `has_time_column`, `temporal_column` (when `has_time_column=True`). See `predictive.md` for the user-type → task_type translation rules. | **For chained questions**, use a `stages` array in `implementation_hint`: @@ -433,11 +434,12 @@ Each suggestion includes a `reasoners` field — an ordered list specifying the |-------------------|---------------------------| | **prescriptive** | `rai-prescriptive-problem-formulation` → `rai-prescriptive-solver-management` → `rai-prescriptive-results-interpretation` | | **graph** | `rai-graph-analysis` | +| **predictive** | `rai-predictive-modeling` → `rai-predictive-training` | | **rules** | `rai-rules-authoring` | For all reasoners, also load `rai-querying` + `rai-pyrel-coding` for v1 syntax, imports, and query patterns. If the selected question is **MODEL_GAP**, load `rai-ontology-design` first to enrich the ontology before the reasoner skill runs (see Enrichment Handoff above). -Discovery covers *what* to ask. The reasoner-specific reference files in this skill (`prescriptive.md` / `graph.md` / `predictive.md` / `rules.md`) translate the user's framing into the technical fields each downstream skill consumes (problem_type / algorithm / rule_type). The downstream coding skills cover *how* to write the PyRel. Skipping the coding-skill load leads to hallucinated APIs and wrong imports. +Discovery covers *what* to ask. The reasoner-specific reference files in this skill (`prescriptive.md` / `graph.md` / `predictive.md` / `rules.md`) translate the user's framing into the technical fields each downstream skill consumes (problem_type / algorithm / task_type / rule_type). The downstream coding skills cover *how* to write the PyRel. Skipping the coding-skill load leads to hallucinated APIs and wrong imports. --- @@ -461,7 +463,7 @@ Discovery covers *what* to ask. The reasoner-specific reference files in this sk |-----------|-------------|------| | Prescriptive | Optimization problem types (resource allocation, network flow, routing, scheduling, pricing) → translate into formulation parameters for `rai-prescriptive-problem-formulation` | [prescriptive.md](references/prescriptive.md) | | Graph | Graph question types (centrality, community, reachability, distance, similarity) → translate into RAI Graph algorithms for `rai-graph-analysis` | [graph.md](references/graph.md) | -| Predictive | Predictive modeling — forecasting, classification, anomaly detection | [predictive.md](references/predictive.md) | +| Predictive | User-facing predictive types (node classification, node regression, link prediction) → translate into GNN `task_type` / `eval_metric` / `has_time_column` for `rai-predictive-modeling` and `rai-predictive-training` | [predictive.md](references/predictive.md) | | Rules | Rule question types (validation, classification, derivation, alerting, reconciliation) → translate into `rule_type` and PyRel patterns for `rai-rules-authoring` | [rules.md](references/rules.md) | --- @@ -473,5 +475,5 @@ Discovery covers *what* to ask. The reasoner-specific reference files in this sk | Prescriptive routing | Discovery scenario walkthrough for optimization problems | [prescriptive_routing.md](examples/prescriptive_routing.md) | | Graph routing | Discovery scenario walkthrough for graph analytics | [graph_routing.md](examples/graph_routing.md) | | Rules routing | Discovery scenario walkthrough for classification, validation, and derivation rules | [rules_routing.md](examples/rules_routing.md) | -| Predictive routing | Discovery scenario walkthrough for predictive modeling | [predictive_routing.md](examples/predictive_routing.md) | +| Predictive routing | Discovery walkthroughs for node classification, node regression, link prediction (GNN mode) and pre-computed predictions | [predictive_routing.md](examples/predictive_routing.md) | | Chained routing | Discovery scenario walkthrough for multi-reasoner pipelines | [chained_routing.md](examples/chained_routing.md) | diff --git a/plugins/rai/skills/rai-discovery/examples/predictive_routing.md b/plugins/rai/skills/rai-discovery/examples/predictive_routing.md index bdd650d..4f291db 100644 --- a/plugins/rai/skills/rai-discovery/examples/predictive_routing.md +++ b/plugins/rai/skills/rai-discovery/examples/predictive_routing.md @@ -19,7 +19,7 @@ Discovery-to-routing walkthroughs for predictive reasoner questions. Each exampl ### Implementation hint ```json -{"type": "classification", "mode": "pre_computed", +{"type": "node_classification", "mode": "pre_computed", "target_concept": "Entity", "target_property": "risk_probability", "output_concept": "RiskPrediction", "output_properties": ["predicted_risk_prob", "risk_tier", "confidence"], @@ -41,3 +41,110 @@ Discovery-to-routing walkthroughs for predictive reasoner questions. Each exampl This prediction output enables prescriptive chains: - "Given predicted risks, how should we re-allocate to minimize cost?" → predictive → prescriptive - "Set reliability threshold at 80% — exclude entities below" → downstream prescriptive uses `RiskPrediction.predicted_risk_prob` as a reliability parameter + +--- + +## "Will this customer churn in the next period?" (GNN node classification) + +### Ontology signals +- `Customer` graph node concept with feature properties (`tenure`, `plan_type`, `monthly_spend`, ...) +- Edges from `Customer` to neighbor entities — e.g. `Interaction` joining customers to support tickets, products, or other customers — provide structural signal +- Historical labeled split tables in Snowflake: `CHURN_TRAIN`, `CHURN_VAL`, `CHURN_TEST` with `customer_id`, optional `as_of_date`, and a `churned` label on train/val +- No pre-computed `predicted_churn_*` table in schema → must train a GNN + +### Reasoner classification: Predictive (`rai_predictive`, node classification) +- Categorical target (`churned`) on a node concept embedded in a graph → node classification +- Graph topology around `Customer` is meaningful → `rai_predictive` (GNN) is a strong fit, not flat-table classification +- NOT rules (no fixed threshold; learned from history) +- NOT graph (predicting a future label, not summarizing current structure) + +### Implementation hint +```json +{"type": "node_classification", "mode": "rai_predictive", + "target_concept": "Customer", "target_property": "churned", + "feature_properties": ["Customer.tenure", "Customer.plan_type", "Customer.monthly_spend"], + "task_type": "binary_classification", "eval_metric": "roc_auc", + "has_time_column": true, + "train_table": "CHURN_TRAIN", "val_table": "CHURN_VAL", "test_table": "CHURN_TEST", + "output_concept": "Customer.predictions", + "output_properties": ["probs", "predicted_labels"]} +``` + +### Reasoner handoff +- → `rai-predictive-modeling`: define `Customer` graph concept, edges to neighbors, task-table concepts, `f"{Customer} at {Any:ts} has {Any:label}"` train/val Relationships +- → `rai-predictive-training`: `GNN(..., task_type="binary_classification", eval_metric="roc_auc", has_time_column=True)`, then `Customer.predictions = gnn.predictions(domain=Test)` + +### Cumulative discovery note +- "Flag customers above 70% predicted churn for retention outreach" → predictive → rules +- "Allocate retention budget to highest-churn-probability segments" → predictive → prescriptive + +--- + +## "What will each unit's output be next period?" (GNN node regression) + +### Ontology signals +- `Unit` graph node concept with continuous and categorical features +- Edges from `Unit` to related entities (e.g., `BelongsTo` → `Site`, `Operates` → `Equipment`) — graph topology informs the prediction +- Historical labeled split tables with `unit_id`, `period`, and a numeric `output_value` on train/val +- No pre-computed forecast table → train a GNN regression model + +### Reasoner classification: Predictive (`rai_predictive`, node regression) +- Numeric target on a graph node concept → node regression +- Graph topology around `Unit` carries signal → `rai_predictive` mode +- NOT forecasting on a time series alone (per-unit prediction, not whole-series) +- NOT prescriptive (predicting a value, not deciding allocation) + +### Implementation hint +```json +{"type": "node_regression", "mode": "rai_predictive", + "target_concept": "Unit", "target_property": "output_value", + "feature_properties": ["Unit.capacity", "Unit.age", "Site.region"], + "task_type": "regression", "eval_metric": "rmse", + "has_time_column": true, + "train_table": "OUTPUT_TRAIN", "val_table": "OUTPUT_VAL", "test_table": "OUTPUT_TEST", + "output_concept": "Unit.predictions", + "output_properties": ["predicted_value"]} +``` + +### Reasoner handoff +- → `rai-predictive-modeling`: `Unit` concept, edges, task-table concepts, `f"{Unit} at {Any:ts} has {Any:value}"` Relationships +- → `rai-predictive-training`: `GNN(..., task_type="regression", eval_metric="rmse", has_time_column=True)` + +### Cumulative discovery note +- "Allocate inputs across units to maximize total predicted output subject to capacity" → predictive → prescriptive (often via aggregation/bridge concept; see `rai-predictive-training` § Aggregation and bridge concepts) + +--- + +## "Which products should we recommend to each user?" (GNN link prediction) + +### Ontology signals +- Two graph node concepts: `User` and `Item`, with feature properties on each +- `Interaction` concept joining `User` × `Item` over time (purchases, views, ratings) — the historical edge set +- Split tables for link-prediction: `LINK_TRAIN(user_id, ts, item_id)`, `LINK_VAL(user_id, ts, item_id)`, `LINK_TEST(user_id, ts)` +- Verified flat format: `item_id` is a scalar column, not a `VARIANT` array — see `rai-predictive-modeling` § Link Prediction — Task Table Format Requirements (VARIANT check) + +### Reasoner classification: Predictive (`rai_predictive`, link prediction) +- "Which Y for each X" / "recommend" / "predict pair" → link prediction, not classification or regression +- Two node concepts joined by historical pair data → `rai_predictive` `link_prediction` (or `repeated_link_prediction` with time) +- NOT graph (graph reasons over current edges; link prediction predicts missing or future edges) +- NOT rules (no deterministic rule for what to recommend) + +### Implementation hint +```json +{"type": "link_prediction", "mode": "rai_predictive", + "target_concept": "User", "link_target_concept": "Item", + "feature_properties": ["User.locale", "User.tenure", "Item.category", "Item.price"], + "task_type": "repeated_link_prediction", "eval_metric": "link_prediction_precision@5", + "has_time_column": true, + "train_table": "LINK_TRAIN", "val_table": "LINK_VAL", "test_table": "LINK_TEST", + "output_concept": "User.predictions", + "output_properties": ["rank", "scores", "predicted_item"]} +``` + +### Reasoner handoff +- → `rai-predictive-modeling`: `User` and `Item` concepts, `Interaction` edges, task-table concepts, `f"{User} at {Any:ts} has {Item}"` train/val Relationships, `DESCRIBE TABLE` on all three split tables to confirm scalar (non-VARIANT) target columns +- → `rai-predictive-training`: `GNN(..., task_type="repeated_link_prediction", eval_metric="link_prediction_precision@5", head_layers=2, num_negative=20, label_smoothing=True)` + +### Cumulative discovery note +- "Assign top-K predicted items per user subject to inventory and per-item exposure caps" → predictive → prescriptive (treat predicted pairs as candidate edges in an assignment problem) +- "Alert when a high-value user has no item with predicted score above 0.8" → predictive → rules diff --git a/plugins/rai/skills/rai-discovery/references/predictive.md b/plugins/rai/skills/rai-discovery/references/predictive.md index d1ca739..734d7b6 100644 --- a/plugins/rai/skills/rai-discovery/references/predictive.md +++ b/plugins/rai/skills/rai-discovery/references/predictive.md @@ -1,5 +1,6 @@ - [Prediction Question Types](#prediction-question-types) +- [User-Type → GNN Task Type Translation](#user-type--gnn-task-type-translation) - [Predictive Implementation Hints](#predictive-implementation-hints) - [When Predictive vs Other Reasoners](#when-predictive-vs-other-reasoners) - [Pre-Computed Predictions Pattern](#pre-computed-predictions-pattern) @@ -9,23 +10,48 @@ ## Prediction Question Types -Predictive reasoning uses historical data patterns to forecast outcomes, classify entities, or detect anomalies. +Predictive reasoning uses historical data patterns to predict labels, values, or links on a graph. **Two modes:** Predictive capabilities can be delivered via **pre-computed prediction tables** (external ML outputs loaded into Snowflake) or via the **RAI predictive pipeline** (GNN-based models trained directly on the knowledge graph — see `rai-predictive-modeling` and `rai-predictive-training`). Discovery should identify both pre-computed predictions already in the data and predictive questions the data could support via GNN training. +The RAI predictive reasoner supports three task families. Anything outside these (anomaly detection, clustering, time-series forecasting on aggregate signals) is not supported natively — surface it only when a pre-computed table already exists. + | Type | Question Pattern | Ontology Signal | |------|-----------------|-----------------| -| **Classification / Risk Scoring** | "Which category / risk tier does X belong to?" | Categorical target, labeled historical data, status/tier fields | -| **Regression** | "How much / what value will X be?" | Numeric target + numeric/categorical features, historical actuals | -| **Forecasting** | "What will happen next period?" | Temporal properties (date, time_period, period index), time-series data | -| **Anomaly Detection** | "What's unusual or unexpected?" | Many numeric properties, historical baselines, status/flag fields | -| **Clustering** | "What natural segments exist?" (unsupervised) | Many numeric/categorical properties, no obvious target variable | +| **Node Classification / Risk Scoring** | "Which category / risk tier does X belong to?" / "Will X churn?" / "Will X be next period?" (per-node binary/multiclass/multilabel) | Categorical target on a graph node concept, labeled historical data, status/tier fields. Temporal flavor: same plus a time column on the node. | +| **Node Regression** | "How much / what value will X be?" / "Forecast X's value next period" (per-node regression, with or without time) | Numeric target on a graph node concept + numeric/categorical features, historical actuals. Temporal flavor adds a time column on the node. | +| **Link Prediction / Recommendation** | "Will X be linked to Y?" / "Which Ys should we recommend for each X?" / "Which pairs will interact next period?" | Two node concepts joined by an interaction/edge concept; historical pair data; optionally a timestamp on the edge | **Disambiguation rules:** - "What will happen?" with historical labeled data → predictive - "What should we do about it?" → predictive → prescriptive chain - Deterministic classification from known thresholds (e.g., "high-value if spend > $10K") → rules or derived property, NOT predictive - "What patterns exist in the network?" → graph, not predictive (unless using graph features for prediction) +- "Will edges form / which pairs are likely to interact / recommend Y for X" → predictive (link prediction), not graph; graph reasons over the **current** topology, link prediction predicts **future or missing** edges +- Per-period numeric prediction ("forecast next month's demand for each unit") → node regression with a time column (`has_time_column=True` on the GNN), not a separate forecasting task type +- Node classification or node regression with rich graph topology around the predicted entity → strong fit for `rai_predictive` (GNN) mode; flat-table classification/regression with no meaningful graph structure can still use `rai_predictive` but loses the GNN's structural advantage + +--- + +## User-Type → GNN Task Type Translation + +Discovery is the translation layer. Classify the user's question into one of three user-facing `type` values, then resolve the granular GNN `task_type` and `eval_metric` from the data signals below. Downstream `rai-predictive-modeling` and `rai-predictive-training` consume the technical fields directly. + +| User-facing `type` | Sub-signal | GNN `task_type` | Default `eval_metric` | `has_time_column` | +|---|---|---|---|---| +| `node_classification` | Two-class label (boolean / 0-1) | `binary_classification` | `roc_auc` | from data | +| `node_classification` | Single label, 3+ mutually exclusive classes | `multiclass_classification` | `accuracy` | from data | +| `node_classification` | Multiple non-exclusive labels per row (e.g. tag set) | `multilabel_classification` | `multilabel_auprc_macro` | from data | +| `node_regression` | Numeric target, no per-row time column | `regression` | `rmse` | `False` | +| `node_regression` | Numeric target, per-period or per-timestamp prediction (a.k.a. "forecasting") | `regression` | `rmse` | `True` | +| `link_prediction` | Predict pairs, no time column | `link_prediction` | `link_prediction_precision@5` | `False` | +| `link_prediction` | Predict pairs over time / "what will X interact with next period" | `repeated_link_prediction` | `link_prediction_precision@5` | `True` | + +Resolution rules: +- **Label cardinality** decides binary vs multiclass vs multilabel — inspect the train table's label column (or the schema description) before emitting `task_type`. +- **`has_time_column`** is true iff the train/val Relationship template carries an `at {Any:ts}` slot — equivalently, iff the task table has a per-row timestamp column the user wants the model to condition on. +- **Forecasting** is not a separate `task_type`. A "forecast next month's value per unit" question is `node_regression` → `regression` + `has_time_column=True` + a `temporal_column`. +- See `rai-predictive-training/references/task-types-and-metrics.md` for the full set of valid `(task_type, eval_metric)` pairs and `rai-predictive-modeling/references/task-relationships.md` for the matching Relationship templates. --- @@ -34,7 +60,11 @@ Predictive reasoning uses historical data patterns to forecast outcomes, classif For each predictive suggestion, provide an implementation hint with these fields: ### type -Problem type: `classification`, `regression`, `forecasting`, `anomaly_detection`, `clustering`. +Problem type — one of `node_classification`, `node_regression`, `link_prediction`. These are the only families the RAI predictive reasoner supports. + +For `rai_predictive` (GNN) mode these map to the training-skill `task_type` values: `node_classification` → `binary_classification` | `multiclass_classification` | `multilabel_classification`; `node_regression` → `regression`; `link_prediction` → `link_prediction` (no time) or `repeated_link_prediction` (with time). See `rai-predictive-training` § Quick Reference. + +Per-period numeric forecasting is `node_regression` with `has_time_column=True`, not a separate type. Anomaly detection and clustering are not supported — only emit them as `pre_computed` if an external ML table exists. ### mode How prediction is delivered: @@ -42,15 +72,20 @@ How prediction is delivered: - **`rai_predictive`**: Build and train a graph neural network (GNN) using the RAI predictive pipeline (**early access** — APIs and behavior may change). See `rai-predictive-modeling` for data modeling and `rai-predictive-training` for training and evaluation. ### target_concept / target_property -What to predict. E.g., `Entity` / `risk_value`, or `Customer` / `churn_flag`. +What to predict. +- **Node classification / regression / forecasting:** `target_concept` is the node concept being predicted on; `target_property` is the label/value (e.g., `Customer` / `churn_flag`, `Entity` / `risk_value`). +- **Link prediction:** `target_concept` is the **source** node (head of the link, e.g., `User`); `target_property` does not apply. Add `link_target_concept` for the linked-to node type. + +### link_target_concept (for link prediction only) +The destination node concept of the predicted link (e.g., `Item`). Maps to the `Target` slot in the GNN task relationship template `f"{Source} has {Target}"` and to the `target_concept` constructor argument when loading a link-prediction model. See `rai-predictive-modeling` § Task Relationships. ### feature_properties (for rai_predictive mode) Which ontology properties serve as model inputs. E.g., `["Entity.reliability_score", "Activity.quantity", "Resource.category"]`. -### temporal_column (for forecasting) -Which property provides the time dimension. E.g., `Activity.time_period`, `Order.order_date`. +### temporal_column (for time-aware tasks) +Which property provides the time dimension. Required when emitting `node_regression` or `node_classification` with `has_time_column=True`, or `repeated_link_prediction`. E.g., `Activity.time_period`, `Order.order_date`. -### prediction_horizon (for forecasting) +### prediction_horizon (optional, for time-aware tasks) What future period to predict. E.g., "next period", "next 30 days". ### output_concept / output_properties @@ -66,7 +101,11 @@ Which schema table contains the predictions. E.g., `RISK_PREDICTION`. - Historical labeled data + "what will happen?" → **predictive** - "What should we do about the prediction?" → **predictive → prescriptive** chain - Deterministic rules for classification (threshold-based, no learning) → **rules** or derived ontology property, not predictive -- "What patterns exist in the network?" → **graph**, not predictive (unless using graph features for prediction) +- "What patterns exist in the network?" (current topology) → **graph**, not predictive +- "Will edges form / which pairs interact next / recommend Y for X" → **predictive link prediction** (`rai_predictive` mode, `task_type=link_prediction` or `repeated_link_prediction`) +- Node-level label/value prediction on entities embedded in a graph → **predictive `rai_predictive` mode** (node classification / node regression) +- Per-period numeric prediction → **node regression with `has_time_column=True`**, not a separate forecasting task +- "Find anomalies / cluster entities" → **not supported by the GNN**; only emit as `pre_computed` if an external scoring/clustering table already exists in the schema - Pre-computed prediction table exists in schema → **predictive (pre_computed mode)** — suggest downstream use **Common chains involving predictive:** @@ -89,7 +128,7 @@ Look for tables or concepts with columns/properties like: ### Discovery behavior for pre-computed predictions When a prediction table is detected: -1. **Classify it** — what kind of prediction is it? (classification, regression, forecasting) +1. **Classify it** — what kind of prediction is it? Map to one of the supported types (`node_classification`, `node_regression`, `link_prediction`). Anomaly scores or cluster labels from external systems are still valid pre-computed inputs even though they aren't supported `rai_predictive` task types — describe them with the closest match (`node_classification` for category labels, `node_regression` for continuous scores). 2. **Identify downstream use** — which other reasoners can consume this data? - Prescriptive: use predicted values as parameters/constraints (e.g., predicted risk probability as reliability threshold) - Rules: use predicted categories for alerting (e.g., flag "HIGH" risk tier entities) @@ -110,10 +149,9 @@ Predictive reasoning (whether pre-computed or via the RAI predictive pipeline) a | Prediction Type | Output Concept | Downstream Use | |----------------|----------------|----------------| -| Risk classification | `RiskPrediction` with probability, risk_tier | Prescriptive: reliability constraint. Rules: risk alerting. | -| Demand forecasting | `DemandForecast` with predicted_quantity | Prescriptive: demand parameter for allocation/inventory. | -| Churn classification | `ChurnProbability` with probability | Prescriptive: optimize retention resource allocation. | -| Anomaly scoring | `AnomalyScore` with score | Rules: flag entities above threshold. | +| Node classification | `Entity.predictions` with `.probs`, `.predicted_labels` (bound on the source concept), or pre-computed concepts like `RiskPrediction` / `ChurnProbability` | Prescriptive: reliability constraint, retention allocation. Rules: risk alerting, threshold flags. | +| Node regression (incl. per-node forecasting) | `Entity.predictions.predicted_value` (bound on the source concept), or pre-computed `DemandForecast`-style concepts | Prescriptive: predicted value as a constraint parameter or objective coefficient (often via aggregation/bridge concept). | +| Link prediction / recommendation | `User.predictions` with `.rank`, `.scores`, `.predicted_` (bound on the source concept) | Prescriptive: top-K predicted pairs as candidate edges in an assignment/matching problem. Rules: flag predicted pairs above a score threshold. | These outputs are available for cumulative discovery — prescriptive problems that need predicted parameters become feasible once prediction data exists. @@ -129,10 +167,17 @@ What ontology patterns indicate prediction potential: - Check if the prediction table links to other ontology concepts via FK (e.g., entity_id linking predictions to Entity concept) ### For rai_predictive mode (GNN training) +- **Graph topology**: At least one edge concept (FK-joined or via an interaction concept) connecting the predicted-on entity to other entities — the GNN's structural advantage comes from this. Without graph structure, node classification/regression still trains but loses most of the GNN signal. - **Feature availability**: Target property with sufficient non-null values; 3+ candidate features with variance -- **Temporal span**: For forecasting, at least 2 full cycles of the target period (period-level prediction needs 2+ periods of history) -- **Label quality**: For classification, labels exist and are reasonably balanced (flag extreme imbalance like 99%/1%) -- **Row count**: Rough minimums (regression 50+, classification 30+ per class, forecasting 2+ full periods) +- **Temporal span**: For temporal node classification/regression and `repeated_link_prediction`, at least 2 full cycles of the target period +- **Label quality**: For node classification, labels exist and are reasonably balanced (flag extreme imbalance like 99%/1%) +- **Pair history (link prediction)**: Historical (source, target [, timestamp]) rows in a flat task table — one row per pair, target column is a scalar ID, not a `VARIANT` array. See `rai-predictive-modeling` § Link Prediction — Task Table Format Requirements (VARIANT check). +- **Row count**: Rough minimums (node regression 50+, node classification 30+ per class, link prediction 100+ pairs) - **Feature-target relationship**: At least some features plausibly related to target (domain signal) -**Minimum viable ontology for prediction:** For pre-computed: a prediction table exists and links to other concepts. For rai_predictive (GNN): at least one concept with a target property (what to predict) and 2+ feature properties (what to predict from), backed by sufficient historical data. See `rai-predictive-modeling` for the full data modeling workflow. +**Minimum viable ontology for prediction:** +- *Pre-computed:* a prediction table exists and links to other concepts. +- *rai_predictive (node classification / node regression):* a graph node concept with a target property (what to predict) and 2+ feature properties, plus at least one edge concept tying it to neighbor entities. +- *rai_predictive (link prediction):* two graph node concepts plus historical (source, target) pair rows in a flat task table; optional timestamp for `repeated_link_prediction`. + +See `rai-predictive-modeling` for the full data modeling workflow and `rai-predictive-training` for the task-type/eval-metric matrix. From 86677c40250ddb588948808be87ea4d41ab57364 Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 14:41:33 -0700 Subject: [PATCH 23/27] predictive: sizing, timing, two-engine model, stuck-diagnostic ladder, discovery precheck MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Filling four gaps validated against the gnn3 venv (relationalai 1.1.1 editable + relationalai_gnns 0.1.5): - rai-predictive-modeling: add a Two-engine model section (Logic for data ingest/queries/exports vs Predictive for fit/predictions) and an Engine sizing section with CPU vs GPU heuristics tied to graph scale. Calls out the CLI-vs-backend allow-list gap on GPU sizes — REASONER_SIZES_AWS in services/reasoners/constants.py lists CPU only, while the AWSEngineSize Literal in config_reasoners_fields.py accepts GPU_NV_S — and points at the async API as the fall-through. - rai-predictive-training: add a Timing expectations table that distinguishes stream_logs=True (default; fit() blocks synchronously via _stream_logs_formatted) from stream_logs=False (returns at submit; predictions() then waits via _wait_obtain_model_run_id). Both modes block in predictions(). Add a short "Training appears stuck" pointer to the new diagnostic ladder in references/known-limitations.md. - references/known-limitations.md: new "Training appears stuck" three-step ladder (GET_REASONER → jobs.list → SHOW EXPERIMENTS) that localizes failure to one component before suspending anything. - rai-health: enhance the existing Predictive-stuck-QUEUED section with the same three-step diagnostic-ladder framing parallel to the Logic / CDC ladders. Recovery (SUSPEND/RESUME) preserved as the second half. - rai-discovery references/predictive.md: precheck note in Data Sufficiency Signals (rai_predictive mode) that classifying a question as rai_predictive-feasible requires confirming the Predictive reasoner is provisioned and READY — most accounts default to Logic only. The general engine-management surface (api.CREATE_REASONER_ASYNC + poll pattern, Predictive row in rai-setup reasoners.md, ban on EXPERIMENTAL.* procs) ships separately as a setup-management PR — that material reaches all reasoner families and isn't predictive-coupled. --- .../rai-discovery/references/predictive.md | 1 + plugins/rai/skills/rai-health/SKILL.md | 14 +++++++- .../skills/rai-predictive-modeling/SKILL.md | 22 ++++++++++++ .../skills/rai-predictive-training/SKILL.md | 16 +++++++++ .../references/known-limitations.md | 36 +++++++++++++++++++ 5 files changed, 88 insertions(+), 1 deletion(-) diff --git a/plugins/rai/skills/rai-discovery/references/predictive.md b/plugins/rai/skills/rai-discovery/references/predictive.md index 734d7b6..2d4cd1d 100644 --- a/plugins/rai/skills/rai-discovery/references/predictive.md +++ b/plugins/rai/skills/rai-discovery/references/predictive.md @@ -167,6 +167,7 @@ What ontology patterns indicate prediction potential: - Check if the prediction table links to other ontology concepts via FK (e.g., entity_id linking predictions to Entity concept) ### For rai_predictive mode (GNN training) +- **Predictive engine provisioned**: `rai_predictive` mode requires a Predictive reasoner — it is opt-in and most accounts default to Logic only. Before classifying a question as `rai_predictive`-feasible, confirm `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` returns `STATUS=READY`. If no Predictive reasoner exists yet, treat the question as feasible-after-provisioning and surface that as the next step (see `rai-predictive-modeling` § Two-engine model + Engine sizing). Pre-computed mode is unaffected — it runs on Logic. - **Graph topology**: At least one edge concept (FK-joined or via an interaction concept) connecting the predicted-on entity to other entities — the GNN's structural advantage comes from this. Without graph structure, node classification/regression still trains but loses most of the GNN signal. - **Feature availability**: Target property with sufficient non-null values; 3+ candidate features with variance - **Temporal span**: For temporal node classification/regression and `repeated_link_prediction`, at least 2 full cycles of the target period diff --git a/plugins/rai/skills/rai-health/SKILL.md b/plugins/rai/skills/rai-health/SKILL.md index 094ea5e..37fed04 100644 --- a/plugins/rai/skills/rai-health/SKILL.md +++ b/plugins/rai/skills/rai-health/SKILL.md @@ -319,7 +319,19 @@ for the full step-by-step recovery checklist, schema reference, and official doc A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The SDK only checks reasoner-pod status before submitting — the in-pod worker queue can be out of sync with that status, and the SDK has no way to detect it (`relationalai_gnns/core/connector.py::_check_engine_availability`). -**Recovery (empirical):** suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Do **not** call `CREATE_GNN_SERVICE()`. +### Diagnostic ladder + +Long-running predictive jobs are usually fine — distinguish stuck from slow before suspending anything. Use the same ladder as `rai-predictive-training` § "Training appears stuck": + +1. `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` → `STATUS=READY`? +2. `client.jobs.list("Predictive", name="")` → is there a `RUNNING` train job (with rising `AGE_MIN`), or a `QUEUED` one going stale? +3. `SHOW EXPERIMENTS IN SCHEMA .` → did a new experiment row append within ~60s of the `RUNNING` train job? + +A `QUEUED` train job that won't advance while the reasoner reports `READY` is the worker-queue desync this section addresses. Genuine long runs progress through (1) READY → (2) RUNNING → (3) new experiment row. + +### Recovery + +Suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Do **not** call `CREATE_GNN_SERVICE()`. ```sql -- 1. Confirm a stuck train job diff --git a/plugins/rai/skills/rai-predictive-modeling/SKILL.md b/plugins/rai/skills/rai-predictive-modeling/SKILL.md index f6aeec5..e7e49fd 100644 --- a/plugins/rai/skills/rai-predictive-modeling/SKILL.md +++ b/plugins/rai/skills/rai-predictive-modeling/SKILL.md @@ -65,6 +65,28 @@ The error is a `PermissionError`, not a generic `RuntimeError` — code that wra The predictive submodule (`relationalai.semantics.reasoners.predictive`) is not in every published `relationalai` release — `from relationalai.semantics.reasoners.predictive import GNN` raises `ModuleNotFoundError` on releases that pre-date it. Pin a release that ships the submodule (or install from the development branch when iterating against unreleased changes). +### Two-engine model: Logic + Predictive + +A GNN workflow runs against **two distinct reasoner engines** that must both be `READY`: + +| Reasoner | Handles | Why it matters here | +|----------|---------|---------------------| +| **Logic** | `model.data()` / `Table().to_schema()` ingest, all PyRel queries (including `select(...)` over `Source.predictions`), data exports back to Snowflake | The data pipeline that feeds the GNN and reads predictions back is Logic-engine work | +| **Predictive** | `gnn.fit()` training, `gnn.predictions()` inference, experiment + model-registry writes | Where the actual GNN training and inference happen | + +When training "hangs" or queries are slow, the first question is *which engine* — they have separate sizes, separate `STATUS`, separate auto-suspend timers. `rai-health` § Predictive train jobs stuck QUEUED covers the Predictive side; the Logic-engine ladder lives in `rai-health` Steps 1–3. + +### Engine sizing + +The Predictive reasoner accepts both CPU (`HIGHMEM_X64_S` / `_M` / `_L`) and GPU (`GPU_NV_S`, …) sizes. The CLI's allow-list trails the backend — `REASONER_SIZES_AWS` in `relationalai/services/reasoners/constants.py` currently lists CPU sizes only, while the backend's `AWSEngineSize` Literal in `config_reasoners_fields.py` accepts `GPU_NV_S`. If `rai reasoners:create` rejects a GPU size, fall through to `CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', PARSE_JSON('{}'))` directly. + +Rough heuristics for choosing CPU vs GPU: + +- **CPU (HIGHMEM_X64_*)**: prototyping, single-task GNNs on graphs under ~100K nodes / ~1M edges, runs where you're iterating on features more than scaling out +- **GPU (`GPU_NV_S`+)**: production-leaning runs at ~1M+ nodes or ~10M+ edges, multi-epoch training over rich feature sets, link-prediction with large negative sampling + +GPU is faster per epoch when the dataset fits in the GPU VM's CPU memory; if the dataset is borderline, CPU `HIGHMEM_X64_L`/`_SL` may finish sooner overall than GPU paging. Confirm current sizing tradeoffs with the RelationalAI team — pool capacity and price points evolve. + --- ## Quick Reference diff --git a/plugins/rai/skills/rai-predictive-training/SKILL.md b/plugins/rai/skills/rai-predictive-training/SKILL.md index c3b37fc..b991e80 100644 --- a/plugins/rai/skills/rai-predictive-training/SKILL.md +++ b/plugins/rai/skills/rai-predictive-training/SKILL.md @@ -185,6 +185,22 @@ For all hyperparameters and tuning guidance, see [references/hyperparameters.md] 2. Model training over `n_epochs` 3. Evaluation on the validation set +### Timing expectations + +`gnn.fit()` and `gnn.predictions()` both submit Snowpark Container Services jobs that can run for many minutes — the long quiet between submission and completion is **expected**, not stuck. + +| Mode | Behavior | +|------|----------| +| `stream_logs=True` (default) | `fit()` blocks until training completes — log streaming runs synchronously inside `fit()` (`relationalai.semantics.reasoners.predictive.estimator._stream_logs_formatted`). The console silence after "Training job submitted" is the streamer waiting on log buffers, not a stalled client. | +| `stream_logs=False` | `fit()` returns shortly after submission with "Job submitted and running in background." `predictions()` then waits — `_wait_obtain_model_run_id` blocks for training completion before submitting the prediction job. | +| In both modes | `predictions()` always blocks until the prediction job completes (`_wait_for_completion`). | + +Treat the run as "long-running" until it crosses **~5× the dataset-prep time printed at Step 1** before suspecting it's stuck. At that point run the diagnostic ladder below before suspending or killing anything. + +### "Training appears stuck" + +Once the run crosses the ~5×-prep-time threshold above, run the three-step diagnostic ladder in [`references/known-limitations.md`](references/known-limitations.md) § "Training appears stuck" — diagnostic ladder before suspending or killing anything: (1) `GET_REASONER('predictive', …)` for pod status, (2) `client.jobs.list("Predictive", …)` for job state, (3) `SHOW EXPERIMENTS` for artifact creation. Each step localizes the failure before the next so you don't suspend the wrong reasoner. + ### Known Limitations & Runtime Troubleshooting GNN training has runtime gotchas that surface as opaque or no-error symptoms in the client. Use this table to recognize each one; load `references/known-limitations.md` for full causes (with SDK source citations), the before/after fallback code for `has_time_column=True` at scale, and the `GET_TRANSACTION_ARTIFACTS` recipe. diff --git a/plugins/rai/skills/rai-predictive-training/references/known-limitations.md b/plugins/rai/skills/rai-predictive-training/references/known-limitations.md index 62fa854..6b8bedb 100644 --- a/plugins/rai/skills/rai-predictive-training/references/known-limitations.md +++ b/plugins/rai/skills/rai-predictive-training/references/known-limitations.md @@ -57,3 +57,39 @@ model.define(Train(Sale, TrainTable.unit_sales)).where(...) | Symptom | Cause | Fix | |---|---|---| | `gnn.fit()` polling for an unreasonable amount of time | `JobMonitor._wait_for_completion` (`job_manager.py:332-340`) polls every 5s with no timeout/retry-cap | Kill the client manually. Recover via the QUEUED runbook, re-instantiate `GNN(...)`, resubmit | + +--- + +## "Training appears stuck" — diagnostic ladder + +Long-running predictive jobs are usually fine, not stuck. Use this ladder before suspending or killing anything. Each step localizes the failure to one component before the next, so you don't suspend the wrong reasoner. + +1. **Reasoner status — is the Predictive engine even up?** + ```sql + CALL RELATIONALAI.API.GET_REASONER('predictive', ''); + ``` + `STATUS=READY` means the reasoner pod is alive. If `SUSPENDED` / `PROVISIONING` / `FAILED`, that's the problem — see `rai-health` § Predictive train jobs stuck QUEUED for recovery. `STATUS=READY` does **not** prove the in-pod worker queue is healthy — step 2 catches that case. + +2. **Job ledger — did the train job actually land on a worker?** + ```python + client.jobs.list("Predictive", name="", only_active=True, limit=10) + ``` + ```sql + -- equivalent SQL + SELECT ID, STATE, JOB_TYPE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN + FROM RELATIONALAI.API.JOBS + WHERE STATE IN ('QUEUED','RUNNING') + AND PAYLOAD LIKE '%"job_type": "train"%' + ORDER BY CREATED_ON ASC; + ``` + - One `train` job in `RUNNING` with `AGE_MIN` rising → training is progressing. Wait. + - `QUEUED` and old (`AGE_MIN` > a few) while reasoner reports `READY` → in-pod worker desync. → `rai-health` § Predictive train jobs stuck QUEUED. + - No train job at all but recent COMPLETED short jobs → `fit()` never made it to job submission. Look for client-side errors above the "Training job submitted" line. + +3. **Experiment ledger — did training start writing artifacts?** + ```sql + SHOW EXPERIMENTS IN SCHEMA .; + ``` + A new experiment row appears within ~60 seconds of a `RUNNING` train job. If the job is `RUNNING` for several minutes and `SHOW EXPERIMENTS` shows no new run, the worker accepted the job but is failing silently before artifact creation — escalate via the QUEUED runbook (suspend + resume the predictive reasoner, then re-instantiate `GNN(...)`). + +If all three checks look healthy and the run is still long, it's a real long run — let it continue, or scale the engine up (see `rai-predictive-modeling` § Engine sizing) on the next attempt. From 3b7eadfb22a4cbed756e3c0c87e72df5a4030c50 Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 15:03:10 -0700 Subject: [PATCH 24/27] predictive: drop CREATE_GNN_SERVICE callouts, recommend GPU + api.CREATE_REASONER_ASYNC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The predictive provisioning + recovery story should point at the supported RELATIONALAI.API.* surface and a GPU compute type, not at EXPERIMENTAL.CREATE_GNN_SERVICE — which is off-surface, GNN-specific, and currently broken on V5 due to an image-mismatch (issues.md ISS-005). Customer-facing guidance frames the answer positively. - rai-predictive-modeling: replace the Engine sizing CPU-vs-GPU heuristic block with a "Provisioning the Predictive reasoner" block. Names GPU_NV_S as the recommended default and shows the canonical CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', OBJECT_CONSTRUCT()) shape with a GET_REASONER poll. Keeps the CLI-vs-backend allow-list note as fall-through context. - rai-predictive-training: drop the standalone "CREATE_GNN_SERVICE() is not the right escalation" paragraph. The new line points at SUSPEND_REASONER / RESUME_REASONER_ASYNC / DELETE_REASONER + CREATE_REASONER_ASYNC('predictive', ..., 'GPU_NV_S', ...) for rebuild. - rai-health: § Predictive train jobs stuck QUEUED Recovery now includes the rebuild-on-GPU path (DELETE_REASONER + CREATE_REASONER_ASYNC with GPU_NV_S) when worker-recycle isn't enough. Drops the long EXPERIMENTAL.CREATE_GNN_SERVICE blockquote; keeps a single-clause "stay on the API surface, not EXPERIMENTAL.*" reminder paired with the positive recovery instructions. --- plugins/rai/skills/rai-health/SKILL.md | 11 +++++++-- .../skills/rai-predictive-modeling/SKILL.md | 23 ++++++++++++++----- .../skills/rai-predictive-training/SKILL.md | 2 +- 3 files changed, 27 insertions(+), 9 deletions(-) diff --git a/plugins/rai/skills/rai-health/SKILL.md b/plugins/rai/skills/rai-health/SKILL.md index 37fed04..243a84f 100644 --- a/plugins/rai/skills/rai-health/SKILL.md +++ b/plugins/rai/skills/rai-health/SKILL.md @@ -331,7 +331,7 @@ A `QUEUED` train job that won't advance while the reasoner reports `READY` is th ### Recovery -Suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Do **not** call `CREATE_GNN_SERVICE()`. +Suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Use only the supported `RELATIONALAI.API.*` surface — do not invoke `RELATIONALAI.EXPERIMENTAL.*` procedures as a workaround. ```sql -- 1. Confirm a stuck train job @@ -349,7 +349,14 @@ CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', ''); CALL RELATIONALAI.API.GET_REASONER('predictive', ''); ``` -> **Do not use `CALL RELATIONALAI.EXPERIMENTAL.CREATE_GNN_SERVICE();` to recover stuck predictive train jobs.** The SDK never invokes it — train submission goes through `.api.exec_job_async('GNN', , ...)` against the predictive reasoner directly (`relationalai_gnns/core/connector.py::exec_job`). `CREATE_GNN_SERVICE` targets a separate code path; if it fails with an image-mismatch error like `Invalid image specified in service spec: image 'rai-gnn-app:' does not exist in current application version`, that does **not** mean GNN training is broken — it just means that orthogonal path can't be brought up. The right escalation is `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` on the predictive reasoner itself. +If recycling doesn't unstick the worker, **rebuild on a fresh GPU predictive reasoner** — the documented path: + +```sql +CALL RELATIONALAI.API.DELETE_REASONER('predictive', ''); +CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', OBJECT_CONSTRUCT()); +-- Poll until STATUS=READY: +CALL RELATIONALAI.API.GET_REASONER('predictive', ''); +``` See `rai-predictive-training` § Worker not ready to accept jobs for the matching client-side symptom and § Stalled train job: SDK polls without a timeout for stalled-job forensics. diff --git a/plugins/rai/skills/rai-predictive-modeling/SKILL.md b/plugins/rai/skills/rai-predictive-modeling/SKILL.md index e7e49fd..e6a9213 100644 --- a/plugins/rai/skills/rai-predictive-modeling/SKILL.md +++ b/plugins/rai/skills/rai-predictive-modeling/SKILL.md @@ -76,16 +76,27 @@ A GNN workflow runs against **two distinct reasoner engines** that must both be When training "hangs" or queries are slow, the first question is *which engine* — they have separate sizes, separate `STATUS`, separate auto-suspend timers. `rai-health` § Predictive train jobs stuck QUEUED covers the Predictive side; the Logic-engine ladder lives in `rai-health` Steps 1–3. -### Engine sizing +### Provisioning the Predictive reasoner -The Predictive reasoner accepts both CPU (`HIGHMEM_X64_S` / `_M` / `_L`) and GPU (`GPU_NV_S`, …) sizes. The CLI's allow-list trails the backend — `REASONER_SIZES_AWS` in `relationalai/services/reasoners/constants.py` currently lists CPU sizes only, while the backend's `AWSEngineSize` Literal in `config_reasoners_fields.py` accepts `GPU_NV_S`. If `rai reasoners:create` rejects a GPU size, fall through to `CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', PARSE_JSON('{}'))` directly. +**Use a GPU compute type for the Predictive reasoner.** The canonical provisioning shape: -Rough heuristics for choosing CPU vs GPU: +```sql +CALL RELATIONALAI.API.CREATE_REASONER_ASYNC( + 'predictive', + '', + 'GPU_NV_S', + OBJECT_CONSTRUCT() -- {} — accept all defaults; or pass auto_suspend_mins, settings, … +); + +-- Poll until STATUS=READY (1–3 minutes typical): +CALL RELATIONALAI.API.GET_REASONER('predictive', ''); +``` + +`GPU_NV_S` is faster per epoch on the GNN training job and is the recommended default for predictive workloads. `HIGHMEM_X64_S` / `_M` / `_L` are also valid sizes for the predictive reasoner, but GPU is the path the platform team recommends; pick it unless you have a specific reason not to. -- **CPU (HIGHMEM_X64_*)**: prototyping, single-task GNNs on graphs under ~100K nodes / ~1M edges, runs where you're iterating on features more than scaling out -- **GPU (`GPU_NV_S`+)**: production-leaning runs at ~1M+ nodes or ~10M+ edges, multi-epoch training over rich feature sets, link-prediction with large negative sampling +The `rai reasoners:create --type Predictive --size GPU_NV_S` CLI form may report an allow-list error (`Allowed sizes: HIGHMEM_X64_S, HIGHMEM_X64_M, HIGHMEM_X64_L`) on older client versions — the validation list (`relationalai/services/reasoners/constants.py::REASONER_SIZES_AWS`) trails the backend's `AWSEngineSize` Literal in `config_reasoners_fields.py`. The SQL `CREATE_REASONER_ASYNC` call above is the canonical fall-through; both reach the same backend. -GPU is faster per epoch when the dataset fits in the GPU VM's CPU memory; if the dataset is borderline, CPU `HIGHMEM_X64_L`/`_SL` may finish sooner overall than GPU paging. Confirm current sizing tradeoffs with the RelationalAI team — pool capacity and price points evolve. +Confirm current sizing options with the RelationalAI team — pool capacity and recommendations evolve. --- diff --git a/plugins/rai/skills/rai-predictive-training/SKILL.md b/plugins/rai/skills/rai-predictive-training/SKILL.md index b991e80..88fd11c 100644 --- a/plugins/rai/skills/rai-predictive-training/SKILL.md +++ b/plugins/rai/skills/rai-predictive-training/SKILL.md @@ -214,7 +214,7 @@ GNN training has runtime gotchas that surface as opaque or no-error symptoms in | Client polls forever with no progress | `JobMonitor._wait_for_completion` has no timeout — kill the client manually + recover via the QUEUED runbook | | `Failed to pull data into index: transaction was aborted (runtime error)` | Opaque wrapper — pull `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` -> `problems.json` for the real error. For the schema-drift / compiled-relation-cache cause: rename `Model(...)` | -`CREATE_GNN_SERVICE()` is **not** the right escalation for any predictive train issue — the SDK submits training in-pod against the predictive reasoner, not via that legacy path (`relationalai_gnns/core/connector.py::exec_job`). See `rai-health` § Predictive train jobs stuck QUEUED. +For predictive train issues, stay on the supported `RELATIONALAI.API.*` surface — `SUSPEND_REASONER` / `RESUME_REASONER_ASYNC` for recovery, `DELETE_REASONER` + `CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', OBJECT_CONSTRUCT())` for a fresh rebuild. See `rai-health` § Predictive train jobs stuck QUEUED. --- From 88b19a42d1d18d0677a860faf886877fde83da70 Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 15:08:09 -0700 Subject: [PATCH 25/27] rai-health: add "Predictive reasoner stuck in data-index init" entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Distinct upstream failure mode from the existing "Predictive train jobs stuck QUEUED" section: the reasoner is still PROVISIONING (not READY), and gnn.fit() appears to hang in Step 1 (dataset prep) while the in-pod data index hydrates from CDC streams. Same per-table CDC stream-sync compounding that affects unwarmed Logic reasoners on first model query, surfaced on the Predictive side. 3-step diagnostic ladder anchored to the existing rai-health surface: - GET_REASONER('predictive', ...) for pod status - relationalai.api.cdc_status for upstream stream health (cross-link to § Step 5 for the quarantine/resume_cdc runbook) - GET_OWN_TRANSACTION_PROBLEMS('') for the specific transaction the client errored against (cross-link to § Step 4 for owner restriction pitfall) Recovery escalates to the QUEUED-section's SUSPEND/RESUME pattern, or rebuild on a fresh GPU reasoner via DELETE_REASONER + CREATE_REASONER_ASYNC('predictive', ..., 'GPU_NV_S', OBJECT_CONSTRUCT()). --- plugins/rai/skills/rai-health/SKILL.md | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/plugins/rai/skills/rai-health/SKILL.md b/plugins/rai/skills/rai-health/SKILL.md index 243a84f..7c3a3fd 100644 --- a/plugins/rai/skills/rai-health/SKILL.md +++ b/plugins/rai/skills/rai-health/SKILL.md @@ -315,6 +315,37 @@ for the full step-by-step recovery checklist, schema reference, and official doc --- +## Predictive reasoner stuck in data-index init + +A freshly created or freshly resumed Predictive reasoner can sit in `STATUS=PROVISIONING` for many minutes while the in-pod data index hydrates from CDC streams — and `gnn.fit()` running against it appears to hang in Step 1 (dataset prep) before reaching "Training job submitted." This is the same per-table CDC stream-sync compounding that affects unwarmed Logic reasoners on first model query, surfacing on the Predictive side as a silent prep-step stall. + +### Diagnose (in order) + +1. **Reasoner status** — is the pod still hydrating or actually ready? + ```sql + CALL RELATIONALAI.API.GET_REASONER('predictive', ''); + ``` + `STATUS=PROVISIONING` → wait (1–3 min typical for warm pools; longer on first-time Snowflake-stream attach). `STATUS=READY` plus a hang → continue to (2) and (3). + +2. **CDC stream health** — is a stream the predictive reasoner depends on suspended or quarantined? + ```sql + SELECT * FROM relationalai.api.cdc_status; + ``` + `engine_status != READY` or `stream_status` not `RUNNING` → see § Step 5 — Diagnose CDC / Data Stream Health for the quarantine-recovery / `resume_cdc` runbook. Predictive jobs cannot proceed until the upstream streams are healthy. + +3. **Transaction problems** — if a specific transaction id appeared in client logs (often the `Failed to pull data into index: transaction was aborted` wrapper), pull its problems: + ```sql + CALL RELATIONALAI.API.GET_OWN_TRANSACTION_PROBLEMS(''); + -- or with admin role: GET_TRANSACTION_PROBLEMS('') + ``` + See § Step 4 — Diagnose a Failed Transaction for the schema reference and owner-restriction pitfall. + +### Recovery + +If CDC is healthy and the reasoner has been `PROVISIONING` for more than ~5 minutes with no client-side progress, treat it the same as a stuck worker — `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` (see § Predictive train jobs stuck QUEUED below). Persistent failure: rebuild on a fresh GPU reasoner via `DELETE_REASONER` + `CREATE_REASONER_ASYNC('predictive', ..., 'GPU_NV_S', OBJECT_CONSTRUCT())`. + +--- + ## Predictive train jobs stuck QUEUED A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The SDK only checks reasoner-pod status before submitting — the in-pod worker queue can be out of sync with that status, and the SDK has no way to detect it (`relationalai_gnns/core/connector.py::_check_engine_availability`). From b556191f3615186fa872fa83e4e68db454bf5e4b Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 15:51:13 -0700 Subject: [PATCH 26/27] rai-predictive-modeling: tighten experiment-schema setup to positive guidance only Drop the negative framings (PermissionError walk-through, shared-DB warning quote, "not a generic RuntimeError" callout) and the specific EXPERIMENTS schema name in favor of a placeholder. The four GRANT statements + "All four grants are required" + the matching GNN constructor args are sufficient guidance. --- .../skills/rai-predictive-modeling/SKILL.md | 25 ++++++------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/plugins/rai/skills/rai-predictive-modeling/SKILL.md b/plugins/rai/skills/rai-predictive-modeling/SKILL.md index e6a9213..f3f9606 100644 --- a/plugins/rai/skills/rai-predictive-modeling/SKILL.md +++ b/plugins/rai/skills/rai-predictive-modeling/SKILL.md @@ -30,37 +30,28 @@ description: Build GNN data models -- concepts, Snowflake data loading, task rel ### Experiment schema setup (one-time, ACCOUNTADMIN) -GNN training writes experiment artifacts to a Snowflake schema. The RELATIONALAI native app must have write access on it. Without this the first `gnn.fit()` raises `PermissionError` (from `relationalai_gnns.core.diagnostics.PermissionDiagnostic`) whose message names the missing grant — typically *"Database does not exist or the GNN RelationalAI Native App lacks permissions"* or *"Schema does not exist or ..."*. - -The diagnostic prescribes exactly four grants on top of the database+schema: +GNN training writes experiment artifacts to a Snowflake schema. Create a database and schema you own, then grant the RELATIONALAI native app the four required privileges: ```sql --- Use a database you own (NOT a Snowflake-shared/marketplace database). --- Shared DBs reject schema creation: "Creating schema on shared database --- '' is not allowed." CREATE DATABASE IF NOT EXISTS ; -CREATE SCHEMA IF NOT EXISTS .EXPERIMENTS; +CREATE SCHEMA IF NOT EXISTS .; -GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; -GRANT USAGE ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; -GRANT CREATE EXPERIMENT ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; -GRANT CREATE MODEL ON SCHEMA .EXPERIMENTS TO APPLICATION RELATIONALAI; +GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; +GRANT USAGE ON SCHEMA . TO APPLICATION RELATIONALAI; +GRANT CREATE EXPERIMENT ON SCHEMA . TO APPLICATION RELATIONALAI; +GRANT CREATE MODEL ON SCHEMA . TO APPLICATION RELATIONALAI; ``` -`GRANT ALL PRIVILEGES ON SCHEMA .EXPERIMENTS` is a working superset if you don't need least-privilege. - -Then in the script: +All four grants are required. Then pass the same database and schema to the GNN constructor: ```python gnn = GNN( exp_database="", - exp_schema="EXPERIMENTS", + exp_schema="", ... ) ``` -The error is a `PermissionError`, not a generic `RuntimeError` — code that wraps `gnn.fit()` can catch it specifically. - ### `relationalai` package version The predictive submodule (`relationalai.semantics.reasoners.predictive`) is not in every published `relationalai` release — `from relationalai.semantics.reasoners.predictive import GNN` raises `ModuleNotFoundError` on releases that pre-date it. Pin a release that ships the submodule (or install from the development branch when iterating against unreleased changes). From aba56e2a234766c41131d3f8fb5c0ca35598a188 Mon Sep 17 00:00:00 2001 From: cafzal Date: Mon, 4 May 2026 15:55:18 -0700 Subject: [PATCH 27/27] rai-predictive-training: fix experiment-schema-grants pitfall row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to the Common Pitfalls row that maps the PermissionDiagnostic error to a fix: 1. Symptom now uses the actual PermissionError message text ("Database does not exist or the GNN RelationalAI Native App lacks permissions" / "Schema does not exist or ...") so agents matching the row to a real error log find it directly. 2. Fix now names the four explicit grants (USAGE on database, USAGE on schema, CREATE EXPERIMENT, CREATE MODEL) and points at rai-predictive-modeling § Prerequisites for the canonical SQL. Drops the GRANT ALL ON SCHEMA recommendation — CREATE EXPERIMENT and CREATE MODEL are not part of the legacy ALL bundle. --- plugins/rai/skills/rai-predictive-training/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/rai/skills/rai-predictive-training/SKILL.md b/plugins/rai/skills/rai-predictive-training/SKILL.md index 88fd11c..eb95c40 100644 --- a/plugins/rai/skills/rai-predictive-training/SKILL.md +++ b/plugins/rai/skills/rai-predictive-training/SKILL.md @@ -506,7 +506,7 @@ User.predictions = gnn.predictions(domain=Test) | `has_time_column=True` fails with "no time column defined in data tables" | The concept carrying `time_col` is an edge, not a node — `time_col` only propagates for node concepts | Use `has_time_column=False` with non-temporal Relationships as workaround | | `has_time_column=True` fails with `ValidationError: Error processing datetime column ''` at scale | Server-side datetime processing rejects the column despite clean data, node-level concept, and correct `datetime`/`time_col` config — second known limitation | Verify the timestamp column type matches the GNN datetime pipeline's expected format (see `rai-predictive-modeling`); fall back to non-temporal Relationships if it persists | | `SnowflakeTableObjectsException: Failed to pull data into index: transaction was aborted (runtime error)` | Opaque client wrapper that hides the actual server-side error (commonly a stale compiled-relation signature after schema drift, but other causes possible) | Pull `problems.json` via `RELATIONALAI.API.GET_TRANSACTION_ARTIFACTS('')` (presigned URL) and read the `report` field for the real error. For the schema-drift case specifically, see § Known Limitations | -| Experiment schema not accessible by the RAI native app | RAI app needs explicit grants to read from the experiment schema | `GRANT USAGE ON DATABASE TO APPLICATION RELATIONALAI; GRANT ALL ON SCHEMA . TO APPLICATION RELATIONALAI` | +| `gnn.fit()` raises `PermissionError` with *"Database does not exist or the GNN RelationalAI Native App lacks permissions"* (or *"Schema does not exist or ..."*) — from `relationalai_gnns.core.diagnostics.PermissionDiagnostic` | RAI app missing one or more of the four required grants on the experiment database/schema | Apply all four grants per `rai-predictive-modeling` § Prerequisites: `USAGE` on the database, `USAGE` on the schema, `CREATE EXPERIMENT` on the schema, `CREATE MODEL` on the schema | ---