diff --git a/.gitignore b/.gitignore index bbe3e69..94279c9 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,5 @@ logs/ integration_tests/.env integration_tests/.user.yml integration_tests/dbt_packages/ -integration_tests/package-lock.yml \ No newline at end of file +integration_tests/package-lock.yml +graphify-out \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index 0738a79..a6b516c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- `enable_versioning` config option — controls Snowflake Cortex Agent versioning; when `true` (default), uses `CREATE AGENT IF NOT EXISTS` + `ALTER AGENT MODIFY LIVE VERSION SET SPECIFICATION` instead of `CREATE OR REPLACE AGENT`, preserving version history across runs. `dbt run --full-refresh` falls back to `CREATE OR REPLACE`, resetting history. Set to `false` in dev environments to skip versioning overhead. +- `auto_commit` config option — when `enable_versioning=true`, automatically snapshot the LIVE version into a new immutable named version (`VERSION$1`, `VERSION$2`, …) after each run. Defaults to `true`. Set to `false` to accumulate spec changes in LIVE without committing, then commit manually via `ALTER AGENT COMMIT`. +- `version_comment` config option — optional comment string attached to the committed version snapshot; only used when `enable_versioning=true` and `auto_commit=true`. +- `snowflake__create_cortex_agent_if_not_exists` macro — `CREATE AGENT IF NOT EXISTS` DDL +- `snowflake__add_cortex_agent_live_version` macro — `ALTER AGENT ADD LIVE VERSION FROM LAST` DDL; recreates the LIVE working copy after `COMMIT` consumes it +- `snowflake__alter_cortex_agent_live_spec` macro — `ALTER AGENT MODIFY LIVE VERSION SET SPECIFICATION` DDL +- `snowflake__commit_cortex_agent_version` macro — `ALTER AGENT COMMIT` DDL - `mcp_server` materialization for Snowflake-managed MCP servers — model body is the raw MCP server YAML specification (a `tools:` array), sent verbatim to `CREATE OR REPLACE MCP SERVER ... FROM SPECIFICATION` - `snowflake__create_mcp_server` macro for DDL generation - `snowflake__get_drop_mcp_server_sql` macro for DROP DDL generation - Integration tests for the MCP server materialization (`mcp_server_test`), verified via `DESCRIBE MCP SERVER` captured into a table, since `GET_DDL` does not support MCP servers - Local development setup: `integration_tests/.env` template, `scripts/run_tests.ps1` runner, and `profiles.yml` SSO support (`externalbrowser` authenticator) +### Changed +- `enable_versioning` now defaults to `true` — versioning is on by default for all `cortex_agent` models; set `enable_versioning: false` in `dbt_project.yml` or per-model config to opt out (e.g. in dev environments) + ### Fixed - Singular integration tests now declare `-- depends_on: {{ ref('cortex_agent_test') }}` so `dbt build` runs models before tests diff --git a/README.md b/README.md index a8df82a..5e1c87e 100644 --- a/README.md +++ b/README.md @@ -83,19 +83,55 @@ SHOW AGENTS IN SCHEMA my_db.my_schema; | `feedback_schema` | string | No | Schema for the feedback table and `AGENT_SUBMIT_FEEDBACK` procedure. Accepts `'SCHEMA'` or `'DB.SCHEMA'`. Defaults to the agent's own database and schema. See [Feedback Tool](#feedback-tool). | | `feedback_table` | string | No | Fully-qualified table name override for user feedback. Defaults to `{feedback_schema}.AGENT_FEEDBACK`. Ignored when `create_feedback_table` is `false`. See [Feedback Tool](#feedback-tool). | | `feedback_execute_as` | string | No | Execution rights for the `AGENT_SUBMIT_FEEDBACK` procedure. `'caller'` (default) captures the end user via `current_user()`. Use `'owner'` if the calling role lacks `INSERT` on the feedback table. See [Feedback Tool](#feedback-tool). | +| `enable_versioning` | bool | No | Whether to use Snowflake agent versioning. Defaults to `true`. When `true`, uses `CREATE AGENT IF NOT EXISTS` + `ALTER AGENT MODIFY LIVE VERSION` instead of `CREATE OR REPLACE AGENT`, preserving version history across runs. Set to `false` in dev environments to skip versioning overhead. `dbt run --full-refresh` always falls back to `CREATE OR REPLACE`, resetting history. | +| `auto_commit` | bool | No | When `enable_versioning=true`, automatically snapshot the LIVE version into a new named version after each run. Defaults to `true`. Set to `false` to accumulate spec changes in LIVE without committing, then commit manually via `ALTER AGENT COMMIT`. | +| `version_comment` | string | No | Comment attached to each committed version snapshot. Only used when `enable_versioning=true` and `auto_commit=true`. | ## How It Works -The model body is passed verbatim as the agent YAML specification to Snowflake's `CREATE OR REPLACE AGENT ... FROM SPECIFICATION $$ ... $$`. Because it's a direct passthrough: +The model body is passed verbatim as the agent YAML specification to Snowflake. Because it's a direct passthrough: - All current and future Snowflake agent YAML options work automatically - No package updates needed when Snowflake adds new features - You can use Jinja (`{{ ref() }}`, `{{ var() }}`, etc.) anywhere in the spec - `{{ ref() }}` calls in `tool_resources` resolve to fully qualified names **and** wire the agent into the dbt DAG — the agent will always run after its upstream semantic views +## Versioning + +By default (`enable_versioning=true`, `auto_commit=true`) every `dbt run`: + +1. Creates the agent if it doesn't exist (`CREATE AGENT IF NOT EXISTS`) +2. Restores a LIVE working copy from the last committed version (`ALTER AGENT ADD LIVE VERSION FROM LAST`) +3. Updates the LIVE copy with the latest compiled spec (`ALTER AGENT MODIFY LIVE VERSION SET SPECIFICATION`) +4. Commits the LIVE copy as a new named version — `VERSION$1`, `VERSION$2`, … (`ALTER AGENT COMMIT`) + +Version history accumulates across runs. `dbt run --full-refresh` falls back to `CREATE OR REPLACE AGENT`, resetting all history. + +**Disable versioning in dev** to avoid the overhead when iterating quickly: + +```yaml +# dbt_project.yml +models: + my_project: + agents: + +enable_versioning: false +``` + +Or per-model: + +```sql +{{ config(materialized='cortex_agent', enable_versioning=false) }} +``` + +**Commit manually** by setting `auto_commit=false` — the materialization keeps the LIVE version open across runs and you commit explicitly when ready: + +```sql +{{ config(materialized='cortex_agent', auto_commit=false) }} +``` + ## Idempotency -Every `dbt run` issues `CREATE OR REPLACE AGENT`, so re-runs are safe and fully idempotent. `dbt run --full-refresh` behaves identically. +With versioning enabled (default), every `dbt run` preserves history — re-runs are safe and create a new committed version. With `enable_versioning=false`, every run issues `CREATE OR REPLACE AGENT`. `dbt run --full-refresh` always uses `CREATE OR REPLACE AGENT`, resetting all version history. ## Supported Tools in Specification diff --git a/integration_tests/models/cortex_agent_test.sql b/integration_tests/models/cortex_agent_test.sql index 2bc72e3..d00dd94 100644 --- a/integration_tests/models/cortex_agent_test.sql +++ b/integration_tests/models/cortex_agent_test.sql @@ -7,7 +7,8 @@ tags=['integration'], agent_grants=['dbt_demo_role'], create_feedback_table=true, - feedback_execute_as='owner' + feedback_execute_as='owner', + enable_versioning=false ) }} diff --git a/integration_tests/models/cortex_agent_versioned_test.sql b/integration_tests/models/cortex_agent_versioned_test.sql new file mode 100644 index 0000000..99d3897 --- /dev/null +++ b/integration_tests/models/cortex_agent_versioned_test.sql @@ -0,0 +1,41 @@ +{{ + config( + materialized='cortex_agent', + alias='cortex_agent_versioned_test', + comment='Versioned integration test agent — exercises enable_versioning, auto_commit, and version_comment', + tags=['integration'], + version_comment='CI test commit' + ) +}} + +models: + orchestration: auto + +orchestration: + budget: + seconds: 30 + tokens: 8000 + +instructions: + response: 'You are a versioning integration test assistant.' + orchestration: 'Use the analyst tool for structured data questions.' + sample_questions: + - question: 'What was total revenue last quarter?' + +tools: + - tool_spec: + type: cortex_analyst_text_to_sql + name: analyst_tool + description: 'Answers structured data questions using a semantic view.' + input_schema: + type: object + properties: + question: + type: string + description: 'A natural language question about the data.' + required: + - question + +tool_resources: + analyst_tool: + semantic_view: '{{ ref('test_semantic_view') }}' diff --git a/integration_tests/models/cortex_agent_versioned_test_versions.sql b/integration_tests/models/cortex_agent_versioned_test_versions.sql new file mode 100644 index 0000000..caa272a --- /dev/null +++ b/integration_tests/models/cortex_agent_versioned_test_versions.sql @@ -0,0 +1,12 @@ +-- SHOW VERSIONS can't be used as a subquery, so this model captures its output into a table +-- using the same pre_hook pattern as mcp_server_test_describe. The ref() in the pre_hook +-- forces dbt to run cortex_agent_versioned_test first. Singular tests assert on this table. +{{ + config( + materialized='table', + pre_hook="show versions in agent {{ ref('cortex_agent_versioned_test') }}" + ) +}} + +select * +from table(result_scan(last_query_id())) diff --git a/integration_tests/models/schema.yml b/integration_tests/models/schema.yml index 06b1fa3..e39fb57 100644 --- a/integration_tests/models/schema.yml +++ b/integration_tests/models/schema.yml @@ -22,5 +22,8 @@ models: alias: mcp_server_test tags: ['integration'] + - name: cortex_agent_versioned_test_versions + description: "Captures SHOW VERSIONS IN AGENT output for cortex_agent_versioned_test so singular tests can assert on committed version count." + - name: mcp_server_test_describe description: "Captures DESCRIBE MCP SERVER output for mcp_server_test so singular tests can assert on the spec (GET_DDL does not support MCP servers)." diff --git a/integration_tests/tests/cortex_agent_versioned_test_has_versions.sql b/integration_tests/tests/cortex_agent_versioned_test_has_versions.sql new file mode 100644 index 0000000..4061242 --- /dev/null +++ b/integration_tests/tests/cortex_agent_versioned_test_has_versions.sql @@ -0,0 +1,6 @@ +-- Verify the versioned agent has at least one committed named version. +-- A successful auto_commit produces VERSION$1 on first run and increments on each subsequent run. +-- Returns 0 rows on success (standard dbt test contract). + +select 'no committed versions found for cortex_agent_versioned_test' as error +where (select count(*) from {{ ref('cortex_agent_versioned_test_versions') }}) < 1 diff --git a/macros/materializations/cortex_agent.sql b/macros/materializations/cortex_agent.sql index f2dc1ab..33e6825 100644 --- a/macros/materializations/cortex_agent.sql +++ b/macros/materializations/cortex_agent.sql @@ -19,6 +19,19 @@ -- Accepts 'caller' (default) or 'owner'. -- Use 'caller' to capture the end user via current_user(). -- Use 'owner' if the calling role lacks INSERT on the feedback table. +-- enable_versioning (bool, optional) : opt into Snowflake agent versioning. Defaults to true. +-- When true, uses CREATE AGENT IF NOT EXISTS + ALTER AGENT +-- MODIFY LIVE VERSION instead of CREATE OR REPLACE AGENT, +-- preserving version history across runs. +-- Set to false in dev environments to skip versioning overhead. +-- dbt run --full-refresh falls back to CREATE OR REPLACE, +-- resetting all version history. +-- auto_commit (bool, optional) : when enable_versioning=true, automatically snapshot the +-- LIVE version into a new named version after each run. +-- Defaults to true. Set to false to accumulate changes in +-- LIVE without committing, then commit manually. +-- version_comment (string, optional) : comment attached to the committed version snapshot. +-- Only used when enable_versioning=true and auto_commit=true. {% materialization cortex_agent, adapter='snowflake' %} @@ -29,6 +42,9 @@ {%- set feedback_schema_config = config.get('feedback_schema', default=none) -%} {%- set feedback_table = config.get('feedback_table', default=none) -%} {%- set feedback_execute_as = config.get('feedback_execute_as', default='caller') -%} + {%- set enable_versioning = config.get('enable_versioning', default=true) -%} + {%- set auto_commit = config.get('auto_commit', default=true) -%} + {%- set version_comment = config.get('version_comment', default=none) -%} {%- set target_relation = api.Relation.create( identifier=this.identifier, @@ -67,10 +83,55 @@ {% endcall %} {%- endif %} + {%- if enable_versioning and not should_full_refresh() %} + + {# Determine whether any committed versions exist. + On first run (agent absent or no committed versions) skip MODIFY LIVE VERSION — + the LIVE working copy does not exist until after the first COMMIT. + On subsequent runs COMMIT has already established LIVE, so we can update it. #} + {%- set _show_agents_sql -%} + SHOW AGENTS LIKE '{{ target_relation.identifier }}' IN SCHEMA {{ target_relation.database }}.{{ target_relation.schema }} + {%- endset -%} + {%- set _agent_rows = run_query(_show_agents_sql) -%} + {%- set _has_committed_versions = false -%} + {%- if _agent_rows | length > 0 -%} + {%- set _show_versions_sql -%} + SHOW VERSIONS IN AGENT {{ target_relation.database }}.{{ target_relation.schema }}.{{ target_relation.identifier }} + {%- endset -%} + {%- set _version_rows = run_query(_show_versions_sql) -%} + {%- set _has_committed_versions = (_version_rows | length) > 0 -%} + {%- endif -%} + + {% call statement('main') %} + {{ dbt_cortex_agent.snowflake__create_cortex_agent_if_not_exists(target_relation, sql, comment, profile) }} + {% endcall %} + + {%- if _has_committed_versions %} + {# COMMIT consumes LIVE and does not recreate it. Explicitly restore LIVE from the + last committed version before modifying, so MODIFY LIVE VERSION has a target. #} + {% call statement('add_live') %} + {{ dbt_cortex_agent.snowflake__add_cortex_agent_live_version(target_relation) }} + {% endcall %} + {%- endif %} + + {% call statement('update_live') %} + {{ dbt_cortex_agent.snowflake__alter_cortex_agent_live_spec(target_relation, sql) }} + {% endcall %} + + {%- if auto_commit %} + {% call statement('commit_version') %} + {{ dbt_cortex_agent.snowflake__commit_cortex_agent_version(target_relation, version_comment) }} + {% endcall %} + {%- endif %} + + {%- else %} + {% call statement('main') %} {{ dbt_cortex_agent.snowflake__create_cortex_agent(target_relation, sql, comment, profile) }} {% endcall %} + {%- endif %} + {{ run_hooks(post_hooks) }} {%- if agent_grants | length > 0 %} diff --git a/macros/relations/cortex_agent/versioned_agent.sql b/macros/relations/cortex_agent/versioned_agent.sql new file mode 100644 index 0000000..932d81f --- /dev/null +++ b/macros/relations/cortex_agent/versioned_agent.sql @@ -0,0 +1,63 @@ +-- Macros supporting Snowflake Cortex Agent versioning. +-- Used by the cortex_agent materialization when enable_versioning=true. + +-- Creates the agent only if it does not already exist. +-- On first run this initialises VERSION$1 and a LIVE working copy. +-- On subsequent runs the CREATE is a no-op; spec changes go through +-- snowflake__alter_cortex_agent_live_spec instead. +{% macro snowflake__create_cortex_agent_if_not_exists(relation, specification, comment, profile) %} + + create agent if not exists + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + {%- if comment is not none %} + comment = '{{ comment }}' + {%- endif %} + {%- if profile is not none %} + profile = '{{ profile }}' + {%- endif %} + from specification + $$ +{{ specification | trim }} + $$ + +{% endmacro %} + + +-- Recreates the LIVE working copy from the last committed version. +-- Required before MODIFY LIVE VERSION on any run after the first COMMIT, +-- because COMMIT consumes the LIVE version and does not recreate it automatically. +{% macro snowflake__add_cortex_agent_live_version(relation) %} + + alter agent + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + add live version from last + +{% endmacro %} + + +-- Pushes the latest compiled spec into the mutable LIVE version. +-- Runs on every dbt run when versioning is enabled, keeping LIVE current. +{% macro snowflake__alter_cortex_agent_live_spec(relation, specification) %} + + alter agent + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + modify live version set specification = + $$ +{{ specification | trim }} + $$ + +{% endmacro %} + + +-- Snapshots the current LIVE version into a new immutable named version +-- (VERSION$2, VERSION$3, …). Called when auto_commit=true. +{% macro snowflake__commit_cortex_agent_version(relation, version_comment) %} + + alter agent + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + commit + {%- if version_comment is not none %} + comment = '{{ version_comment }}' + {%- endif %} + +{% endmacro %}