diff --git a/CHANGELOG.md b/CHANGELOG.md index a6b516c..5c32f0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,9 +20,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `snowflake__get_drop_mcp_server_sql` macro for DROP DDL generation - Integration tests for the MCP server materialization (`mcp_server_test`), verified via `DESCRIBE MCP SERVER` captured into a table, since `GET_DDL` does not support MCP servers - Local development setup: `integration_tests/.env` template, `scripts/run_tests.ps1` runner, and `profiles.yml` SSO support (`externalbrowser` authenticator) +- `cortex_search_service` materialization for [Snowflake Cortex Search Services](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) — unlike `cortex_agent`/`mcp_server`, the model body is a normal dbt `SELECT` query (not a passthrough YAML spec) that becomes the `AS ` clause; supports the single-index (`ON` + `ATTRIBUTES`) syntax only +- `search_column`, `attributes`, and `target_lag` required config options for `cortex_search_service` — Snowflake has no default for any of these +- `warehouse`, `primary_key`, `embedding_model`, `refresh_mode`, `initialize`, `full_index_build_interval_days`, `request_logging`, `auto_suspend`, and `comment` optional config options for `cortex_search_service` +- `search_service_grants` config option — list of role names to grant `USAGE` on the search service after creation, mirrors `agent_grants` +- Refresh behavior: `cortex_search_service` issues `CREATE OR REPLACE CORTEX SEARCH SERVICE` only on the first run for a given service or on `dbt run --full-refresh`; subsequent runs use `ALTER CORTEX SEARCH SERVICE ... SET` to update mutable properties in place, avoiding a full index rebuild on every `dbt run` +- `snowflake__create_cortex_search_service` macro for CREATE DDL generation +- `snowflake__alter_cortex_search_service` macro for ALTER ... SET DDL generation — emits one `ALTER CORTEX SEARCH SERVICE ... SET = ;` statement per scalar mutable property (target_lag, warehouse, comment, auto_suspend, request_logging, full_index_build_interval_days) rather than combining them into a single `SET` clause, since Snowflake rejected a combined multi-property `SET` clause with a syntax error; `primary_key` and `attributes` each use their own dedicated `SET PRIMARY KEY = (...)` / `SET ATTRIBUTES (...)` statement forms per Snowflake's syntax reference +- `snowflake__get_drop_cortex_search_service_sql` macro for DROP DDL generation +- `snowflake__grant_cortex_search_service_usage` macro for GRANT DDL generation +- Integration tests for the Cortex Search Service materialization (`test_search_service`), verified via `DESCRIBE CORTEX SEARCH SERVICE` captured into a table, since `GET_DDL` support for search services is not documented ### Changed - `enable_versioning` now defaults to `true` — versioning is on by default for all `cortex_agent` models; set `enable_versioning: false` in `dbt_project.yml` or per-model config to opt out (e.g. in dev environments) +- `base_table` integration test model no longer creates `TEST_SEARCH_SERVICE` via a raw-SQL `post_hook`; it's now a separate `test_search_service` model using the `cortex_search_service` materialization ### Fixed - Singular integration tests now declare `-- depends_on: {{ ref('cortex_agent_test') }}` so `dbt build` runs models before tests diff --git a/README.md b/README.md index 5e1c87e..7b04821 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ A custom dbt materialization for deploying [Snowflake Cortex Agents](https://doc - dbt >= 1.0.0 - Snowflake adapter (`dbt-snowflake`) - Snowflake account with Cortex Agents enabled +- Snowflake account with Cortex Search enabled (for `cortex_search_service` models) ## Installation @@ -293,6 +294,76 @@ Notes specific to MCP servers: Refer to the [Snowflake CREATE MCP SERVER docs](https://docs.snowflake.com/en/sql-reference/sql/create-mcp-server) for the full and up-to-date specification reference. +## Cortex Search Services + +The package also ships a `cortex_search_service` materialization for deploying [Snowflake Cortex Search Services](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) — the backing search index used by a `cortex_search` tool on a Cortex Agent, or a `CORTEX_SEARCH_SERVICE_QUERY` tool on an MCP server. + +Unlike `cortex_agent` and `mcp_server`, this materialization is **not** a YAML passthrough. The model body is a normal dbt `SELECT` query, which becomes the `AS ` clause of `CREATE CORTEX SEARCH SERVICE`. All other DDL clauses (search column, attributes, warehouse, target lag, etc.) come from `config()`. + +Only the single-index syntax (`ON ` + `ATTRIBUTES`) is supported. If you need to search over multiple text fields, blend them into one column in the model's `SELECT` (e.g. `short_description || ' ' || long_description || ' ' || synonyms as search_text`) and keep the individual columns as `ATTRIBUTES` for filtering/display. + +> `PRIMARY KEY` columns must be `TEXT` — Cortex Search Service rejects numeric types for `PRIMARY KEY` regardless of precision/scale, failing with `Invalid column type NUMBER(n,0) for source query column `. Cast numeric ID columns to a string type, e.g. `id::varchar`, before using them in `primary_key`. + +```sql +-- models/search/product_search.sql +{{ config( + materialized='cortex_search_service', + search_column='search_text', + attributes=['id', 'title', 'category'], + target_lag='1 hour', + primary_key=['id'], + comment='Product search index for the shopping assistant agent' +) }} + +select + id, + title, + category, + short_description || ' ' || long_description || ' ' || synonyms as search_text +from {{ ref('products') }} +``` + +Run it: + +```bash +dbt run --select product_search +``` + +Verify in Snowflake: + +```sql +SHOW CORTEX SEARCH SERVICES IN SCHEMA my_db.my_schema; +DESCRIBE CORTEX SEARCH SERVICE my_db.my_schema.product_search; +``` + +### Config Options + +| Option | Type | Required | Description | +|---|---|---|---| +| `materialized` | string | Yes | Must be `'cortex_search_service'` | +| `search_column` | string | Yes | Column that Cortex Search indexes and searches over. Immutable after creation — changing it requires `dbt run --full-refresh`. | +| `attributes` | list | Yes | Non-empty list of column names to index as filterable/returnable attributes. Snowflake requires at least one. | +| `target_lag` | string | Yes | Maximum staleness of the index relative to the source query, e.g. `'1 hour'`, `'7 days'`. No Snowflake default. | +| `warehouse` | string | No | Warehouse used to refresh and build the index. Defaults to the warehouse in your dbt `target`. | +| `primary_key` | list | No | Column(s) uniquely identifying each row; enables optimized incremental refresh. | +| `embedding_model` | string | No | Vector embedding model, e.g. `'snowflake-arctic-embed-m-v1.5'`. Immutable after creation. | +| `refresh_mode` | string | No | `INCREMENTAL` (default) or `FULL`. Immutable after creation. | +| `initialize` | string | No | `ON_CREATE` (default, synchronous) or `ON_SCHEDULE` (deferred). Immutable after creation. | +| `full_index_build_interval_days` | number | No | Soft target for periodic full index rebuilds. Only meaningful with `primary_key` set. | +| `request_logging` | bool | No | Enables request logging for monitoring queries. Defaults to `false`. | +| `auto_suspend` | number | No | Seconds of inactivity before suspending. Minimum `1800` (30 minutes). | +| `comment` | string | No | Descriptive text visible in Snowflake. | +| `search_service_grants` | list | No | Role names to grant `USAGE` on the search service, e.g. `['my_role']`. | + +### How It Works + +- **First run for a service, or `dbt run --full-refresh`:** issues `CREATE OR REPLACE CORTEX SEARCH SERVICE ... AS ` with the full set of configured options. +- **Every other run:** issues `ALTER CORTEX SEARCH SERVICE ... SET` to update only the mutable properties (`target_lag`, `warehouse`, `comment`, `auto_suspend`, `request_logging`, `full_index_build_interval_days`, `attributes`, `primary_key`) in place. + +This split matters because `CREATE OR REPLACE` forces Snowflake to fully rebuild the search index from scratch. Unlike `mcp_server`, which issues `CREATE OR REPLACE` on every run because MCP server definitions are cheap to redefine, `cortex_search_service` avoids doing that on steady-state runs — recreating the service on every `dbt run` would discard Cortex Search's own incremental refresh and be wasteful and slow for anything beyond trivial data volumes. + +`search_column`, `embedding_model`, `refresh_mode`, `initialize`, and the defining query itself are immutable once the service is created. To change any of those, run `dbt run --full-refresh`. + ## Local Development ### Prerequisites @@ -331,6 +402,8 @@ SNOWFLAKE_TEST_AUTHENTICATOR=externalbrowser This script stages a clean copy of the package to avoid a Windows path-length issue caused by dbt's recursive local package installation, then runs `dbt deps` and `dbt build` from `integration_tests/`. +> CI always runs against a freshly created schema, so it only ever exercises the `cortex_search_service` CREATE path. To verify the ALTER-in-place path, run `.\scripts\run_tests.ps1` twice in a row against the same persistent dev schema and confirm the second run issues `ALTER CORTEX SEARCH SERVICE ... SET` instead of another `CREATE OR REPLACE`. + To install packages only (no build): ```powershell diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 5356837..0f4faaa 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -18,6 +18,6 @@ models: +schema: INTEGRATION_TESTS vars: - # Points at the Cortex Search Service created by base_table's post_hook. + # Points at the Cortex Search Service created by the test_search_service model. # The schema suffix matches dbt's custom-schema behaviour (target.schema ~ '_INTEGRATION_TESTS'). test_cortex_search_service: "{{ target.database }}.{{ target.schema }}_INTEGRATION_TESTS.TEST_SEARCH_SERVICE" diff --git a/integration_tests/models/base_table.sql b/integration_tests/models/base_table.sql index 0bc7bc5..9485de3 100644 --- a/integration_tests/models/base_table.sql +++ b/integration_tests/models/base_table.sql @@ -1,10 +1,9 @@ {{- config( materialized='table', - alias='BASE_TABLE', - post_hook="create or replace cortex search service {{ this.database }}.{{ this.schema }}.TEST_SEARCH_SERVICE on description attributes id warehouse = {{ target.warehouse }} target_lag = '7 days' as (select id, description from {{ this }})" + alias='BASE_TABLE' ) -}} --- Minimal base table used by test_semantic_view and TEST_SEARCH_SERVICE in the integration test suite. +-- Minimal base table used by test_semantic_view and test_search_service in the integration test suite. select * from values (1, 'Revenue figures show strong growth in the enterprise segment last quarter.', 100.00), (2, 'Chargeback rates increased by 2% among small business clients in Q3.', 200.00), diff --git a/integration_tests/models/schema.yml b/integration_tests/models/schema.yml index e39fb57..654026d 100644 --- a/integration_tests/models/schema.yml +++ b/integration_tests/models/schema.yml @@ -2,11 +2,24 @@ version: 2 models: - name: base_table - description: "Minimal base table used by test_semantic_view." + description: "Minimal base table used by test_semantic_view and test_search_service." - name: test_semantic_view description: "Minimal semantic view referenced via ref() in cortex_agent_test." + - name: test_search_service + description: "Comprehensive Cortex Search Service exercising every config option — search_column, attributes, target_lag, primary_key, auto_suspend, comment, and grants. `warehouse` is intentionally left unset to test the fallback to target.warehouse. auto_suspend is set to the minimum (1800s) to limit serving compute cost on this test object." + config: + alias: test_search_service + search_column: description + attributes: ['id'] + target_lag: '7 days' + primary_key: ['id'] + auto_suspend: 1800 + comment: 'Full integration test search service — exercises every config option' + tags: ['integration'] + search_service_grants: ['dbt_demo_role'] + - name: cortex_agent_test description: "Comprehensive agent exercising every config and spec option — comment, profile, alias, tags, orchestration budget, all instruction fields, both tool types with full input_schema, tool_resources, and grants." config: @@ -27,3 +40,6 @@ models: - name: mcp_server_test_describe description: "Captures DESCRIBE MCP SERVER output for mcp_server_test so singular tests can assert on the spec (GET_DDL does not support MCP servers)." + + - name: test_search_service_describe + description: "Captures DESCRIBE CORTEX SEARCH SERVICE output for test_search_service so singular tests can assert on its properties (GET_DDL support for search services is not documented)." diff --git a/integration_tests/models/test_search_service.sql b/integration_tests/models/test_search_service.sql new file mode 100644 index 0000000..506953b --- /dev/null +++ b/integration_tests/models/test_search_service.sql @@ -0,0 +1,22 @@ +{{ + config( + materialized='cortex_search_service', + alias='TEST_SEARCH_SERVICE', + search_column='description', + attributes=['id'], + target_lag='7 days', + primary_key=['id'], + auto_suspend=1800, + comment='Full integration test search service — exercises every config option', + tags=['integration'], + search_service_grants=['dbt_demo_role'] + ) +}} + +-- `warehouse` is intentionally omitted from config above to exercise the +-- materialization's "defaults to target.warehouse" fallback. +-- +-- id is cast to a text type because Cortex Search Service's PRIMARY KEY clause only +-- accepts TEXT columns, not numeric types (confirmed via "Invalid column type +-- NUMBER(n,0) for source query column ID" regardless of precision/scale). +select id::varchar as id, description from {{ ref('base_table') }} diff --git a/integration_tests/models/test_search_service_describe.sql b/integration_tests/models/test_search_service_describe.sql new file mode 100644 index 0000000..2855899 --- /dev/null +++ b/integration_tests/models/test_search_service_describe.sql @@ -0,0 +1,20 @@ +-- GET_DDL support for Cortex Search Services is not documented, so we verify them via +-- DESCRIBE CORTEX SEARCH SERVICE. DESCRIBE can't be used as a subquery, so this model +-- captures its output into a table: the pre-hook runs DESCRIBE on the connection +-- immediately before the SELECT, and result_scan(last_query_id()) reads that DESCRIBE's +-- result set. The ref(...) in the pre-hook also forces dbt to create the search service +-- first. Singular tests assert on this table. +{{ + config( + materialized='table', + pre_hook="describe cortex search service {{ ref('test_search_service') }}" + ) +}} + +select + "search_column" as search_column, + "attribute_columns" as attribute_columns, + "target_lag" as target_lag, + "warehouse" as warehouse, + "comment" as comment +from table(result_scan(last_query_id())) diff --git a/integration_tests/tests/cortex_search_service_test_exists.sql b/integration_tests/tests/cortex_search_service_test_exists.sql new file mode 100644 index 0000000..a048208 --- /dev/null +++ b/integration_tests/tests/cortex_search_service_test_exists.sql @@ -0,0 +1,9 @@ +-- Fails if the search service's DESCRIBE capture returned no rows, i.e. the service +-- does not exist in Snowflake. +-- Returns 0 rows on success (standard dbt test contract). +-- depends_on: {{ ref('test_search_service_describe') }} + +select 'cortex search service does not exist' as error +where ( + select count(*) from {{ ref('test_search_service_describe') }} +) = 0 diff --git a/integration_tests/tests/cortex_search_service_test_spec_content.sql b/integration_tests/tests/cortex_search_service_test_spec_content.sql new file mode 100644 index 0000000..e0e0be0 --- /dev/null +++ b/integration_tests/tests/cortex_search_service_test_spec_content.sql @@ -0,0 +1,21 @@ +-- Validates that key DDL properties are present on the search service, read from the +-- DESCRIBE CORTEX SEARCH SERVICE output captured by test_search_service_describe: +-- - search column, attributes, target_lag, comment +-- Returns one row per failed assertion; 0 rows = all pass (standard dbt test contract). +-- depends_on: {{ ref('test_search_service_describe') }} + +with described as ( + select * from {{ ref('test_search_service_describe') }} +), + +assertions as ( + select 'search column not set to DESCRIPTION' as error from described where upper(search_column) != 'DESCRIPTION' + union all + select 'attribute columns missing ID' as error from described where not contains(upper(attribute_columns), 'ID') + union all + select 'target_lag not set to 7 days' as error from described where not contains(lower(target_lag), '7 day') + union all + select 'comment missing expected text' as error from described where not contains(comment, 'Full integration test search service') +) + +select * from assertions diff --git a/macros/materializations/cortex_search_service.sql b/macros/materializations/cortex_search_service.sql new file mode 100644 index 0000000..3581f23 --- /dev/null +++ b/macros/materializations/cortex_search_service.sql @@ -0,0 +1,100 @@ +-- Custom dbt materialization for Snowflake Cortex Search Services. +-- Unlike cortex_agent/mcp_server, the model body is a NORMAL dbt SELECT query — it +-- becomes the `AS ` clause of the search service, not a passthrough YAML spec. +-- Supports single-index syntax only (ON + ATTRIBUTES) — TEXT/VECTOR multi-index +-- definitions are not supported. +-- +-- Refresh behaviour: CREATE OR REPLACE only runs on the first dbt run for a given +-- service, or when `dbt run --full-refresh` is passed. On every subsequent normal run, +-- mutable properties (target_lag, warehouse, comment, auto_suspend, request_logging, +-- full_index_build_interval_days, attributes, primary_key) are updated in place via +-- ALTER ... SET, so the search index is never gratuitously rebuilt. +-- +-- Config options: +-- search_column (string, required) : column CORTEX SEARCH SERVICE indexes and searches over +-- attributes (list, required) : non-empty list of filterable/returnable column names +-- target_lag (string, required) : e.g. '1 hour', '7 days' — no Snowflake default +-- warehouse (string, optional) : defaults to target.warehouse +-- primary_key (list, optional) : enables optimized incremental refresh +-- embedding_model (string, optional) : immutable after create +-- refresh_mode (string, optional) : INCREMENTAL|FULL, immutable after create +-- initialize (string, optional) : ON_CREATE|ON_SCHEDULE, immutable after create +-- full_index_build_interval_days (int, optional) +-- request_logging (bool, optional) +-- auto_suspend (int, optional) : seconds, minimum 1800 +-- comment (string, optional) +-- search_service_grants (list, optional) : role names to grant USAGE on the service +-- +-- IMPORTANT for consumers: if you change a model away from cortex_search_service +-- materialization, dbt will attempt DROP VIEW IF EXISTS, which may silently no-op +-- rather than dropping the search service. Drop it manually before switching: +-- DROP CORTEX SEARCH SERVICE IF EXISTS ..; + +{% materialization cortex_search_service, adapter='snowflake' %} + + {%- set search_column = config.get('search_column', default=none) -%} + {%- set attributes = config.get('attributes', default=none) -%} + {%- set warehouse = config.get('warehouse', default=target.warehouse) -%} + {%- set target_lag = config.get('target_lag', default=none) -%} + {%- set primary_key = config.get('primary_key', default=none) -%} + {%- set embedding_model = config.get('embedding_model', default=none) -%} + {%- set refresh_mode = config.get('refresh_mode', default=none) -%} + {%- set initialize = config.get('initialize', default=none) -%} + {%- set full_index_build_interval_days = config.get('full_index_build_interval_days', default=none) -%} + {%- set request_logging = config.get('request_logging', default=none) -%} + {%- set auto_suspend = config.get('auto_suspend', default=none) -%} + {%- set comment = config.get('comment', default=none) -%} + {%- set search_service_grants = config.get('search_service_grants', default=[]) -%} + + {%- if search_column is none %} + {{ exceptions.raise_compiler_error("cortex_search_service models require a `search_column` config option (the column CORTEX SEARCH SERVICE will index and search over).") }} + {%- endif %} + {%- if attributes is none or attributes | length == 0 %} + {{ exceptions.raise_compiler_error("cortex_search_service models require a non-empty `attributes` config option (list of column names) — ATTRIBUTES is required by Snowflake's CREATE CORTEX SEARCH SERVICE syntax.") }} + {%- endif %} + {%- if target_lag is none %} + {{ exceptions.raise_compiler_error("cortex_search_service models require a `target_lag` config option (e.g. '1 hour', '7 days') — Snowflake has no default and requires this value on every CREATE CORTEX SEARCH SERVICE.") }} + {%- endif %} + + -- dbt has no native 'cortex search service' relation type. 'view' is used as a + -- placeholder for graph tracking only — the actual DDL is CREATE/ALTER CORTEX SEARCH SERVICE. + {%- set target_relation = api.Relation.create( + identifier=this.identifier, + schema=this.schema, + database=this.database, + type='view' + ) -%} + + {{ run_hooks(pre_hooks) }} + + {%- set _show_services_sql -%} + SHOW CORTEX SEARCH SERVICES LIKE '{{ target_relation.identifier }}' IN SCHEMA {{ target_relation.database }}.{{ target_relation.schema }} + {%- endset -%} + {%- set _service_rows = run_query(_show_services_sql) -%} + {%- set _service_exists = (_service_rows | length) > 0 -%} + + {%- if _service_exists and not should_full_refresh() %} + + {% call statement('main') %} + {{ dbt_cortex_agent.snowflake__alter_cortex_search_service(target_relation, attributes, warehouse, target_lag, primary_key, full_index_build_interval_days, request_logging, auto_suspend, comment) }} + {% endcall %} + + {%- else %} + + {% call statement('main') %} + {{ dbt_cortex_agent.snowflake__create_cortex_search_service(target_relation, sql, search_column, attributes, warehouse, target_lag, primary_key, embedding_model, refresh_mode, initialize, full_index_build_interval_days, request_logging, auto_suspend, comment) }} + {% endcall %} + + {%- endif %} + + {{ run_hooks(post_hooks) }} + + {%- if search_service_grants | length > 0 %} + {% call statement('grants') %} + {{ dbt_cortex_agent.snowflake__grant_cortex_search_service_usage(target_relation, search_service_grants) }} + {% endcall %} + {%- endif %} + + {{ return({'relations': [target_relation]}) }} + +{% endmaterialization %} diff --git a/macros/relations/cortex_search_service/alter_search_service.sql b/macros/relations/cortex_search_service/alter_search_service.sql new file mode 100644 index 0000000..35e50a1 --- /dev/null +++ b/macros/relations/cortex_search_service/alter_search_service.sql @@ -0,0 +1,49 @@ +-- Generates and executes ALTER CORTEX SEARCH SERVICE ... SET DDL, updating the mutable +-- properties of an existing Cortex Search Service in place without rebuilding its index. +-- Called by the cortex_search_service materialization on every run after the first, +-- as long as --full-refresh was not passed. +-- +-- Emits one full ALTER statement per configured property (e.g. +-- `alter cortex search service if exists set target_lag = '1 hour';`) rather +-- than combining multiple scalar properties into a single SET clause — Snowflake +-- rejected a combined `SET target_lag = ... warehouse = ... comment = ...` clause with +-- a syntax error on the first property, while the single-property form matches +-- Snowflake's own documented examples. Multiple `;`-separated statements in one macro +-- call is the same pattern already used by snowflake__grant_cortex_agent_usage for +-- multi-role grants. +-- +-- PRIMARY KEY and ATTRIBUTES are each their own dedicated ALTER form per Snowflake's +-- syntax reference (not part of the general SET clause above), confirmed via docs: +-- ALTER CORTEX SEARCH SERVICE mysvc SET PRIMARY KEY = (region, agent_id); +-- ALTER CORTEX SEARCH SERVICE mysvc SET ATTRIBUTES (category, region); +-- +-- Args mirror snowflake__create_cortex_search_service, minus the immutable properties +-- (search_column, embedding_model, refresh_mode, initialize, and the defining query itself) +-- which cannot be changed without CREATE OR REPLACE. +{% macro snowflake__alter_cortex_search_service(relation, attributes, warehouse, target_lag, primary_key, full_index_build_interval_days, request_logging, auto_suspend, comment) %} + {%- set _relation_name = relation.database ~ '.' ~ relation.schema ~ '.' ~ relation.identifier -%} + {%- if target_lag is not none %} + alter cortex search service if exists {{ _relation_name }} set target_lag = '{{ target_lag }}'; + {%- endif %} + {%- if warehouse is not none %} + alter cortex search service if exists {{ _relation_name }} set warehouse = {{ warehouse }}; + {%- endif %} + {%- if full_index_build_interval_days is not none %} + alter cortex search service if exists {{ _relation_name }} set full_index_build_interval_days = {{ full_index_build_interval_days }}; + {%- endif %} + {%- if request_logging is not none %} + alter cortex search service if exists {{ _relation_name }} set request_logging = {{ request_logging | lower }}; + {%- endif %} + {%- if auto_suspend is not none %} + alter cortex search service if exists {{ _relation_name }} set auto_suspend = {{ auto_suspend }}; + {%- endif %} + {%- if comment is not none %} + alter cortex search service if exists {{ _relation_name }} set comment = '{{ comment }}'; + {%- endif %} + {%- if primary_key is not none and primary_key | length > 0 %} + alter cortex search service if exists {{ _relation_name }} set primary key = ({{ primary_key | join(', ') }}); + {%- endif %} + {%- if attributes is not none and attributes | length > 0 %} + alter cortex search service if exists {{ _relation_name }} set attributes ({{ attributes | join(', ') }}); + {%- endif %} +{% endmacro %} diff --git a/macros/relations/cortex_search_service/create_search_service.sql b/macros/relations/cortex_search_service/create_search_service.sql new file mode 100644 index 0000000..2afa289 --- /dev/null +++ b/macros/relations/cortex_search_service/create_search_service.sql @@ -0,0 +1,56 @@ +-- Generates and executes CREATE OR REPLACE CORTEX SEARCH SERVICE DDL for a Snowflake +-- Cortex Search Service, using the single-index (ON + ATTRIBUTES) syntax only. +-- Called by the cortex_search_service materialization on the first run for a given +-- service, or on any `dbt run --full-refresh`. +-- Args: +-- relation : the target relation object (database, schema, identifier) +-- query : compiled SELECT query from the model body (the AS clause) +-- search_column : required — column to search over +-- attributes : required — list of attribute column names +-- warehouse : required — warehouse name used to (re)build the index +-- target_lag : required — e.g. '1 hour', '7 days' +-- primary_key : optional list of primary key column names +-- embedding_model : optional — immutable after create +-- refresh_mode : optional — INCREMENTAL|FULL, immutable after create +-- initialize : optional — ON_CREATE|ON_SCHEDULE, immutable after create +-- full_index_build_interval_days : optional int +-- request_logging : optional bool +-- auto_suspend : optional int (seconds, min 1800) +-- comment : optional string +{% macro snowflake__create_cortex_search_service(relation, query, search_column, attributes, warehouse, target_lag, primary_key, embedding_model, refresh_mode, initialize, full_index_build_interval_days, request_logging, auto_suspend, comment) %} + + create or replace cortex search service + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + on {{ search_column }} + {%- if primary_key is not none and primary_key | length > 0 %} + primary key ({{ primary_key | join(', ') }}) + {%- endif %} + attributes {{ attributes | join(', ') }} + warehouse = {{ warehouse }} + target_lag = '{{ target_lag }}' + {%- if embedding_model is not none %} + embedding_model = '{{ embedding_model }}' + {%- endif %} + {%- if refresh_mode is not none %} + refresh_mode = {{ refresh_mode }} + {%- endif %} + {%- if initialize is not none %} + initialize = {{ initialize }} + {%- endif %} + {%- if full_index_build_interval_days is not none %} + full_index_build_interval_days = {{ full_index_build_interval_days }} + {%- endif %} + {%- if request_logging is not none %} + request_logging = {{ request_logging | lower }} + {%- endif %} + {%- if auto_suspend is not none %} + auto_suspend = {{ auto_suspend }} + {%- endif %} + {%- if comment is not none %} + comment = '{{ comment }}' + {%- endif %} + as ( +{{ query }} + ) + +{% endmacro %} diff --git a/macros/relations/cortex_search_service/drop_search_service.sql b/macros/relations/cortex_search_service/drop_search_service.sql new file mode 100644 index 0000000..92a7e22 --- /dev/null +++ b/macros/relations/cortex_search_service/drop_search_service.sql @@ -0,0 +1,8 @@ +-- Generates DROP CORTEX SEARCH SERVICE DDL for a Snowflake Cortex Search Service. +-- Called by dbt internals when a cortex_search_service model is removed or replaced. +{% macro snowflake__get_drop_cortex_search_service_sql(relation) %} + + drop cortex search service if exists + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + +{% endmacro %} diff --git a/macros/relations/cortex_search_service/grant_search_service.sql b/macros/relations/cortex_search_service/grant_search_service.sql new file mode 100644 index 0000000..6c33af7 --- /dev/null +++ b/macros/relations/cortex_search_service/grant_search_service.sql @@ -0,0 +1,12 @@ +-- Generates GRANT USAGE ON CORTEX SEARCH SERVICE DDL for each role in the grants list. +-- Called by the cortex_search_service materialization when the `search_service_grants` config is set. +-- Args: +-- relation : the target relation object (database, schema, identifier) +-- roles : list of role names to grant USAGE to +{% macro snowflake__grant_cortex_search_service_usage(relation, roles) %} + {%- for role in roles %} + grant usage on cortex search service + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + to role {{ role }}; + {%- endfor %} +{% endmacro %}