Add cortex_search_service materialization for Snowflake Cortex Search

CHANGELOG.md

@@ -20,9 +20,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `snowflake__get_drop_mcp_server_sql` macro for DROP DDL generation
 - Integration tests for the MCP server materialization (`mcp_server_test`), verified via `DESCRIBE MCP SERVER` captured into a table, since `GET_DDL` does not support MCP servers
 - Local development setup: `integration_tests/.env` template, `scripts/run_tests.ps1` runner, and `profiles.yml` SSO support (`externalbrowser` authenticator)
+- `cortex_search_service` materialization for [Snowflake Cortex Search Services](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) — unlike `cortex_agent`/`mcp_server`, the model body is a normal dbt `SELECT` query (not a passthrough YAML spec) that becomes the `AS <query>` clause; supports the single-index (`ON` + `ATTRIBUTES`) syntax only
+- `search_column`, `attributes`, and `target_lag` required config options for `cortex_search_service` — Snowflake has no default for any of these
+- `warehouse`, `primary_key`, `embedding_model`, `refresh_mode`, `initialize`, `full_index_build_interval_days`, `request_logging`, `auto_suspend`, and `comment` optional config options for `cortex_search_service`
+- `search_service_grants` config option — list of role names to grant `USAGE` on the search service after creation, mirrors `agent_grants`
+- Refresh behavior: `cortex_search_service` issues `CREATE OR REPLACE CORTEX SEARCH SERVICE` only on the first run for a given service or on `dbt run --full-refresh`; subsequent runs use `ALTER CORTEX SEARCH SERVICE ... SET` to update mutable properties in place, avoiding a full index rebuild on every `dbt run`
+- `snowflake__create_cortex_search_service` macro for CREATE DDL generation
+- `snowflake__alter_cortex_search_service` macro for ALTER ... SET DDL generation — emits one `ALTER CORTEX SEARCH SERVICE ... SET <property> = <value>;` statement per scalar mutable property (target_lag, warehouse, comment, auto_suspend, request_logging, full_index_build_interval_days) rather than combining them into a single `SET` clause, since Snowflake rejected a combined multi-property `SET` clause with a syntax error; `primary_key` and `attributes` each use their own dedicated `SET PRIMARY KEY = (...)` / `SET ATTRIBUTES (...)` statement forms per Snowflake's syntax reference
+- `snowflake__get_drop_cortex_search_service_sql` macro for DROP DDL generation
+- `snowflake__grant_cortex_search_service_usage` macro for GRANT DDL generation
+- Integration tests for the Cortex Search Service materialization (`test_search_service`), verified via `DESCRIBE CORTEX SEARCH SERVICE` captured into a table, since `GET_DDL` support for search services is not documented
 
 ### Changed
 - `enable_versioning` now defaults to `true` — versioning is on by default for all `cortex_agent` models; set `enable_versioning: false` in `dbt_project.yml` or per-model config to opt out (e.g. in dev environments)
+- `base_table` integration test model no longer creates `TEST_SEARCH_SERVICE` via a raw-SQL `post_hook`; it's now a separate `test_search_service` model using the `cortex_search_service` materialization
 
 ### Fixed
 - Singular integration tests now declare `-- depends_on: {{ ref('cortex_agent_test') }}` so `dbt build` runs models before tests

README.md

@@ -7,6 +7,7 @@ A custom dbt materialization for deploying [Snowflake Cortex Agents](https://doc
 - dbt >= 1.0.0
 - Snowflake adapter (`dbt-snowflake`)
 - Snowflake account with Cortex Agents enabled
+- Snowflake account with Cortex Search enabled (for `cortex_search_service` models)
 
 ## Installation
 
@@ -293,6 +294,76 @@ Notes specific to MCP servers:
 
 Refer to the [Snowflake CREATE MCP SERVER docs](https://docs.snowflake.com/en/sql-reference/sql/create-mcp-server) for the full and up-to-date specification reference.
 
+## Cortex Search Services
+
+The package also ships a `cortex_search_service` materialization for deploying [Snowflake Cortex Search Services](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) — the backing search index used by a `cortex_search` tool on a Cortex Agent, or a `CORTEX_SEARCH_SERVICE_QUERY` tool on an MCP server.
+
+Unlike `cortex_agent` and `mcp_server`, this materialization is **not** a YAML passthrough. The model body is a normal dbt `SELECT` query, which becomes the `AS <query>` clause of `CREATE CORTEX SEARCH SERVICE`. All other DDL clauses (search column, attributes, warehouse, target lag, etc.) come from `config()`.
+
+Only the single-index syntax (`ON <search_column>` + `ATTRIBUTES`) is supported. If you need to search over multiple text fields, blend them into one column in the model's `SELECT` (e.g. `short_description || ' ' || long_description || ' ' || synonyms as search_text`) and keep the individual columns as `ATTRIBUTES` for filtering/display.
+
+> `PRIMARY KEY` columns must be `TEXT` — Cortex Search Service rejects numeric types for `PRIMARY KEY` regardless of precision/scale, failing with `Invalid column type NUMBER(n,0) for source query column <col>`. Cast numeric ID columns to a string type, e.g. `id::varchar`, before using them in `primary_key`.
+
+```sql
+-- models/search/product_search.sql
+{{ config(
+    materialized='cortex_search_service',
+    search_column='search_text',
+    attributes=['id', 'title', 'category'],
+    target_lag='1 hour',
+    primary_key=['id'],
+    comment='Product search index for the shopping assistant agent'
+) }}
+
+select
+    id,
+    title,
+    category,
+    short_description || ' ' || long_description || ' ' || synonyms as search_text
+from {{ ref('products') }}
+```
+
+Run it:
+
+```bash
+dbt run --select product_search
+```
+
+Verify in Snowflake:
+
+```sql
+SHOW CORTEX SEARCH SERVICES IN SCHEMA my_db.my_schema;
+DESCRIBE CORTEX SEARCH SERVICE my_db.my_schema.product_search;
+```
+
+### Config Options
+
+| Option | Type | Required | Description |
+|---|---|---|---|
+| `materialized` | string | Yes | Must be `'cortex_search_service'` |
+| `search_column` | string | Yes | Column that Cortex Search indexes and searches over. Immutable after creation — changing it requires `dbt run --full-refresh`. |
+| `attributes` | list | Yes | Non-empty list of column names to index as filterable/returnable attributes. Snowflake requires at least one. |
+| `target_lag` | string | Yes | Maximum staleness of the index relative to the source query, e.g. `'1 hour'`, `'7 days'`. No Snowflake default. |
+| `warehouse` | string | No | Warehouse used to refresh and build the index. Defaults to the warehouse in your dbt `target`. |
+| `primary_key` | list | No | Column(s) uniquely identifying each row; enables optimized incremental refresh. |
+| `embedding_model` | string | No | Vector embedding model, e.g. `'snowflake-arctic-embed-m-v1.5'`. Immutable after creation. |
+| `refresh_mode` | string | No | `INCREMENTAL` (default) or `FULL`. Immutable after creation. |
+| `initialize` | string | No | `ON_CREATE` (default, synchronous) or `ON_SCHEDULE` (deferred). Immutable after creation. |
+| `full_index_build_interval_days` | number | No | Soft target for periodic full index rebuilds. Only meaningful with `primary_key` set. |
+| `request_logging` | bool | No | Enables request logging for monitoring queries. Defaults to `false`. |
+| `auto_suspend` | number | No | Seconds of inactivity before suspending. Minimum `1800` (30 minutes). |
+| `comment` | string | No | Descriptive text visible in Snowflake. |
+| `search_service_grants` | list | No | Role names to grant `USAGE` on the search service, e.g. `['my_role']`. |
+
+### How It Works
+
+- **First run for a service, or `dbt run --full-refresh`:** issues `CREATE OR REPLACE CORTEX SEARCH SERVICE ... AS <query>` with the full set of configured options.
+- **Every other run:** issues `ALTER CORTEX SEARCH SERVICE ... SET` to update only the mutable properties (`target_lag`, `warehouse`, `comment`, `auto_suspend`, `request_logging`, `full_index_build_interval_days`, `attributes`, `primary_key`) in place.
+
+This split matters because `CREATE OR REPLACE` forces Snowflake to fully rebuild the search index from scratch. Unlike `mcp_server`, which issues `CREATE OR REPLACE` on every run because MCP server definitions are cheap to redefine, `cortex_search_service` avoids doing that on steady-state runs — recreating the service on every `dbt run` would discard Cortex Search's own incremental refresh and be wasteful and slow for anything beyond trivial data volumes.
+
+`search_column`, `embedding_model`, `refresh_mode`, `initialize`, and the defining query itself are immutable once the service is created. To change any of those, run `dbt run --full-refresh`.
+
 ## Local Development
 
 ### Prerequisites
@@ -331,6 +402,8 @@ SNOWFLAKE_TEST_AUTHENTICATOR=externalbrowser
 
 This script stages a clean copy of the package to avoid a Windows path-length issue caused by dbt's recursive local package installation, then runs `dbt deps` and `dbt build` from `integration_tests/`.
 
+> CI always runs against a freshly created schema, so it only ever exercises the `cortex_search_service` CREATE path. To verify the ALTER-in-place path, run `.\scripts\run_tests.ps1` twice in a row against the same persistent dev schema and confirm the second run issues `ALTER CORTEX SEARCH SERVICE ... SET` instead of another `CREATE OR REPLACE`.
+
 To install packages only (no build):
 
 ```powershell

integration_tests/dbt_project.yml

@@ -18,6 +18,6 @@ models:
     +schema: INTEGRATION_TESTS
 
 vars:
-  # Points at the Cortex Search Service created by base_table's post_hook.
+  # Points at the Cortex Search Service created by the test_search_service model.
   # The schema suffix matches dbt's custom-schema behaviour (target.schema ~ '_INTEGRATION_TESTS').
   test_cortex_search_service: "{{ target.database }}.{{ target.schema }}_INTEGRATION_TESTS.TEST_SEARCH_SERVICE"

integration_tests/models/base_table.sql

@@ -1,10 +1,9 @@
 {{- config(
     materialized='table',
-    alias='BASE_TABLE',
-    post_hook="create or replace cortex search service {{ this.database }}.{{ this.schema }}.TEST_SEARCH_SERVICE on description attributes id warehouse = {{ target.warehouse }} target_lag = '7 days' as (select id, description from {{ this }})"
+    alias='BASE_TABLE'
 ) -}}
 
--- Minimal base table used by test_semantic_view and TEST_SEARCH_SERVICE in the integration test suite.
+-- Minimal base table used by test_semantic_view and test_search_service in the integration test suite.
 select * from values
     (1, 'Revenue figures show strong growth in the enterprise segment last quarter.',      100.00),
     (2, 'Chargeback rates increased by 2% among small business clients in Q3.',           200.00),

integration_tests/models/schema.yml

@@ -2,11 +2,24 @@ version: 2
 
 models:
   - name: base_table
-    description: "Minimal base table used by test_semantic_view."
+    description: "Minimal base table used by test_semantic_view and test_search_service."
 
   - name: test_semantic_view
     description: "Minimal semantic view referenced via ref() in cortex_agent_test."
 
+  - name: test_search_service
+    description: "Comprehensive Cortex Search Service exercising every config option — search_column, attributes, target_lag, primary_key, auto_suspend, comment, and grants. `warehouse` is intentionally left unset to test the fallback to target.warehouse. auto_suspend is set to the minimum (1800s) to limit serving compute cost on this test object."
+    config:
+      alias: test_search_service
+      search_column: description
+      attributes: ['id']
+      target_lag: '7 days'
+      primary_key: ['id']
+      auto_suspend: 1800
+      comment: 'Full integration test search service — exercises every config option'
+      tags: ['integration']
+      search_service_grants: ['dbt_demo_role']
+
   - name: cortex_agent_test
     description: "Comprehensive agent exercising every config and spec option — comment, profile, alias, tags, orchestration budget, all instruction fields, both tool types with full input_schema, tool_resources, and grants."
     config:
@@ -27,3 +40,6 @@ models:
 
   - name: mcp_server_test_describe
     description: "Captures DESCRIBE MCP SERVER output for mcp_server_test so singular tests can assert on the spec (GET_DDL does not support MCP servers)."
+
+  - name: test_search_service_describe
+    description: "Captures DESCRIBE CORTEX SEARCH SERVICE output for test_search_service so singular tests can assert on its properties (GET_DDL support for search services is not documented)."

integration_tests/models/test_search_service.sql

@@ -0,0 +1,22 @@
+{{
+    config(
+        materialized='cortex_search_service',
+        alias='TEST_SEARCH_SERVICE',
+        search_column='description',
+        attributes=['id'],
+        target_lag='7 days',
+        primary_key=['id'],
+        auto_suspend=1800,
+        comment='Full integration test search service — exercises every config option',
+        tags=['integration'],
+        search_service_grants=['dbt_demo_role']
+    )
+}}
+
+-- `warehouse` is intentionally omitted from config above to exercise the
+-- materialization's "defaults to target.warehouse" fallback.
+--
+-- id is cast to a text type because Cortex Search Service's PRIMARY KEY clause only
+-- accepts TEXT columns, not numeric types (confirmed via "Invalid column type
+-- NUMBER(n,0) for source query column ID" regardless of precision/scale).
+select id::varchar as id, description from {{ ref('base_table') }}

integration_tests/models/test_search_service_describe.sql

@@ -0,0 +1,20 @@
+-- GET_DDL support for Cortex Search Services is not documented, so we verify them via
+-- DESCRIBE CORTEX SEARCH SERVICE. DESCRIBE can't be used as a subquery, so this model
+-- captures its output into a table: the pre-hook runs DESCRIBE on the connection
+-- immediately before the SELECT, and result_scan(last_query_id()) reads that DESCRIBE's
+-- result set. The ref(...) in the pre-hook also forces dbt to create the search service
+-- first. Singular tests assert on this table.
+{{
+    config(
+        materialized='table',
+        pre_hook="describe cortex search service {{ ref('test_search_service') }}"
+    )
+}}
+
+select
+    "search_column" as search_column,
+    "attribute_columns" as attribute_columns,
+    "target_lag" as target_lag,
+    "warehouse" as warehouse,
+    "comment" as comment
+from table(result_scan(last_query_id()))

integration_tests/tests/cortex_search_service_test_exists.sql

@@ -0,0 +1,9 @@
+-- Fails if the search service's DESCRIBE capture returned no rows, i.e. the service
+-- does not exist in Snowflake.
+-- Returns 0 rows on success (standard dbt test contract).
+-- depends_on: {{ ref('test_search_service_describe') }}
+
+select 'cortex search service does not exist' as error
+where (
+  select count(*) from {{ ref('test_search_service_describe') }}
+) = 0

integration_tests/tests/cortex_search_service_test_spec_content.sql

@@ -0,0 +1,21 @@
+-- Validates that key DDL properties are present on the search service, read from the
+-- DESCRIBE CORTEX SEARCH SERVICE output captured by test_search_service_describe:
+--   - search column, attributes, target_lag, comment
+-- Returns one row per failed assertion; 0 rows = all pass (standard dbt test contract).
+-- depends_on: {{ ref('test_search_service_describe') }}
+
+with described as (
+  select * from {{ ref('test_search_service_describe') }}
+),
+
+assertions as (
+  select 'search column not set to DESCRIPTION' as error from described where upper(search_column) != 'DESCRIPTION'
+  union all
+  select 'attribute columns missing ID'         as error from described where not contains(upper(attribute_columns), 'ID')
+  union all
+  select 'target_lag not set to 7 days'         as error from described where not contains(lower(target_lag), '7 day')
+  union all
+  select 'comment missing expected text'        as error from described where not contains(comment, 'Full integration test search service')
+)
+
+select * from assertions