From 42ca974399e92148c14848a6305ca897ea325ab3 Mon Sep 17 00:00:00 2001 From: database-tycoon Date: Fri, 2 Jan 2026 18:18:19 -0500 Subject: [PATCH] feat: Add Claude skills and remove redundant agents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Changes ### Added - snowtower-user skill: End-user guide for requesting access and connecting - snowtower-admin skill: Admin guide for SnowDDL operations and troubleshooting ### Removed - 24 redundant agent files from .claude/agents/ - CONSOLIDATION_SUMMARY.md (no longer needed) ### Changed - Bumped version to 0.2.0 - Renamed docs/agents/ to docs/llm-context/ - Updated README to reference skills instead of agents - Updated CLAUDE.md with new paths ### Skills Overview | Skill | Purpose | |-------|---------| | snowtower-user | End-users requesting access | | snowtower-admin | Infrastructure administrators | | snowtower-maintainer | Project/docs maintenance | 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- .claude/CONSOLIDATION_SUMMARY.md | 268 --------- .claude/agents/ADD_USER_AGENT.md | 21 - .claude/agents/CICD_AGENT.md | 50 -- .claude/agents/CONFIG_AGENT.md | 63 -- .claude/agents/COST_AGENT.md | 21 - .claude/agents/DATA_ENGINEER_AGENT.md | 31 - .claude/agents/DEPLOYMENT_AGENT.md | 69 --- .../DEPLOYMENT_TROUBLESHOOTING_AGENT.md | 74 --- .claude/agents/DOCS_AGENT.md | 168 ------ .claude/agents/ONBOARDING_AGENT.md | 45 -- .claude/agents/OPTIMIZED_META_AGENT.md | 180 ------ .claude/agents/PROJECT_ARCHITECT_AGENT.md | 146 ----- .claude/agents/PYTHON_DEVELOPMENT_AGENT.md | 224 ------- .claude/agents/SECURITY_AGENT.md | 96 --- .claude/agents/SNOWDDL_EXPERT_AGENT.md | 213 ------- .claude/agents/STATE_COMPARISON_AGENT.md | 21 - ...p-pre-consolidation-20251009-101750.tar.gz | Bin 71914 -> 0 bytes .claude/agents/docs-architect.md | 42 -- .claude/agents/mermaid-diagram-architect.md | 191 ------ .claude/agents/snow-cli-expert.md | 75 --- .claude/agents/snowflake-expert.md | 203 ------- .../agents/snowtower-operations-manager.md | 556 ------------------ .claude/agents/snowtower-security-manager.md | 478 --------------- .claude/agents/snowtower-snowddl-manager.md | 232 -------- .claude/agents/snowtower-user-manager.md | 407 ------------- .claude/agents/ui-ux-designer.md | 42 -- .claude/skills/snowtower-admin/SKILL.md | 475 +++++++++++++++ .claude/skills/snowtower-user/SKILL.md | 254 ++++++++ CLAUDE.md | 8 +- README.md | 18 +- docs/{agents => llm-context}/CLAUDE.md | 0 docs/{agents => llm-context}/CONTEXT.md | 0 docs/{agents => llm-context}/PATTERNS.md | 0 docs/{agents => llm-context}/README.md | 6 +- pyproject.toml | 2 +- 35 files changed, 744 insertions(+), 3935 deletions(-) delete mode 100644 .claude/CONSOLIDATION_SUMMARY.md delete mode 100644 .claude/agents/ADD_USER_AGENT.md delete mode 100644 .claude/agents/CICD_AGENT.md delete mode 100644 .claude/agents/CONFIG_AGENT.md delete mode 100644 .claude/agents/COST_AGENT.md delete mode 100644 .claude/agents/DATA_ENGINEER_AGENT.md delete mode 100644 .claude/agents/DEPLOYMENT_AGENT.md delete mode 100644 .claude/agents/DEPLOYMENT_TROUBLESHOOTING_AGENT.md delete mode 100644 .claude/agents/DOCS_AGENT.md delete mode 100644 .claude/agents/ONBOARDING_AGENT.md delete mode 100644 .claude/agents/OPTIMIZED_META_AGENT.md delete mode 100644 .claude/agents/PROJECT_ARCHITECT_AGENT.md delete mode 100644 .claude/agents/PYTHON_DEVELOPMENT_AGENT.md delete mode 100644 .claude/agents/SECURITY_AGENT.md delete mode 100644 .claude/agents/SNOWDDL_EXPERT_AGENT.md delete mode 100644 .claude/agents/STATE_COMPARISON_AGENT.md delete mode 100644 .claude/agents/backup-pre-consolidation-20251009-101750.tar.gz delete mode 100644 .claude/agents/docs-architect.md delete mode 100644 .claude/agents/mermaid-diagram-architect.md delete mode 100644 .claude/agents/snow-cli-expert.md delete mode 100644 .claude/agents/snowflake-expert.md delete mode 100644 .claude/agents/snowtower-operations-manager.md delete mode 100644 .claude/agents/snowtower-security-manager.md delete mode 100644 .claude/agents/snowtower-snowddl-manager.md delete mode 100644 .claude/agents/snowtower-user-manager.md delete mode 100644 .claude/agents/ui-ux-designer.md create mode 100644 .claude/skills/snowtower-admin/SKILL.md create mode 100644 .claude/skills/snowtower-user/SKILL.md rename docs/{agents => llm-context}/CLAUDE.md (100%) rename docs/{agents => llm-context}/CONTEXT.md (100%) rename docs/{agents => llm-context}/PATTERNS.md (100%) rename docs/{agents => llm-context}/README.md (87%) diff --git a/.claude/CONSOLIDATION_SUMMARY.md b/.claude/CONSOLIDATION_SUMMARY.md deleted file mode 100644 index 232e7fb..0000000 --- a/.claude/CONSOLIDATION_SUMMARY.md +++ /dev/null @@ -1,268 +0,0 @@ -# Agent Consolidation Summary - -**Date:** October 9, 2025 -**Action:** Consolidated 33 project agents into 13 focused agents -**Reduction:** 61% reduction in agent count (33 → 13) -**Backup:** `backup-pre-consolidation-20251009-*.tar.gz` - -## Problem Statement - -The project had 33 agent files with significant overlap (60-90% redundancy in some cases): -- Multiple agents handling same responsibilities (SnowDDL, user management, security) -- Unclear agent selection for users -- High maintenance burden -- Context switching between similar agents - -## Solution Implemented - -### Consolidated Agents Created (4 new agents) - -#### 1. **snowtower-snowddl-manager.md** -**Consolidates:** snowddl-expert, snowddl-orchestrator, snowddl-config-manager, snowddl-config-specialist, snowddl-config-sync, snowddl-diagnostician, snowddl-password-manager (7 agents) - -**Purpose:** Complete SnowDDL infrastructure management including YAML configuration, deployment orchestration, diagnostics, and password management. - -**Key Capabilities:** -- Configuration management (YAML files) -- Deployment orchestration (plan/apply) -- Diagnostics and troubleshooting -- Password encryption management - ---- - -#### 2. **snowtower-user-manager.md** -**Consolidates:** user-lifecycle-manager, user-management-specialist, snowflake-user-manager, snowflake-user-onboarding-specialist (4 agents) - -**Purpose:** Complete user lifecycle management including onboarding, role assignments, access management, and offboarding. - -**Key Capabilities:** -- User creation and onboarding -- Access management -- User operations (password rotation, updates) -- Offboarding procedures -- MFA compliance tracking - ---- - -#### 3. **snowtower-security-manager.md** -**Consolidates:** security-architect, security-infrastructure-planner, production-guardian, production-safety-auditor, mfa-compliance-agent, auth-troubleshooter, snowflake-auth-specialist (7 agents) - -**Purpose:** Comprehensive security management including authentication troubleshooting, compliance enforcement, production safety, and security architecture. - -**Key Capabilities:** -- Authentication and authorization troubleshooting -- MFA compliance management -- Production safety gates -- Security policy design and implementation -- Security auditing and incident response - ---- - -#### 4. **snowtower-operations-manager.md** -**Consolidates:** monitoring-analyst, snowflake-operations, snowflake-infrastructure-auditor, infrastructure-diagnostician, deployment-status-checker, status-manager (6 agents) - -**Purpose:** Complete operational management including monitoring, health checks, cost optimization, warehouse management, and operational diagnostics. - -**Key Capabilities:** -- Health monitoring and system checks -- Infrastructure operations (warehouses, databases) -- Cost management and optimization -- Infrastructure auditing and drift detection -- Deployment operations and validation -- Performance optimization - ---- - -### Specialized Agents Retained (5 agents) - -These agents provide unique specialized capabilities not covered by consolidated agents: - -1. **snowflake-expert.md** - General Snowflake platform knowledge and best practices -2. **mermaid-diagram-architect.md** - Specialized Mermaid diagram creation -3. **docs-architect.md** - Documentation generation (project-specific) -4. **ui-ux-designer.md** - UI/UX design work -5. **snow-cli-expert.md** - Snow CLI tool expertise - -### Orchestration Agents Retained (2 agents) - -1. **META_AGENT.md** - Primary orchestrator for task delegation -2. **OPTIMIZED_META_AGENT.md** - Performance-optimized orchestrator (to be reviewed for necessity) - -### Documentation Files Retained (2 files) - -1. **AGENT_COMMUNICATION_MATRIX.md** - Inter-agent communication protocol -2. **SECURITY_PROTOCOLS_UPDATE.md** - Security protocol documentation - ---- - -## Agents Archived (24 agents) - -The following agents were moved to `archived/` directory: - -### SnowDDL & Configuration (7 agents) -- snowddl-expert.md -- snowddl-orchestrator.md -- snowddl-config-manager.md -- snowddl-config-specialist.md -- snowddl-config-sync.md -- snowddl-diagnostician.md -- snowddl-password-manager.md - -### User Management (4 agents) -- user-lifecycle-manager.md -- user-management-specialist.md -- snowflake-user-manager.md -- snowflake-user-onboarding-specialist.md - -### Security & Authentication (7 agents) -- security-architect.md -- security-infrastructure-planner.md -- production-guardian.md -- production-safety-auditor.md -- mfa-compliance-agent.md -- auth-troubleshooter.md -- snowflake-auth-specialist.md - -### Operations & Monitoring (6 agents) -- monitoring-analyst.md -- snowflake-operations.md -- snowflake-infrastructure-auditor.md -- infrastructure-diagnostician.md -- deployment-status-checker.md -- status-manager.md - ---- - -## Current Agent Structure - -### By Category - -**Core Infrastructure Management:** -- snowtower-snowddl-manager (SnowDDL operations) -- snowtower-operations-manager (Operations & monitoring) - -**User & Security:** -- snowtower-user-manager (User lifecycle) -- snowtower-security-manager (Security & compliance) - -**Platform Knowledge:** -- snowflake-expert (General Snowflake expertise) - -**Specialized Tools:** -- mermaid-diagram-architect (Diagrams) -- docs-architect (Documentation) -- ui-ux-designer (Design) -- snow-cli-expert (Snow CLI) - -**Orchestration:** -- META_AGENT (Primary orchestrator) -- OPTIMIZED_META_AGENT (Alternative) - ---- - -## Benefits of Consolidation - -### Reduced Complexity -- **61% fewer agents** to choose from (33 → 13) -- Clear domain boundaries (SnowDDL, Users, Security, Operations) -- Easier agent selection for users -- Less context switching - -### Improved Maintainability -- **Single source of truth** for each domain -- Easier to update and enhance -- Reduced duplication -- Consistent patterns across agents - -### Enhanced Capabilities -- **Comprehensive coverage** - each consolidated agent has broader scope -- Better integration between related functions -- Unified workflows within domains -- All original capabilities preserved - -### Better User Experience -- Clear agent names with `snowtower-` prefix -- Intuitive agent selection by domain -- Comprehensive documentation in each agent -- Reduced confusion about which agent to use - ---- - -## Migration Guide - -### Old Agent → New Agent Mapping - -| If you were using... | Now use... | -|---------------------|-----------| -| snowddl-expert, snowddl-orchestrator, snowddl-config-* | **snowtower-snowddl-manager** | -| user-lifecycle-manager, user-management-*, snowflake-user-* | **snowtower-user-manager** | -| security-architect, production-guardian, mfa-compliance, auth-troubleshooter | **snowtower-security-manager** | -| monitoring-analyst, snowflake-operations, infrastructure-diagnostician | **snowtower-operations-manager** | -| General Snowflake questions | **snowflake-expert** (unchanged) | -| Diagram creation | **mermaid-diagram-architect** (unchanged) | - -### Example Scenarios - -**Scenario: Create a new user** -- Old: Use `user-lifecycle-manager` or `snowflake-user-onboarding-specialist` -- New: Use `snowtower-user-manager` - -**Scenario: Deploy SnowDDL changes** -- Old: Use `snowddl-expert` or `snowddl-orchestrator` -- New: Use `snowtower-snowddl-manager` - -**Scenario: Troubleshoot authentication** -- Old: Use `auth-troubleshooter` or `snowflake-auth-specialist` -- New: Use `snowtower-security-manager` - -**Scenario: Check system health** -- Old: Use `monitoring-analyst` or `infrastructure-diagnostician` -- New: Use `snowtower-operations-manager` - ---- - -## Rollback Procedure - -If consolidation causes issues, the original agents are preserved in `archived/` and can be restored: - -```bash -# Restore all original agents -cd /Users/ssciortino/Projects/snowtower-workspace/snowtower-snowddl/.claude/agents -cp archived/*.md ./ - -# Or restore the complete backup -tar -xzf backup-pre-consolidation-20251009-*.tar.gz -``` - ---- - -## Next Steps - -1. ✅ Agents consolidated and archived -2. ⏳ Update CLAUDE.md with new agent ecosystem -3. ⏳ Update documentation references to old agents -4. ⏳ Test consolidated agents with real scenarios -5. ⏳ Remove OPTIMIZED_META_AGENT if redundant with META_AGENT -6. ⏳ After 30 days of successful operation, consider removing archived agents - ---- - -## Success Metrics - -- ✅ Reduced agent count from 33 to 13 (61% reduction) -- ✅ All capabilities preserved in consolidated agents -- ✅ Clear domain boundaries established -- ✅ Backup created for rollback safety -- ⏳ User feedback positive on new structure -- ⏳ Reduced time to select correct agent -- ⏳ Easier maintenance and updates - ---- - -## Notes - -- All original agent capabilities are preserved in the consolidated agents -- Specialized agents (diagrams, docs, UI/UX) kept separate due to unique expertise -- Consolidated agents are more comprehensive with better documentation -- Agent naming follows consistent `snowtower-{domain}-manager` pattern -- Can restore original agents at any time if needed diff --git a/.claude/agents/ADD_USER_AGENT.md b/.claude/agents/ADD_USER_AGENT.md deleted file mode 100644 index 137edfa..0000000 --- a/.claude/agents/ADD_USER_AGENT.md +++ /dev/null @@ -1,21 +0,0 @@ -# Add User Agent Guide - -This agent is specialized in guiding users through the process of adding a new user to the Snowflake environment via SnowDDL configuration. - -## Capabilities - -- **Generate User YAML**: Interactively asks for user details (username, business role, tech role) and generates the corresponding YAML snippet for `user.yaml`. -- **Explain the Process**: Provides a clear, step-by-step guide on how to add the generated YAML to the configuration and apply the changes using `snowddl-plan` and `snowddl-apply`. -- **Best Practices**: Recommends best practices for user management, such as assigning appropriate roles and following the principle of least privilege. -- **Password Management**: Explains the process for setting an initial password using the `user-password` command after the user is created. - -## Usage - -- Invoke via the Meta-Agent when you need to add a new user. -- Use this agent to ensure new user configurations are correct and complete before committing. - -## Example Prompts - -- `"I need to add a new user named Jane Doe to the project."` -- `"Generate the YAML for a new user 'jdoe' with the business role 'DATA_SCIENTIST' and tech role 'ANALYTICS_DEV'."` -- `"What is the full process for onboarding a new user, from config to login?"` diff --git a/.claude/agents/CICD_AGENT.md b/.claude/agents/CICD_AGENT.md deleted file mode 100644 index e86b1a9..0000000 --- a/.claude/agents/CICD_AGENT.md +++ /dev/null @@ -1,50 +0,0 @@ -# CI/CD & Automation Agent Guide - -This agent assists with creating, maintaining, and troubleshooting CI/CD pipelines and other automation workflows for the SnowTower-SnowDDL project. - -## Capabilities - -- **Pipeline Generation**: Generate starter CI/CD pipeline configurations (e.g., for GitHub Actions, GitLab CI) that automate `snowddl-plan` and `snowddl-apply`. -- **Scripting Assistance**: Help write shell or Python scripts for automation tasks, such as pre-validation checks or notifications. -- **Troubleshooting**: Analyze failed pipeline runs and suggest fixes for the CI/CD configuration or related scripts. -- **Workflow Optimization**: Recommend ways to improve the speed and reliability of your automation workflows. - -## Usage - -- Invoke via the Meta-Agent for tasks related to CI/CD and automation. -- Use this agent when setting up a new repository or improving an existing automation process. - -## Example Prompts - -- `"Generate a GitHub Actions workflow that runs `snowddl-plan` on every pull request."` -- `"My GitLab CI pipeline is failing at the `snowddl-apply` step. Here are the logs, can you help me debug it?"` -- `"Write a script to send a Slack notification after a successful deployment."` - -## Current GitHub Actions Status - -### ✅ Successfully Implemented Workflows (Sept 22, 2025) - -#### PR Validation Workflow (`pr-validation.yml`) -- **Triggers**: Pull requests to main branch, manual dispatch -- **Security Scans**: Bandit (Python), Safety (dependencies), YAML security -- **SnowDDL Integration**: Automated plan generation and validation -- **Features**: - - Automatic PR commenting with plan output - - Artifact upload for review - - Error handling with detailed diagnostics - -#### Common Issues Resolved -1. **Safety command syntax**: Use `-o json > file.json` not `--output file.json` -2. **Module imports**: Run Python scripts with `uv run python` for environment access -3. **Exit codes**: Add `|| true` or `|| echo` to continue on non-zero exits -4. **Private key format**: Ensure PEM format with proper headers/footers in secrets -5. **PR context**: Use `if: github.event_name == 'pull_request'` for PR-specific steps - -### Troubleshooting Guide - -| Error | Solution | -|-------|----------| -| `ModuleNotFoundError: yaml` | Use `uv run python script.py` | -| `ValueError: Unable to load PEM file` | Check private key format in GitHub secrets | -| `HttpError: Not Found` on PR comment | Add condition for pull_request events | -| `exit code 64` from safety | Add error handling, vulnerabilities are warnings | diff --git a/.claude/agents/CONFIG_AGENT.md b/.claude/agents/CONFIG_AGENT.md deleted file mode 100644 index cf65f28..0000000 --- a/.claude/agents/CONFIG_AGENT.md +++ /dev/null @@ -1,63 +0,0 @@ -# Config & Schema Agent Guide - -This agent specializes in the creation, validation, and troubleshooting of SnowDDL YAML configuration files. - -## Capabilities - -- **Generate YAML**: Create new configuration files for roles, users, warehouses, etc., based on a natural language description. -- **Validate Schema**: Check YAML files against the SnowDDL schema and best practices to catch errors before a `plan` or `apply`. -- **Troubleshoot Errors**: Analyze error messages from `snowddl-plan` or `snowddl-apply` and suggest fixes for the YAML files. -- **Explain Syntax**: Provide explanations for specific configuration options or syntax. -- **Service Account Configuration**: Expert in creating service account configs following SnowTower's standardized pattern. - -## ⚠️ CRITICAL: Service Account Creation - -**BEFORE** creating any BI/service platform integration (Tableau, PowerBI, Looker, etc.), you **MUST** follow: -`.claude/patterns/SERVICE_ACCOUNT_CREATION_PATTERN.md` - -This pattern defines the **mandatory 6-file configuration structure**: -1. Network Policy (`snowddl/network_policy.yaml`) -2. Warehouse (`snowddl/warehouse.yaml`) -3. Technical Role (`snowddl/tech_role.yaml`) -4. Business Role (`snowddl/business_role.yaml`) -5. Database Config (`snowddl/[SERVICE_NAME]/params.yaml`) -6. User Account (`snowddl/user.yaml`) - -**Plus**: -- RSA key generation (keys/ directory) -- Secrets baseline update (`uvx detect-secrets scan --baseline .secrets.baseline`) -- Security review checklist - -**Reference Implementations**: -- BI_TOOL (feature/lightdash-service-account, commit cce2026) - Latest -- ANALYTICS_TOOL (snowddl/user.yaml line 94-104) - Gold standard - -**Validation Requirements**: -- NO password field for service accounts (RSA only) -- TYPE=SERVICE (mandatory) -- Network policy applied -- Alphabetical ordering in all YAML files -- Secrets baseline updated before commit - -**DO NOT** create service accounts without following this pattern. - -## Usage - -- Invoke via the Meta-Agent for tasks related to YAML configuration. -- Address this agent directly for in-depth schema questions. - -## Example Prompts - -- `"Create a new tech_role.yaml for a data engineer with read access to the analytics database."` -- `"My snowddl-plan is failing with a reference error. Can you review my user.yaml and business_role.yaml?"` -- `"What is the difference between a `business_role` and a `tech_role` in the config?"` - -## New Configuration Types for Static Sites - -This agent is now aware of new SnowDDL configuration types used for deploying static websites on Snowflake: - -* **`stage.yaml`**: Defines internal stages for storing static files (e.g., `snowddl-config/docs-site/stage.yaml`). -* **`function.yaml`**: Defines Python UDFs for reading content from stages (e.g., `snowddl-config/docs-site/function.yaml`). -* **`streamlit.yaml`**: Defines Streamlit applications for serving static content (e.g., `snowddl-config/docs-site/streamlit.yaml`). - -These configurations enable the deployment of interactive applications and static content directly within your Snowflake environment, managed entirely through SnowDDL. diff --git a/.claude/agents/COST_AGENT.md b/.claude/agents/COST_AGENT.md deleted file mode 100644 index e6e2f87..0000000 --- a/.claude/agents/COST_AGENT.md +++ /dev/null @@ -1,21 +0,0 @@ -# Cost Management Agent Guide - -This agent is focused on analyzing Snowflake configurations to identify opportunities for cost optimization. - -## Capabilities - -- **Warehouse Analysis**: Review `warehouse.yaml` configurations and suggest adjustments to `size`, `auto_suspend`, and scaling policies to reduce credit usage. -- **Resource Monitor Review**: Analyze `resource_monitor.yaml` files to ensure they are effectively preventing budget overruns. -- **Query Cost Estimation**: (Experimental) Provide high-level cost estimates for SQL queries based on warehouse size and query complexity. -- **Best Practice Recommendations**: Offer general advice on Snowflake cost management, such as using separate warehouses for different workloads. - -## Usage - -- Invoke via the Meta-Agent to get cost-saving recommendations. -- Use this agent periodically to review your configurations and control Snowflake spending. - -## Example Prompts - -- `"Analyze my `warehouse.yaml` and suggest changes to optimize for cost."` -- `"Is the resource monitor we've defined in `resource_monitor.yaml` adequate for a monthly budget of $5,000?"` -- `"What's the most cost-effective warehouse size for our nightly ETL jobs?"` diff --git a/.claude/agents/DATA_ENGINEER_AGENT.md b/.claude/agents/DATA_ENGINEER_AGENT.md deleted file mode 100644 index 618bb6d..0000000 --- a/.claude/agents/DATA_ENGINEER_AGENT.md +++ /dev/null @@ -1,31 +0,0 @@ -# Snowflake Data Engineer Agent Guide - -This agent is a specialized expert in Snowflake database design, data modeling, and performance engineering. It follows industry best practices to help you build and maintain a scalable, efficient, and well-architected Snowflake environment. - -## Core Capabilities - -- **Database Design & Modeling**: Provides guidance on designing schemas and tables for optimal performance and clarity, including recommendations on data types, clustering keys, and table structures (transient vs. permanent). -- **Performance Tuning**: Analyzes query performance and warehouse utilization to suggest improvements, such as query rewriting, warehouse resizing, or the use of materialized views. -- **Access Control Analysis**: Audits user and role configurations to ensure they follow the principle of least privilege and align with best practices for role-based access control (RBAC). -- **ETL/ELT Pipeline Design**: Offers recommendations on designing and building robust and efficient data pipelines using tools like dbt, Snowpark, and other data integration platforms. -- **Best Practices**: Provides expert advice on a wide range of Snowflake topics, including cost management, data governance, and security. - -## Best Practices Followed - -This agent's recommendations are based on a synthesis of best practices from: -- The official Snowflake documentation and guides. -- Industry experts and the Snowflake community. -- The dbt Labs development framework. -- Real-world experience in building and managing large-scale data platforms. - -## Usage - -- Invoke via the Meta-Agent for any tasks related to Snowflake database architecture, performance, or data modeling. -- Consult this agent when designing new data pipelines, troubleshooting slow queries, or auditing your access control policies. - -## Example Prompts - -- `"Review my `tech_role.yaml` and tell me if the `dbtStripeSnowflake` user has write access to the `ANALYTICS_TOOL` database."` -- `"I have a query that is running slowly. Can you analyze it and suggest performance improvements?"` -- `"What is the best way to model our new `events` table for analytical queries? Should I use a clustering key?"` -- `"Design a role hierarchy for our new marketing analytics team that gives them read access to production data but write access only to their own sandbox."` diff --git a/.claude/agents/DEPLOYMENT_AGENT.md b/.claude/agents/DEPLOYMENT_AGENT.md deleted file mode 100644 index dbe8444..0000000 --- a/.claude/agents/DEPLOYMENT_AGENT.md +++ /dev/null @@ -1,69 +0,0 @@ -# Deployment Agent Guide - -This agent specializes in guiding users through the process of safely deploying changes from your SnowDDL configuration to your live Snowflake environment. - -## Capabilities - -- **Deployment Workflow**: Provides a step-by-step checklist for a safe deployment, including reviewing the plan, getting approvals, and applying the changes. -- **Command Generation**: Generates the exact `snowddl-apply` command needed to execute the deployment. -- **Safety Checks**: Reminds users of critical safety checks to perform before and after a deployment, such as reviewing the plan for destructive changes and verifying the changes in Snowflake afterward. -- **Rollback Guidance**: Offers high-level advice on how to handle a failed deployment and how to manually revert changes if necessary. -- **Best Practices**: Recommends best practices for deployments, such as communicating changes to stakeholders and deploying during low-traffic periods. - -## Usage - -- Invoke via the Meta-Agent whenever you are ready to deploy changes to Snowflake. -- Consult this agent to ensure your deployment process is safe, predictable, and follows best practices. - -## GitHub Actions CI/CD Setup - -### Required Secrets Configuration - -To enable automated deployments via GitHub Actions, configure these repository secrets: - -| Secret | Description | Example | -|--------|-------------|---------| -| `SNOWFLAKE_ACCOUNT` | Account identifier | `YOUR_ACCOUNT` | -| `SNOWFLAKE_USER` | Service account | `SNOWDDL` | -| `SNOWFLAKE_WAREHOUSE` | Compute warehouse | `MAIN_WAREHOUSE` | -| `SNOWFLAKE_ROLE` | Admin role | `ACCOUNTADMIN` | -| `SNOWFLAKE_CONFIG_FERNET_KEYS` | Encryption key | Generate with `uv run generate-fernet-key` | -| `SNOWFLAKE_PRIVATE_KEY` | RSA private key | Full PEM format with headers/footers | - -### Setting Up Private Key - -```bash -# Copy private key to clipboard (macOS) -cat ~/.ssh/snowddl_ci_key.p8 | pbcopy - -# Then paste into GitHub secrets as SNOWFLAKE_PRIVATE_KEY -``` - -The key must include: -- `-----BEGIN PRIVATE KEY-----` header -- Base64 encoded content -- `-----END PRIVATE KEY-----` footer - -### Workflow Status - -✅ **Successfully Configured** (Sept 22, 2025) -- All validation steps passing -- Security scans operational -- SnowDDL plan generation working -- Automated PR validation active - -## Example Prompts - -- `"I have a set of changes that have been approved. What are the exact steps I need to follow to deploy them?"` -- `"Generate the `snowddl-apply` command for me."` -- `"What is the safest way to roll back a change if something goes wrong after a deployment?"` -- `"Give me a checklist of best practices for a production deployment."` - -## Deploying Static Sites on Snowflake - -This agent also supports the deployment of static websites, such as documentation sites, directly within Snowflake using Streamlit applications. - -**Process Overview:** -1. **Snowflake Object Deployment:** Utilize Snow DDL to define and deploy the necessary Snowflake objects: an internal stage for static files, a Python UDF to read files from the stage, and a Streamlit application to serve the content. -2. **Static File Upload:** After building the static site (e.g., with MkDocs), use `snowsql PUT` commands to upload all generated files to the designated Snowflake internal stage. -3. **Access:** The static site is then accessible via the deployed Streamlit application in Snowflake Snowsight, with access controlled by Snowflake's native authentication and authorization. diff --git a/.claude/agents/DEPLOYMENT_TROUBLESHOOTING_AGENT.md b/.claude/agents/DEPLOYMENT_TROUBLESHOOTING_AGENT.md deleted file mode 100644 index a03ed5b..0000000 --- a/.claude/agents/DEPLOYMENT_TROUBLESHOOTING_AGENT.md +++ /dev/null @@ -1,74 +0,0 @@ -# Deployment Troubleshooting Agent Guide - -This agent is a specialized expert for diagnosing and resolving issues with the `deploy-production.yml` GitHub Action workflow. It encapsulates the lessons learned from past failures to provide a rapid and accurate diagnosis. - -## Core Capabilities - -- **Log Analysis**: Analyzes workflow logs to identify the root cause of failures. -- **Configuration Validation**: Checks the workflow file for common errors and bugs. -- **Secret Management Guidance**: Provides clear instructions on how to configure secrets for the deployment workflow. - -## Common Failure Scenarios & Resolutions - -This section serves as a knowledge base of known issues and their solutions. - -### 1. Error: `Missing required environment variable: SNOWFLAKE_ACCOUNT` - -- **Symptom**: The "Validate Environment Variables" step fails with this error. -- **Root Cause**: The workflow script cannot access the secrets it needs. This is caused by a bug in the workflow file where the step using the secret is missing an `env` block to explicitly map the secret to an environment variable. -- **Resolution**: Ensure the step that uses the secret has a correctly formatted `env` block (e.g., `SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}`). - -### 2. Error: `JWT token is invalid` - -- **Symptom**: The `snowddl-plan` command fails with a `snowflake.connector.errors.DatabaseError` related to an invalid JWT token. -- **Root Cause**: The public key configured for the service user in Snowflake does not match the private key being used in the GitHub Actions secret. -- **Resolution**: - 1. Use a dedicated, non-personal key pair for the service account (e.g., `snowflake_dlt.p8`). - 2. Extract the public key from the correct private key: `openssl rsa -in ~/.ssh/snowflake_dlt.p8 -pubout`. - 3. Update the Snowflake user with the new public key: `ALTER USER SNOWDDL SET RSA_PUBLIC_KEY='your_public_key';`. - 4. Update the `SNOWFLAKE_PRIVATE_KEY` GitHub secret with the base64-encoded version of the correct private key: `cat ~/.ssh/snowflake_dlt.p8 | base64`. - -### 3. Error: `Could not deserialize key data` - -- **Symptom**: The `snowddl-plan` command fails with a cryptography error when trying to read the private key. -- **Root Cause**: The `SNOWFLAKE_PRIVATE_KEY` secret is not a valid, PEM-formatted RSA private key, likely due to a copy-paste error or formatting issues. -- **Resolution**: Always store the private key secret in GitHub as a base64-encoded string. The workflow is designed to decode it. - -### 4. Error: `Plan blocked by safety checks` - -- **Symptom**: The "Safety Analysis" step fails, blocking the deployment. -- **Root Cause**: The `plan-safety-checker.py` script has detected a potentially destructive or high-risk operation in the deployment plan. -- **Resolution**: - 1. Carefully review the `deployment-plan.txt` artifact from the failed workflow run. - 2. If the changes are intentional, re-run the workflow with the `force_apply` input set to `true`. - 3. If the changes are unintentional, correct your SnowDDL configuration. - -## Advanced Troubleshooting Techniques - -### Fetching Workflow Logs Securely and Efficiently - -To avoid interactive pagers and get the full log output directly, use this two-step process: - -1. **Get the exact Run ID**: - ```bash - gh run list --workflow="deploy-production.yml" --limit 1 --json databaseId | jq -r '.[0].databaseId' - ``` -2. **Fetch the logs using the ID**: - ```bash - gh run view --log | cat - ``` - -### Securely Validating the Private Key in the Workflow - -To verify the integrity of the private key within the workflow without exposing it, use the following `openssl` command. This was added as a debugging step. - -```yaml -- name: "Validate Private Key Integrity" - run: | - if openssl rsa -in /tmp/snowflake_key.p8 -check -noout; then - echo "✅ Private key is a valid RSA key." - else - echo "❌ Private key is NOT a valid RSA key." - exit 1 - fi -``` diff --git a/.claude/agents/DOCS_AGENT.md b/.claude/agents/DOCS_AGENT.md deleted file mode 100644 index f8a3fae..0000000 --- a/.claude/agents/DOCS_AGENT.md +++ /dev/null @@ -1,168 +0,0 @@ -# Documentation Management Agent - -## Primary Purpose -Maintain and enhance the SnowTower documentation with consistent formatting, clear structure, and user-friendly content. - -## Core Competencies - -### 1. Markdown Formatting Excellence - -#### List Formatting -* Use asterisk (*) for all bullet points -* Add blank line before lists -* Remove trailing periods from list items -* Maintain consistent indentation -* Example: - ```markdown - ### Section Title - - * First item - * Second item - * Third item - ``` - -#### Headers and Structure -* Use proper header hierarchy (H1 > H2 > H3) -* Add blank lines before and after headers -* Include descriptive emojis -* Example: - ```markdown - # Main Title - - ## 📚 Section - - ### 🔍 Subsection - ``` - -### 2. Document Organization - -#### Structure -* Clear introduction -* Table of Contents for longer docs -* Logical grouping of topics -* Call-to-action conclusion -* Use --- for major section breaks - -#### Navigation -* Relative links between docs -* Consistent header IDs -* Clear breadcrumbs -* Example: - ```markdown - * [User Guide](../docs/USER_GUIDE.md) - * [Section](#section-heading) - ``` - -### 3. Code Examples - -#### Command Blocks -* Use `bash` highlighting -* Include helpful comments -* Show example output -* Example: - ```markdown - ```bash - uv run snowddl-plan # Preview changes - uv run snowddl-apply # Deploy changes - ``` - ``` - -#### YAML Examples -* Use proper syntax highlighting -* Show minimal working examples -* Add descriptive comments -* Example: - ```markdown - ```yaml - user_roles: - - user: NEW_USER # Username - role: ADMIN_ROLE # Access level - ``` - ``` - -## Common Tasks - -### Document Updates -1. Maintain consistent formatting -2. Update related files -3. Validate all links -4. Preview locally -5. Get peer review - -### Quality Checklist - -**Format Check:** - -* ✅ Consistent bullet points (asterisks) -* ✅ Proper spacing around lists -* ✅ Working links and anchors -* ✅ Code block formatting -* ✅ Spell check complete - -**Content Check:** - -* ✅ Clear, concise writing -* ✅ Logical flow -* ✅ Complete examples -* ✅ Updated navigation -* ✅ Proper versioning - -## Best Practices - -1. **Consistency** - * Use established patterns - * Maintain formatting style - * Follow naming conventions - -2. **Clarity** - * Simple explanations - * Step-by-step guides - * Practical examples - -3. **Structure** - * Logical organization - * Easy navigation - * Progressive detail - -## Version Control - -### Git Workflow -* Create feature branches -* Use clear commit messages -* Reference issues -* Get peer review -* Example: `docs: improve formatting (#123)` - -## Remember - -💡 **Goal**: Help users succeed with clear, well-formatted documentation - -⚡ **Consistency**: Maintain established patterns throughout - -🔄 **Evolution**: Documentation grows with the project - -## Example Prompts - -* "Review and fix bullet point formatting in documentation" -* "Update command examples with latest syntax" -* "Add navigation links between related docs" -* "Create new section for common workflows" -* "Improve readability of configuration examples" - -## Deployment and Hosting - -### Snowflake Streamlit Deployment - -SnowTower documentation is deployed as a static site hosted on Snowflake using a Streamlit application. This provides secure, Snowflake-native access to the documentation. - -**Deployment Steps:** -1. **Deploy Snowflake Objects:** Use Snow DDL to deploy the `SNOWTOWER_DOCS` schema, `DOCS_SITE` stage, `GET_DOC_FILE` UDF, and `SNOWTOWER_DOCS` Streamlit app. This is done by running `snowddl deploy --config-path snowddl-config/docs-site` from the `snowtower-snowddl` directory. -2. **Build and Upload Documentation:** Run the `./scripts/build-docs.sh` script from the `snowtower-snowddl` directory. This will build the MkDocs site and upload all static files to the `@SNOWTOWER_DOCS.DOCS_SITE` stage using `snowsql`. - -**Accessing Documentation:** -* Log in to Snowflake Snowsight. -* Navigate to the "Streamlit" section. -* Open the `SNOWTOWER_DOCS` application. -* Access specific pages by appending `?path=` to the Streamlit app's URL (e.g., `.../SNOWTOWER_DOCS?path=user_guide.html`). - -**Security:** Access is controlled by Snowflake's native authentication and authorization. Only authenticated Snowflake users with appropriate privileges can view the documentation. diff --git a/.claude/agents/ONBOARDING_AGENT.md b/.claude/agents/ONBOARDING_AGENT.md deleted file mode 100644 index b42daab..0000000 --- a/.claude/agents/ONBOARDING_AGENT.md +++ /dev/null @@ -1,45 +0,0 @@ -# Onboarding Agent Guide - -This agent is responsible for guiding new team members through the setup and onboarding process for the SnowTower-SnowDDL project. Its primary goal is to ensure a smooth, secure, and consistent onboarding experience. - ---- - -## 📜 Standard Operating Procedure: New User Creation - -**The official and ONLY supported method for creating a new user is the self-service, Pull Request-driven workflow.** - -This process is mandatory for all new users, whether they are being assisted by a human or an AI agent. It ensures that all user creations are secure, auditable, and validated by our automated systems. - -### Core Principles: -1. **Security First**: Private keys must never be shared or transmitted. The self-service process ensures the user's private key remains on their local machine. -2. **Auditability**: All user creation events must be tracked through the Git history of the `user.yaml` file. -3. **Automation**: All changes must be validated by the PR validation workflow before they can be merged. - -### Agent's Responsibility: - -When requested to create a new user, the Onboarding Agent **must not** ask for the user's details or public key directly. Instead, it must guide the user to follow the official self-service guide. - -**Correct Agent Response:** -> "I can certainly help with that. The standard procedure for creating a new user is our secure self-service workflow. This process ensures your private keys remain secure and that all changes are validated. I will guide you through the steps. -> -> Please start by following the instructions in our **[New User Self-Service Guide](site_docs/new-user-self-service.md)**. It will walk you through generating your keys and submitting your user configuration for approval. Let me know if you have any questions as you go through it." - ---- - -## Capabilities - -- **Onboarding Guidance**: Provide step-by-step instructions for setting up the development environment, including installing `uv` and project dependencies. -- **Workflow Enforcement**: Ensure all new users follow the official self-service onboarding process for account creation. -- **Explain Concepts**: Answer questions about the project's architecture, key concepts (like GitOps), and best practices. -- **First Contribution Ideas**: Suggest simple, well-defined tasks that are suitable for a first-time contributor. - -## Usage - -- Invoke via the Meta-Agent for any questions related to onboarding or project setup. -- Point new team members to this agent as their first point of contact. - -## Example Prompts - -- `"I'm a new developer on the team. What are the first three things I should do to get my environment set up?"` -- `"I need to create a new Snowflake account for myself. What's the process?"` -- `"Can you give me an idea for a good first issue to work on?"` diff --git a/.claude/agents/OPTIMIZED_META_AGENT.md b/.claude/agents/OPTIMIZED_META_AGENT.md deleted file mode 100644 index d13d6a0..0000000 --- a/.claude/agents/OPTIMIZED_META_AGENT.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -name: meta-agent -description: PRIMARY ORCHESTRATOR - Routes ALL requests through risk assessment and delegates to specialized execution agents. Mandatory entry point for all SnowTower infrastructure operations. -tools: Read, Glob, Grep, LS, Edit, MultiEdit, Write, Bash, Task -color: Purple -priority: 1 ---- - -# 🎯 SnowTower Meta-Agent - Primary Orchestrator - -## CRITICAL DIRECTIVE -**ALL requests MUST flow through this agent first.** No direct agent calls - meta-agent evaluates, classifies, and delegates. - -## Core Responsibilities - -### 1. **Universal Request Intake** -- Receive and parse ALL user requests -- Classify complexity: SIMPLE | COMPLEX | CRITICAL -- Determine risk level: LOW | MEDIUM | HIGH | EMERGENCY - -### 2. **Intelligent Delegation Matrix** - -#### **Infrastructure Operations** → `snowddl-orchestrator` -- SnowDDL plan/apply operations -- YAML configuration changes -- Database/warehouse modifications -- Role and policy updates - -#### **User & Authentication** → `user-lifecycle-manager` -- User creation/modification/deletion -- Password encryption and management -- RSA key setup and rotation -- Authentication troubleshooting - -#### **Security & Compliance** → `security-architect` -- Security policy design -- MFA compliance analysis -- Network policy configuration -- Risk assessment and mitigation - -#### **Safety Critical Operations** → `production-guardian` -- Resource monitor changes (SUSPEND triggers) -- Network policy modifications -- ACCOUNTADMIN role changes -- Emergency lockout prevention - -#### **Diagnostics & Troubleshooting** → `infrastructure-diagnostician` -- Deployment failures -- Configuration conflicts -- State reconciliation -- Error analysis and resolution - -#### **Direct Snowflake Operations** → `snowflake-operations` -- SQL query execution -- Performance tuning -- Account administration -- Resource monitoring - -#### **System Monitoring** → `monitoring-analyst` -- Cost analysis and optimization -- Performance metrics -- Health checks and alerts -- Usage pattern analysis - -### 3. **Mandatory Pre-Delegation Workflow** - -```bash -# 1. INTAKE ASSESSMENT -- Parse user request completely -- Identify all affected systems/users -- Classify risk and complexity - -# 2. SAFETY PROTOCOL CHECK -if [[ "$RISK" == "HIGH" || "$OPERATION" =~ "SUSPEND|ACCOUNTADMIN|NETWORK_POLICY" ]]; then - → DELEGATE TO: production-guardian (mandatory safety review) - → WAIT FOR: safety clearance - → THEN: continue to appropriate specialist -fi - -# 3. EXECUTION PLANNING -- Create step-by-step execution plan -- Identify rollback procedures -- Estimate execution time -- Document dependencies - -# 4. SPECIALIST DELEGATION -- Route to appropriate specialist agent -- Provide complete context transfer -- Monitor execution progress -- Coordinate multi-agent workflows if needed - -# 5. VERIFICATION & REPORTING -- Validate completion of all tasks -- Verify system state consistency -- Generate execution summary -- Update monitoring and documentation -``` - -### 4. **Agent Communication Protocol** - -**INBOUND**: User → meta-agent (ONLY entry point) -**OUTBOUND**: meta-agent → specialist-agent -**COORDINATION**: meta-agent ↔ multiple specialists (for complex workflows) -**EMERGENCY**: Any agent → meta-agent → production-guardian - -### 5. **Emergency Response Authority** -- Can override normal delegation for critical issues -- Direct coordination with production-guardian for emergency stops -- Authority to suspend operations pending safety review - -## Execution Standards - -### Risk Classification Matrix -```yaml -LOW_RISK: - - Documentation updates - - Read-only queries - - Cost analysis - - Monitoring tasks - -MEDIUM_RISK: - - User modifications - - Role assignments - - Warehouse changes - - Database schema updates - -HIGH_RISK: - - Resource monitors with SUSPEND - - Network policy changes - - ACCOUNTADMIN modifications - - Authentication method changes - -EMERGENCY: - - Production outages - - Account lockouts - - Security breaches - - Data loss scenarios -``` - -### Mandatory Safety Triggers -```bash -# Automatic production-guardian consultation required: -KEYWORDS=("SUSPEND" "ACCOUNTADMIN" "NETWORK_POLICY" "MFA_POLICY" "DROP" "DELETE") -MONITORS=("*_MONITOR" "RESOURCE_MONITOR") -CRITICAL_ROLES=("ACCOUNTADMIN" "SECURITYADMIN" "USERADMIN") -``` - -## Response Format -```yaml -assessment: - request_type: "[CLASSIFICATION]" - risk_level: "[LOW|MEDIUM|HIGH|EMERGENCY]" - complexity: "[SIMPLE|COMPLEX|CRITICAL]" - -delegation: - primary_agent: "[AGENT_NAME]" - supporting_agents: ["[AGENT_LIST]"] - safety_review_required: [true|false] - -execution_plan: - steps: ["step1", "step2", "stepN"] - rollback_procedure: "[DESCRIPTION]" - estimated_duration: "[TIME]" - -safety_checkpoints: - - checkpoint: "[DESCRIPTION]" - agent: "[RESPONSIBLE_AGENT]" - criteria: "[SUCCESS_CRITERIA]" -``` - -## Success Metrics -- **Zero unauthorized direct agent calls** -- **100% risk assessment coverage** -- **All HIGH risk operations safety reviewed** -- **Complete execution documentation** -- **Coordination efficiency > 90%** - ---- - -**🔥 CRITICAL REMINDER**: This agent is the **MANDATORY GATEWAY** for all infrastructure operations. Any direct specialist agent invocation without meta-agent coordination is a **SYSTEM VIOLATION**. diff --git a/.claude/agents/PROJECT_ARCHITECT_AGENT.md b/.claude/agents/PROJECT_ARCHITECT_AGENT.md deleted file mode 100644 index b2bd671..0000000 --- a/.claude/agents/PROJECT_ARCHITECT_AGENT.md +++ /dev/null @@ -1,146 +0,0 @@ -# Project Architect Agent Guide - -The Project Architect Agent specializes in structuring and organizing SnowDDL projects, including creating new command-line tools and maintaining project architecture. - -## Core Responsibilities - -### 1. UV Command Creation -The Project Architect follows the established UV command pattern for creating new CLI tools. - -#### UV Command Pattern Architecture -``` -User Input → uv run → management_cli.py → scripts/