feat: PostgreSQL database migration with Drizzle ORM - #165
Conversation
…s, and application patterns - Add Drizzle ORM and postgres dependencies - Create database schema for application_patterns, effect_patterns, jobs, and relations - Implement repository layer with CRUD and search operations - Add migration script to parse existing JSON/MDX/MD files - Update toolkit to support both file-based and database-based storage - Add integration tests for all repositories - Configure drizzle.config.ts for migrations Database tables: - application_patterns: High-level pattern categories - effect_patterns: Concrete code examples - jobs: Jobs-to-be-Done entries - pattern_jobs: Many-to-many pattern-job relationships - pattern_relations: Related patterns linking New scripts: - bun run db:generate - Generate migrations - bun run db:push - Push schema to database - bun run db:migrate - Run data migration from files - bun run db:verify - Verify migration results - bun run db:studio - Open Drizzle Studio
- Create DatabaseService with Effect.Service wrapper for repositories - Add high-level database operations (searchEffectPatterns, findEffectPatternBySlug) - Update MCP server PatternsService to use database instead of file loading - Convert database EffectPattern types to legacy Pattern types for compatibility - Add migration testing guide Database service provides: - DatabaseLayer for dependency injection - Effect-based repository access - Type conversion from DB schema to legacy Pattern schema - Error handling and logging integration
- Replace file-based loading with database queries - Load application patterns and effect patterns from PostgreSQL - Find actual file paths by searching filesystem - Maintain backward compatibility with existing file structure - Improve path resolution accuracy
CLI Updates: - Update search, list, and show commands to use database - Replace JSON file loading with database queries - Add proper error handling and database connection management - Support related patterns query from database Documentation Updates: - Update DATA_MODEL.md to reflect PostgreSQL as primary source - Add database schema documentation - Update ARCHITECTURE.md with database technologies - Add database configuration section - Update package descriptions to mention database - Add migration guide references
Testing Infrastructure: - Add test-db.ts - Comprehensive database test runner - Add test-db-quick.ts - Quick database connectivity test - Add db-helpers.ts - Test utilities (setup, cleanup, seeding) - Update repositories.test.ts to use test helpers Documentation: - Add DATABASE_TESTING.md - Complete testing guide - Update MIGRATION_TESTING.md with testing steps Scripts: - bun run test:db - Run full database test suite - bun run test:db:quick - Quick connectivity test - bun run test:db:repositories - Run repository integration tests
- Add validated and validatedAt fields to application_patterns, effect_patterns, and jobs tables - Implement lock/unlock methods in all repositories - Add locked error types (EffectPatternLockedError, ApplicationPatternLockedError, JobLockedError) - Enforce read-only behavior for locked entities in update/delete/upsert operations - Add lock and unlock commands to ep-admin CLI - Fix postgres import in database client (use default import instead of namespace) - Fix generate.ts to group patterns by applicationPatternId from database - Generate database migration for locking fields - Export locked error types from toolkit package
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| createEffectPatternRepository, | ||
| createJobRepository, | ||
| } from "../repositories/index.js" | ||
| import { eq } from "drizzle-orm" |
Check notice
Code scanning / CodeQL
Unused variable, import, function or class Note test
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
To fix the problem, remove the unused import from the file so that every imported symbol is referenced somewhere in the code. This eliminates confusion, keeps the test file minimal, and silences the static analysis warning.
Concretely, in packages/toolkit/src/__tests__/repositories.test.ts, delete the line that imports eq from "drizzle-orm" (line 22). No other changes are needed: there are no reported usages to update, and we are not altering any behavior because the symbol is never used. No new methods, imports, or definitions are required.
| @@ -19,7 +19,6 @@ | ||
| createEffectPatternRepository, | ||
| createJobRepository, | ||
| } from "../repositories/index.js" | ||
| import { eq } from "drizzle-orm" | ||
| import { | ||
| setupTestDatabase, | ||
| cleanDatabase, |
| } from "../db/schema/index.js" | ||
| import type { Pattern } from "../schemas/pattern.js" | ||
| import { ToolkitLogger, ToolkitLoggerLive } from "./logger.js" | ||
| import { ToolkitConfig, ToolkitConfigLive } from "./config.js" |
Check notice
Code scanning / CodeQL
Unused variable, import, function or class Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
To fix the problem, remove the unused ToolkitConfig symbol from the import statement while keeping the used ToolkitConfigLive. This eliminates the unused value import without affecting runtime behavior or typings.
Concretely, in packages/toolkit/src/services/database.ts, locate the line:
import { ToolkitLogger, ToolkitLoggerLive } from "./logger.js"
import { ToolkitConfig, ToolkitConfigLive } from "./config.js"and change it so that only ToolkitConfigLive is imported from ./config.js. No other code changes are required because no code references ToolkitConfig.
| @@ -29,7 +29,7 @@ | ||
| } from "../db/schema/index.js" | ||
| import type { Pattern } from "../schemas/pattern.js" | ||
| import { ToolkitLogger, ToolkitLoggerLive } from "./logger.js" | ||
| import { ToolkitConfig, ToolkitConfigLive } from "./config.js" | ||
| import { ToolkitConfigLive } from "./config.js" | ||
|
|
||
| /** | ||
| * Convert database EffectPattern to legacy Pattern format |
| * Create initial step state | ||
| */ | ||
| function _createInitialStepState( | ||
| function createInitialStepState( |
Check notice
Code scanning / CodeQL
Unused variable, import, function or class Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
In general, to fix an "unused function" finding, either delete the function or start using it where appropriate. To avoid changing runtime behavior, we should not alter the logic of existing, used code; instead, we remove only code that truly has no effect: the unused createInitialStepState helper and its associated comment.
Concretely, in scripts/migrate-state.ts, remove the entire createInitialStepState function definition (lines 93–105 in the snippet) and its JSDoc comment, leaving the rest of the file unchanged. No additional imports, methods, or definitions are required because nothing else depends on this function.
| @@ -91,20 +91,6 @@ | ||
| // --- HELPER FUNCTIONS --- | ||
|
|
||
| /** | ||
| * Create initial step state | ||
| */ | ||
| function createInitialStepState( | ||
| status: StepState['status'] = 'pending', | ||
| ): StepState { | ||
| return { | ||
| status, | ||
| attempts: 0, | ||
| checkpoints: [], | ||
| errors: [], | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * Create pattern state with all steps completed | ||
| */ | ||
| function createCompletedPatternState( |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the final PR Bugbot will review for you during this billing cycle
Your free Bugbot reviews will reset on January 17
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| | :--- | :--- | :--- | | ||
| | [Stream Pattern 1: Transform Streams with Map and Filter](./content/published/patterns/streams/stream-pattern-map-filter-transformations.mdx) | 🟢 **Beginner** | Use Stream.map and Stream.filter to transform and select stream elements, enabling data pipelines that reshape and filter data in flight. | | ||
| | [Sink Pattern 1: Batch Insert Stream Records into Database](./content/published/patterns/streams/sink-pattern-batch-insert-stream-records-into-database.mdx) | 🟡 **Intermediate** | Use Sink to batch stream records and insert them efficiently into a database in groups, rather than one-by-one, for better performance and resource usage. | | ||
| | [Sink Pattern 2: Write Stream Events to Event Log](./content/published/patterns/streams/sink-pattern-write-stream-events-to-event-log.mdx) | 🟡 **Intermediate** | Use Sink to append stream events to an event log with metadata and causal ordering, enabling event sourcing and audit trail patterns. | |
There was a problem hiding this comment.
README links to non-existent Sink Pattern files
The README adds links to Sink Pattern 1 and 2 that point to incorrect file paths. The links reference ./content/published/patterns/streams/sink-pattern-batch-insert-stream-records-into-database.mdx and ./content/published/patterns/streams/sink-pattern-write-stream-events-to-event-log.mdx, but the actual files exist at ./content/published/patterns/streams/sinks/batch-insert-stream-records-into-database.mdx and ./content/published/patterns/streams/sinks/write-stream-events-to-event-log.mdx. The links use the wrong directory (streams/ instead of streams/sinks/) and incorrect filenames (added sink-pattern- prefix). Users clicking these links will get 404 errors.
* feat: add PostgreSQL database with Drizzle ORM for patterns, use cases, and application patterns - Add Drizzle ORM and postgres dependencies - Create database schema for application_patterns, effect_patterns, jobs, and relations - Implement repository layer with CRUD and search operations - Add migration script to parse existing JSON/MDX/MD files - Update toolkit to support both file-based and database-based storage - Add integration tests for all repositories - Configure drizzle.config.ts for migrations Database tables: - application_patterns: High-level pattern categories - effect_patterns: Concrete code examples - jobs: Jobs-to-be-Done entries - pattern_jobs: Many-to-many pattern-job relationships - pattern_relations: Related patterns linking New scripts: - bun run db:generate - Generate migrations - bun run db:push - Push schema to database - bun run db:migrate - Run data migration from files - bun run db:verify - Verify migration results - bun run db:studio - Open Drizzle Studio * feat: add database service layer and update MCP server to use database - Create DatabaseService with Effect.Service wrapper for repositories - Add high-level database operations (searchEffectPatterns, findEffectPatternBySlug) - Update MCP server PatternsService to use database instead of file loading - Convert database EffectPattern types to legacy Pattern types for compatibility - Add migration testing guide Database service provides: - DatabaseLayer for dependency injection - Effect-based repository access - Type conversion from DB schema to legacy Pattern schema - Error handling and logging integration * feat: update publishing pipeline to use database - Replace file-based loading with database queries - Load application patterns and effect patterns from PostgreSQL - Find actual file paths by searching filesystem - Maintain backward compatibility with existing file structure - Improve path resolution accuracy * feat: update CLI tools and documentation for database migration CLI Updates: - Update search, list, and show commands to use database - Replace JSON file loading with database queries - Add proper error handling and database connection management - Support related patterns query from database Documentation Updates: - Update DATA_MODEL.md to reflect PostgreSQL as primary source - Add database schema documentation - Update ARCHITECTURE.md with database technologies - Add database configuration section - Update package descriptions to mention database - Add migration guide references * feat: add comprehensive database testing utilities and guides Testing Infrastructure: - Add test-db.ts - Comprehensive database test runner - Add test-db-quick.ts - Quick database connectivity test - Add db-helpers.ts - Test utilities (setup, cleanup, seeding) - Update repositories.test.ts to use test helpers Documentation: - Add DATABASE_TESTING.md - Complete testing guide - Update MIGRATION_TESTING.md with testing steps Scripts: - bun run test:db - Run full database test suite - bun run test:db:quick - Quick connectivity test - bun run test:db:repositories - Run repository integration tests * feat: add entity locking mechanism for validated patterns - Add validated and validatedAt fields to application_patterns, effect_patterns, and jobs tables - Implement lock/unlock methods in all repositories - Add locked error types (EffectPatternLockedError, ApplicationPatternLockedError, JobLockedError) - Enforce read-only behavior for locked entities in update/delete/upsert operations - Add lock and unlock commands to ep-admin CLI - Fix postgres import in database client (use default import instead of namespace) - Fix generate.ts to group patterns by applicationPatternId from database - Generate database migration for locking fields - Export locked error types from toolkit package * feat: update pipeline state and database services with latest changes
2 participants
This PR adds PostgreSQL database support with Drizzle ORM for patterns, use cases, and application patterns. Includes comprehensive database testing utilities and migration guides.
Note
Database integration
Coding standards
Effect.Service; forbidas any; avoidLive/Default/ImplsuffixesDocs and content
README.mdnavigation (removes Schema section, reorganizes Sinks links)Dependencies
drizzle-orm,postgres,drizzle-kit,@types/pg; bumpsai,turbo; large lockfile updatesWritten by Cursor Bugbot for commit 4ae95e5. This will update automatically on new commits. Configure here.