Skip to content

Commit 97de508

Browse files
authored
feat: PostgreSQL database migration with Drizzle ORM (#165)
* feat: add PostgreSQL database with Drizzle ORM for patterns, use cases, and application patterns - Add Drizzle ORM and postgres dependencies - Create database schema for application_patterns, effect_patterns, jobs, and relations - Implement repository layer with CRUD and search operations - Add migration script to parse existing JSON/MDX/MD files - Update toolkit to support both file-based and database-based storage - Add integration tests for all repositories - Configure drizzle.config.ts for migrations Database tables: - application_patterns: High-level pattern categories - effect_patterns: Concrete code examples - jobs: Jobs-to-be-Done entries - pattern_jobs: Many-to-many pattern-job relationships - pattern_relations: Related patterns linking New scripts: - bun run db:generate - Generate migrations - bun run db:push - Push schema to database - bun run db:migrate - Run data migration from files - bun run db:verify - Verify migration results - bun run db:studio - Open Drizzle Studio * feat: add database service layer and update MCP server to use database - Create DatabaseService with Effect.Service wrapper for repositories - Add high-level database operations (searchEffectPatterns, findEffectPatternBySlug) - Update MCP server PatternsService to use database instead of file loading - Convert database EffectPattern types to legacy Pattern types for compatibility - Add migration testing guide Database service provides: - DatabaseLayer for dependency injection - Effect-based repository access - Type conversion from DB schema to legacy Pattern schema - Error handling and logging integration * feat: update publishing pipeline to use database - Replace file-based loading with database queries - Load application patterns and effect patterns from PostgreSQL - Find actual file paths by searching filesystem - Maintain backward compatibility with existing file structure - Improve path resolution accuracy * feat: update CLI tools and documentation for database migration CLI Updates: - Update search, list, and show commands to use database - Replace JSON file loading with database queries - Add proper error handling and database connection management - Support related patterns query from database Documentation Updates: - Update DATA_MODEL.md to reflect PostgreSQL as primary source - Add database schema documentation - Update ARCHITECTURE.md with database technologies - Add database configuration section - Update package descriptions to mention database - Add migration guide references * feat: add comprehensive database testing utilities and guides Testing Infrastructure: - Add test-db.ts - Comprehensive database test runner - Add test-db-quick.ts - Quick database connectivity test - Add db-helpers.ts - Test utilities (setup, cleanup, seeding) - Update repositories.test.ts to use test helpers Documentation: - Add DATABASE_TESTING.md - Complete testing guide - Update MIGRATION_TESTING.md with testing steps Scripts: - bun run test:db - Run full database test suite - bun run test:db:quick - Quick connectivity test - bun run test:db:repositories - Run repository integration tests * feat: add entity locking mechanism for validated patterns - Add validated and validatedAt fields to application_patterns, effect_patterns, and jobs tables - Implement lock/unlock methods in all repositories - Add locked error types (EffectPatternLockedError, ApplicationPatternLockedError, JobLockedError) - Enforce read-only behavior for locked entities in update/delete/upsert operations - Add lock and unlock commands to ep-admin CLI - Fix postgres import in database client (use default import instead of namespace) - Fix generate.ts to group patterns by applicationPatternId from database - Generate database migration for locking fields - Export locked error types from toolkit package * feat: update pipeline state and database services with latest changes
1 parent 6ccf5e6 commit 97de508

71 files changed

Lines changed: 10500 additions & 1477 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 2 additions & 158 deletions
Large diffs are not rendered by default.

‎bun.lock‎

Lines changed: 111 additions & 205 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
description: Always use Effect.Service pattern for service definitions. Never use Context.Tag or Context.GenericTag.
2+
globs: "**/*.ts"
3+
alwaysApply: true
4+
5+
# Always Use Effect.Service for Service Definitions
6+
**Rule:** Always use the `Effect.Service` pattern for defining services. Never use `Context.Tag` or `Context.GenericTag`.
7+
8+
### Why This Rule Exists
9+
10+
The `Effect.Service` pattern is the modern, recommended way to define services in Effect-TS. It provides:
11+
- Automatic service provision - services are available when you yield them
12+
- Better type safety and inference
13+
- Cleaner dependency management
14+
- Consistent service definition patterns across the codebase
15+
- Simplified testing - no need to manually provide services
16+
17+
### Anti-Pattern (Forbidden)
18+
19+
```typescript
20+
// ❌ FORBIDDEN: Using Context.Tag for service definition
21+
import { Context } from "effect";
22+
23+
export class MyService extends Context.Tag("MyService")<
24+
MyService,
25+
{
26+
readonly doSomething: () => Effect.Effect<void>;
27+
}
28+
>() {}
29+
30+
// ❌ FORBIDDEN: Using Context.GenericTag for service definition
31+
export const MyService = Context.GenericTag<MyServiceInterface>("MyService");
32+
33+
// ❌ FORBIDDEN: Manual layer creation with Context.Tag
34+
const MyServiceLive = Layer.succeed(MyService, {
35+
doSomething: () => Effect.succeed(undefined),
36+
});
37+
```
38+
39+
### Correct Pattern (Required)
40+
41+
#### 1. Synchronous Service (No Dependencies)
42+
43+
```typescript
44+
// ✅ CORRECT: Use Effect.Service with sync for simple services
45+
import { Effect } from "effect";
46+
47+
interface MyServiceInterface {
48+
readonly doSomething: () => Effect.Effect<void>;
49+
}
50+
51+
const makeMyService = (): MyServiceInterface => ({
52+
doSomething: () => Effect.succeed(undefined),
53+
});
54+
55+
export class MyService extends Effect.Service<MyService>()("MyService", {
56+
sync: () => makeMyService(),
57+
}) {}
58+
59+
// Usage: Just yield the service - Effect handles provision automatically
60+
const program = Effect.gen(function* () {
61+
const service = yield* MyService;
62+
// Use the service
63+
});
64+
```
65+
66+
#### 2. Service with Dependencies
67+
68+
```typescript
69+
// ✅ CORRECT: Use Effect.Service with effect and dependencies
70+
export class DatabaseService extends Effect.Service<DatabaseService>()(
71+
"DatabaseService",
72+
{
73+
effect: Effect.gen(function* () {
74+
const config = yield* ConfigService; // Effect.Service automatically infers ConfigService as a dependency
75+
return createDatabase(config);
76+
}),
77+
// ✅ CORRECT: No dependencies array needed - Effect.Service infers dependencies automatically
78+
}
79+
) {}
80+
81+
// Usage: Just yield the service - Effect handles provision automatically
82+
```
83+
84+
#### 3. Service with Scoped Resources
85+
86+
```typescript
87+
// ✅ CORRECT: Use Effect.Service with scoped for resource management
88+
export class DatabaseService extends Effect.Service<DatabaseService>()(
89+
"DatabaseService",
90+
{
91+
scoped: Effect.gen(function* () {
92+
const config = yield* ConfigService; // Effect.Service automatically infers ConfigService as a dependency
93+
return yield* Effect.acquireRelease(
94+
() => createConnection(config),
95+
(connection) => connection.close()
96+
);
97+
}),
98+
// ✅ CORRECT: No dependencies array needed - Effect.Service infers dependencies automatically
99+
}
100+
) {}
101+
```
102+
103+
#### 4. Service with Accessors
104+
105+
```typescript
106+
// ✅ CORRECT: Enable accessors for convenient static access
107+
export class LoggerService extends Effect.Service<LoggerService>()(
108+
"LoggerService",
109+
{
110+
accessors: true,
111+
sync: () => ({
112+
log: (message: string) => Effect.sync(() => console.log(message)),
113+
}),
114+
}
115+
) {}
116+
117+
// Usage: LoggerService.log("message") - static accessor available
118+
```
119+
120+
### Migration from Context.Tag
121+
122+
When migrating existing services:
123+
124+
1. **Replace Context.Tag/GenericTag with Effect.Service class**
125+
2. **Move implementation to `sync`, `effect`, or `scoped` property**
126+
3. **Remove manual Layer creation** - Effect.Service handles it automatically
127+
4. **Remove all `.Default` usage** - Never explicitly use `.Default`
128+
5. **Just yield services** - Services are automatically available when you yield them
129+
130+
**Example Migration:**
131+
132+
```typescript
133+
// Before (❌ FORBIDDEN)
134+
export const StateStore = Context.GenericTag<StateStoreService>("StateStore");
135+
const makeStateStore = (): StateStoreService => { /* ... */ };
136+
export const StateStoreLive = Layer.succeed(StateStore, makeStateStore());
137+
138+
// Usage (❌ FORBIDDEN)
139+
Effect.provide(program, StateStoreLive);
140+
141+
// After (✅ CORRECT)
142+
const makeStateStore = (): StateStoreService => { /* ... */ };
143+
export class StateStore extends Effect.Service<StateStore>()("StateStore", {
144+
sync: () => makeStateStore(),
145+
}) {}
146+
147+
// Usage (✅ CORRECT)
148+
// Just yield the service - Effect handles provision automatically
149+
const program = Effect.gen(function* () {
150+
const store = yield* StateStore;
151+
// Use the service
152+
});
153+
Effect.runPromise(program); // Service is automatically provided
154+
```
155+
156+
### Exception: Test Mocks
157+
158+
Test mocks may use `Context.Tag` for isolated test scenarios, but production service definitions must always use `Effect.Service`:
159+
160+
```typescript
161+
// ✅ ACCEPTABLE: Test-only mock using Context.Tag
162+
const MockDisplayService = Context.Tag<{ isMocked: true }>();
163+
```
164+
165+
**Explanation:**
166+
The `Effect.Service` pattern is the standard way to define services in Effect-TS. It provides better type safety, automatic layer generation, and consistent patterns across the codebase. Using `Context.Tag` or `Context.GenericTag` directly is deprecated in favor of the `Effect.Service` pattern.
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
description: Never use 'as any' type assertions - they bypass TypeScript's type safety.
2+
globs: "**/*.ts"
3+
alwaysApply: true
4+
5+
# Never Use 'as any' Type Assertions
6+
**Rule:** Never use `as any` type assertions. They bypass TypeScript's type safety and are an anti-pattern.
7+
8+
### Why This Is Forbidden
9+
10+
Using `as any` disables TypeScript's type checking, which defeats the purpose of using a type-safe language. It can lead to:
11+
- Runtime errors that could have been caught at compile time
12+
- Loss of IDE autocomplete and type hints
13+
- Hidden bugs that surface in production
14+
- Reduced code maintainability
15+
16+
### Anti-Pattern (Forbidden)
17+
18+
```typescript
19+
// ❌ FORBIDDEN: Using 'as any' to bypass type checking
20+
const result = someFunction() as any;
21+
const value = (someOption as any).value;
22+
const effect = someEffect.gen(function* () {
23+
// ...
24+
}) as any;
25+
```
26+
27+
### Correct Approaches
28+
29+
#### 1. Fix Type Mismatches Properly
30+
31+
If you encounter type errors, fix the underlying issue:
32+
33+
```typescript
34+
// ✅ CORRECT: Fix the type definition or use proper type narrowing
35+
const result: ExpectedType = someFunction();
36+
const value = Option.isSome(someOption) ? someOption.value : defaultValue;
37+
```
38+
39+
#### 2. Use Type Guards
40+
41+
```typescript
42+
// ✅ CORRECT: Use type guards for runtime type checking
43+
function isExpectedType(value: unknown): value is ExpectedType {
44+
return typeof value === "object" && value !== null && "property" in value;
45+
}
46+
47+
if (isExpectedType(value)) {
48+
// TypeScript knows value is ExpectedType here
49+
console.log(value.property);
50+
}
51+
```
52+
53+
#### 3. Use Proper Type Assertions
54+
55+
If you must assert a type, use a more specific assertion:
56+
57+
```typescript
58+
// ✅ CORRECT: Use specific type assertions when you're certain
59+
const result = someValue as SpecificType;
60+
// Or use type predicates
61+
const result = someValue as unknown as SpecificType;
62+
```
63+
64+
#### 4. Fix Dependency Version Conflicts
65+
66+
If type errors are caused by multiple versions of the same library:
67+
68+
```typescript
69+
// ✅ CORRECT: Resolve dependency conflicts at the package level
70+
// Update package.json to use consistent versions
71+
// Use workspace hoisting or resolution strategies
72+
```
73+
74+
#### 5. Use Effect's Type System Properly
75+
76+
For Effect-specific type issues:
77+
78+
```typescript
79+
// ✅ CORRECT: Use Effect's type system features
80+
const effect = Effect.gen(function* () {
81+
const value = yield* someEffect;
82+
return value;
83+
});
84+
85+
// ✅ CORRECT: Use Effect.map, Effect.flatMap, etc. for type transformations
86+
const transformed = effect.pipe(
87+
Effect.map((value) => value.property)
88+
);
89+
```
90+
91+
**Explanation:**
92+
Type safety is one of TypeScript's core benefits. Using `as any` throws away this protection and should never be used. Always fix the underlying type issue instead of bypassing it.

0 commit comments

Comments
 (0)