feat: Korean-aware ingestion with resolvable citations — closes #3 - #20
Merged
Conversation
… citations Replaces the blank-line-split stub with a real, deterministic segmentation pipeline and closes every acceptance criterion of the ingestion epic: - Heading-aware segmentation: Markdown (#), Korean chapter conventions (제N장/절/편/부/과), numbered (1. / 1.1 / 2)), roman (I.), and 가나다 (가. 나.) headings split the document into sections; paragraphs split within sections. Zero dependencies, fully deterministic. - Citation offsets on every unit with the hard invariant rawText.slice(citationStart, citationEnd) === content — the source is never normalized or rewritten, so citations always resolve, including under CRLF line endings. - No more fabricated ids: units no longer carry a placeholder "pending-source-id"; assigning real ids is the persister's job. - New @study-os/db persistIngestionResult(): atomic transaction writing StudySource + StudyUnits with real foreign keys and citation offsets; integration-tested against the migrated Postgres in CI (skipped locally without DATABASE_URL), including a rollback-atomicity test. - Validation: empty userId/title/rawText rejected, 500-unit cap, title truncation at 80 chars. - 16 ingestion unit tests over Korean fixtures (span resolution, heading conventions, CRLF, determinism, validation). - CI: migrate + seed moved before tests so DB integration tests run against the migrated database. Closes #3 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… import resolves Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the blank-line-split ingestion stub with a real, deterministic, Korean-aware segmentation pipeline, closing every acceptance criterion of epic #3.
Closes #3
Acceptance criteria → implementation
#,제N장/절/편/부/과, numbered1. / 1.1 / 2), romanI., and가. 나.headings; paragraph split within sections. Zero deps, fully deterministic.rawText.slice(citationStart, citationEnd) === content. Source text is never normalized/rewritten (CRLF-safe).pending-source-id)persistIngestionResult()in@study-os/dbwrites StudySource + StudyUnits in one transaction with real foreign keys — integration-tested against the migrated Postgres in CI, incl. rollback atomicity.IngestionValidationError; 500-unit cap;orderIndexby document position; determinism test.Notes
1.1 배경without a trailing dot wasn't recognized) — fixed with a dedicated multi-level pattern.DATABASE_URLthey skip cleanly.Verified
37 unit tests pass locally (+2 DB integration tests that run in CI); lint/typecheck/build/API-smoke all green.
🤖 Generated with Claude Code