Skip to content

feat: Add opt-in article descriptions and keywords - #20

Merged
Hyaxia merged 8 commits into
Hyaxia:mainfrom
DamienSchreurs:codex/pr18-keywords-description
Aug 31, 2026
Merged

feat: Add opt-in article descriptions and keywords#20
Hyaxia merged 8 commits into
Hyaxia:mainfrom
DamienSchreurs:codex/pr18-keywords-description

Conversation

@DamienSchreurs

@DamienSchreurs DamienSchreurs commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

This follows up on #18 and builds on the work by @hermes-lol and @twaithalek-ai.

  • add nullable articles.description and articles.keywords columns through the existing additive migration path
  • keep description and keyword collection disabled unless explicitly enabled for scan
  • keep the existing articles output unchanged unless the corresponding display flags are requested
  • preserve gofeed as the primary RSS, Atom, and JSON Feed parser
  • extract repeated XML keywords in a supplementary, failure-isolated pass only when keyword storage is enabled
  • cap keyword-mode buffering at 10 MiB of decompressed feed data
  • store the first meaningful description paragraph, with content fallback and a configurable Unicode-safe character limit
  • prefer nearby Unicode whitespace or punctuation when truncating, with an exact character-boundary fallback for CJK and long tokens

Opt-in interfaces

Storage:

blogwatcher scan --store-descriptions
blogwatcher scan --store-keywords
blogwatcher scan --store-descriptions --description-max-chars 500

Environment equivalents:

BLOGWATCHER_STORE_DESCRIPTIONS=true
BLOGWATCHER_STORE_KEYWORDS=true
BLOGWATCHER_DESCRIPTION_MAX_CHARS=1000

Output:

blogwatcher articles --show-descriptions
blogwatcher articles --show-keywords

Flags override environment variables. Description storage defaults to a 1,000-character limit when enabled; 0 means unlimited. Existing rows are not backfilled, and already stored values are not removed when storage is later disabled.

Compatibility

  • a default scan writes NULL for both optional fields
  • blogwatcher articles without either display flag is byte-for-byte compatible with v0.0.3 in an isolated fixture comparison
  • the same database was exercised old → candidate → old after metadata population; v0.0.3 output remained byte-for-byte identical
  • old binaries safely ignore the additive nullable columns
  • no production database, scheduler, or news-curation integration is changed here

Verification

  • gofmt
  • go vet ./...
  • go test -count=1 ./...
  • candidate build
  • exact-limit and oversized keyword-buffer tests
  • CJK, Unicode punctuation, distant-whitespace, and exact character-boundary tests
  • isolated SQLite migration and PRAGMA quick_check
  • isolated CLI output and configuration smoke tests
  • v0.0.3 versus candidate default-output comparison on equivalent fixture databases

Closes no issue automatically; #18 is left open for the maintainer to manage.

hermes-lol and others added 4 commits August 26, 2026 12:05
- Add Keywords and Description fields to Article model
- Implement custom XML parser to capture multiple <keyword> tags
- Add database migration for new columns
- Update scanner to pass new fields to database

(cherry picked from commit 351bbe3)
@DamienSchreurs DamienSchreurs changed the title Add opt-in article descriptions and keywords feat: Add opt-in article descriptions and keywords Aug 26, 2026
@DamienSchreurs
DamienSchreurs marked this pull request as draft August 26, 2026 10:27
@DamienSchreurs
DamienSchreurs marked this pull request as ready for review August 26, 2026 10:59
@Hyaxia
Hyaxia merged commit 58acfeb into Hyaxia:main Aug 31, 2026
@Hyaxia

Hyaxia commented Aug 31, 2026

Copy link
Copy Markdown
Owner

thanks @DamienSchreurs!
i've made small changes there (making sure the oversized feed wont crash items for example).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants