Skip to content

Enhance feed-history utility with date index support and fixes#73

Merged
TBosak merged 13 commits into
deployfrom
main
May 15, 2026
Merged

Enhance feed-history utility with date index support and fixes#73
TBosak merged 13 commits into
deployfrom
main

Conversation

@TBosak

@TBosak TBosak commented May 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Feed date preservation for web and API feeds — Items in web-scraping and API feeds no longer receive a fresh on every scrape. A per-feed JSON date index
    (feed-history/{feedId}.dates.json) records each item's first-seen timestamp by a stable SHA-256 fingerprint of its content (excluding date). Subsequent scrapes restore the original date for
    recognised items, preventing RSS clients from treating existing items as new on every poll.
  • Stable GUID fallback — The GUID fallback for feeds without an explicit GUID selector previously hashed the full item including the volatile date, so GUIDs changed every scrape. It now uses
    the same stable fingerprint, making GUIDs consistent across scrapes.
  • Enclosure URL XML escaping fix — Enclosure URLs containing & in query strings (e.g. image CDN URLs) were written unescaped into XML attributes, producing invalid XML (EntityRef: expecting
    ';'). The URL is now sanitized for XML output while the raw URL is still used for the content-length fetch.
  • Biome linting — Added Biome with recommended rules; bun run lint and bun run test scripts added to package.json. ~100 auto-fixable violations applied (node: import protocol, parseInt radix,
    import type, template literals, optional chaining, useConst).

Migration note

On the first scrape after deploying, feeds without an explicit GUID selector configured will have new GUIDs generated for all existing items (one-time only — the algorithm changed from a
date-dependent hash to a stable one). Subscribers may see all items appear as unread once.

TBosak and others added 13 commits May 14, 2026 14:08
…tility

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace top-level-only key sort with recursive stableStringify so
  nested objects (enclosure, category) hash identically regardless of
  field insertion order
- Rename unused `date` binding to `_date` to silence lint warnings
- Re-throw write errors in saveDateIndex so callers see disk/permission
  failures instead of them being silently swallowed
- Strengthen makeItemKey test to use two objects with different insertion
  orders rather than the same reference twice
- Add test for corrupt/invalid JSON in loadDateIndex index file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… import

Wrap saveDateIndex in its own try/catch so a transient disk error on the
date-index file no longer suppresses the webhook or marks the feed update
as failed. Also promote getPreviousFeedHistory and storeFeedHistory to the
existing static import of feed-history.utility, eliminating the redundant
dynamic import inside the webhook block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add biome.json (recommended rules, excludes frontend/ and public/)
- Add lint and test scripts to package.json
- Apply auto-fixes: node: import protocol, parseInt radix, import type,
  template literals, optional chaining, useConst, useLiteralKeys
- 215 → 119 remaining violations (95 noExplicitAny, 12 noControlCharactersInRegex,
  others require manual review)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enclosure URLs with query string ampersands (e.g. ?foo=1&bar=2) were
written raw into XML attributes, producing invalid XML. Use fetchUrl
for the content-length HEAD request and sanitizeURLForXML for the
feed output, matching the existing behaviour in buildRSSFromApiData.

Fixes: "EntityRef: expecting ';'" parse error on feeds with image enclosures.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ntrolled format string'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…ntrolled format string'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Enhance feed-history utility with date index support and fixes
@TBosak
TBosak merged commit df91496 into deploy May 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant