Better World Books OPDS feed importer

[ronibhakta1]

Closes #12844

refactor BWB script

Technical

Testing

Screenshot

Stakeholders

@cdrini @mekarpeles

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds Trusted Book Provider (TBP) infrastructure (feed registry + acquisitions) and extends the Better World Books OPDS importer to record purchase acquisitions and persist incremental progress via the registry.

Changes:

• Introduces acquisitions + tbp_feed_registry tables and corresponding Python model/service layers (Acquisition, FeedRegistry) with tests.
• Extends scripts/bwb_opds_imports.py to extract buy URLs/acquisition data, upsert acquisitions for existing editions, and use tbp_feed_registry.last_updated as the primary incremental cursor.
• Expands OPDS importer tests to cover acquisition extraction, acquisition upsert mapping, and registry cursor normalization.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/tests/test_bwb_opds_imports.py	Adds coverage for acquisition extraction/collection, acquisition upserts, and registry cursor normalization.
scripts/bwb_opds_imports.py	Extracts buy URL/acquisition data, collects acquisitions during crawl, upserts to DB, and uses the TBP registry cursor.
openlibrary/tests/core/test_tbp.py	Adds unit tests for FeedRegistry CRUD/idempotent registration/advance.
openlibrary/tests/core/test_acquisitions.py	Adds unit tests for acquisition lookup, upsert behavior, and merge remapping helpers.
openlibrary/core/tbp.py	Implements the TBP feed registry interface.
openlibrary/core/schema.sql	Adds acquisitions and tbp_feed_registry tables + indexes.
openlibrary/core/acquisitions.py	Implements the acquisitions table interface, including upsert and edition-merge remapping.

[mekarpeles]

Thanks for the PR, @ronibhakta1!

🤖 Copilot has been assigned for an initial review.

The linked issue (#12844) hasn't been triaged yet — triage happens on Mondays and Fridays. There are currently 74 open non-draft PRs ahead of yours.

Possible improvements for this PR

• PR description is incomplete — the Technical, Testing, and Screenshot sections are unfilled. For future PRs, please follow the PR template and provide details. Here's a draft based on the diff and linked issue:

  This PR implements the first three subtasks of the TBP ingestion pipeline (#12844): it introduces a tbp_feed_registry table and FeedRegistry class for tracking provider feeds, an acquisitions table and Acquisition class for storing per-edition provider metadata, and extends bwb_opds_imports.py to extract acquisition records from the BWB OPDS feed and upsert them into the new table. Unit tests are included for all three new modules.

• No proof of testing — the Testing section is empty and no test output is shown. A brief description of how you ran or validated the changes (e.g. running the test suite locally, a dry-run of the importer) would help reviewers move faster. See how to attach files if you'd like to include output.

PR triage checklist (maintainers / Pam)

• PR description — not empty; explains what the change does and how to verify it
• References an issue — PR body contains a #NNN reference
  • Linked issue is triaged — has a Priority: * label (not just Needs: Triage)
  • Linked issue is assigned — has at least one assignee
• Commit history clean — no WIP/fixup/conflict noise; commit messages are meaningful (note: history includes a merge commit from 12844/feature/tbp-feed-registry; consider rebasing)
• CI passing — no failing check-runs
• Test cases present — if the change touches substantive logic, test coverage exists or is explained
• Proof of testing — PR body includes a description of what was tested, a screenshot, or a video

Note

This comment was automatically generated by Pam, Open Library's Project AI Manager, on behalf of @mekarpeles. Pam is designed to provide status visibility, perform basic project management functions and relevant codebase research, and provide actionable feedback so contributors aren't left waiting.

[cdrini]

Pushed up CR feedback from #12851 ; apologies likely causes some issues with some of the other code! Continuing to review/push up fixes over the weekend. I will review #12846 next, and push up any small CR fixes here, before reviewing this one and giving everything a test.