Open Library Solr Bulk Indexing BWB & Lenny OPDS (price)

[mekarpeles]

[P2/P1] Feature: Index and harvest a curated list of OPDS feeds on a schedule, update Open Library and Solr

Problem

Open Library needs a mechanism to index/harvest a manually curated list of OPDS URL feeds. Currently, there is no system to repeatedly check and update these feeds within Open Library and Solr.

Solution Proposal

• Allow Open Library administrators to maintain a curated list of OPDS feed URLs
• Implement a daily cron job that:
  • Checks each OPDS feed for updates (define: hash, modified date, or other)
  • Detects changes since last check
  • Updates Open Library and pushes changes to Solr

Requirements

• Method to determine when an OPDS feed has changed (e.g., ETag, Last-Modified, checksum, etc.)
• Efficient strategy for update detection (should not re-harvest if unchanged)
• Update or insert new entries into Open Library and Solr index

Open Questions

• What criteria should be used to determine if an OPDS feed has changed?
• Should this only support Atom OPDS feeds, or also RSS?
• Where to store the status/history of the last harvested state per feed?

---

Priority: P2/P1 (High)

[mekarpeles]

In this ecosystem, we have:

An App that reads OPDS (reader.archive.org)
A universal book catalog (one ring to rule them all) that generates a master OPDS feed (openlibrary.org)

Stakeholders have seen:

• Archive.org as the Library that lends books
• BetterWorldBooks.org as the Vendor that distributes ownable digital books

The ecosystem now has:

• LennyForLibraries.org that may enable Libraries (plural) to run nodes w/ OPDS that could connect to OpenLibrary.org
• BRIET is a great example (like BetterWorldBooks.org) that runs more like a BookServer node, publishes its items in OPDS

What if everyone could run a BRIET node -- some folks are investigating how something like a plugin (think shopify / wordpress) that may help independent publishers achieve something like this in a turn-key way:

• Stores files
• Handles payment & fulfillment
• Generate OPDS feeds

Almost all of these things already exist in systems like wordpress (maybe the opds generation takes a bit of work, but we're talking about a prototype opencode could 1-shot in 15 minutes).

The question becomes, if we look 2 years into the future (or less), how does a patron benefit from a world where there are e.g. 100 of these independent bookseller nodes w/ OPDS on the web?

Open Library has always existed as a catalog of every book published. One of its functions has been to facilitate access through individual libraries, like the Internet Archive, and increasingly it has grown to support a dozen or so https://openlibrary.org/trusted-book-providers.

Increasingly, as libraries, publishers, etc adopt OPDS and make their books available for purchase or borrow discoverable in the format, it makes sense that (in the same way Open Library imports MARC, ONIX) Open Library would begin to harvest/aggregate/integrate a set of known OPDS feeds, including:

• BRIET
• Lenny instances
• Shopify/Wordpress like Plugins

So that any patron can discover these sources from a single catalog (i.e. OpenLibrary.org).

This will mean (and the topic of this issue) Open Library becoming better at storing "holdings" (i.e. what materials are purchasable or borrowable, from whom, and under what terms) at scale.