Book Availability in Solr

[mekarpeles]

Right now, when Open Library wants to find the availability of a book, we hit Archive.org's Bulk Availability Service (which doesn't currently do much cacheing). The performance implications aren't great for either party.

Question

Is there any internal changes API for loans which can report: "what loans have been made in the last ~1 minute"?

Rational

Why 1 min? Back of napkin numbers; If we somehow reached 100M loans per year (way beyond what we do today) , that would be about 200 loans a minute (on avg; discounts temporal locality / popularity spikes). This number is cheap enough to calculate and process at that time increment.

Opportunity / Benefits

With such a loans/changes API in place, we could:

• add accurate availability into Open Library solr and hopefully drastically lower OL dependency on IA ES @ximm
  • We may go from making 10s of thousands of requests to ES --> a few requests to this API / minute.
• significantly improve performance & ux of Open Library pages which make several Bulk Availability calls to IA
  • search, carousels, books, my books

Risks & Mitigations

1. Would this change but too much stress on solr-updater? Editing 1k bulk solr records each minute or so? @cdrini
   • ?
2. How do we handle cold-starts? (e.g. a solr-reindex which has no knowledge of anonymized active loans)
   • On OL Books Page & borrow action, we already hit ground truth loan API. We could use this to update availability_status in solr
   • We could use the Bulk Availability API to warm cache (just for records OL solr records with no availability_status)

Stakeholders

@bfalling @ximm @mekarpeles