Index Lenny instance(s) holdings in Open Library

[cdrini]

Feature Request

In order to allow for searching of a specific lenny instance or across all lenny instances, we want to be able to store the holding in Open Library's solr.

Proposal

POST /api/lenny/synchronize?check=true
> Response {
    "status": "ok",
    "last_synchronization": "2024-01-02T12:00:00Z"
}

POST /api/lenny/synchronize
> Do the actual synchronization
> Body {
    "catalog": [
        "/books/OL123M",
    ]
}
> Response {
    "added": [ ... ],
    "removed": [ ... ],
    "missing": [ ... ]
}

1. Lenny on startup/every 30 minutes (with jitter!) does a heartbeat, and gets back the last time OL updated its catalog for this lenny instance.
   • Two ways: OL can store this timestamp in infogami/db (?)
2. If the returned timestamp is less than its internal last updated timestamp, it will send its entire catalog for synchronization.
3. When OL receives a synchronize request, it will query solr to find all editions with lenny_host:{host}, intersect that against the provided catalog and add missing using a partial add-distinct update to lenny_host, and delete no longer supplied catalog values.
   • Note: We ideally also want the ebook-access; blocked on Add Solr acquisitions field for BookServer integration #11264

Full reindex will delete all lenny_host:{host} values for editions, which will cause the next lenny heartbeat to send its catalog in entirety.

When a solr document is reindexed normally, we will copy-forward the existing lenny_host:{host} values from the existing solr document to the new one.

Risks:

• It's expensive to synchronize, since no partial updates are easily possible
• Race-conditions if multiple synchronize calls happen at once

Breakdown

Related files

Refer to this map of common Endpoints:
*

Requirements Checklist

Checklist of requirements that need to be satisfied in order for this issue to be closed:

• [ ]

Stakeholders

@mekarpeles @ronibhakta1

---

Instructions for Contributors

• Before creating a new branch or pushing up changes to a PR, please first run these commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

[Aayushdev18]

I’m interested in implementing this feature. Can I work on it?

[mekarpeles]

For now, we're just looking for a way to do testing with a few participating nodes, rather than this being a feature that is mainstreamed in Open Library + Lenny.

For a v1, the idea would just be to give Lenny instances a way to trigger a sync manually (which will likely be < 1000 items). This would also unblock demos for Maria Bustillos would has a few dozen newsletters she'd like to enable libraries to lend.

@cdrini + @pidgezero-one