Get READMEs working on plugins and tools pages again

[simonw]

Split from:

• Finish moving hosting to Fly #163

I'd like this to be less risky. The way it worked before kept hitting GitHub rate limits (and had an annoying bug where badge URLs often broken).

I think I should scrape all the READMEs into a separate repo to avoid having to fetch them at build time. Generally split the "fetch the READMEs" process from the "build and deploy the site" process.

Maybe a simonw/datasette.io-readmes repo?

[simonw]

I created https://github.com/datasette/stashed-readmes which caches the READMEs and their rendered HTML.

It also rewrites the HTML to fix the thing where the <img src="" was to a camo.githubusercontent.com URL that expired and left broken images lying around.

[simonw]

Used o4-mini-high to write the script that fetches the READMEs: https://chatgpt.com/share/6826c1ec-2cec-8006-8fea-6c16ab1fffb6

And Claude to write the code to rewrite the image URLs: https://claude.ai/share/5bed6579-3f19-4359-985b-c1d9e9850922

And LLM on the CLI to write the GitHub Actions workflow: https://gist.github.com/simonw/f1a5b6ae5b811751792cec479b4c436f

[simonw]

OK, that repo is doing a pretty good job now. It's nearly collected everything (needs to run a few times thanks to rate limits). Next step is to integrate it into the datasette.io build scripts.

[simonw]

It has:

find .  | grep README.html | wc -l
     133

It needs 199. Manually triggering twice more might get them.

[simonw]

OK got all 199 of them. That mechanism for only fetching if the pushed_at has changed works too: