Transform local dumped compressed copy of articles (xml.bz2) + index.bz2) to epub

[silbinarywolf]

What would change in wiki2book?

Sub Issue 1
Add support for generating an epub from Wikimedia database dump XML and associated index file. (Links at bottom of issue)

wikipedia-host configured to point at file:/enwiki-20260701-pages-articles-multistream.xml.bz2, it then also looks for an associated enwiki-20260701-pages-articles-multistream-index.txt.bz2 file automatically.

{
  // Loads `./enwiki-20260701-pages-articles-multistream.xml.bz2` and `enwiki-20260701-pages-articles-multistream-index.xml.bz2``
  "wikipedia-host": "file://enwiki-20260701-pages-articles-multistream.xml.bz2"
  // List articles you want as normal
  // "articles": []
}

Sub Issue 2
Related to this, I also want a directive to convert all articles into 1 epub. Maybe this can be limited to localhost servers and files only. This might be unreasonable but I don't think Sub Issue 1.

{
  "articles": [":all_articles"]
}

What's the motivation behind this change?

Why is this useful to others?

It allows local database dumps of wikipedia to be converted to ebooks as needed. Being able to do this regardless of API limits / whether the internet is down or up.

Why do I personally want it?

• I have a Kindle Paperwhite Gen 12 w/ 32GB (has ~25GB of free space right now)
• I want to at least attempt to convert as much of Wikipedia as possible (if not all of it) to an epub, then azw3, then just put it on my Kindle.

What existing parts of wiki2book might be influenced by this change?

• Sub-Issue 1: Config needs updating to support pointing at files instead of hosts: wikipedia-host
• Sub-Issue 2: Add directive for targetting all articles

What alternatives are there?

Not from my googling, but I'm very happy to be proven wrong.

Additional context

Data dump of all of wikipedia

• https://dumps.wikimedia.org/enwiki/20260701/
  • Index file for each article: enwiki-20260701-pages-articles-multistream-index.txt.bz2 269.7 MB
  • The data: enwiki-20260701-pages-articles-multistream.xml.bz2 24.7 GB

[hauke96]

Thanks for the feature request. Making wiki2book work completely offline would be quite nice (even though maybe not the main use-case), but it's also quite difficult and not completely possible. However, it sounds like a good idea to be able to use local files/dumps for a lager project. It would allow people to create larger eBooks from non-Wikipedia files as well.

First of all a general note on rate limits: Yes, they slow down the process of creating an eBook and yes, reducing the load on their API would always be a good thing. However, I never experienced any problems (like a ban for the rest of the day or something like that) and wiki2book respects the best-practices, i.e. it sets the correct headers, respects HTTP 429 responses and caches all responses and data so that the API is not used at all, when the cache is still full and the eBook is re-created. The size and parameter of the cache can be configured in case the default are too small for you.

In contrast to #114 the XML dumps contain the raw wikitext (that's good and a fundamental requirement) and the articles should - in theory - be usable by wiki2book. But I have to disappoint you: as of now, the amount of API calls would not significantly decrease. This is, because templates, math expressions and images must still be rendered/downloaded from the API and these things account for the majority of requests. Articles usually contain dozens or even hundreds of templates leading to the same number of requests.

So you'll still run into rate limits. But some things can be done here or at least here are some thoughts:

Yes, the dump also contains the template pages, but wiki2book has no logic for template expansion, which is the process of turning them into wikitext or HTML. This is a logic that I want to avoid implementing, since wiki2book is not supposed to be a kind of mediawiki-clone or -engine, and the effort would probably be enormous. However, I think it would be easy to make the template expansion configurable so that a local CLI-tool or script could be used instead of the API. Of course that would only work in case there is such a CLI-tool or script (maybe wikitextprocessor works well? Never tried it and maybe some scripting would be needed to create a nice CLI-API for wiki2book). If you (or someone else reading this) can share experiences with locally expanding templates, that would be great.

Another solution for the template expansion problem would be to host a local mediawiki server instance. It offers the possibility to import dumps so maybe that's not too much work. In case this works, having a detailed guide in this repo on how to set up such a local server would be nice. Ideally it's just a simple docker-compose file mounting a dump and that's it. It's probably more difficult than that 😉

To spread some optimism: Math expressions are much easier. They are basically LaTeX (iirc with some limitations), which could be rendered locally (with a TeXlive setup or maybe there's even a docker-container for that). Making the math rendering completely configurable is something that came to my mind a few times now. I think about this and might open a separate issue.

An last but not least the images. I think there's no solution here, since you can't simply clone the whole Wikimedia Commons database of all images. Luckily there are - usually - not so many images per article, so running into rate limits due to images should be rare. I don't know if it's a use-case to provide the path to a local directory with images instead of using the API. I also think about this and might open a separate issue.

Making the dump files usable is possible but there's a lot of work ahead. Let me know what you think. Also if you're interested in working on some part of this, let me know, we can discuss details in a separate issue for that part.