Skip to content

Commit c840ac3

Browse files
chore(tools): reorganize scripts into domain subdirs; add model-download route
Move root-level dev scripts into tools/{audio,bible,inspection,archive} (git-recognized renames, content unchanged except interlinear/lexicon scrapers' import paths now pointing at tools/bible/). Update the one live subprocess path in src/scraper_client.zig to tools/bible/scraper.py, and the docs/MAINTENANCE.md references. Add tools/metanoia_server/routes/ models.py (model-download progress endpoint) and wire it into main.py. Gitignore tools/reference/ (nested qwen3-tts-apple-silicon checkout).
1 parent f30c3b1 commit c840ac3

35 files changed

Lines changed: 124 additions & 26 deletions

.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,11 @@ check_child_id.zig
112112
# the repo root.
113113
vendor/
114114

115+
# tools/reference/ — nested reference checkout of the qwen3-tts-apple-silicon
116+
# project (its own full .git dir + unpacked clone). Dev reference material,
117+
# never to be added as files or a gitlink. Same rationale as vendor/.
118+
tools/reference/
119+
115120
# aikit/examples/tommy_test.zig's reference clip — small (703KB), needed
116121
# for `zig build run-example` to actually run, same reasoning as the main
117122
# repo's data/*.wav exceptions above.

data/ollama_req.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{"model":"minicpm5:latest","prompt":"System: You are a precise biblical scholar. Use ONLY the provided data to explain the verse.\n\nVerse: John 3:1\nText: \"There was a man of the Pharisees named Nicodemus, a ruler of the Jews.\"\n\nHistorical Context: Author: Apostle John, Date: c. 85-95 AD, Audience: General / Believers. Context: Written to prove Jesus is the Son of God through signs and the 'I Am' statements. Provides high-christology and unique theological discourse not found in synoptics.\n\nChapter Summary: Jesus' discourse with Nicodemus about being born again, the love of God manifested in the Son, and the transition from the ministry of John the Baptist to Christ.\n\nLexicon & Morphology:\nWord: Ἦν (There was), Lemma: , Morph: V-IIA-3S, Definition: , Usage: \nWord: δὲ (now), Lemma: , Morph: Conj, Definition: , Usage: \nWord: ἄνθρωπος (a man), Lemma: , Morph: N-NMS, Definition: , Usage: \nWord: ἐκ (of), Lemma: , Morph: Prep, Definition: , Usage: \nWord: τῶν (the), Lemma: , Morph: Art-GMP, Definition: , Usage: \nWord: Φαρισαίων (Pharisees), Lemma: , Morph: N-GMP, Definition: , Usage: \nWord: Νικόδημος (Nicodemus), Lemma: , Morph: N-NMS, Definition: , Usage: \nWord: ὄνομα (name), Lemma: , Morph: N-NNS, Definition: , Usage: \nWord: αὐτῷ (to him), Lemma: , Morph: PPro-DM3S, Definition: , Usage: \nWord: ἄρχων (a ruler), Lemma: , Morph: N-NMS, Definition: , Usage: \nWord: τῶν (of the), Lemma: , Morph: Art-GMP, Definition: , Usage: \nWord: Ἰουδαίων (Jews), Lemma: , Morph: Adj-GMP, Definition: , Usage: \n\n\nCross References:\nNo direct cross-references found.\n\nInstruction: Provide a concise scholarly insight. Explain how the historical setting and chapter summary (literary context) inform the original word meanings to clarify the verse's intent. Connect it to a cross-reference.","stream":false}
1+
{"model":"minicpm5:latest","prompt":"System: You are a precise biblical scholar. Use ONLY the provided data to explain the verse.\n\nVerse: Matthew 19:1\nText: \"Now it came to pass, when Jesus had finished these sayings, that He departed from Galilee and came to the region of Judea beyond the Jordan.\"\n\nHistorical Context: No historical metadata found for this book.\n\nChapter Summary: No summary available.\n\nLexicon & Morphology:\nWord: Καὶ (And), Lemma: , Morph: Conj, Definition: , Usage: \nWord: ἐγένετο (it came to pass), Lemma: , Morph: V-AIM-3S, Definition: , Usage: \nWord: ὅτε (when), Lemma: , Morph: Adv, Definition: , Usage: \nWord: ἐτέλεσεν (had finished), Lemma: , Morph: V-AIA-3S, Definition: , Usage: \nWord: ὁ (-), Lemma: , Morph: Art-NMS, Definition: , Usage: \nWord: Ἰησοῦς (Jesus), Lemma: , Morph: N-NMS, Definition: , Usage: \nWord: τοὺς (the), Lemma: , Morph: Art-AMP, Definition: , Usage: \nWord: λόγους (words), Lemma: , Morph: N-AMP, Definition: , Usage: \nWord: τούτους (these), Lemma: , Morph: DPro-AMP, Definition: , Usage: \nWord: μετῆρεν (he withdrew), Lemma: , Morph: V-AIA-3S, Definition: , Usage: \nWord: ἀπὸ (from), Lemma: , Morph: Prep, Definition: , Usage: \nWord: τῆς (-), Lemma: , Morph: Art-GFS, Definition: , Usage: \nWord: Γαλιλαίας (Galilee), Lemma: , Morph: N-GFS, Definition: , Usage: \nWord: καὶ (and), Lemma: , Morph: Conj, Definition: , Usage: \nWord: ἦλθεν (came), Lemma: , Morph: V-AIA-3S, Definition: , Usage: \nWord: εἰς (to), Lemma: , Morph: Prep, Definition: , Usage: \nWord: τὰ (the), Lemma: , Morph: Art-ANP, Definition: , Usage: \nWord: ὅρια (region), Lemma: , Morph: N-ANP, Definition: , Usage: \nWord: τῆς (-), Lemma: , Morph: Art-GFS, Definition: , Usage: \nWord: Ἰουδαίας (of Judea), Lemma: , Morph: N-GFS, Definition: , Usage: \nWord: πέραν (beyond), Lemma: , Morph: Prep, Definition: , Usage: \nWord: τοῦ (the), Lemma: , Morph: Art-GMS, Definition: , Usage: \nWord: Ἰορδάνου (Jordan), Lemma: , Morph: N-GMS, Definition: , Usage: \n\n\nCross References:\nNo direct cross-references found.\n\nInstruction: Provide a concise scholarly insight. Explain how the historical setting and chapter summary (literary context) inform the original word meanings to clarify the verse's intent. Connect it to a cross-reference.","stream":false}

docs/MAINTENANCE.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ doc is about the caching architecture, the testing strategy, and what's next.
1616
## The original-language caching bug (why "verses aren't cached" happened)
1717

1818
Original-language (Hebrew OT / Greek NT) interlinear + lexicon data is
19-
scraped from BibleHub by two Python scripts (`tools/interlinear_scraper.py`,
20-
`tools/lexicon_scraper.py`) run as subprocesses from
19+
scraped from BibleHub by two Python scripts (`tools/bible/interlinear_scraper.py`,
20+
`tools/bible/lexicon_scraper.py`) run as subprocesses from
2121
`src/scraper_client.zig`, triggered once from `src/services/llm_engine.zig`
2222
when a verse has no cached lexicon context yet. Three independent bugs
2323
combined to make this "just sometimes doesn't work, silently, forever":
@@ -143,11 +143,11 @@ not just "it exists on someone's machine."
143143

144144
Follow-up to the caching-bug section above: `src/scraper_client.zig`'s
145145
`scrape_interlinear`/`scrape_lexicon` no longer shell out to
146-
`uv run python tools/interlinear_scraper.py`/`tools/lexicon_scraper.py`
147-
they call a native Zig port, `src/native_scraper.zig`, directly. (`tools/scraper.py`,
146+
`uv run python tools/bible/interlinear_scraper.py`/`tools/bible/lexicon_scraper.py`
147+
they call a native Zig port, `src/native_scraper.zig`, directly. (`tools/bible/scraper.py`,
148148
the separate verse-text scraper reachable via `scrape_verses`, is untouched —
149-
out of scope for this pass.) `tools/interlinear_scraper.py`,
150-
`tools/lexicon_scraper.py`, and `tools/scraper_common.py` are left in place
149+
out of scope for this pass.) `tools/bible/interlinear_scraper.py`,
150+
`tools/bible/lexicon_scraper.py`, and `tools/bible/scraper_common.py` are left in place
151151
(nothing else references them, and removing them wasn't asked for) but are no
152152
longer invoked by the app.
153153

@@ -160,7 +160,7 @@ longer invoked by the app.
160160
- `parseInterlinearHtml` / `parseLexiconHtml` — pure functions (`[]const u8`
161161
in, no `std.Io`), so they're unit-tested against fixture strings with zero
162162
network access, per the brief's TDD preference.
163-
- `fetchWithRetry` — mirrors `tools/scraper_common.py`'s `fetch_with_retry`
163+
- `fetchWithRetry` — mirrors `tools/bible/scraper_common.py`'s `fetch_with_retry`
164164
exactly: 3 attempts, 1s/2s/4s backoff, retries timeout/connection-failure/5xx,
165165
does not retry 4xx. The real per-attempt 15s timeout is implemented via
166166
`std.Io.Select` racing the fetch against a timer and canceling the loser —
@@ -181,12 +181,12 @@ longer invoked by the app.
181181
see native_scraper.zig's docstring and inline comments for detail):**
182182
1. On today's live BibleHub markup, Hebrew (`tablefloatheb`) word tables use
183183
`class="strongsnt"` for their Strong's-number span, not `class="pos"`/
184-
`class="strongs"` like Greek tables — so `tools/interlinear_scraper.py`'s
184+
`class="strongs"` like Greek tables — so `tools/bible/interlinear_scraper.py`'s
185185
(and this port's) `find(class_=["pos","strongs"])` never matches for
186186
Hebrew, and OT strongs numbers come back empty on a fresh scrape. Same gap,
187187
same class list, both languages. (Confirmed by fetching a real page,
188188
`curl -A "Mozilla/5.0" https://biblehub.com/interlinear/obadiah/1.htm`.)
189-
2. `tools/lexicon_scraper.py`'s `scrape_strongs()` only populates `definition`
189+
2. `tools/bible/lexicon_scraper.py`'s `scrape_strongs()` only populates `definition`
190190
(and only if the page has `class="strongs"` and/or `class="strongsnt"`
191191
elements it never finds on today's real `/greek/N.htm`/`/hebrew/N.htm`
192192
pages); `lemma`/`transliteration`/`usage` are dead code, always `""`.
@@ -228,7 +228,7 @@ the empty lemma/definition on the very first live-fetched lexicon entry is
228228
live rather than just inferred from `data/bible.db`'s existing rows.
229229

230230
**Next-session TODO:**
231-
- Consider whether `tools/interlinear_scraper.py`/`lexicon_scraper.py`/
231+
- Consider whether `tools/bible/interlinear_scraper.py`/`lexicon_scraper.py`/
232232
`scraper_common.py` should now be deleted, since nothing in the app invokes
233233
them anymore (kept for this pass only because deleting wasn't explicitly
234234
asked for).

src/scraper_client.zig

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ pub fn scrape_verses(engine: std.Io, book: []const u8, chapter: i32) !void {
1919
const ch_str = try std.fmt.allocPrint(std.heap.page_allocator, "{d}", .{chapter});
2020
defer std.heap.page_allocator.free(ch_str);
2121

22-
try runScraperScript(engine, &.{ "uv", "run", "python", "tools/scraper.py", book, ch_str });
22+
try runScraperScript(engine, &.{ "uv", "run", "python", "tools/bible/scraper.py", book, ch_str });
2323
}
2424

2525
/// Fetches original-language interlinear data for one chapter and caches it

0 commit comments

Comments
 (0)