[ddehilster]

parse-en-us now runs its passes after zaplines once per text zone (paragraph). Each zone's tree is freed when its passes finish, so memory follows the largest paragraph instead of the whole document.

Pride and Prejudice (738 KB)	Peak memory	Time	Tags
whole document	473 MB	275 s
zoned (this PR)	150 MB	321 s	99.995% of 151,646 the same

The two runs were on different builds (x64 earlier, x86 now), so the times are only roughly comparable.

Requires engine 4.2.47 or later, or 4.2.49 or later when compiled (VisualText/nlp-engine#807). Please merge after 4.2.49 is released: older engines don't know the zone line.

What changed

• analyzer.seq:
  • a zone _TEXTZONE line after zaplines;
  • an endzone line, after which docfin, trav_dict, ne_xml, fin and displayKB run once, after the last zone.
• New docfin pass: the end-of-document work of two per-zone passes, pos_out_noscore's postofile() and xmlrecursive's xmlend()/closefile().
• These passes now write their headers once per document, guarded by G() flags. G() resets per input file, so the guards work the same with or without zoning:
  • rule_out
  • ne_out
  • buff_out
  • xmlrecursive
  • pos_out_noscore (its POS-list reset and treebank blurb)
• pos_out_noscore carries its line state from zone to zone, so the blank lines in tags.txt come out the same.

Same output

On business.txt, doj.txt and test.txt, compared with an unzoned run on the same engine:

• byte-identical: tags.txt, pos.txt, np.txt, rule.txt and the final KB dump;
• doj.txt and test.txt: every output file the same;
• business.txt: KB facts that later passes learn in one paragraph are now known to the next paragraph's early passes, so AMRP and TAI get stems instead of being unknown. That changes clause_ne.txt, out.xml and unknown.txt, and removes three "Unhandled period" and three "More than one POS in caps list" messages;
• trace files (dump.txt, wsd.txt) list the same lines, interleaved per zone.

The KB dump is now ana137.kbb, not ana136.kbb, because docfin adds a pass.

Trade-off in the debugger

With -DEV, passes after the zone line write trees only for the first three zones (z001-ana###.tree, z002-…, z003-…), and final.tree keeps each paragraph as a stub. Clicking such a pass in VS Code (4.3.18+) opens the first zone's tree. On a multi-paragraph input you no longer see a later paragraph at every pass. To debug one, delete the zone and endzone lines; the analyzer runs the same either way.

🤖 Generated with Claude Code