Skip to content

KBFuncs and UtilFuncs no longer wrap a computed strpiece position - #247

Merged
ddehilster merged 1 commit into
VisualText:mainfrom
ddehilster:fix/strpiece-length-arithmetic
Oct 19, 2026
Merged

ddehilster merged 1 commit into
VisualText:mainfrom
ddehilster:fix/strpiece-length-arithmetic

Conversation

@ddehilster

Copy link
Copy Markdown
Member

Since engine 4.2.15 a negative strpiece position counts from the end. Code that computes a position as strlength(s) - k now goes wrong when s is shorter than k: the position comes out negative and wraps round to the end, where it used to run off the start. strpiece(",", 0, strlength(",") - 2) gives "," instead of "", and a trim loop written that way hung a production analyzer on a one-character value.

What changed

spec/KBFuncs.nlp

  • StripEndDigits started its scan one character early. A one-letter name only came back right because -1 wrapped, and a name with no trailing digits lost its last letter. Rewritten so positions only count up from 0.

  • SeparateConCount computed the base as strlength(name) - strlength(num) - 1. Its else branch was the no-op L("stop");, so it never stopped at a letter. It now takes only the trailing digits, with no subtraction that can go negative.

  • JsonKBRecurse wrote each separator after a child:

    • An array followed by a plain sibling was not JSON (a stray }, and no comma after the ]).
    • Two arrays side by side merged into one.
    • An element with no attributes or children ended in a trailing comma.

    Separators now go out before each child. Every shape that was already valid JSON is written byte for byte as before. The debug write of hier.txt on every call is gone. New helper: JsonHasAttributes.

  • The jsonwrite builtin copied these bugs on purpose. It's fixed the same way in jsonwrite writes valid, separate arrays (4.2.37) nlp-engine#796 (4.2.37), so JsonKB and jsonwrite still give identical output.

spec/UtilFuncs.nlp

  • prosifyrec trimmed with strpiece(txt, 0, del - 1) in two branches without the del > 0 guard its third branch got in 2011. At del 0 the end is -1, which took the whole text instead of none of it.
  • Every other computed position in spec/, misc/ and languages/ subtracts only 1 or sits behind a length test, so it can't wrap. Checked by hand: the morph helpers (behind len > 4), the suffix confidence code (behind len > 2, and unreachable), numth, the percentage formatter, and the word remover with its # Sanity. checks.

Help

  • strpiece.md: what changed for computed positions, with the "," example and the -k idiom. Also that out-of-range positions return "" but write strpiece: Error. to err.log, except a start past the end, which doesn't log.
  • jsonwrite.md and New_in_Current_Version.md (new 4.2.15 and 4.2.37 entries) describe both fixes.

Before and after

Run with a one-pass test analyzer calling the functions; old = current main, new = this branch.

Name StripEndDigits old → new SeparateConCount old → new
"" 0 → 0 0 → 0
a a → a (old was right only because -1 wrapped) 0 → 0
12 0 → 0 0 → 0
abc12 abc → abc abc, 12 → abc, 12
item3 item → item item, 3 → item, 3
item ite → item 0 → 0
a1b2 a1b → a1b a1, 12 → a1b, 2

The JSON shapes below were written by JsonKBRecurse and by jsonwrite. Old = engine 4.2.36 with current KBFuncs; new = the #796 engine with this KBFuncs.

Children Old New Builtin == NLP++
item1 item2 name invalid valid yes / yes
item1 item2 thing1 thing2 valid, one merged array valid, two arrays yes / yes
a1b1 a1b2 valid, array a1 valid, array a1b yes / yes
name item1 item2 valid valid, byte-identical to old yes / yes
row1 (bare), row2→child, tail invalid valid yes / yes
nested arrays, then a plain sibling invalid valid yes / yes

No output change in real analyzers. I patched the KBFuncs copies in VisualText/analyzer-templates, analyzers, nlp-tutorials, nlpfix-analyzers and parse-en-us (companion PRs). Then I ran each analyzer that carries a copy on its inputs before and after: 18 analyzers, plus parse-en-us. All output was unchanged; only the analyzer's own path in init.log differed. With UtilFuncs, prosify gives the same result for every limit from 1 to 30. Help/check-help-docs.ps1 passes; its one orphan, New_in_Current_Version, was already missing from index.md.

🤖 Generated with Claude Code

Since engine 4.2.15 a negative strpiece position counts from the end, so
a position computed as strlength(s) - k wraps round instead of running
off the start when s is short. Fixed in the shared library:

KBFuncs.nlp
- StripEndDigits started its scan one character early: a one-letter name
  only came back right because position -1 wrapped, and a name with no
  trailing digits lost its last letter ("item" -> "ite"). Rewritten as a
  scan that only counts up from 0.
- SeparateConCount computed the base as strlength(name) -
  strlength(num) - 1, and never stopped at a letter (its else branch was
  the no-op `L("stop");`), so "a1b2" was base "a1", count "12". It now
  takes only the trailing digits, with positions counted up from 0.
- JsonKBRecurse wrote each separator after a child, so an array followed
  by a plain sibling was not JSON (stray "}", no comma after "]"), two
  arrays side by side merged, and an element with no attributes ended in
  a trailing comma. Separators are now written before each child. It
  also wrote a hier.txt debug file on every call; removed. Every shape
  that was valid JSON is written byte for byte as before. The jsonwrite
  builtin had the same bugs and is fixed to match (nlp-engine 4.2.37,
  VisualText/nlp-engine#796), so JsonKB and jsonwrite still agree.

UtilFuncs.nlp
- prosifyrec trimmed with strpiece(txt, 0, del - 1) in two branches
  without the del > 0 guard its third branch got in 2011; at del 0 the
  end is -1 and took the whole text instead of none of it.

The other computed positions in spec/, misc/ and languages/ subtract
only 1 or sit behind a length test, and are left alone.

Help: strpiece.md explains what changed for computed positions, with the
"," example and the -k idiom, and that out-of-range positions log an
error; jsonwrite.md and New_in_Current_Version.md (4.2.15, 4.2.37)
describe both fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant