[ddehilster]

strindexof and strlastindexof answered in bytes, strlength in UTF-16 units, and strpiece and strchar in characters. On text with a character outside ASCII ahead of the match, a position from one function was wrong in the next.

What it broke. A grammar that cut a label off at the first colon until none was left:

while (strcontains(":",L("s"))) {
    L("i") = strindexof(L("s"),":");
    ...strpiece(L("s"),L("i") + 1, ...)
}

never ended on Registrar’s Signature ... NAME: .... After the curly apostrophe (3 bytes), the index pointed one character past the colon, so the cut came back unchanged. Every turn interned a new string until the process was killed at 2 GB. In production that took down a document extractor five times in thirty minutes, on one birth-certificate translation.

strpiece also had a limit: it copied into a MAXSTR (512) buffer and refused any piece of 512 characters or more (strpiece: Error.). The compiled Arun::strpiece sized its buffer in characters but copied bytes, so multi-byte text overran it.

Changes

• str_chars() and str_char_index() in str.cpp count code points. strlength (interpreted and compiled), strindexof and strlastindexof use them.
• fnStrpiece and Arun::strpiece size the buffer from the string's bytes. str_piece now bounds the copy in bytes; before, it compared the number of characters asked for with the buffer size.
• New fixture str-positions, run on Windows, Linux and macOS. It covers each function, plus the loop above capped at 20 turns, so a regression fails the job instead of hanging it. On 4.2.39 the fixture prints colon:28, tail:EIXEIRA, emoji-len:6 and long-piece:0.

Behaviour change. strlength of a string with a character outside the BMP (an emoji) is now 1 per character instead of 2. It now matches what strpiece and strchar already did.

Checked

• 52 of the 54 Windows CI steps pass locally. The two that don't need CI's environment: compiled-bridge needs packaged compile libraries, and test-nlp needs the analyzers checkout.
• 881 fixture inputs of 17 production analyzers give byte-identical output.json on 4.2.39 and this build.
• The production analyzer that hung finishes in 2 seconds, unchanged.

🤖 Generated with Claude Code