KBFuncs and UtilFuncs no longer wrap a computed strpiece position

Help/markdown/New_in_Current_Version.md

@@ -20,6 +20,10 @@ The ANSI version shares the same source code with the Unicode version, but repli
 
 ## Miscellaneous
 
+(4.2.37) **FIXED:** [jsonwrite](jsonwrite.md) wrote an array followed by a plain sibling -- a list of items, then a field -- with a stray `}` and no comma after the `]`, so the file was not JSON. Adjacent arrays with different base names (`item1`, `item2`, `thing1`) were merged into one, an array element with no attributes or children ended in a trailing comma, and a name with interior digits was split in the wrong place (`a1b2` became array `a1`, id `12`). `JsonKB`, `JsonKBRecurse` and `SeparateConCount` in `KBFuncs.nlp` had the same bugs and were fixed with it, so the builtin and the NLP++ functions still write identical output. In `KBFuncs.nlp` too, `StripEndDigits` no longer drops the last letter of a name that does not end in digits (`item` stayed `ite`), and `JsonKBRecurse` no longer writes a stray `hier.txt` debug file. Refresh an analyzer's copy of `KBFuncs.nlp` to get these.
+
+(4.2.15) **CHANGED:** a negative [strpiece](strpiece.md) position now counts from the end of the string: `strpiece("hello", 0, -2)` is `hell`. Code that computes a position as `strlength(s) - k` changes meaning when `s` is shorter than `k`: the position comes out negative and now wraps round to the end instead of running off the start, so `strpiece(",", 0, strlength(",") - 2)` is `,` rather than empty, and a loop that trims a trailing character this way never ends. Write the position from the end directly, `strpiece(s, 0, -2)`, which is the same for any string long enough and empty for one too short. `KBFuncs.nlp` and `UtilFuncs.nlp` were checked for this pattern and fixed where it could wrap (4.2.37).
+
 (4.1.2) **FIXED:** in an analyzer run by [callanalyzer](callanalyzer.md), a word that the calling analyzer's own dictionary already defined got none of the attributes in the called analyzer's lazily loaded `*full` dictionaries. For example, `hello` kept the caller's `greeting=1` but got no part of speech. The called analyzer's lazy files now add their attributes to such words too, looking each word up once. An analyzer's own lazy files still leave alone a word that its own dictionary defines.
 
 (4.1.1) **FIXED:** in an analyzer run by [callanalyzer](callanalyzer.md), passes with `@NODES _ROOT` or `@PATH _ROOT` matched nothing, because the root they were given was the passed node under its own name. A called analyzer now gets a root of its own named `_ROOT`, holding the node's children, so it selects and traverses exactly as it does on a text of its own. **FIXED:** a word already in the knowledge base with no attributes of its own was never looked up in a lazily loaded `*full` dictionary, so it got no part of speech. Such bare words are common, because every attribute name is a dictionary word, and in a called analyzer the knowledge base is shared with the caller. They are now looked up once.

Help/markdown/jsonwrite.md

@@ -40,7 +40,11 @@ The serialization is:
 | an attribute | a `"name": "value"` member |
 | counted sibling concepts (`item1`, `item2`, ...) | an array under the shared base name, each element carrying an `"id"` |
 
-Two long-standing quirks of the NLP++ original are reproduced exactly, because existing analyzers' output files depend on them: indentation for level *n* is `max(1,n)` steps rather than *n*, and a concept name containing interior digits is split at the wrong place when deciding whether it is a counted sibling.
+Only the digits at the end of a name count: `a1b1`, `a1b2` are the array `a1b` with ids `1` and `2`. Adjacent counted siblings with different base names (`item1`, `item2`, `thing1`) make separate arrays. An all-digit name such as `12` is an ordinary object.
+
+One long-standing quirk of the NLP++ original is reproduced exactly, because existing analyzers' output files depend on it: indentation for level *n* is `max(1,n)` steps rather than *n*.
+
+**FIXED 4.2.37:** an array followed by a plain sibling (`item1`, `item2`, `name`) was written with a stray `}` and no comma after the `]`, so the file was not JSON. Adjacent arrays with different base names were merged into one, an array element with no attributes or children ended in a trailing comma, and a name with interior digits was split in the wrong place (`a1b2` became array `a1`, id `12`). `JsonKB` in `KBFuncs.nlp` was fixed the same way, so the two still produce identical output; refresh an analyzer's copy of `KBFuncs.nlp` to get it.
 
 ## Example
 

Help/markdown/strpiece.md

@@ -32,6 +32,16 @@ Positions count characters, not bytes, so accented letters and emoji count as on
 
 A negative position counts from the end: `-1` is the last character, `-2` the one before it. `strpiece("hello", 1, -1)` is `ello`, and `strpiece("hello", 0, -2)` is `hell`. **NEW 4.2.15**
 
+A position outside the string, or an end before the start, gives an empty string: `strpiece("abc", -5, -1)`, `strpiece("abc", 2, 0)` and `strpiece("ab", 0, -3)` are all empty. A negative position out of range, an end before the start, or an empty string also writes `strpiece: Error.` to err.log; a start past the end does not.
+
+**Older code that computes a position can change meaning.** Before 4.2.15 a negative position ran off the start of the string. Now it counts from the end, so a position worked out as `strlength(s) - k` goes wrong when `s` is shorter than `k`: the result is negative and wraps round to the end instead. `strpiece(",", 0, strlength(",") - 2)` was meant to be empty and is now `,` -- and a loop that trims a trailing character this way never ends on a one-character string. Write the from-the-end position directly instead:
+
+```
+L("trimmed") = strpiece(L("s"), 0, -2);    # not strpiece(L("s"), 0, strlength(L("s")) - 2)
+```
+
+It gives the same result for any string long enough, and an empty one for a string too short. `strlength(s) - 1` is still safe, because it is -1 only for an empty string, where every position gives an empty result. Any larger subtraction can wrap, including one that subtracts another length, such as `strlength(s) - strlength(suffix) - 1`. If short strings are expected, test the length first; that also keeps err.log clean.
+
 ## Example
 
 ```

spec/KBFuncs.nlp

@@ -184,13 +184,14 @@ CountName(L("con"),L("root"),L("pad")) {
 }
 
 StripEndDigits(L("name")) {
+	# "item12" -> "item".  A name with no trailing digits comes back whole,
+	# an all-digit name gives 0.  Positions only count up from 0, so a short
+	# name never reaches a negative strpiece position (which counts from the
+	# end since engine 4.2.15).
 	if (strisdigit(L("name"))) return 0;
-	L("len") = strlength(L("name")) - 1;
-	L("i") = L("len") - 1;
-	L("str") = strpiece(L("name"),L("i"),L("len"));
-	while (strisdigit(L("str")) && L("i")) {
+	L("i") = strlength(L("name")) - 1;
+	while (L("i") > 0 && strisdigit(strpiece(L("name"),L("i"),L("i")))) {
 		L("i")--;
-		L("str") = strpiece(L("name"),L("i"),L("len"));
 	}
 	return strpiece(L("name"),0,L("i"));
 }
@@ -426,65 +427,89 @@ JsonKB(L("file"),L("con")) {
 # own for escaping a string as a JSON literal.  They also document exactly
 # what jsonwrite reproduces, byte for byte.
 JsonKBRecurse(L("file"),L("parent"),L("level")) {
+	# The separator is written before each child, once it is known what the
+	# child is.  Writing it after put the comma inside an array that a plain
+	# sibling then closed, which was not JSON.  A counted child with a new
+	# base name (item2 then thing1) starts its own array.
 	L("con") = down(L("parent"));
 	L("first") = 1;
-	L("has concount") = 0;
+	L("in array") = 0;
+	L("array base") = 0;
 	while (L("con")) {
 		L("name") = conceptname(L("con"));
 		L("concount") = SeparateConCount(L("name"));
 		L("connum") = G("connum");
+		L("continues") = 0;
+		if (L("concount") && L("in array")) {
+			if (L("concount") == L("array base"))
+				L("continues") = 1;
+		}
+
+		if (!L("first")) {
+			if (L("continues")) {
+				L("file") << ",\n";
+			} else if (L("in array")) {
+				L("file") << "\n" << SpacesStr(L("level")+1) << "],\n";
+				L("in array") = 0;
+			} else if (L("concount")) {
+				L("file") << ",";
+			} else {
+				L("file") << ",\n";
+			}
+		}
 
 		if (L("concount")) {
-			"hier.txt" << SpacesStr(L("level")+1) << " ====> " << L("connum") << "\n";
-			if (!L("has concount")) {
+			if (!L("continues")) {
 				L("file") << SpacesStr(L("level")+1) << JsonStr(L("concount")) << ": [\n";
-			} else {
-				L("file") << "\n";
+				L("in array") = 1;
+				L("array base") = L("concount");
 			}
 			L("file") << SpacesStr(L("level")+2) << "{\n";
-			L("file") << SpacesStr(L("level")+3) << "\"id\": \"" << L("connum") << "\",\n";
-			L("first") = !JsonAttributes(L("file"),L("con"),L("level")+2);
-			L("has concount") = 1;
-
+			L("file") << SpacesStr(L("level")+3) << "\"id\": \"" << L("connum") << "\"";
+			L("has attrs") = JsonHasAttributes(L("con"));
+			if (L("has attrs")) {
+				L("file") << ",\n";
+				JsonAttributes(L("file"),L("con"),L("level")+2);
+			}
 			if (down(L("con"))) {
-				if (!L("first")) L("file") << ",\n";
+				L("file") << ",\n";
 				JsonKBRecurse(L("file"),L("con"),L("level")+2);
 			} else {
 				L("file") << "\n" << SpacesStr(L("level")+2) << "}";
 			}
 
 		} else {
-			if (L("has concount")) {
-				L("file") << "\n" << SpacesStr(L("level")+2) << "}";
-				L("file") << "\n" << SpacesStr(L("level")+1) << "]";
-				L("has concount") = 0;
-			}
-			if (!L("first"))
-				L("file") << "\n";
 			L("file") << SpacesStr(L("level")+1) << JsonStr(L("name")) << ": {\n";
-			L("first") = !JsonAttributes(L("file"),L("con"),L("level")+1);
-
+			L("has attrs") = JsonAttributes(L("file"),L("con"),L("level")+1);
 			if (down(L("con"))) {
-				if (!L("first")) L("file") << ",\n";
+				if (L("has attrs")) L("file") << ",\n";
 				JsonKBRecurse(L("file"),L("con"),L("level")+1);
-			}
-			else if (!L("has concount")) {
+			} else {
 				L("file") << "\n" << SpacesStr(L("level")+1) << "}";
 			}
 		}
 
 		L("con") = next(L("con"));
-		if (L("con")) {
-			L("file") << ",";
-		}
 		L("first") = 0;
 	}
-	if (L("has concount")) {
+	if (L("in array")) {
 		L("file") << "\n" << SpacesStr(L("level")+1) << "]";
 	}
 	L("file") << "\n" << SpacesStr(L("level")) << "}";
 }
 
+JsonHasAttributes(L("con")) {
+	# Would JsonAttributes write anything?  An array element writes its "id"
+	# first and needs to know whether a comma follows it.
+	L("attrs") = findattrs(L("con"));
+	while (L("attrs")) {
+		L("count con") = IsCountCon(attrname(L("attrs")),L("con"));
+		if (!L("count con")) return 1;
+		L("attrs") = nextattr(L("attrs"));
+	}
+	return 0;
+}
+
 JsonAttributes(L("file"),L("con"),L("level")) {
 	L("attrs") = findattrs(L("con"));
 	if (!L("attrs")) {
@@ -550,26 +575,21 @@ JsonStr(L("str")) {
 }
 
 SeparateConCount(L("conname")) {
+	# "item12" -> "item", with G("connum") = "12".  Only the digits at the
+	# end count: "a1b2" is "a1b" and "2".  Returns 0 when the name does not
+	# end in digits or is all digits.
+	G("connum") = 0;
 	if (strisdigit(L("conname"))) return 0;
 
-	L("i") = strlength(L("conname")) - 1;
-	G("connum") = 0;
-	L("stop") = 0;
-	while (!L("stop") && L("i") > 0) {
-		L("c") = strpiece(L("conname"),L("i"),L("i"));
-		if (strisdigit(L("c"))) {
-			L("num") = L("c") + L("num");
-		} else {
-			L("stop");
-		}
+	L("last") = strlength(L("conname")) - 1;
+	L("i") = L("last");
+	while (L("i") > 0 && strisdigit(strpiece(L("conname"),L("i"),L("i")))) {
 		L("i")--;
 	}
-	if (L("num")) {
-		G("connum") = L("num");
-		L("base") = strpiece(L("conname"),0,strlength(L("conname"))-strlength(L("num"))-1);
-		return L("base");
-	}
-	return 0;
+	if (L("i") == L("last")) return 0;
+
+	G("connum") = strpiece(L("conname"),L("i")+1,L("last"));
+	return strpiece(L("conname"),0,L("i"));
 }
 
 IsCountCon(L("str"),L("con")) {

spec/UtilFuncs.nlp

@@ -2084,8 +2084,11 @@ if (L("field"))
 	  else if (L("tot") > G("MAX"))	# TRACK AND LIMIT.
 	    {
 		L("del") = G("MAX") - G("LEN");
-		L("xx") = strpiece(L("txt"),0,L("del")-1);
-		L("str") = L("str") + L("xx");
+		if (L("del") > 0)	# A 0 end of -1 would take the whole text.
+		  {
+		  L("xx") = strpiece(L("txt"),0,L("del")-1);
+		  L("str") = L("str") + L("xx");
+		  }
 		G("LEN") = G("MAX");	# TRACK AND LIMIT.
 		return L("str");
 		}
@@ -2140,8 +2143,8 @@ else if (L("ch") != "_" || L("nm") == "_")  # Literal. Alpha, num, punct, etc.
 	else if (L("tot") > G("MAX"))	# HUGE TOKEN!
 	  {
 		L("del") = G("MAX") - G("LEN");
-		L("xx") = strpiece(L("nm"),0,L("del")-1);
-		L("str") = L("xx");
+		if (L("del") > 0)	# A 0 end of -1 would take the whole token.
+		  L("str") = strpiece(L("nm"),0,L("del")-1);
 		G("LEN") = G("MAX");	# TRACK AND LIMIT.
 		return L("str");
 	  }