From 15186a51768449a0a6559190e04ebb9d966d8691 Mon Sep 17 00:00:00 2001 From: David de Hilster Date: Mon, 3 Aug 2026 21:57:50 -0400 Subject: [PATCH] docs(help): jsonwrite and jsonparse, and rewire JsonKB to the builtin Engine side: VisualText/nlp-engine#712. spec/KBFuncs.nlp JsonKB becomes a one-line wrapper around the jsonwrite builtin. Its output is byte-identical, so nothing downstream changes, but the pass gets about 3x faster: serializing the KB is where a knowledge-base analyzer spends most of its time. On the date-time analyzer the thirteen-line output pass that calls JsonKB and SaveKB was 43.9% of total runtime, all of it inside interpreted NLP++. Kept as a wrapper rather than editing every analyzer's output.nlp. Note that analyzers carry their OWN copy of KBFuncs.nlp, so an existing analyzer picks this up when its copy is refreshed; newly created ones get it from the template automatically. JsonKBRecurse, JsonAttributes and JsonStr are left in place, with a note saying why: an analyzer may call them directly (JsonStr is useful on its own for escaping a string), and they document exactly what the builtin reproduces byte for byte. New pages jsonwrite.md, jsonparse.md, including the two long-standing quirks the builtin reproduces deliberately for byte compatibility -- the indent for level n is max(1,n) steps, and a concept name with interior digits is split at the wrong place when deciding whether it is a counted sibling. Also records the 3.8.1 python pass fix in the version notes. Co-Authored-By: Claude Opus 5 (1M context) --- Help/markdown/New_in_Current_Version.md | 2 + Help/markdown/Table_of_File_Functions.md | 2 + Help/markdown/index.md | 2 + Help/markdown/jsonparse.md | 70 ++++++++++++++++++++++++ Help/markdown/jsonwrite.md | 60 ++++++++++++++++++++ spec/KBFuncs.nlp | 34 +++++++----- 6 files changed, 156 insertions(+), 14 deletions(-) create mode 100644 Help/markdown/jsonparse.md create mode 100644 Help/markdown/jsonwrite.md diff --git a/Help/markdown/New_in_Current_Version.md b/Help/markdown/New_in_Current_Version.md index f3b5cd4..b4316a5 100644 --- a/Help/markdown/New_in_Current_Version.md +++ b/Help/markdown/New_in_Current_Version.md @@ -20,6 +20,8 @@ The ANSI version shares the same source code with the Unicode version, but repli ## Miscellaneous +(3.8.2) **JSON in and out as built-in functions.** [jsonwrite](jsonwrite.md) replaces the `JsonKB` family in `KBFuncs.nlp` with a built-in that produces byte-identical output. Serializing the knowledge base is where a knowledge-base analyzer spends most of its time -- on the date-time analyzer the thirteen-line output pass that calls `JsonKB` and `SaveKB` was 43.9% of total runtime -- and the built-in is about 3x faster on that pass. `JsonKB` is now a one-line wrapper, so an analyzer picks up the speedup by refreshing its copy of `KBFuncs.nlp`; newly created analyzers get it automatically. [jsonparse](jsonparse.md) is the inverse and reads a JSON document into concepts, which previously required the `json2kbb` Python library pass -- that needs Python on the machine, which the npm and pypi distributions do not provide, and it silently skips rebuilding when a stale `.kbb` is present. **FIXED:** a python pass shelled out to the literal command `python`, absent on macOS 12.3+ and most Linux distributions, and reported failure only when `system()` returned a negative value -- which never happens for command-not-found, so a failing pass was silent and its analyzer ran on with no data (3.8.1). + (3.8.0) **User functions may now shadow built-in functions.** Defining a function in a [@DECL](DECL.md) region whose name is also a built-in used to be a hard error, so adding any new built-in broke every existing analyzer that had defined a function of that name. Now the user's definition wins, the analyzer builds, and a warning names the shadowed function. This is what lets [arrayreverse](arrayreverse.md) and [push](push.md) become built-ins without disturbing the hand-written versions that have long been in `visualText/spec/UtilFuncs.nlp`, and it means future built-ins can be added safely. (3.8.0) **String and array utilities.** [split](split.md) had no inverse, so building a delimited string meant a loop and a "first time through" flag: [strjoin](strjoin.md) now does it in one call. There was also no way to get the position of a substring — [strchr](strchr.md) takes a single character and returns a tail rather than an index, and [strcontains](strcontains.md) answers only yes or no — so [strindexof](strindexof.md) and [strlastindexof](strlastindexof.md) have been added. Every existing sort was knowledge-base or parse-tree oriented, leaving plain arrays with none, so there are now [arraysort](arraysort.md) (lexical or numeric, ascending or descending, stable), [arrayunique](arrayunique.md), [arrayreverse](arrayreverse.md) and [arrayslice](arrayslice.md). [push](push.md), previously listed as UNIMPLEMENTED, now works. All of these leave the source array alone and return a new one. **FIXED:** [str](str.md) converted signed values with an unsigned format, so every negative number printed as roughly 1.8e19 — `str(-1)` gave "18446744073709551615", which also made the documented -1 returns of [filesize](filesize.md) and the new index functions unreadable. The same fault was in the compiled-analyzer path and in the code generator, so compiled analyzers emitted the wrong numeric literals too. diff --git a/Help/markdown/Table_of_File_Functions.md b/Help/markdown/Table_of_File_Functions.md index cff6887..7017614 100644 --- a/Help/markdown/Table_of_File_Functions.md +++ b/Help/markdown/Table_of_File_Functions.md @@ -14,6 +14,8 @@ The return types include: **STR** (string), **INT** (integer), and **BOOL** (Boo | [**fileexists(path_str)**](fileexists.md) | **BOOL(1,0)** | True if the path is an existing regular file. | | [**filesize(path_str)**](filesize.md) | **INT** | Size of a regular file in bytes, or -1 if not a readable file. | | [**mkdir(dir_str)**](mkdir.md) | **BOOL(1,0)** | Create a directory. Operating system-dependent. | +| [**jsonparse(json_str, parent_con)**](jsonparse.md) | **BOOL(1,0)** | Parse a JSON document into knowledge base concepts. | +| [**jsonwrite(path_str, concept)**](jsonwrite.md) | **BOOL(1,0)** | Write a concept tree to a file as JSON. | | [**readfile(path_str)**](readfile.md) | **STR** | Read a whole text file into a string. | | [**readlines(path_str)**](readlines.md) | **STR ARRAY** | Read a text file as an array of its lines, terminators stripped. | diff --git a/Help/markdown/index.md b/Help/markdown/index.md index 9877908..4121498 100644 --- a/Help/markdown/index.md +++ b/Help/markdown/index.md @@ -269,6 +269,8 @@ to a topic. - [dirlist](dirlist.md) - [fileexists](fileexists.md) - [filesize](filesize.md) + - [jsonparse](jsonparse.md) + - [jsonwrite](jsonwrite.md) - [readfile](readfile.md) - [readlines](readlines.md) - [Formatting and I/O Functions](Table_of_Formatting_and_I_O_Functions.md) diff --git a/Help/markdown/jsonparse.md b/Help/markdown/jsonparse.md new file mode 100644 index 0000000..b5955f6 --- /dev/null +++ b/Help/markdown/jsonparse.md @@ -0,0 +1,70 @@ +[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md) + +# jsonparse + +## Purpose + +Parse a JSON document into knowledge base concepts. + +## Syntax + +``` +returnedBool = jsonparse(jsonStr, parentConcept) +``` + +``` +returnedBool - type: bool (1,0) + +jsonStr - type: str + +parentConcept - type: concept +``` + +## Returns + +1 if the document parsed, else 0. + +## Remarks + +The inverse of [jsonwrite](jsonwrite.md). The members of the JSON object become children and attributes of parentConcept. + +Pair it with [readfile](readfile.md) to load a file: + +``` +jsonparse(readfile(G("$apppath") + "/data/config.json"), G("kb")); +``` + +Before this existed, the way to get JSON into an analyzer was the `json2kbb` Python library pass, which converts `kb/user/*.json` to `.kbb` before the tokenizer. That still works, but it requires Python on the machine — which the npm and pypi distributions of the engine do not provide — and it will not overwrite an existing `.kbb`, so editing the `.json` appears to do nothing until you delete the `.kbb`. `jsonparse` has neither problem. + +The mapping is the one `json2kbb` documents: + +| JSON | knowledge base | +| --- | --- | +| an object | a concept | +| `"key": ` | an attribute `key=value` | +| `"key": { ... }` | a child concept named `key` | +| `"key": [ ... ]` | counted children `key1`, `key2`, ... | +| a primitive array element | `keyN` with a `value` attribute | + +An integer becomes a num, a number containing `.`, `e` or `E` becomes a float, `true` and `false` become 1 and 0, and `null` becomes an empty string. String escapes are resolved, including `\uXXXX` with surrogate pairs combined. + +A top-level object is required, and trailing content after it is an error, so a truncated or doubled document is reported rather than half-loaded. Inside an array element, an `"id"` member is skipped: [jsonwrite](jsonwrite.md) generates it from the concept's name, so keeping it would duplicate it on the next write. + +## Example + +``` +@CODE + +L("text") = readfile(G("$apppath") + "/data/settings.json"); +G("cfg") = makeconcept(findroot(),"cfg"); + +if (jsonparse(L("text"),G("cfg"))) { + G("threshold") = numval(findvals(findconcept(G("cfg"),"tuning"),"threshold")); +} + +@@CODE +``` + +## See Also + +[jsonwrite](jsonwrite.md), [readfile](readfile.md), [makeconcept](makeconcept.md), [Knowledge Base Functions](Table_of_Knowledge_Base_Functions.md) diff --git a/Help/markdown/jsonwrite.md b/Help/markdown/jsonwrite.md new file mode 100644 index 0000000..33681c9 --- /dev/null +++ b/Help/markdown/jsonwrite.md @@ -0,0 +1,60 @@ +[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md) + +# jsonwrite + +## Purpose + +Write a knowledge base concept tree to a file as JSON. + +## Syntax + +``` +returnedBool = jsonwrite(fileNameStr, concept) +``` + +``` +returnedBool - type: bool (1,0) + +fileNameStr - type: str + +concept - type: concept +``` + +## Returns + +1 if the file was written, else 0. + +## Remarks + +This is the built-in form of the `JsonKB` function in `KBFuncs.nlp`, with the same argument order and **byte-identical output**, so it is a drop-in replacement. `JsonKB` is now a one-line wrapper around it. + +Serializing the knowledge base is where a knowledge-base analyzer spends most of its time. Measured on the date-time analyzer, its `output.nlp` — thirteen lines calling `JsonKB` and `SaveKB` — was **43.9%** of total runtime, all of it inside interpreted NLP++. The built-in is roughly **3x faster** on that pass. + +A bare filename is written into the current input's log directory, the same place the [output operator](NLP_PP_Stuff/Operators_and_Expressions.md) `<<` writes. An absolute path is used as given. + +The serialization is: + +| knowledge base | JSON | +| --- | --- | +| a concept | an object | +| an attribute | a `"name": "value"` member | +| counted sibling concepts (`item1`, `item2`, ...) | an array under the shared base name, each element carrying an `"id"` | + +Two long-standing quirks of the NLP++ original are reproduced exactly, because existing analyzers' output files depend on them: indentation for level *n* is `max(1,n)` steps rather than *n*, and a concept name containing interior digits is split at the wrong place when deciding whether it is a counted sibling. + +## Example + +``` +@CODE + +G("results") = makeconcept(findroot(),"results"); +# ... passes accumulate into G("results") ... + +jsonwrite("output.json",G("results")); + +@@CODE +``` + +## See Also + +[jsonparse](jsonparse.md), [readfile](readfile.md), [makeconcept](makeconcept.md), [Knowledge Base Functions](Table_of_Knowledge_Base_Functions.md) diff --git a/spec/KBFuncs.nlp b/spec/KBFuncs.nlp index ffdefd5..e0d55f2 100644 --- a/spec/KBFuncs.nlp +++ b/spec/KBFuncs.nlp @@ -399,20 +399,26 @@ PadStr(L("num str"),L("pad str"),L("pad len")) { ############################################### JsonKB(L("file"),L("con")) { - L("file") << "{\n" << SpacesStr(1) << "\"" << conceptname(L("con")) << "\": {\n"; - # Emit the top concept's OWN attributes before its child concepts. - # JsonKBRecurse only descends into children, so any attribute on the - # top concept was silently dropped. Emitting them here lets a top - # concept carry scalar fields alongside its nested objects (e.g. a - # "document_type" next to nested "data" objects). When the top concept - # has no attributes, output is unchanged. - L("hasattrs") = JsonAttributes(L("file"),L("con"),1); - if (down(L("con")) && L("hasattrs")) - L("file") << ",\n"; - JsonKBRecurse(L("file"),L("con"),1); - L("file") << "\n}\n"; -} - + # Delegates to the jsonwrite builtin (engine 3.8.2 and later). + # + # This used to be ~130 lines of NLP++ across JsonKB, JsonKBRecurse, + # JsonAttributes and JsonStr. Serializing the KB is where a knowledge + # base analyzer spends most of its time -- measured on the date-time + # analyzer, the 13-line output pass that calls JsonKB and SaveKB was + # 43.9% of total runtime. The builtin produces byte-identical output + # and is about 3x faster on that pass. + # + # Kept as a wrapper rather than changing every analyzer's output.nlp, + # so existing analyzers pick up the speedup unchanged. An analyzer + # that defines its own JsonKB still wins: a user function shadows a + # builtin, and this is an ordinary NLP++ function like any other. + jsonwrite(L("file"),L("con")); +} +# JsonKBRecurse, JsonAttributes and JsonStr below are no longer reached from +# JsonKB, which now calls the jsonwrite builtin. They are kept because an +# analyzer may call them directly -- JsonStr in particular is useful on its +# own for escaping a string as a JSON literal. They also document exactly +# what jsonwrite reproduces, byte for byte. JsonKBRecurse(L("file"),L("parent"),L("level")) { L("con") = down(L("parent")); L("first") = 1;