docs: callanalyzer and callconcept (engine 4.1.0)

Help/markdown/New_in_Current_Version.md

@@ -20,6 +20,8 @@ The ANSI version shares the same source code with the Unicode version, but repli
 
 ## Miscellaneous
 
+(4.1.0) **One analyzer can call another.** [callanalyzer](callanalyzer.md) runs another analyzer's passes on the subtree under a parse node, so one analyzer can be built from several separately written ones that all share a single parse tree and knowledge base. The called analyzer changes the caller's tree directly and puts its results under the concept it was given, which it gets with [callconcept](callconcept.md). It is loaded on the first call, together with its dictionaries and `.kbb` files, and stays in memory, so later calls pay nothing to load it. Its `dicttok` pass does not retokenize; it looks up the node's words, including in lazily loaded `*full` dictionaries, and gives them their dictionary attributes. Each call starts with empty `G()` variables. An analyzer cannot call itself, directly or through another analyzer. This completes what [findana](findana.md) started in 1.6.
+
 (3.8.2) **JSON in and out as built-in functions.** [jsonwrite](jsonwrite.md) replaces the `JsonKB` family in `KBFuncs.nlp` with a built-in that produces byte-identical output. Serializing the knowledge base is where a knowledge-base analyzer spends most of its time -- on the date-time analyzer the thirteen-line output pass that calls `JsonKB` and `SaveKB` was 43.9% of total runtime -- and the built-in is about 3x faster on that pass. `JsonKB` is now a one-line wrapper, so an analyzer picks up the speedup by refreshing its copy of `KBFuncs.nlp`; newly created analyzers get it automatically. [jsonparse](jsonparse.md) is the inverse and reads a JSON document into concepts, which previously required the `json2kbb` Python library pass -- that needs Python on the machine, which the npm and pypi distributions do not provide, and it silently skips rebuilding when a stale `.kbb` is present. **FIXED:** a python pass shelled out to the literal command `python`, absent on macOS 12.3+ and most Linux distributions, and reported failure only when `system()` returned a negative value -- which never happens for command-not-found, so a failing pass was silent and its analyzer ran on with no data (3.8.1).
 
 (3.8.0) **User functions may now shadow built-in functions.** Defining a function in a [@DECL](DECL.md) region whose name is also a built-in used to be a hard error, so adding any new built-in broke every existing analyzer that had defined a function of that name. Now the user's definition wins, the analyzer builds, and a warning names the shadowed function. This is what lets [arrayreverse](arrayreverse.md) and [push](push.md) become built-ins without disturbing the hand-written versions that have long been in `visualText/spec/UtilFuncs.nlp`, and it means future built-ins can be added safely.

Help/markdown/Table_of_Special_Functions.md

@@ -8,6 +8,8 @@ The return types include:  **STR** (string), **INT** (integer), and **BOOL** (B
 
 | **FUNCTION NAME** | **RETURNS** | **DESCRIPTION** |
 | --- | --- | --- |
+| [**callanalyzer(pnode, concept, analyzer_str)**](callanalyzer.md) | **BOOL(1,0)** | Run another analyzer's passes on the subtree under pnode, sharing this parse tree and knowledge base. The analyzer is loaded once and stays in memory. **(New in 4.1.0)** |
+| [**callconcept()**](callconcept.md) | **CONCEPT** | In an analyzer run by callanalyzer, the concept it was given for its results. **(New in 4.1.0)** |
 | [**debug()**](debug.md) | **BOOL(1,0)** | No op. Convenient breakpoint for debugging from C++ compiler. |
 | [**exitpass()**](exitpass.md) | **NONE** | Exit from the current pass file immediately, without performing rule matching (if any). |
 | [**fail()**](fail.md) | **NONE** | In CHECK Region, abort the rule that has just matched. Noop elsewhere. See succeed(). |

Help/markdown/callanalyzer.md

@@ -0,0 +1,91 @@
+[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md)
+
+# callanalyzer
+
+## Purpose
+
+Run another analyzer's passes on part of the current parse tree, using the current knowledge base.
+
+## Syntax
+
+```
+returnedBool = callanalyzer(pnode, concept, analyzerStr)
+```
+
+```
+returnedBool - type: bool (1,0)
+
+pnode - type: pnode
+
+concept - type: concept
+
+analyzerStr - type: str
+```
+
+## Returns
+
+1 if the analyzer ran, else 0. When it returns 0, the reason is written to the calling analyzer's `err.log`.
+
+## Remarks
+
+`callanalyzer` lets you build one analyzer out of several separately written analyzers, all working on the same parse tree and the same knowledge base. For example, an analyzer that finds addresses can hand each address node to an address analyzer, and an analyzer that finds sentences can hand each sentence node to a parser.
+
+**What the called analyzer works on.** The called analyzer has no text or parse tree of its own. Its passes run on the subtree under `pnode`: its rules match `pnode`'s children the way an analyzer's rules normally match the children of the root, and `pnroot()` returns `pnode`. It changes the caller's tree directly, so any nodes it builds, renames or gives variables are still there when the call returns. It also uses the caller's knowledge base, and [callconcept](callconcept.md) returns `concept`, the concept it was given for its results. Anything it adds under that concept is in the caller's knowledge base immediately.
+
+**What is loaded once.** The first call builds the called analyzer's passes and reads its `kb/user` dictionary (`.dict`) and `.kbb` files into the knowledge base. Its lazily loaded `*full.dict` and `*full.kbb` files are registered with the knowledge base rather than read. The analyzer then stays in memory, so later calls read nothing again. Loading is often most of an analyzer's running time, so calling the same analyzer on hundreds of nodes costs little more than the analysis itself. Its `.kb` files are not read: they are a complete saved knowledge base, and reading one into a knowledge base that already exists would add every attribute value a second time.
+
+**What runs on each call.** The called analyzer's passes run in order on the subtree, with these differences from running on a text of its own:
+
+- Tokenizer passes are skipped, because the tree already exists.
+- A `dicttok` or `dicttokz` pass does not tokenize. Instead it looks up every word under `pnode` in the dictionaries, including the lazily loaded `*full` files, which only load a word once it is seen. Each word gets its dictionary attributes (such as `pos`), and multi-word dictionary phrases are matched among `pnode`'s children.
+- Global variables (`G`) start empty on every call and are discarded afterwards, so the caller's global variables are neither visible nor changed.
+- `G("$apppath")` is the called analyzer's folder.
+- No per-pass `.tree` log files are written.
+
+Error messages from the called analyzer go to the caller's `err.log`, numbered by the called analyzer's own passes.
+
+**Writing an analyzer to be called.** Start its sequence with a `dicttok` pass. When it runs on its own, the pass tokenizes its input as usual; when it is called, the pass runs its dictionaries on the node it was given. Put its results under `callconcept()`, which also returns a concept when the analyzer runs on its own, so it can be developed and tested without a caller.
+
+**Finding the analyzer.** `analyzerStr` is the name of the analyzer's folder. The folder is looked for beside the calling analyzer's folder, which is where analyzers sit in a VisualText workspace. A relative path is taken from that same place, and an absolute path is used as given. If an analyzer with that name is already loaded, it is reused.
+
+**Calls that are refused.** An analyzer cannot call itself. It also cannot call an analyzer that is already running further up a chain of calls (for example, A calls B and B then calls A). In both cases `callanalyzer` returns 0.
+
+When the NLP++ debugger is attached, it does not stop inside a called analyzer. The whole call is stepped over as one statement of the caller.
+
+## Example
+
+The calling analyzer hands each `_sent` node to an analyzer named `colors`:
+
+```
+@CODE
+G("results") = makeconcept(findroot(),"results");
+@@CODE
+
+@POST
+L("sent") = makeconcept(G("results"),"sent");
+callanalyzer(N(1), L("sent"), "colors");
+
+@RULES
+_xNIL <- _sent @@
+```
+
+The `colors` analyzer sits in a folder beside the caller. Its `analyzer.seq` runs a `dicttok` pass and then this pass, which turns every word its dictionary marks as a color into a `_color` node and records the color:
+
+```
+@CHECK
+if (!N("color",1))
+	fail();
+
+@POST
+addstrval(callconcept(),"color",strtolower(N("$text",1)));
+single();
+
+@RULES
+_color <- _xALPHA @@
+```
+
+After each call, the color words in that sentence are `_color` nodes in the caller's parse tree, and the `sent` concept holds their colors.
+
+## See Also
+
+[callconcept](callconcept.md), [findana](findana.md), [makeconcept](makeconcept.md), [Special Functions](Table_of_Special_Functions.md)

Help/markdown/callconcept.md

@@ -0,0 +1,43 @@
+[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md)
+
+# callconcept
+
+## Purpose
+
+In an analyzer run by [callanalyzer](callanalyzer.md), get the concept it was given for its results.
+
+## Syntax
+
+```
+returnedConcept = callconcept()
+```
+
+```
+returnedConcept - type: concept
+```
+
+## Returns
+
+The concept passed to [callanalyzer](callanalyzer.md) by the analyzer that called this one.
+
+When the analyzer is not being called, it returns the concept named `callanalyzer` under the root of the knowledge base, which is created if it does not exist.
+
+## Remarks
+
+A called analyzer shares its caller's knowledge base, so `callconcept()` is the caller's concept itself, not a copy. Whatever the called analyzer adds under it is in place as soon as the call returns.
+
+When the analyzer runs on its own, from VisualText or the command line, `callconcept()` still returns a concept, so you can develop and test the analyzer and inspect its results in the knowledge base before another analyzer calls it.
+
+## Example
+
+```
+@POST
+addstrval(callconcept(),"color",strtolower(N("$text",1)));
+
+@RULES
+_xNIL <- _xALPHA @@
+```
+
+## See Also
+
+[callanalyzer](callanalyzer.md), [findroot](findroot.md), [Special Functions](Table_of_Special_Functions.md)

Help/markdown/findana.md

@@ -28,7 +28,7 @@ True if named analyzer is already loaded into memory.
 
 Analyzer is named by its project (i.e., folder) name, and is assumed to reside in the area defined by the $APPS environment variable.
 
-This is the first of a set of functions under development for loading an analyzer once and running it many times.  While such functionality exists in the programmer's API, it is under development for running within VisualText as well.
+This was the first of a set of functions for loading an analyzer once and running it many times. [callanalyzer](callanalyzer.md) now does that from NLP++: it loads the named analyzer the first time it is called, and from then on findana returns true for that analyzer.
 
 ## Example
 
@@ -38,4 +38,4 @@ L("ana present") = findana("TAIParse");
 
 ## See Also
 
-[system](system.md), [interactive](interactive.md), [Special Functions](Table_of_Special_Functions.md)
+[callanalyzer](callanalyzer.md), [system](system.md), [interactive](interactive.md), [Special Functions](Table_of_Special_Functions.md)

Help/markdown/index.md

@@ -429,6 +429,8 @@ to a topic.
         - [varinlist](varinlist.md)
       - [Special Functions](Table_of_Special_Functions.md)
         - [Table of Special Functions](Table_of_Special_Functions.md)
+        - [callanalyzer](callanalyzer.md)
+        - [callconcept](callconcept.md)
         - [debug](debug.md)
         - [exitpass](exitpass.md)
         - [fail](fail.md)