docs(help): document the 3.8.0 builtins, shadowing, and two fixes

Help/markdown/DECL.md

@@ -20,6 +20,18 @@ An example follows.
 
 This function call returns and outputs the number "6" to the file "output.txt".
 
+## Shadowing a built-in function
+
+If you define a function whose name is also that of a built-in NLP++ function, **your definition wins**. The analyzer still builds, and a warning is written to the build log:
+
+```
+[Warning: user function shadows built-in NLP++ function name='push'. The user function will be used.]
+```
+
+This used to be a hard error, which meant that adding a new built-in function could break any existing analyzer that happened to have defined a function of that name. Shadowing makes the language safe to extend: your analyzer keeps doing exactly what it did before, whatever gets added to the engine later.
+
+If you want the built-in behavior instead, remove or rename your own definition. Check the warning if a function does not behave the way its help page describes — a shadowing definition elsewhere in the analyzer is the usual reason.
+
 ## See Also
 
 [Declare Zone](Declare.md)

Help/markdown/New_in_Current_Version.md

@@ -20,6 +20,12 @@ The ANSI version shares the same source code with the Unicode version, but repli
 
 ## Miscellaneous
 
+(3.8.0) **User functions may now shadow built-in functions.** Defining a function in a [@DECL](DECL.md) region whose name is also a built-in used to be a hard error, so adding any new built-in broke every existing analyzer that had defined a function of that name. Now the user's definition wins, the analyzer builds, and a warning names the shadowed function. This is what lets [arrayreverse](arrayreverse.md) and [push](push.md) become built-ins without disturbing the hand-written versions that have long been in `visualText/spec/UtilFuncs.nlp`, and it means future built-ins can be added safely.
+
+(3.8.0) **String and array utilities.** [split](split.md) had no inverse, so building a delimited string meant a loop and a "first time through" flag: [strjoin](strjoin.md) now does it in one call. There was also no way to get the position of a substring — [strchr](strchr.md) takes a single character and returns a tail rather than an index, and [strcontains](strcontains.md) answers only yes or no — so [strindexof](strindexof.md) and [strlastindexof](strlastindexof.md) have been added. Every existing sort was knowledge-base or parse-tree oriented, leaving plain arrays with none, so there are now [arraysort](arraysort.md) (lexical or numeric, ascending or descending, stable), [arrayunique](arrayunique.md), [arrayreverse](arrayreverse.md) and [arrayslice](arrayslice.md). [push](push.md), previously listed as UNIMPLEMENTED, now works. All of these leave the source array alone and return a new one. **FIXED:** [str](str.md) converted signed values with an unsigned format, so every negative number printed as roughly 1.8e19 — `str(-1)` gave "18446744073709551615", which also made the documented -1 returns of [filesize](filesize.md) and the new index functions unreadable. The same fault was in the compiled-analyzer path and in the code generator, so compiled analyzers emitted the wrong numeric literals too.
+
+(3.8.0) **Reading files, listing directories, and real regular expressions.** NLP++ could write files but had no way to read one, so a pass could not load a config file, a lookup table, or its own earlier output. New: [readfile](readfile.md) reads a whole text file into a string, [readlines](readlines.md) reads one as an array of lines, and [dirlist](dirlist.md) lists a directory as full forward-slash paths, sorted, with an optional `*`/`?` glob. Also new are three true regular-expression functions using ECMAScript syntax — [rematch](rematch.md), [refind](refind.md) (with capture groups) and [resubst](resubst.md) (replaces every match, with `$1` backreferences) — which unlike the older [regexp](regexp.md) support character classes, anchors, quantifiers, alternation and groups, and can be called from anywhere rather than only from a `@PRE` region. **FIXED:** [regexp](regexp.md) and [regexpi](regexpi.md) had been returning false for every input since 2.x — the "no literal yet" marker in the pattern compiler was changed from a null pointer to an empty string, so the pattern's literal never bound and nothing ever matched. [regexpi](regexpi.md) additionally lost its case-insensitivity after the first pattern element. Both are fixed, and the help now records that these patterns must match the whole string.
+
 (2.7.2.0) Rework of crawling functions eg [urltofile](urltofile.md), to handle HTTPS as well as HTTP. (2.7.0.9) New function [pnmove](pnmove.md) to enable moving nodes within the parse tree. (2.7.0.0) Overhauls, optimizations, and bug fixes focusing on the recursive NLP++ pass algorithm. (2.6.0.3) New function [dejunk](dejunk.md) to clean out non-ASCII and extended ASCII characters (sometimes replacing with a question mark character). (2.6.0.0) Important fix to NLP++ pattern matcher re feature-based match with wildcards.  **dicttokz** pass, a variant of dicttok that removes whitespace from parse tree at the outset, puts in attributes NOSP (no space before), NL (new line before).  Adds capitalization attributes to parse tree as well.  (This will be an ongoing development in 2.6.x.x). (2.5.0.0) New [dicttok black-box tokenization pass](VisualText_Basics/About_the_analyzer_sequence.md) that uses KB from TAIParse to put part-of-speech tags on parse tree automatically. See [DictTok_Sample](DictTok_Sample.md) analyzer. (2.4.x.x) Maintenance releases, various fixes in feature-based matching, compiled analyzers, etc. (2.3.2.0) In text windows ctrl-A now is Select All. ctrl-G is Find Again. (2.3.1.9) Maintenance release. Fixes to error reporting and recovery, compiled vs interpreted analyzers.. [group](group.md) now returns the new "reduce" node that it creates, an important convenience.  [group](group.md) also reports when it is including the [lookahead](lookahead.md) node -- an error.  [listadd](listadd.md) no longer treats the list node as the "reduce" node. (2.3.1.8) [xmlstr](xmlstr.md) now converts characters to { where 123 is a character number, rather than to strings such as ¼. (2.3.1.6) Added special rule elements **[_xLET](_xLET.md)** and **[_xCAPLET](_xCAPLET.md)** to conveniently match single-character words. (2.3.1.0) VISUAL BASIC .NET API for calling VisualText analyzers.  See the sample program called [VB_to_VT](VB_to_VT.md), which also lists the API functions. (2.3.0.0) Compiling analyzers now generates separate code files for each pass, enabling large analyzers to be compiled into C++.  For example, the latest versions of TAIParse should now compile.  Note that a Microsoft Visual Studio project may now need to have files added to it, within an analyzer's RUN folder. (2.2.2.13) New PRE actions called [regexp](regexp.md) for matching using regular expressions, with [regexpi](regexpi.md) case insensitive version. (2.2.2.11) New function for sorting a knowledge base concept's hierarchy and children: [sorthier](sorthier.md) and [sortchilds](sortchilds.md). (2.2.2.8) Compiling analyzers once again generates PRETTY, or human-readable, code.  We tried very long lines to circumvent the 65K file line limit of some compilers, but too-long lines are an issue as well.  In subsequent versions, we'll be splitting files as needed.  For now, huge analyzers may need their generated C++ code (in the RUN folder) to be tweaked manually in order to compile. (2.2.2.5) EXITPASS, SUCCEED, and FAIL.  These functions have been restored in that they are allowed in practically any context without triggering error messages.  The compiled analyzer version uses C++ exception handling mechanisms in order to instantly handle these in deeply embedded contexts (function within a function, etc.)  Parse tree display ("yellow box") now places node variables in two columns for convenient viewing of long variable lists. Parse tree nodes now point back to rules that renamed them via [xrename](xrename.md) and [pnrename](pnrename.md).  Compile-during-analyzer-load popup of MS DOS window has been removed.  Interpreted user functions improperly returned a non-zero value in the absence of a return statement. (2.2.2.4) Incorrect VisualText TAB SIZE in Vista has been fixed. SAVE KB and COMPILE KB default popup answer is now NO.  (Unless you are explicitly trying to build a KB, then you don't want to save the KB.)  Parse tree for lone underscore character is now a terminal punctuation char.  Listadd in recursive passes when building a list right-to-left now fixed (compiled version hang fixed). [excise](excise.md) forbids deleting a lookahead node. (2.2.2.2) The functions [exitpass](exitpass.md), [succeed](succeed.md), [fail](fail.md), can appear only at the top level of code regions in compiled analyzers (**FIXED**. See 2.2.2.5 above).  Fixes to NLP++ syntax relating to negation.  A fix to float conversion.  Further fixes to optimized recursive pass algorithm. (2.2.2.1) Further fixes to align compiled and interpreted analyzers (fixes to zeroed items in relational operators and others).  Fix of an annoying VisualText crash when brought into focus from the background.  A substantial fix to the optimized recursive NLP++ pass algorithm in both the interpreted and compiled analyzer frameworks. (2.2.1.1) New function [pninsert](pninsert.md) for programmatically inserting nonliteral nodes into the parse tree. (2.2.1.0) NEW. A parse tree view mode that reflects items removed from the parse tree, such as HTML tags.  See Tree Text Mode in the View Menu. Fix to some NLP++ compiled functions to enable zero (0) as a null string.  E.g., strsubst("ab,c",",",0); will replace commas by empty string in a compiled analyzer. Note that this behavior already works as expected in interpreted analyzers, e.g., those that run inside VisualText. (2.2.0.0) A substantially cleaned up VisualText release (see 2.1.0.9 notes below).  Tree Text view does not leave everything expanded on reentering VisualText, but remembers what folders have been closed up in the view. The RUN NEXT button behaves better. (2.1.0.9) A CLEANUP RELEASE.  ALIGNED INTERPRETED AND COMPILED ANALYZERS, FIXED MISC BUGS AND LEAKS.  Fixed a substantial bug that hobbled compiled analyzers.  Memory leaks plugged in eltnode and lasteltnode.  The PROG project or sample driver for analyzers now sets the BATCHSTART flag at the start of a run, to enable a compiled analyzer to test the NLP++ batchstart() function/flag. (So that an analyzer can perform initializations when it runs the first file of a batch of files.) DEFAULT configuration of VisualText is now full screen, kb safe mode off, toolbars in a standard configuration.  TIP OF THE DAY offers real suggestions. Fixed bugs in the PROG driver of each analyzer, for the analysis of multiple files and folders. (2.1.0.8) Fix to functions like resolveurl, strchr, strrchr. Fix to parse tree nodes with large string-valued variables.  Fixes to Tab Window and Find Window show/hide bugs.  Change to VisualText naming in the Windows registry.  Fixed a memory leak in rmattrs. (2.1.0.6) Fixes to file and directory handling.  (See the PROG project of a newly created analyzer project.) (2.1.0.3) Misc fixes, including parse tree display and return of arrays in NLP++ user-defined functions. (2.1.0.0) In order to support huge knowledge bases, a VisualTextQ version is provided that utilizes the QDBM database management freeware under the hood.  For backward compatibility, one can treat the knowledge base built in this way as the "compiled" knowledge base.  Loading the interpreted knowledge base will read kb command files into a QDBM file system.  "Compiling" the kb will save the latest version of the QDBM files -- not compiling the changes will discard the latest modifications to the kb. Other notes: Removed the Windows MS-DOS Command Prompt popups from most system calls. (2.0.2.9) The [levenshtein](levenshtein.md) edit-distance computation is provided. (2.0.2.8) Optimizations to string manipulation using L (local) variables.  These are not interned, as with S,X,G,N variables.  Locally assigned strings are completely freed when a scope (function, @CODE, @CHECK) is exited, while globally assigned strings are interned and persist throughout a text analysis.  Analyzers can be optimized by using local L variables wherever feasible. (2.0.2.7) [fltval](fltval.md) knowledge base value fetch. (2.0.2.5) Linux upgrade. (2.0.2.4) New variables **[G**("$passnum")](%24passnum.md) and **[G**("$rulenum")](%24rulenum.md) e.g. to track rule context and performance. (2.0.2.3) New NLP++ function [eltnode](eltnode.md), analogous to lasteltnode. (2.0.2.2) Recovery from very long tokens. (2.0.2.1) Refix of an infrequent crash in using $treetext. (2.0.2.0) Optimization to recursive passes. (2.0.1.1) Fixes to the **Find in Files** popup menu.  Bugs were introduced in VisualText 2.0: (a) missing patterns that start at the beginning of a line, (b) incorrect display of capitalization in found patterns.
 
 # NEW IN CURRENT VERSION (2.0 NON-UNICODE BETA)

Help/markdown/Table_of_Array_Functions.md

@@ -10,7 +10,26 @@ Arrays can also be assigned and manipulated with parse tree functions and kb fun
 
 | **FUNCTION NAME** | **RETURNS** | **DESCRIPTION** |
 | --- | --- | --- |
-| **push(var)** | ARRAY | Push a value to the front of a variable's value(s). Allows mixed types (string, num, flt, concept...), allows redundant values, does not check uniqueness. NOTE: This will alter the array indexing, pushed item will be the 0th element of array index. **UNIMPLEMENTED** |
-|   |   |   |
+| [**arraylength(array)**](arraylength.md) | **INT** | Number of elements in an array. |
+| [**arrayreverse(array)**](arrayreverse.md) | **ARRAY** | The same elements, last to first. |
+| [**arrayslice(array, start_num [, end_num])**](arrayslice.md) | **ARRAY** | Elements from start up to but not including end. Zero based; omit end to run to the end. |
+| [**arraysort(array [, numeric_bool [, descending_bool]])**](arraysort.md) | **ARRAY** | Sort an array. Lexical and ascending by default; stable. |
+| [**arrayunique(array)**](arrayunique.md) | **ARRAY** | Keep the first of each distinct element, in the original order. |
+| [**push(value, array)**](push.md) | **ARRAY** | Push a value to the front of an array. Allows mixed types (string, num, flt, concept...), allows redundant values, does not check uniqueness. NOTE: This alters the array indexing; the pushed item becomes the 0th element. |
+| [**split(str, char_str)**](split.md) | **ARRAY** | Split a string into an array on a separator character. |
+| [**strjoin(array, separator_str)**](strjoin.md) | **STR** | Join an array's elements into one string. The inverse of split. |
+
+These functions never modify the array given to them; each returns a new array.
+
+### Testing for an empty result
+
+The functions that can find nothing — `arrayslice`, `arrayunique`, [refind](refind.md), [dirlist](dirlist.md), [readlines](readlines.md) — return an **empty array**. Note that assigning an empty array to a variable collapses it back to no value, and `arraylength` reports **1** for a variable holding no value. So guard a loop by testing the first element, not the count:
+
+```
+L("hits") = arrayslice(L("rows"),1);
+if (strlength(L("hits")[0])) {
+    # ... safe to loop over L("hits") ...
+}
+```
 
 ## See Also

Help/markdown/Table_of_File_Functions.md

@@ -10,11 +10,14 @@ The return types include: **STR** (string), **INT** (integer), and **BOOL** (Boo
 | --- | --- | --- |
 | [**deletefile(path_str)**](deletefile.md) | **BOOL(1,0)** | Delete a regular file. Never removes directories. |
 | [**direxists(path_str)**](direxists.md) | **BOOL(1,0)** | True if the path is an existing directory. |
+| [**dirlist(dir_str [, pattern_str])**](dirlist.md) | **STR ARRAY** | Full paths of the entries of a directory, sorted, optionally filtered by a `*`/`?` glob. |
 | [**fileexists(path_str)**](fileexists.md) | **BOOL(1,0)** | True if the path is an existing regular file. |
 | [**filesize(path_str)**](filesize.md) | **INT** | Size of a regular file in bytes, or -1 if not a readable file. |
 | [**mkdir(dir_str)**](mkdir.md) | **BOOL(1,0)** | Create a directory. Operating system-dependent. |
+| [**readfile(path_str)**](readfile.md) | **STR** | Read a whole text file into a string. |
+| [**readlines(path_str)**](readlines.md) | **STR ARRAY** | Read a text file as an array of its lines, terminators stripped. |
 
-For reading and writing file contents, see the file-stream functions [openfile](openfile.md), [closefile](closefile.md), and [fileout](fileout.md) in the [Formatting and I/O Functions](Table_of_Formatting_and_I_O_Functions.md) table, and [urltofile](urltofile.md) in the [Web Functions](Table_of_Web_Functions.md) table.
+For writing file contents, see the file-stream functions [openfile](openfile.md), [closefile](closefile.md), and [fileout](fileout.md) in the [Formatting and I/O Functions](Table_of_Formatting_and_I_O_Functions.md) table, and [urltofile](urltofile.md) in the [Web Functions](Table_of_Web_Functions.md) table.
 
 ## See Also
 

Help/markdown/Table_of_String_Functions.md

@@ -13,6 +13,9 @@ The return types include:  **STR** (string), **STR ARRAY** (string array), **IN
 | [**flt(str)**](flt.md) | **FLOAT** | Convert string or int to float, if possible. |
 | [**levenshtein(str1,str2)**](levenshtein.md) | **INT** | Levenshtein edit-distance between two strings.** New in 2.0.2.9** |
 | [**num(str)**](num.md) | **INT** | Convert string to number, if possible. (Also accepts num arg). |
+| [**refind(str, pattern_str [, flags_str])**](refind.md) | **STR ARRAY** | Regular expression match: element 0 is the whole match, 1..n the capture groups. |
+| [**rematch(str, pattern_str [, flags_str])**](rematch.md) | **BOOL** | True if the regular expression matches the string. |
+| [**resubst(str, pattern_str, replacement_str [, flags_str])**](resubst.md) | **STR** | Replace every regular expression match, with `$1` backreferences. |
 | [**split(str, char_str)**](split.md) | **STR ARRAY** | Split a string using the given char as separator. |
 | [**stem(str)**](stem.md) | **STR** | Compute stem of noun and verb. Return lowercase string. |
 | [**strchar(str, num)**](strchar.md) | **STR** | Index to numth char of string |
@@ -23,6 +26,9 @@ The return types include:  **STR** (string), **STR ARRAY** (string array), **IN
 | [**strcontainsnocase(str1,str2)**](strcontainsnocase.md) | **BOOL** | Check if str1 is contained in str2, ignoring letter case. |
 | [**strendswith(str, suffix_str)**](strendswith.md) | **BOOL** | Does given str ends with given suffix_str |
 | [**strhaspunct(str)**](strhaspunct.md) | **BOOL** | Check if str contains a punctuation character. |
+| [**strindexof(str, substr)**](strindexof.md) | **INT** | Zero-based position of the first occurrence of substr, or -1. |
+| [**strlastindexof(str, substr)**](strlastindexof.md) | **INT** | Zero-based position of the last occurrence of substr, or -1. |
+| [**strjoin(array, separator_str)**](strjoin.md) | **STR** | Join an array's elements into one string. The inverse of split. |
 | [**strstartswith(str, suffix_str)**](strstartswith.md) | **BOOL** | Does given str starts with given suffix_str |
 | [**strequal(str1,str2)**](strequal.md) | **BOOL** | Check if str1 is identical to str2. |
 | [**strisalpha(str)**](strisalpha.md) | **BOOL** | Check if str is all alphabetic chars. |

Help/markdown/arrayreverse.md

@@ -0,0 +1,44 @@
+[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md)
+
+# arrayreverse
+
+## Purpose
+
+Reverse the order of an array's elements.
+
+## Syntax
+
+```
+returnedArray = arrayreverse(array)
+```
+
+```
+returnedArray - type: array
+
+array - type: array
+```
+
+## Returns
+
+A new array holding the same elements last to first. Returns an empty array if the argument is not an array.
+
+## Remarks
+
+The source array is not modified; a new array comes back.
+
+Combine with [arraysort](arraysort.md) only when you need something other than a plain descending sort — for a descending sort, pass 1 as arraysort's third argument instead of reversing afterwards.
+
+## Example
+
+```
+@CODE
+
+L("abc") = split("a b c"," ");
+"out.txt" << strjoin(arrayreverse(L("abc")),",") << "\n";   # c,b,a
+
+@@CODE
+```
+
+## See Also
+
+[arraysort](arraysort.md), [arrayunique](arrayunique.md), [arrayslice](arrayslice.md), [Array Functions](Table_of_Array_Functions.md)