Measure: python tools/toolclass.py tools, which derives the population from
os.listdir and refuses to print a table unless every script on disk is in
exactly one class and every script in a class is on disk.
$ python tools/toolclass.py tools
13 tools, 5 classes, each tool in exactly one class.
## the installation, as an object
instcensus.py 4,769 files: path, size, mtime, optional SHA-1, extension
recorded both as spelt and case-folded
dupes.py groups the census by SHA-1: what the object stores more than
once, and where a repeated name is not a repeated content
## the video
bk2.py all 1,035 Bink 2 headers, 44 bytes each, with the
declared-size field checked against `stat` on every one
## the text, at both levels
sheet.py the five SpreadsheetML workbooks: DocumentProperties,
declared and counted geometry, keys, duplicate keys
closure.py the two-directional closure test between 1,321 CaptionData
files and the workbooks, as spelt and case-folded
bridge.py video stems against timing stems, four ways
## the binaries
pe.py PE header, sections, resource tree, VS_VERSIONINFO node walk
clrmeta.py ECMA-335 metadata: the 28 tables, #Strings and #US,
no decompiler
paths.py build-machine and source-tree paths left in shipped files
unity.py the 47 Unity serialized headers -- and where this document stops
## the document itself
leftovers.py what should not be here, in bytes, against two denominators
toolclass.py this file
checkscore.py counts the verdicts in the scoring chapter from its own tables
1,693 lines of Python, no dependency outside the standard library, thirteen files. Every tool takes the installation directory as an argument; no absolute path to the object appears in any default.
The rule that has held for three sessions: running an old tool on a new object
measures the tool as much as the object. It is how the alignment bug that had
been in exepack.py since 1990 was found — by an object its own subject could
not have revealed. So all four portable tools from the two previous
repositories were run on this object unmodified, before any of them was
touched.
python ../pc-megaman-doc/tools/census.py "<G>"
python ../pc-megaman-doc/tools/strdump.py "<G>/.../Assembly-CSharp.dll" --min 12
python ../pc-megaman3-doc/tools/toolclass.py tools
python ../pc-megaman3-doc/tools/checkscore.py docs/00-predictions.md
Output in notes/old-tools.txt. Four ran. Two of the four told the truth
about this object; one told the truth and it had stopped meaning anything; and
one under-reported by design and said so.
It works. Same SHA-1 for TeslaEffect.exe as this session's own census
(4b9b9aa5…), same sizes, correct per-extension totals.
But it ends with this:
files with an odd seconds field: 2 of 2 ['TeslaEffect.exe', 'bink2w32.dll']
That check exists because FAT stores modification time on a two-second grid, so every file written by DOS has an even seconds field, and an odd one is evidence that something rewrote the file. On the 1990 and 1992 objects it fired on nothing and that was the finding. Here it fires on almost everything, and it is right, and it means nothing — because 04 established that the entire timestamp layer is a rewrite. A check designed to detect one rewritten file cannot say anything useful when every file has been rewritten.
The tool is not wrong and does not need changing. What it needs is a reader who knows that a check can outlive the condition that made it informative. That is worth more than a bug.
strdump.py exists because a string scanner that silently returns fewer strings
than a file contains is the specific failure that cost the sixteenth session
three findings. So it prints two counts derived two different ways and shows
the gap.
$ python ../pc-megaman-doc/tools/strdump.py Assembly-CSharp.dll --min 12 --quiet
runs of length >= 12 : 6311
bytes inside those runs : 119101
printable bytes in the region : 576056
accounted for by the run scan : 20.7% (the rest are runs shorter than 12)
20.7 %. Four fifths of the printable bytes in this file are invisible to it,
and the reason is structural: the #US heap of a .NET assembly is UTF-16LE,
so every other byte is \0 and a byte-oriented run scanner sees runs of length
one. A tool built for 8-bit DOS strings cannot read a 2014 managed assembly, and
tools/clrmeta.py — reading the same file through its metadata — finds 5,002
literals and 146,706 characters that strdump.py structurally cannot see.
This is the result the rule exists to produce. The tool did not crash, did not lie, and told the reader the size of what it was missing in its own output. Twenty-four years of encoding history, measured as a percentage, by a program written for a different problem.
Ran on this repository unmodified and did exactly what it is for:
!! named in a class but not on disk (12): zipcensus.py, clocks.py, exepack.py, ...
!! on disk but in no class (9): bk2.py, bridge.py, closure.py, clrmeta.py, ...
!! totals disagree: 13 classified, 10 on disk
refusing to print the table until every tool is classified exactly once.
It refused, correctly, because it was still carrying 1992's class list. Then it
was copied here and only the CLASSES table was edited — no code change, in
the third repository in a row. Two repositories back, in pc-tubularworlds-doc,
this file held a hard-coded list of 167 script names and would not run anywhere
else. Portability is a property, and this one has now been measured three times.
Ran on docs/00-predictions.md and reported clauses 0, which is correct: a
predictions file has clauses but no verdicts, and the tool requires an id, a
verdict and a tag before it counts a row. It is used for real in
20.
The previous session had to widen its id pattern from B to B\d* because that
session's predictions carried two named inferences and the old pattern dropped
the second. This session also has two, B and B2, and the widened pattern
counts both — verified by running it and comparing against a hand count.
instcensus.py — because find + sha1sum would not have recorded the
extension as spelt, and 1,150 .XML against 176 .xml is the single most
consequential naming fact in the object.
dupes.py — because 4,570 distinct hashes for 4,769 files is a headline and
"which of the repeated names are also repeated contents" is the actual finding.
bk2.py — because 1,035 headers at 44 bytes describe 14.48 GB for 45 KB of
reads, and because the size field makes every one of those reads self-checking.
94 lines.
sheet.py — because the script of this game is an Excel workbook and
xml.etree plus ss:Index handling is all it takes to read one. It reports the
declared geometry beside the counted geometry, and the gap between them is
what revealed the 1,404 blank rows.
closure.py — the central tool. Reads the 1,321 timing files with a regular
expression rather than a parser, on purpose: four of them are not well-formed
XML and one is called .txt. Reports as-spelt and case-folded side by side and
never quietly folds.
bridge.py — strict and normalised stem matching, so the cost of the naming
mess is a number (26 joins) rather than an impression.
pe.py — 202 lines to reach two facts: the linker timestamp and the version
resource. The VS_VERSIONINFO reader walks the length-prefixed node tree
properly; the first version guessed key/value pairs from the UTF-16 runs and
mispaired every one of them. See 19.
clrmeta.py — 364 lines, no decompiler, no dependency. Walks all 28 present
metadata tables in order because index widths depend on row counts, and prints
!! stopped at unimplemented table 0x.. rather than guessing an offset. On this
file it never prints that line.
paths.py — exists because the first attempt at this measurement was a
shell heredoc, the heredoc ate one backslash from the character class [\\/],
the class collapsed to [/], and the search reported zero source-tree paths
in files that visibly contain them. A tool in a file has a backslash count that
can be checked. Its three guards against false positives are each documented
with the false positive that produced them — the first version matched every
http:// in the object as drive p:.
unity.py — 94 lines that read 20 bytes per file and then stop, with
the docstring stating what going further would require.
leftovers.py — refuses to add its classes without printing the overlap,
and prints every figure against two denominators.
Three of the eleven tools above exist because a previous version of the same
measurement was wrong: paths.py because a heredoc ate a backslash,
pe.py's version reader because run-splitting mispaired every key, and
closure.py's regex because a strict parser refuses four of the files it has to
read. In each case the fix is in the docstring next to the code, which is where
a format description belongs and where a mistake is cheapest to find again.
notes/ holds these tools' output verbatim, with one mechanical
substitution. Three of the fourteen files — pe.txt, old-tools.txt and
clr-assembly-csharp.txt — print the path of the file they read, and that path
is an absolute location on a drive this repository does not own. In those three
the installation directory has been replaced by <G>, the placeholder the
chapters use. Nothing else was edited: every number, count and string is as the
tool emitted it.
$ grep -ril "SteamLibrary" notes/ docs/ README.md tools/
(no output)
The build paths quoted in 10 —
C:/buildslave/fmod/…, d:\users\nbtester\… — are not subject to that
substitution and never will be. They are absolute paths found inside a shipped
binary, which makes them material rather than contamination. That distinction is
the whole rule: a path this session's own tooling would have written is removed;
a path the object contains is a finding.