Maintain bounded exact-key indexes through construction and coding

Documentation/Architecture/Release1.1/BenchmarkReview.md

@@ -0,0 +1,56 @@
+# Initial exact-key lookup review
+
+This increment filters the tail of a linear lookup using existing immutable key
+fingerprints. Objects below 32 members retain the scalar loop. Wide objects compare
+the first eight members scalarly, then hash the query once under the active meter.
+Every candidate hit still confirms exact UTF-8 identity. No object representation,
+publication path, duplicate detector, capacity weight or public API changes here.
+The array-backed AVL and controlled hash prototypes remain experiment subjects.
+
+The matched production-stage comparison used MacBookPro18,2, Apple M1 Max, 32 GiB,
+macOS 27.0, Xcode 27.2 / SDK 27.2 and Swift 6.4 in language mode 6, release `-O`.
+`Tools/Release1_1Probes/LookupBenchmark.swift` covers ten widths, three key shapes,
+first/middle/last/missing queries and the first filtered member. Each run has three
+warmups and nine measured samples, twenty lookups with result checks per sample:
+129 query cells and 1161 recorded samples. The balanced/hash prototype measurements
+are a separate corpus; their timing is not presented as production improvement.
+
+Two sequential matched production pairs were retained, together with an earlier
+candidate measurement. The table uses the repeated pair. Times are medians in
+milliseconds for twenty lookups, including identical result checks; work counts
+are owned package units, excluding the benchmark's result-check work.
+
+| Shape / width / query | Scalar ms | Filter ms | Change | Scalar / filter work |
+| --- | ---: | ---: | ---: | ---: |
+| ASCII / 8 / last | 0.018041 | 0.018083 | +0.23% | 1620 / 1620 |
+| ASCII / 16 / middle | 0.020166 | 0.020167 | approximately 0% | 1820 / 1820 |
+| ASCII / 32 / first filtered | 0.020209 | 0.023750 | +17.52% | 1820 / 2020 |
+| ASCII / 32 / miss | 0.017584 | 0.019375 | +10.19% | 1280 / 1400 |
+| Prefix / 32 / first filtered | 0.230042 | 0.252583 | +9.80% | 24320 / 27020 |
+| ASCII / 1024 / last | 1.657416 | 0.525750 | -68.28% | 144420 / 42140 |
+| Prefix / 1024 / last | 24.807875 | 0.848333 | -96.58% | 2704420 / 67140 |
+| ASCII / 10000 / last | 16.837083 | 5.013375 | -70.22% | 1422220 / 401180 |
+| Prefix / 10000 / last | 248.313000 | 6.156417 | -97.52% | 26422220 / 426180 |
+
+The unchanged 10% review trigger applies. Accept the observed boundary cost for
+this focused draft increment: query hashing adds ten work units to the first
+filtered ASCII hit and six to the short missing key, approximately 0.18 and 0.09
+microseconds per lookup in the repeated pair. Later wide hits materially reduce
+owned comparisons. This is an explicit tradeoff, not an assertion that every
+lookup improves. Keep the scalar path and qualify adaptive metadata separately.
+
+Timings are host observations. An initial width-16 ASCII timing outlier disappeared
+in the unchanged-code repeat, and wide scalar timings varied between runs; retain
+both raw sets and variance instead of attributing those changes to the algorithm.
+Empty-loop timings are at timer/call-overhead scale and establish no throughput
+claim. This stage retains no owned buffer and reports zero logical retained bytes;
+it measures neither process RSS nor allocator counts. Construction, whole-root
+admission, native keyed encoding, thermal/energy effects and mobile crossovers
+remain unqualified R1/R5 work. Algorithm costs may change resource failure points;
+all finite profiles and counter version/weights remain unchanged.
+
+The historical stream median 1.0749626803 seconds versus 0.968493986 seconds
+(+10.9932%) retains its existing accepted disposition and 10% trigger. This lookup
+stage gives no stream attribution or fresh full-workload release qualification.
+Raw samples, source/input hashes, command receipts and medians/variance are retained
+in the milestone return bundle, with compiler, SDK, CI and fixture evidence.

Documentation/Architecture/Release1.1/Decisions.md

@@ -96,3 +96,21 @@ bundle cardinalities, three targeted Thread Sanitizer cases, 49 exact cap probes
 independent corpus counts and exclusions remain intact. R0's filtered run requires
 exactly seven cases. New platform declarations must later update the exact
 macOS-only manifest expectation in the same focused change.
+
+## First production increment
+
+Select only the narrow cached-fingerprint filter described in
+[BenchmarkReview](BenchmarkReview.md). The below-32-member scalar path and first
+eight scalar probes remain available. A query hash, each member probe, fingerprint
+comparison and exact candidate comparison consume the same noncopyable meter;
+there is no new table, cached mutable state or owned index allocation. The two
+additional behavior cases exercise public presence/Pointer/equality, insertion
+order, forced collision confirmation in the production comparator, inclusive and
+cumulative work exhaustion and cancellation during actual lookup work.
+
+The current 10% performance review trigger is unchanged; accept the measured
+first-filtered-hit and short-miss cost for this limited draft change. Full adaptive
+index publication/construction and keyed-builder/mobile measurement remain R1/R5,
+not completed by these tests. CI now expects JSONTests 63 and total 131 functions;
+the seven-case R0 subset, baseline parameterized invocations and all other exact
+cardinality/49-cap checks remain unchanged.

Documentation/Architecture/Release1.1/Index.md

@@ -0,0 +1,48 @@
+# Exact-key construction and publication
+
+Ordered members remain the public collection and writer authority. Each published
+object also owns immutable exact-key metadata. The index stores decoded JSONKey
+identity and insertion positions, never canonically equating String dictionary keys.
+
+The R0 release-native comparisons favored controlled open addressing for wide
+objects. The current candidate uses a linear index below 32 keys, then a power-of-two
+table with occupancy at most one half. Lookups keep the scalar path for small
+objects and the first eight members. A wide lookup hashes its query once, probes
+the table and confirms exact UTF-8 on every candidate hit. Collision scans are
+finite and metered; this is no adversarial constant-time guarantee.
+
+Each metadata capacity slot retains the existing Version 1 `indexSlot` weight of
+32 logical bytes. Unused capacity is included. New capacity is admitted while old
+capacity remains live; copying, initialization, rehashing and exact probes consume
+the same meter. Rejected growth releases its replacement reservation and leaves
+prior metadata valid. Logical capacities do not assert allocator or RSS ceilings.
+
+The shared grammar maintains its own duplicate index at decoded key recognition.
+A duplicate still fails at the second encoded key's start before its colon/value.
+Materialized objects maintain separate metadata as keys arrive. Checked
+construction and the accumulator maintain their index before publication; consuming
+nonthrowing `finish` transfers it without search, allocation or a fresh budget.
+Editing, native materialization and schema inspection/output build metered metadata
+before returning a value. Empty Merge Patch targets use an explicit empty index.
+
+Grammar duplicate capacity and DOM metadata are distinct live owners. Full owned
+graph admission includes the published index's actual logical capacity and member
+slots separately. The metric and weight meanings stay at Version 1; capacity/work
+counts legitimately differ with the algorithm, and defaults are not raised.
+
+Native keyed encoding now keeps one invocation-owned mutable storage object and
+maintained exact index. Updating a field no longer extracts and rewrites the entire
+member array. Member growth admits old/new capacities together. Materialization
+charges key traversal and builds immutable output metadata under the same meter.
+This source observation alone makes no claim of one allocation per field.
+
+`IndexPublicationTests` covers all JSON publication routes, scalar-distinct keys,
+forced collisions, failed rehash, duplicate timing and cancellation. The separate
+release construction benchmark measures checked, parser, accumulator and native
+keyed paths over 120 cells, using the matched finite offline profile. It records
+resource refusals separately, and allocator snapshots separately from timing and
+logical retention. A successful SDK build does not establish a mobile crossover.
+
+Full R1 closure requires the actual final native/safety/consumer measurements and
+review receipts. The test-only AVL/scalar alternatives remain qualification
+references; no table, platform or throughput result is inferred from this prose.

Documentation/Architecture/Release1.1/IndexMeasurements.md

@@ -0,0 +1,93 @@
+# Production index stage measurements
+
+The matched predecessor core is `5d01f88` (mobile branch `bf0ff41` changes only
+resource packaging/platform documentation). The metadata/keyed-storage core is
+`bf37941`. Swift 6.4, SDK 27.2, macOS 27.0, M1 Max/32 GiB, release optimization,
+finite offline profile, three warmups and nine measured samples apply to both.
+Exact benchmark source and static-library digests accompany the private native
+receipts. Documentation/test-only successors reuse these bytes explicitly.
+
+Construction measures 120 cells: four paths, ten widths 0–10000, three key shapes.
+The predecessor has 1044 successful samples and four explicit long-prefix
+width-10000 work refusals. The candidate has 1080 successful samples. Never use a
+refusal's elapsed time as successful throughput. Accumulator timing is available;
+its public facade has no accounting report, so its work/retention is unavailable.
+
+| Stage / ASCII width | Before median ms | After median ms | Change | Before / after logical peak bytes |
+| --- | ---: | ---: | ---: | ---: |
+| Checked 8 | 0.0060 | 0.0039 | -35.9% | 808 / 808 |
+| Checked 32 | 0.0510 | 0.0155 | -69.5% | 3136 / 5096 |
+| Checked 1024 | 39.5464 | 0.4029 | -99.0% | 99360 / 132128 |
+| Parse 1024 | 39.8769 | 1.1994 | -97.0% | 132160 / 197696 |
+| Native keyed 1024 | 79.3653 | 1.4056 | -98.2% | 238624 / 336928 |
+| Native keyed 10000 | 6930.3080 | 15.7802 | -99.8% | 2330032 / 4464688 |
+
+Lookup measures 129 query cells with 1161 samples per side, 20 lookups/sample.
+The final serialized pair, after the overlapping compiler job completed, gives:
+
+| Query | Before / after median ms per 20 lookups | Change | Before / after work |
+| --- | ---: | ---: | ---: |
+| ASCII 8 last | 0.018042 / 0.018791 | +4.15% | 1620 / 1620 |
+| ASCII 32 first filtered | 0.023833 / 0.022792 | -4.37% | 2020 / 2020 |
+| ASCII 32 miss | 0.019375 / 0.007542 | -61.07% | 1400 / 580 |
+| ASCII 1024 last | 0.526334 / 0.018541 | -96.48% | 42140 / 1540 |
+| Prefix 10000 last | 6.195458 / 0.247250 | -96.01% | 426180 / 26580 |
+
+Retain and review every cell above the unchanged 10% trigger, including tiny
+measurements near clock/call overhead. The disposition selects this candidate for
+wide construction/lookup benefit with explicit additional spare-capacity retention.
+It does not claim an improvement for every tiny case or a mobile crossover.
+Construction/root admission/ordered rebuilding remain separate from lookup.
+
+Nine separately instrumented native-keyed cells record libmalloc live blocks/bytes
+at selected field boundaries. Per-sample allocator observations and cumulative
+`ru_maxrss` are process-wide; they are not one allocation per field or isolated
+workload allocation counts. Logical weights remain Version 1 and are not RSS.
+
+First lookup samples were lost to a local receipt/output filename collision and
+were rerun with distinct sample paths. An earlier retained pair overlapped compiler
+work; the final serialized pair is used above. All actual failure/command records
+remain evidence. These stage results precede the reader's shared-meter and ISO
+correction; affected integrated performance must be refreshed on the final source.
+The historical approximately 11% streaming disposition remains separate.
+
+## Complete trigger review
+
+All 11 construction and 16 lookup cells above 10% are retained below.
+Construction absolute increases are at most 1.541 microseconds per operation;
+lookup increases are at most 1.625 microseconds per twenty queries. These are
+observed regressions, without a statistical noise or causal attribution claim.
+Accept the small-object metadata/ownership cost for the measured wide-object
+benefit, retaining the scalar lookup path below 32. Empty measurements establish
+no throughput claim. Four predecessor refusals remain excluded from timing ratios.
+The threshold stays 10%; mobile crossovers remain unqualified.
+
+| Operation / shape / width / query | Before / after microseconds | Change | Disposition |
+| --- | ---: | ---: | --- |
+| accumulator / prefix / 0 / — | 0.584 / 0.666 | +14.04% | No throughput claim; retain observation |
+| checked / prefix / 0 / — | 0.792 / 0.875 | +10.48% | No throughput claim; retain observation |
+| checked / unicode / 0 / — | 0.709 / 0.833 | +17.49% | No throughput claim; retain observation |
+| checked / ascii / 1 / — | 1.125 / 1.417 | +25.96% | Accept measured small-case tradeoff |
+| native-keyed / unicode / 0 / — | 2.125 / 2.416 | +13.69% | No throughput claim; retain observation |
+| native-keyed / ascii / 1 / — | 4.041 / 4.542 | +12.40% | Accept measured small-case tradeoff |
+| native-keyed / unicode / 1 / — | 3.667 / 4.042 | +10.23% | Accept measured small-case tradeoff |
+| parse / ascii / 1 / — | 2.625 / 2.958 | +12.69% | Accept measured small-case tradeoff |
+| parse / prefix / 1 / — | 11.792 / 13.333 | +13.07% | Accept measured small-case tradeoff |
+| parse / unicode / 1 / — | 2.542 / 2.834 | +11.49% | Accept measured small-case tradeoff |
+| parse / unicode / 4 / — | 6.500 / 7.500 | +15.38% | Accept measured small-case tradeoff |
+| lookup / ascii / 0 / miss | 0.083 / 0.167 | +101.20% | No throughput claim; retain observation |
+| lookup / prefix / 0 / miss | 0.042 / 0.167 | +297.62% | No throughput claim; retain observation |
+| lookup / unicode / 0 / miss | 0.042 / 0.166 | +295.24% | No throughput claim; retain observation |
+| lookup / ascii / 1 / miss | 0.625 / 0.875 | +40.00% | Accept measured small-case tradeoff |
+| lookup / unicode / 1 / first | 3.542 / 3.916 | +10.56% | Accept measured small-case tradeoff |
+| lookup / unicode / 1 / last | 3.542 / 3.916 | +10.56% | Accept measured small-case tradeoff |
+| lookup / unicode / 1 / middle | 3.542 / 3.916 | +10.56% | Accept measured small-case tradeoff |
+| lookup / unicode / 1 / miss | 0.583 / 0.875 | +50.09% | Accept measured small-case tradeoff |
+| lookup / ascii / 4 / first | 2.875 / 3.208 | +11.58% | Accept measured small-case tradeoff |
+| lookup / ascii / 4 / miss | 2.334 / 2.834 | +21.42% | Accept measured small-case tradeoff |
+| lookup / unicode / 4 / miss | 2.333 / 2.833 | +21.43% | Accept measured small-case tradeoff |
+| lookup / ascii / 8 / miss | 4.542 / 5.292 | +16.51% | Accept measured small-case tradeoff |
+| lookup / unicode / 8 / miss | 4.542 / 5.292 | +16.51% | Accept measured small-case tradeoff |
+| lookup / ascii / 16 / first | 2.916 / 3.208 | +10.01% | Accept measured small-case tradeoff |
+| lookup / ascii / 16 / miss | 8.625 / 10.250 | +18.84% | Accept measured small-case tradeoff |
+| lookup / unicode / 16 / miss | 8.916 / 9.959 | +11.70% | Accept measured small-case tradeoff |

Documentation/Qualification/README.md

@@ -8,7 +8,7 @@ Safety and DocC lanes run separately. Unsupported hosts fail preflight.
 The 1.0 baseline has 122 Swift Testing functions across five bundles and
 173 parameterized invocations. The first 1.1 qualification milestone adds seven
 authored prototype cases in JSONTests (54 to 61), for 129 functions, with the existing 173 parameterized invocations retained
-and seven additional nonparameterized cases. No baseline case or corpus outcome is removed. One public cap test exercises all 49 metrics;
+and seven additional nonparameterized cases. The first production lookup increment adds two further cases (JSONTests 63, total 131 functions). No baseline case or corpus outcome is removed. One public cap test exercises all 49 metrics;
 the independent runner adds nine budget/report vectors. There are three separate
 JSON/JSONSchema consumer tests. All discoveries must be nonzero.
 

Scripts/ci.py

@@ -6,7 +6,7 @@
 from preflight import collect
 
 ROOT = pathlib.Path(__file__).resolve().parents[1]
-SHARED_TESTS = 'independentCodecFacadesAndExactValuesCanBeSharedAcrossTasks|concurrentSchemaEvaluationsHaveExclusiveDynamicRegexAndAnnotationState|sharedRegexProgramHasExclusiveConcurrentSearchState'
+SHARED_TESTS = 'independentCodecFacadesAndExactValuesCanBeSharedAcrossTasks|concurrentSchemaEvaluationsHaveExclusiveDynamicRegexAndAnnotationState|sharedRegexProgramHasExclusiveConcurrentSearchState|immutableIndexedObjectCanBeSharedAcrossTasks'
 R0_TESTS = 'readerProbe|indexProbes|duplicateRecognitionPrecedesValueAndClosingSyntax'
 
 def snapshot():
@@ -68,7 +68,7 @@ def check_result(command, code, log):
     counts = [int(n) for n in re.findall(r'Test run with (\d+) tests? in[^\n]*passed', log)]
     passed = code == 0
     if command[:2] == ['swift', 'test']:
-        expected = [7] if R0_TESTS in command else [3] if 'Tools/MCPReadiness' in command else [1] if 'Tools/Qualification' in command else [1, 1, 1] if 'thread' in command else [8, 11, 12, 37, 61]
+        expected = [7] if R0_TESTS in command else [3] if 'Tools/MCPReadiness' in command else [1] if 'Tools/Qualification' in command else [1, 1, 2] if 'thread' in command else [9, 11, 12, 38, 68]
         passed = passed and sorted(counts) == expected
         if 'Tools/Qualification' in command:
             probes = re.findall(r'^CAP_PROBE (\w+) (\S+)$', log, re.MULTILINE)