Repository navigation
Port Windows dev to Linux - #4
Open
silver2127 wants to merge 73 commits into
Open
silver2127 wants to merge 73 commits into
silver2127 wants to merge 73 commits into
Conversation
…ff by default) The Workshop minimap cannot work past 50 km or 5,000 lines, so this one is new. A GUI-only game script (embedded in the DLL, written to res/config/game_script at start and removed when off) owns the window, legend, markers and camera circle. An ImageView token "bigmap:minimap?..." is intercepted at ImageView::SetImage(string); the plugin renders that world rectangle from CTerrain::BaseGetVertices with the engine's colour ramp and hillshade and uploads it as RGBA through ImageView::SetImage(raw). Twelve sites are byte-verified before anything is hooked. - Roads, tracks and stations are drawn natively from a one-time network payload (anti-aliased Hermite strokes), cached by id; layer and company toggles re-send only the header. - Multiplayer companies mode: owners map to the lobby roster and take the lobby chip colours. - Industry types with cargo icons: from production, else the mod's own construction file, else the stock table, else an icon that exists (the plain map_town/map_industry names are not files; ui.zip ships only @2x). - UI units measured at run time (setMinimumSize is UI units x scale; AbsoluteLayout, getContentRect and getMousePos are screen pixels). tools/test_minimap.py: parser, renderer against an engine layout model, accessor, detour, network drawing, thunk, script sync, installer against the Steam exe, and the Lua script under a mocked UI and world. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges 1 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
The minimap guessed which engine player is which company from creation order and the lobby config, which is wrong after a company switch, a load or an in-game company, and it labelled companies "Company N". The multiplayer mod now writes mp_company_map.txt (me=, then cid=pid=name); the minimap reads it beside mp_company_cfg.txt and colours and names companies from it, keeping the old guess only when the file is absent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges 1 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: claude.
Owner
Author
|
Automated port of Windows branch
Windows commits
Rounds
Agent reportPort of Windows dev d99c054 to LinuxMerged
Ported (what and how)
Not ported (and why)
Tests
|
…ement for content dedup) docs/terrain-cow-sharing.md section 0c refuted copy-on-write sharing: during a save load both CTerrain versions allocate their 65,536 tiles independently from the same save data, so the 8.25 GiB (256x256) is not a copy that can be skipped. The remaining lever is content dedup, and it is only worth building if the two versions' tiles really are byte-identical. This is the cheap test the doc asks for, off by default. TerrainPager::Probe hashes every live allocation and counts hash groups: a packed slot (cold, or resident read-only and unwritten since its restore) contributes the hash its blob already carries, a writable resident slot is hashed through a private alias so a blocked public view is never touched, and a slot mid-encode or mid-decode is skipped. The all-zero group is reported on its own because during a load it is tiles not yet filled, not duplication. The worker logs "terrain dedup probe:" every 10 s while loading and every 2 min otherwise when terrain_dedup_probe=1. tools/test_terrain_pager.cpp covers resident + cold + read-only-restored duplicates, the zero group, group sizes and the busy skip. Both pager tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…us the probe that measured the need docs/terrain-cow-sharing.md section 0c refuted copy-on-write sharing: during a save load both CTerrain versions allocate their tiles independently, so there is no copy to skip. It named content dedup as the remaining lever, on the condition that the two versions' tiles really are byte-identical. Measured (terrain_dedup_probe, off by default): TerrainPager::Probe hashes every live allocation from the pager worker and counts hash groups. One 256x256 load: at 131,072 live tiles, 64,922 exact pairs, 719 still-unfilled zero tiles, 65,432 distinct; after the load, 65,536 live and all distinct. Every tile of the released version was a twin of one in the surviving version. Section 0d has the table. Built (terrain_dedup, on in the shipped cfg): Evict hashes the tile before encoding (the codec's hash plus an independent second one) and looks the pair up in an index of stored blobs. A hit shares the blob through the existing Clone mechanism (immutable, reference counted, dropped by the first write) and skips the encode; a miss encodes and registers. Open addressing with tombstones, rebuilt from the slot table when half full; a blob leaves the index with its last owner. Resident backing is unchanged: a twin is still evicted only when the budget says so. tools/test_terrain_pager.cpp: probe counts (resident, cold, read-only restored, zero group, busy skip) and dedup (hit shares, write privatizes, reuse eviction, last-owner removal, rebuild, eight writers allocating twins against a forced evictor). Both pager tests pass. In-game hit count and load time not yet measured. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ictions encode, 53% share) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g to hot_mb after a load MEASURED 2026-09-17 on a 103,680-tile world: when the load ended the resident target snapped from 13.4 GiB to the 3.2 GiB hot budget, the pager evicted 67,000 tiles inside a minute, and the engine, which keeps ~36,000 tiles (4.6 GiB) in use on that map, faulted 5,900 evicted tiles per second back in, each through a decode on the engine thread: intermittent freezes, the same mechanism as the 2026-09-15 stutter on the 256x256 map at a 1 GiB target. TerrainBudgetSteady, applied once per second outside loading and commit-tight seconds: the target drops by at most 1/16 per second instead of snapping; at 300 or more cold restores per second it grows by 1/8 (a 10 s throttled log line says so), at 100 or more it holds, below that it drifts down; the ceiling is hot_mb plus half of what is free above a quarter reserve (min of free RAM and free commit), never above 65536 MiB, and hot_mb stays the floor. Cold restores are faults minus soft rescues, taken from the pager snapshot each second. tools/test_terrain_compression.py covers ramp, floor, hold, grow, the free- memory ceiling, the absolute clamp and the minimum growth step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nd, ramp the material budget too Three findings from the 2026-09-17 LONGMAPSAVE runs (103,680 tiles per version, two instances on a 94 GiB machine with no page file, so the commit limit is physical memory): 1. Crash at the commit limit. Minidump 13c03792: a write access violation at an address ending in 0x020 (the first sample of a pager tile) inside a memcpy in tpf2_bigmap.dll, with failures=1 on the pager line just before. A tile had been evicted, the engine wrote into it through the bit-exact block copy, and the restore could not create a section because the commit charge was at its limit; the fault handler handed the violation back and the game died with "An error just occurred". The eviction threads are freeing sections at exactly that moment, so the handler now retries for up to ~2 s (500 x 4 ms) before giving up; restore_retries and restore_giveups are logged. A test hook refuses the next N section creations; the pager test reads and writes a cold tile through three and two refusals. 2. Stutter from eviction rate, not from faults. In the run after the terrain ramp landed, terrain was quiet (3 faults/s) but the material pager snapped from 6.7 GiB to 532 MiB and evicted 93,000 cells in 30 s: ~3,100 protect+encode+unmap cycles per second on background threads, the same rate the 2026-09-15 note recorded as stutter. Both pagers now cap evictions (and soft blocks) per second outside loading bursts and memory pressure: terrain_cache_evict_per_s and material_cache_evict_per_s, default 1000, 0 for unlimited; rate_limited counts the Tick passes cut short. Pressure (over three times the budget, which the commit-tight back-off produces) and loading keep the old unlimited behaviour. 3. The material worker gets the same steady-state controller as terrain (ramp down by 1/16 per second, hold at 100+ cold restores/s, grow at 300+, ceiling from free memory). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ng; commit-tight at 10 GiB Two loads of LONGMAPSAVE (103,680 tiles per version) on 2026-09-17 ended in the game's own assert `false && "Out of memory"` (Application.cpp:813, minidumps ad70a72e and ea5a0065) after the pager had survived its own restore failures (restore_retries=6, giveups=0). The machine has 94 GiB and no page file, so the commit limit is physical memory, and ~44 GiB of it is committed by other processes and the kernel before the game starts. The loading allowance kept up to ~20 GiB of tile sections resident with a quarter of free memory (2 GiB floor) in reserve; the engine's own load-time allocations then found no commit. Sections commit in full at creation and cannot be returned faster than they encode, so the room has to exist before the engine's burst. The reserve is now half of what is free with an 8 GiB floor, for the loading allowance and for the steady-state ceiling alike, and the commit-tight back-off fires at 10 GiB of free commit instead of 6. Policy tests updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The out-of-memory asserts of 2026-09-17 came with 6-8 GiB of free commit still showing a second earlier, so the engine's load-time burst is larger than that. The reserve the loading allowance leaves untouched is now half of free memory with a 12 GiB floor (was 8 GiB an hour ago, 2 GiB before). On a 16 GiB machine that leaves the warm budget as the floor, which is what the old flat 12 GiB reserve was blamed for; the difference is that the warm floor is still granted, only the growth above it is not. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MEASURED 2026-09-17, first load that survived the 12 GiB reserve: terrain stayed under the cap (570 evictions/s, rate_limited climbing), but the material pager evicted 57,288 cells in 30 s (1,900/s). Its allowance is small next to what a load allocates, so "over three times the budget" was true from the first second and that test exempted it from the cap. The exemption is now SetUrgent, which the worker sets only while the commit charge is tight; over-budget encodes directly as before but at the capped rate. Pager test updated: three per second over the budget, unlimited once urgent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…talls The fixed 1,000/s cap was a guess for one machine. The rate now adapts once a second in each pager's worker (EvictRateStep): - Cost: every eviction and soft block is timed in Tick (evictMicros, evictOps). Evictions get a quarter of one core: 250 ms of that work per second, so a machine that needs 800 us per eviction gets ~300/s and one that needs 50 us gets the ceiling. Never below 100/s. - Stutter: the terrain worker samples the menu DLL's last-gameplay-frame stamp every 25 ms (UiStallMeter); two consecutive observed stamps 60 ms or more apart are a stall. A stall halves the rate; after five stall-free seconds it grows by a quarter per second. The material worker reads the same count. Without the multiplayer DLL there is no frame signal and only the cost rule applies. A backoff is logged (10 s throttle) with the last second's evictions and their cost; the 30 s line carries evict_rate= and evict_us=. The cfg keys terrain_cache_evict_per_s and material_cache_evict_per_s become the ceiling (default 4000, 0 = unlimited). Loading bursts and a tight commit charge still ignore the rate. Offline: 12 step cases (start, cost cap, stall halving, floor, quiet growth, ceiling, no-signal, unlimited) and 5 stall-meter sequences in tools/test_terrain_compression.py; the pager test checks the timing counters. In the pager test one eviction measured 136 us. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…quiet seconds MEASURED 2026-09-17 on the first run with the adaptive rate: evictions cost 170-280 us, which caps the rate near 1,250/s, yet the 30 s line showed evict_rate=4000/s. A second with no evictions measured no cost, the cap fell back to the ceiling, and the quiet-growth rule climbed the rate to it; the next busy second then started with a burst before the measurement pulled it down. The state now keeps a running average of the cost (three parts old, one new) that a quiet second leaves untouched. Tests cover the first measurement, the blend, the hold and the cap it still applies. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nute limits as cfg knobs
The limits are two float cells in .rdata (Steam 35924), found by a
RIP-relative scan of .text for readers of 1200.0f and 6000.0f:
0x3094978 1200.0f PathFactory::Compute for entities without SimCargo
(people), destination_util scoring (people), and the
destination_util task that runs
path_finder::GetReachableStationsBFS(node, maxTime, ...,
CargoTypeId, ...) for every cargo type, passengers
included: which stations a cargo type can reach from a
station within the limit.
0x309497c 6000.0f PathFactory::Compute for entities with SimCargo, and
stocklistsystem GetRandomTarget.
Each cell is read only by those functions, so a new value is a byte-verified
4-byte rewrite. Values are game seconds (730 s = one game year at 1x), 0 keeps
stock, others clamp to 60..86400. Shipped cfg keeps stock; the keys are for
big maps where 20 minutes at 1x does not reach the next town.
tools/test_travel_time.py checks the stock cells in the real executable, the
rewrites, the guards and the clamp.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…first touch, not at allocation MEASURED 2026-09-17 on a 103,680-tile save load (memwatch.py, read-only region census of the native instance): the engine allocates every tile of both versions first, all zero, then fills them over the next 20-30 s (one dedup probe saw 112,678 of 131,072 live tiles still zero). A section per allocation committed 19 GiB of tile sections within ten seconds while the target said 8 GiB, because eviction sheds at most a few thousand tiles a second; the commit charge reached zero and the game asserted "Out of memory" (three times today, once with a single instance). With terrain_lazy_zero=1 (default on) Allocate hands out a cold slot on a shared canonical zero blob: no section, no commit. The first touch goes through the ordinary cold-restore path; a read maps a read-only zero section on the blob, a write a private one, and the decode is skipped because a fresh section is already zero. The blob is encoded once by EnableLazyZero, holds one reference for the pager's lifetime, and sits in the dedup index so evictions of still-zero tiles share it. Untouched tiles release without ever having had a section. The 30 s line reports lazy=. Pager test: no section until the first read, read-only on the blob, private after a write, untouched release, write-first, reused eviction of a read-only zero tile, and the switch off again. Both pager tests and the policy tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ough the terrain pager (terrain_blocks)
MEASURED 2026-09-17 with an ETW VirtualAllocation trace of the 18:10 load that
ended in the game's "Out of memory" assert (tools/re/etl_alloc.py pairs each
VirtualAlloc with its stack and its release): at the peak the process had
31.6 GiB live, and two call sites inside ecs::TerrainAlignmentSystem::
UpdateSubterrains held 25.8 GiB of it, every block alive at once:
19.2 GiB 99,014 x 132,098 B vector<uint16_t>(n) 0x310230 <- terrain_util::
GetBlock (return 0x3c4ba7) <- BaseGetHeightmapRefined
6.7 GiB 37,354 x 132,098 B vector resize 0x1d5c50 <- UpdateSubterrains (return 0xaac4d9)
The load computes a 257x257 block and an alignment result for every tile and
keeps all of them until the publication pass has copied each into its tile's
height cache. Both are the pager's own tile size.
terrain_blocks=1 (default on, needs terrain_cache_compress=1): the constructor
is detoured (19-byte steal, byte-verified) and hands GetBlock's one-tile
vectors a pager slot; the resize hook accepts 0xaac4d9 as a second eligible
return address. Both then start as lazy zero slots and compress or evict under
commit pressure like tiles. Their release comes through the CRT: the aligned
delete reads the raw pointer at [-8] and calls free, which the game imports
(api-ms-win-crt-heap-l1-1-0!free, IAT slot 0x2f0b5b8); that slot is pointed at
a detour that returns arena addresses to the pager (TerrainPager::ReleaseAny,
by base or data pointer) and passes everything else to ucrtbase!free
unchanged. The slot is patched only after it is verified to hold
ucrtbase!free, before the constructor hook, and is restored if that hook
fails. Arena addresses that are not live slots are dropped, never handed to
the CRT.
tools/test_terrain_blocks.py checks the stock bytes and both call sites in the
real executable, the import slot's identity, the constructor fall-through,
frees of a pager slot by data and by base pointer, a stray drop and a heap
pass-through. Pager and policy tests pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in the log MEASURED 2026-09-17, first load with terrain_blocks (LONGBOI, 207,360 tiles): the game asserted "Out of memory" again with private commit at 34.2 GiB and the pager at 15.6 GiB of filled tiles resident. Lazy zero moves the section to the fill, but the fill runs at up to ~30,000 sections per second (the trace's burst was 4.3 GB/s) and eviction sheds a few thousand, so the commit charge runs out between two worker ticks whatever the target says. Throttle: while the worker sees the commit charge tight it sets a flag, and a fault that would create a NEW section (a cold slot: lazy zero, evicted, or a clone) first waits, in 4 ms steps up to 2 s, while the pool is over budget. Slots that already have a section never wait, those are protection changes. The burst becomes a stream the evictors, unlimited under urgency, can absorb; the load slows instead of dying. throttle_waits= and throttle_ms= are logged. The 30 s line also carries blocks=, block_releases= and block_stray= from terrain_blocks, and the first routed block logs once, so the next run shows whether the constructor detour is reached at all (this run's lazy count stayed at the tile count, which the log could not explain). Pager test: a cold write under the throttle waits the bound and proceeds, a slot with a section does not wait, the throttle off is immediate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ount the result resizes MEASURED 2026-09-17, first load with the throttle (LONGBOI): the pager held only 2.4 GiB of tiles resident (26 throttle waits, 39 s in total, no restore failures) yet the game's private commit still reached 34.6 GiB, and blocks=0: the block constructor detour routed nothing. The 30 s line now carries block_calls= (every call of the constructor), block_sized= (calls with one tile's 66,049 samples) and result_resizes= (resize calls returning to 0xaac4d9), and the first eight distinct return addresses of one-tile-sized constructions are logged once each, so the next load shows whether the constructor is reached, whether the size matches, and which caller it is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, compressed, decommitting spans MEASURED 2026-09-17 on LONGBOI (207,360 tiles) with the terrain_blocks counters: a save load resizes 3.3 million alignment result vectors (UpdateSubterrains, return 0xaac4d9) and constructs ~10 million work vectors (terrain_util::GetBlock), none of them one tile in size, averaging ~16 KB, and holds them all until the publication pass has copied each into its tile. That is the game's own 34 GiB private peak; with the tile pager's throttle it survived the load at 35.2 GiB (18:56) but only because the rest of the machine left exactly that much commit. src/small_codec.h: the tile codec's rANS machinery for variable-length blocks with a previous-sample predictor (the width is not known at the allocator), one static table, escapes, hash-checked decode. Terrain-like blocks encode to 17.7% of raw. src/small_pager.h: a 64 GiB reserved arena of 4 KiB pages. An allocation is a span of whole pages (32-byte aligned-allocation header, raw pointer at [-8]) handed out RESERVED ONLY; the first touch commits it zero. Eviction protects the span read-only, encodes it outside the lock and DECOMMITS the pages, so the commit charge goes back to the OS; a write during the encode cancels it. A fault on a decommitted span commits it again and decodes into it. Release decommits and recycles the span in a free list per page count. Resident spans queue in commit order in a ring; Tick evicts from the old end while over budget. The commit retries while the OS refuses, and the commit-tight throttle applies as in the tile pager. terrain_blocks.h now routes GetBlock's constructor call (any size up to the codec's maximum) and the 0xaac4d9 resize to the small pager; the CRT free import detour returns spans of either pager to their owner. The terrain worker and its helpers tick the small pager, give it the same loading allowance from small_cache_hot_mb / small_cache_warm_mb (default 1024 / the terrain warm budget), drain it to 64 MiB under a tight commit charge, and log a "small pager:" line every 30 s. tools/test_small_pager.cpp: codec sizes 1..71,824 with truncation, wrong count and corruption rejected; lazy spans, recycling, evict/restore/ decommit, sticky incompressible spans, ring policy, throttle, and eight writers against a forced evictor with no failures. test_terrain_blocks.py adds a small-pager span freed by data pointer and by raw base. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…blocks resident during the load) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…f tiles (compute, publish, free per batch) The load-time memory peak is ecs::TerrainAlignmentSystem's update: its caller (0xaac810) hands the std::set of tiles dirtied since the last frame (this+0xa0, 8-byte tile keys) to UpdateSubterrains (0xaad6c0), which for every tile in the set gathers the alignments, computes a work block and a result block per region on the thread pool (LoopImpl at 0xaadc50), and only then publishes them all into the height caches (0x33cd10 at 0xaadcf5) and frees them (0x3b08f0). In play the set is a few tiles; on a save load it is the whole map. MEASURED 2026-09-17 on LONGBOI (207,360 tiles): 3.3 million result vectors and ~10 million work vectors alive at once, the game's own 34 GiB private peak, and the "Out of memory" assert on a 94 GiB machine with no page file. The detour (17-byte steal, byte-verified together with the caller's call site and the iterator's isnil test) walks the game's set read-only with the same MSVC _Tree successor step the function uses, copies the keys, and calls the original once per batch of alignment_batch_tiles (default 512) with a degenerate tree of its own (a left chain, head as nil) holding that batch. The function reads only the head, the nodes' link and isnil fields and the key; the caller erases its own set afterwards, untouched. The live set is one batch's worth and the per-frame behaviour the engine already has. terrain_blocks (the small pager routing) is now off by default: with the accumulation gone it would cost three system calls per block for nothing. It stays as the fallback. tools/test_alignment_batch.cpp: the walk matches std::set iteration on 10,007 random keys (the same STL layout the game uses), batches concatenate to the set in order, pass-through for sets under the batch size and batch 0, chains of one and exact multiples, the game's set untouched, and the byte anchors in the real executable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…alignment items; copy the whole 32-byte value MEASURED 2026-09-17: the first version copied only the 8-byte coordinate; the set held 1,658,880 entries (eight blocks per tile), each with a std::vector of 16-byte items at +0x28 that UpdateSubterrains passes to CreateTileBlock, which asserted box.bounds[0].x <= box.bounds[1].x on the uninitialised copy. The chain nodes now carry the coordinate and the vector's three pointers (shallow, read-only for the callee; the caller's erase frees them). Test: std::set of entries with vectors, the same MSVC layout, items read back through the copied pointers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…h.md) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ain (g_alignmentTerrain) for the sidecar Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…read; asserts the exposed handle Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s aligned 1 m cache on load (not yet wired)
The load-time alignment pass rebuilds the 1 m height cache every load (refine
+ cut every road/track/construction). Batching removed its memory spike; what
remains is its time. The finished cache is a pure function of the save, so
storing it beside the .sav and restoring it on load of the same save skips
both the refine and the pass.
This lands the standalone, tested core only, so it collides with nothing that
is wired: a new src/terrain_sidecar.h (nothing includes it yet) and its test.
- Format: TERR v1, FileHeader{magic,version,fingerprint,nx,ny,tiles,headerHash}
then per tile {recordIndex,bytes}+BlockCodec blob. headerHash catches a torn
write; each blob's own 64-bit content hash catches a flipped byte.
- Grid walk over CTerrain+0x18 -> {x0,y0,nx,ny,records}, 40-byte records,
height vector at record+8 (RE from GetTile 0x33d580 / AddTile 0x33cb60).
Write compresses every Samples-long tile; Apply restores only into a record
whose vector is already that size, so it never allocates a tile.
- Apply refuses a file whose magic/version/headerHash/fingerprint/nx/ny do not
match (foreign, stale or plugin-less save -> no-op, the pass runs) and
returns -1 on a corrupt-but-matching file.
- tools/test_terrain_sidecar.cpp builds a synthetic grid to the exact layout:
full round trip, exact restore, untouched non-stored tiles, and the
foreign/dimension/absent/truncated/corrupt refusal paths. Measured 17.6% of
raw on terrain-like tiles (~1.2 GiB for a 207,360-tile map).
Pending integration (needs the alignment pass, owned by alignment_batch.h):
the .sav-hash fingerprint via SaveGame 0x2e97c0 / LoadGame 0x2e5ec0, the write
trigger over g_alignmentTerrain, and the read-side short-circuit. See
docs/terrain-sidecar.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o consume BeginApply opens the sidecar, validates it against the live grid's fingerprint and dimensions, indexes every stored tile and verifies each blob decodes, so a later Has()==true guarantees ApplyTile() succeeds and the pass can skip the compute for that tile without a fallback. The compressed file is held in memory (~1.2 GiB for a full map) until EndApply. Has/ApplyTile are lock-free reads of the immutable index; ApplyTile takes the caller's own DecodeScratch, so it is safe from any pass thread. Test extended: BeginApply reports exactly the stored tiles, Has() is true only for them, ApplyTile restores each exactly and refuses a non-stored tile, EndApply clears state, and a foreign fingerprint holds nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e's tile index The pass owner needs to turn a tile coordinate into the record index that Has()/ApplyTile() expect. RecordIndex(grid, x, y) = (x - x0) + (y - y0) * nx, the same value CTerrain::GetTile (0x33d580) and AddTile (0x33cb60) compute, -1 outside the window; IndexOfRecord(grid, record) for a caller that already holds the engine's record pointer. Grid now exposes x0()/y0(). Tested on a windowed grid (x0=100, y0=-50): origin, +x, +y, corners, out-of-window, and agreement with IndexOfRecord across the grid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
.github/workflows/build-msi.yml runs installer\build_msi.ps1 -AcceptWixEula -Validate on a windows-2022 runner for every push to dev or main and every pull request, and keeps TpF2BigMaps-<version>.msi and SHA256SUMS.txt as the run's artifact. A v* tag also creates a draft GitHub release with them attached; the tag must equal installer\VERSION or the run fails before building. WiX v7 comes from `dotnet tool install --global wix` (the script already looks in %USERPROFILE%\.dotnet\tools); build.bat finds the runner's Visual Studio through vswhere as it does on any machine. Passing -AcceptWixEula in CI is the repository owner accepting the WiX OSMF terms for those builds; the README says so. The runner has no tpf2-multiplayer checkout beside this one, so the script's own PluginHost.wxs drift check would be skipped. The workflow fetches the multiplayer copy at the release named in installer\vendor\VENDORED.md and compares it (line endings aside) instead, so the shared component fragment is still checked on every build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caps, the sticky throttle, the working-set floor, the log lines and the knobs -- and what a big map costs; a note on virus-scanner findings Replaces the 'Experimental gameplay RAM reduction' section, which still described the early fixed budgets and the 2 m experiment (now a pointer to gameplay-memory.md). Numbers from 2026-09-20: 15 GiB private after generating a large stock map before the caches held a byte, 20 GB on a 32 GiB machine without a cap, 74 throttle flips in one session on a commit-tight rig. The scanner note answers a mod site's 'Unsigned Image Loaded Into LSASS' match, whose event named the sandbox's own hook. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…instead of refusing to load Depth 12/13 replace the root descent prologue and the level builder, and both replacements are Steam-35924 code shapes; on GOG neither is at a verified address. The plugin returned ERR_FAILED there, so a GOG user whose config asked for 13 -- which the shipped cfg does -- got a plugin that refused to load and said nothing useful. EffectiveOctreeDepth() is now the single answer to what this build can install, and EffectiveMaxTiles() reads it: with the ceiling still computed from the raw octree_depth, the menu would have offered 2048 tiles over a 512-tile root, which is the node duplication the widening exists to prevent. The degraded path byte-verifies the GOG site and takes the existing depth-11 branch. Validated in game on GOG 2024-12-12: all three sites byte-verify, the root widens at RVA 0x230718, a 57x57 km map was created, played and reloaded with zero 'Duplicate base nodes found'. Steam behaviour is unchanged. BigmapTestOctreeCeiling exposes the ceiling for offline checks, and docs/octree-depth12.md no longer claims the shipped config is 11/512. Also: build.bat -deploy now refuses when a game-folder install exists, because the host scans the datadir first and the two copies drift. The check has to sit outside the if block -- cmd expands a parenthesised block before running it, so a call inside it expanded GAMEDIR to nothing and the check passed either way.
…ates The package could not be installed on a GOG copy of the game at all. tpf2ca.dll's folder check knows only the Steam 35924 executable values, so CheckGameDir showed GOG_MSG and set TPF2_GAMEDIR_OK=0 -- the wizard's Next button then did nothing whatever the user did. The only way in was TPF2_SKIP_GAMEDIR_CHECK=1 on the msiexec command line, which did not check the folder, it stopped checking it. The CA now recognises a second known build, and the same package can ask for it. There is no separate Wizard to maintain: Package.wxs grows a GogVariant preprocessor branch, the GOG MSI sets TPF2_ALLOW_GOG=1 and adds a registry search for GOG's install path, and everything else -- the UI, the sequence, the component GUIDs, the UpgradeCode -- stays byte-identical, so the two packages upgrade each other as one product. A standard install can still opt in with TPF2_ALLOW_GOG=1 on the command line. installer\vendor\tpf2ca.dll is rebuilt from the tpf2-multiplayer commit recorded in VENDORED.md. That DLL is embedded as a Binary, not installed as a file, so it shares no component GUID and the two repositories' copies are free to differ; alut.dll and tpf2_pluginhost.dll keep the v0.6.1 bytes they must keep. The multiplayer release still carries the older CA and refuses a GOG folder, exactly as it should. Verified on a GOG install with the built package: without the property the folder is refused (1603) with 'is the GOG build; refused (no TPF2_ALLOW_GOG)', with it the install completes -- InstallFiles, PreserveStockAlut and InstallFinalize all run. A first attempt at this test proved nothing: it passed INSTALLFOLDER on the command line, but AppSearch re-reads the remembered folder from the registry and overwrites it, so the test had been running against the real game folder.
The GOG package changes behaviour a user can observe, so the packages built from this branch need a version of their own: with 0.5.2 kept, an installed copy and this one are indistinguishable, and nothing tells a bug report which of the two it came from.
It searched HKLM\...\GOG.com\Games\* for a 'path' value. That wildcard matches EVERY GOG game on the machine, not this one -- GOG writes one row per product id, and RegLocator returns one match, not necessarily the game's. So the default folder it pre-filled could have been a different game's, and nobody would have noticed until the folder check refused it. It was also the only part of this change I could not verify without installing: AppSearch does not run during an administrative install, so the wildcard's behaviour was assumed, not measured. The gain was one fewer click on a first install. Without a search hit, INSTALLFOLDER stays on the Steam default, the folder dialog shows it, and the user points at the right folder once -- after which the remembered-folder search handles every later install. Not worth a guess that can be wrong. The GOG package now differs from the standard one by exactly one property, TPF2_ALLOW_GOG=1, and nothing else.
…binaries vendored from TpF2 Multiplayer 0.6.1.14 PluginHost.wxs and the three shared binaries (alut.dll, tpf2_pluginhost.dll, tpf2ca.dll) come from the released TpF2Multiplayer 0.6.1.14 MSI, which adds the FindGameDir custom action: after the registry searches, when the default folder holds no TransportFever2.exe, the first Steam library (libraryfolders.vdf, any drive) that has the game becomes INSTALLFOLDER. This release also ships the pager policy changes made since 0.5.2 (sticky commit throttle, machine- independent caps, the working-set floor, simulate_physical_mb). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…it can be The README opened with 'GOG is not supported (since 0.2.0)' and called the GOG addresses in bigmap.cpp untested and unmaintained. That stopped being true in this branch: the plugin detects both builds, keeps a pair of constants per site, and all three GOG sites byte-verify at load. A reader deciding whether to try it would have turned away. It also still said the depth 12/13 work was 'not yet validated in a running game'. It is now: a 57 x 57 km map was created, played and reloaded on the GOG build with no duplicate street nodes and no assertion. The fallback that makes this safe is written out, because it is the reason the shipped config can keep octree_depth=13 for both builds instead of shipping the lowest common denominator. The install section gains a GOG part: which package to use, what is different about it (one property, not a second wizard), and the one thing a GOG user will notice -- the folder dialog starts on the Steam default, so it has to be pointed at the game once.
…d per release Same change as tpf2-multiplayer: no builds on dev or pull requests, docs/** and **/*.md ignored, and a main push whose commit a tag already built is skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…f that was never wired
The serving half shipped in 0.5.0 (AddTile hook, terrain_sidecar=1) but nothing ever
called WriteForSave or ArmForLoad, so no .terr file existed on any machine and the setting
was inert. On the dedicated server the two alignment passes it would shorten are 48-64 s
each, most of a load (2026-09-21).
src/terrain_sidecar_io.h detours both, byte-verified:
- SaveGame (0x2e97c0; the SaveGameId is the 8th argument behind the hidden return
pointer). The terrain is captured at the detour's ENTRY into "<save>.terr.tmp" with
fingerprint 0: the world is frozen for the length of the call, so the capture is what
the save holds. When the engine returns and the .sav's write time has moved, the file is
stamped with the new .sav's hash (Refingerprint) and renamed over "<save>.terr". A
failed save leaves no sidecar. Capturing after the return would race the first alignment
of the resumed simulation.
- LoadGame (0x2e5ec0; the id is r9). Clears g_alignmentTerrain, resolves the path, hashes
the .sav, arms the sidecar; the first AddTile opens it as before.
- The path: SaveGameId { wstring path; string name; string ns } -> <path>\<name>.sav, and
for an id without a path the directory StandardSaveGameBackend keeps for ns (app
0xbb23c0, backend 0xbb2db0, directory lookup 0x2471640, vftable checked against
0x38e5758), as GetSavegameInfo (0x2471830) builds it. Engine calls sit in SEH frames.
- Found by fingerprint: multiplayer loads a COPY of the host's save as mp_shared.sav, so
when "<save>.terr" is absent or foreign the loader takes any .terr in that folder whose
header carries the save's hash (32 bytes read per candidate).
- The terrain is g_alignmentTerrain, trusted only when its grid is sane and every full
tile's cache lies in the terrain pager's arena.
- Autosaves rotate: after each write, a .terr (or stale .terr.tmp) whose .sav is gone is
deleted.
terrain_sidecar.h: paths are UTF-8 (the engine's) and opened wide -- the narrow CRT read
them in the ANSI code page; Refingerprint, FingerprintOf, FindByFingerprint.
WIN32_LEAN_AND_MEAN keeps rpcndr.h's `#define small char` out of the offline test.
Settings: terrain_sidecar_write=1; terrain_sidecar_max_tiles=0 (no limit: the owner's
call, a slow save is fine). Cost on synthetic tiles: 0.17 ms and 22.7 KiB per tile --
8,712 tiles 1.5 s / 190 MiB, 65,536 tiles 11 s / 1.4 GiB, on every save.
tools/test_sidecar_io.py: 16 checks through the DLL's test exports, in a folder whose
name is neither ASCII nor ANSI (stamp + exact restore, refusals, found under another
name, SaveGameId -> path for SSO and heap strings, the orphan sweep). The existing
sidecar and serve tests pass. NOT yet run in a game: what a served load saves is the
publication into served tiles; the compute still runs, so the gain has to be measured.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nment system's last The dedicated server's first autosave with the sidecar (2026-09-21 20:25) captured nothing: 'no live terrain to capture (the last seen CTerrain is not this world's)'. A load holds two CTerrain versions and frees one; on an idle world the alignment system's last-seen pointer was the freed one, and its tiles failed the arena check. The AddTile hook now remembers every CTerrain this load populated (four slots, cleared at LoadGame). At a save, PickTerrain validates each candidate and the alignment system's last, takes the one with the most full tiles, and requires it to hold 99% of the pager's live tiles so a half-overwritten stale grid is never captured. The refusal log names every candidate's verdict (-1 null, -2 no grid, -3 dimensions, -4 outside the arena, -5 unreadable) and the pager's live count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…at record+8 MEASURED 2026-09-21 in the running dedicated server (/proc/<pid>/mem, read-only): all 8,712 tile records hold a 66,049-sample std::vector<uint16_t> at *(record+8)+0, and *(record+8) - *(record+0x10) == 0x10 on every one -- record+8 is a std::shared_ptr, +8 the object, +0x10 its control block (make_shared puts the object 0x10 past it). Both CTerrain versions point at the same vector objects (copy-on-write). The code read the vector at *(record+8)+0x10: the vector's end pointer and the next object's bytes. No tile was ever eligible, so every save logged 'no live terrain to capture (candidates: ...=0 ...=0; pager live=8712)' and the serving half could never have applied a tile either. The fake terrains in test_terrain_sidecar, test_terrain_serve and test_sidecar_io encoded the same wrong layout and passed; they now build the make_shared layout the game has. The serve hook marks the pager slot through VectorOf as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A multiplayer host froze in a live join: its UI thread and six render workers all sat in TerrainPager::Fault's throttle loop (Sleep(4) up to ThrottleMaxMs). The counters said throttle_waits=1957, throttle_ms=3,862,828 -- about 2 s per tile the camera needed. The PC was at 80 of 94 GB committed with no page file (other programs held most of it; the pager's own backing was 259 MiB), so commit_tight stayed on and every cold restore waited for eviction that the over-budget check never let finish. The throttle exists for load bursts (~30,000 new sections a second outrunning eviction). It is now set only while commit is tight AND a load is in progress (terrain: busy/bulk/warm-up loading; material: busy/bulk; the small pager follows terrain). Eviction stays urgent under a tight commit charge; only the sleep is limited to loading. Pager tests pass (terrain, small, material). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Merges 20 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
Merges 20 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
Merges 7 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
…layer MSI - native\build.bat bigmap (and all) builds bigmap\out\tpf2_bigmap.dll through bigmap\build.bat; build_msi.ps1 builds it and packages plugins\tpf2_bigmap.dll + plugins\tpf2_bigmap.cfg (the cfg versioned, as the Big Maps MSI had it), and the files zip for the Proton installer carries both. - The MSI runs the plugin's base_mod.lua restore on a full uninstall (not on an upgrade), as the Big Maps MSI did. - An installed TpF2 Big Maps product is removed by this MSI (its UpgradeCode in an Upgrade row), so the plugin has one owner. The shared proxy and host components are refcounted and Big Maps' own base_mod restore does not run during an upgrade removal. - The Big Maps copy of the plugin ABI header is gone; it includes native\src\plugin\tpf2mp_plugin.h (they were byte-identical). - Removed: bigmap\installer (its MSI, the duplicate PluginHost.wxs and the vendored alut / plugin host / custom-action binaries), vendor_host.ps1 and its CI workflow. test_config_msi.py checks the multiplayer Package.wxs. - auto_install.ps1 / deploy_shipping.ps1 build and ship bigmap\ instead of the sibling tpf2-bigmap checkout; the cfg ships with the DLL. - The HostDll default in Package.wxs held a literal TAB for \t (unused: build_msi.ps1 always passes HostDll); fixed. Tested: build_msi.ps1 -Validate (the extracted MSI holds plugins\tpf2_bigmap.dll and .cfg), build_files_zip.py, test_install_sh.py, every bigmap tools\test_*.py, test_alignment_batch and test_terrain_serve. Not tested: an install over an existing Big Maps MSI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@3aaaf3d
… the id's name) Every autosave's sidecar was discarded with "the save was not written": the file the engine writes is autosave_<game>_<date>.sav, not <id.name>.sav, so the expected file never changed. Manual saves were fine. When the expected .sav did not change, the one .sav in that folder written during the call takes the sidecar; two or more take none, so a sidecar never carries another save's hash (e.g. the lobby copying mp_shared.sav at the same moment). The discard line now names the expected path and why. Not sent to joiners, on purpose (docs/terrain-sidecar.md): 1.6 GB beside a 755 MB save for a share of a ~10 s alignment pass in a 185 s world entry. Tested: tools/test_sidecar_io.py (new section 5), test_terrain_sidecar. Not tested in a game. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@60d237c
…ne (0.7) The owner, 2026-09-22: "turn it on for everyone, make sure the version stays as .7". FOR THE LINUX PORT: bigmap/linux/tpf2_bigmap.cfg ships terrain_cache_compress=0 and the Linux plugin treats it as opt-in. Set terrain_cache_compress=1 in that cfg and make 1 the built-in default when the key is missing; keep =0 as the off switch and the stock fallback when userfaultfd setup fails. Update bigmap/docs/linux/PORT.md and INSTALL.md (on by default; the kernel-fault risk; SIGBUS -> set it to 0). No version change: installer/VERSION and LOBBY_VERSION stay 0.7 (compression does not change the simulation). Why: the native dedicated server was OOM-killed at 29.3 GB loading a ~52,000-tile map that Windows holds at 8-10 GB. bigmap/docs/terrain-compression.md "Native Linux: ON by default". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@747fd4d
…ncy eviction needed) FOR THE LINUX PORT: the native pager evicts round-robin and its automatic budget (RAM/30, ~1 GiB) is far too small: the dedicated server faulted ~790 tiles/s and ran at half speed. Evict by recency like Windows, size the budget from the machine and live tiles, and log faults/s. Details in bigmap/docs/terrain-compression.md, "The Linux pager thrashes". Version stays 0.7. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@ea35eb8
…instead of refusing to load Depth 12/13 replace the root descent prologue and the level builder, and both replacements are Steam-35924 code shapes; on GOG neither is at a verified address. The plugin returned ERR_FAILED there, so a GOG user whose config asked for 13 -- which the shipped cfg does -- got a plugin that refused to load and said nothing useful. EffectiveOctreeDepth() is now the single answer to what this build can install, and EffectiveMaxTiles() reads it: with the ceiling still computed from the raw octree_depth, the menu would have offered 2048 tiles over a 512-tile root, which is the node duplication the widening exists to prevent. The degraded path byte-verifies the GOG site and takes the existing depth-11 branch. Validated in game on GOG 2024-12-12: all three sites byte-verify, the root widens at RVA 0x230718, a 57x57 km map was created, played and reloaded with zero 'Duplicate base nodes found'. Steam behaviour is unchanged. BigmapTestOctreeCeiling exposes the ceiling for offline checks, and docs/octree-depth12.md no longer claims the shipped config is 11/512. Also: build.bat -deploy now refuses when a game-folder install exists, because the host scans the datadir first and the two copies drift. The check has to sit outside the if block -- cmd expands a parenthesised block before running it, so a call inside it expanded GAMEDIR to nothing and the check passed either way. Synced-from: silver2127/tpf2-multiplayer@df1436c
…x86) path Follow-up to Kalemillion's GOG change (PR #5, previous commit): with the game under "C:\Program Files (x86)", every build -- not only -deploy -- failed after BUILD BIGMAP OK with "\Steam\steamapps\common\Transport was unexpected at this time" (exit 255, so native\build.bat bigmap/all and the MSI build failed): the ")" of (x86) closed the -deploy block where %GAMEDIR% is echoed, and %ProgramFiles(x86)% closed the fallback for-list. The deploy steps are a subroutine now and the fallback list uses a pre-expanded variable. On a GOG machine the registry lookup answered first, so it never showed there. Checked: native build.bat bigmap exits 0; -deploy with a game-folder install refuses (exit 1) as intended. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@aaae03f
Kalemillion's GOG support, merged as submitted: - octree_depth=12/13 degrade to the depth-11 root on the GOG build instead of refusing to load (EffectiveOctreeDepth / EffectiveMaxTiles, test included); - build.bat -deploy refuses a datadir copy that would shadow a game-folder install; - the GOG package variant (TPF2_ALLOW_GOG=1) with the GOG-aware tpf2ca.dll from silver2127/tpf2-multiplayer#8 (conflicts in installer/vendor taken from this PR); version 0.5.3. One fix folded in: with the game under "C:\Program Files (x86)" every build failed after BUILD BIGMAP OK ("\Steam\steamapps\common\Transport was unexpected at this time"): the ")" of (x86) closed a parenthesised block early. The -deploy steps are a subroutine now and the fallback folder list uses a pre-expanded variable. It never showed on GOG, where the registry answered first. Big Maps now lives in silver2127/tpf2-multiplayer (bigmap/); this merge is carried there too. Co-authored-by: Kalemillion <174786230+Kalemillion@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The PR is merged in silver2127/tpf2-bigmap (926113b); this brings that merge, with Kalemillion's five commits, into the Big Maps copy that now ships here. The plugin change and the build.bat (x86) fix were already on dev (df1436c, aaae03f), so the only content change is the README's GOG section (Kalemillion's text). The separate Big Maps installer files stay deleted: the unified MSI replaced them and already carries the GOG switch (TPF2_ALLOW_GOG, PR #8). Co-authored-by: Kalemillion <174786230+Kalemillion@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@7469fce
Merges 1 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
The owner: the higher octree values have been tested and work fine. README, the shipped cfg's comments and docs/octree-depth12.md no longer call depths 12/13 experimental or say that live/save-reload/multiplayer validation is missing: depth 13 has been created, played, saved and reloaded on Steam, in single player and in multiplayer with the native Linux dedicated server. Every peer needs the same octree_depth. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@63a3b8d
A "Performance optimizations" section: memory (terrain pager, material paging, alignment batching, lazy zero, dedup, instance shrink, generation buffer reuse) and load time/CPU (sidecar, SSE2 refine/minmax/align, material-index loop, placement attempts, save_fast, native minimap), each with its cfg switch, shipped default, mechanism, measured gain and design doc. Notes that all but placement_attempts are lossless, so settings cannot desync peers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@4616c16
…dev push Big Maps stays downloadable on its own, and bigmap/ here stays its one source. tools/bigmap_sync/sync.py replays each dev commit that touches Big Maps onto tpf2-bigmap's dev (same author, date and message, a Synced-from trailer). The standalone tree is bigmap/ at the root, with the standalone repo's own installer, CI and MSI tooling kept, the plugin ABI header copied from native/src/plugin, its includes repointed, and the README's <!-- standalone:NAME --> blocks swapped for tools/bigmap_sync/readme/. Deterministic: a rerun makes the same commits, an up-to-date sync none. .github/workflows/sync-bigmap.yml runs it on dev pushes touching bigmap/, the ABI header or the sync, through a write deploy key on tpf2-bigmap only (secret BIGMAP_SYNC_DEPLOY_KEY). The README's install, uninstall and MSI-build text for the multiplayer side still described the old standalone package; it now describes the unified MSI, and the standalone wording lives in the fragments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@c9ac009
…oading a big save) A player's crash dump (2026-09-26, 0.7.0.5): tpf2_bigmap.dll+0x2e74, the vector free inside TerrainSidecar::EndApply, reading the header of a block already released, called from the alignment pass's done hook. Every time: a save of a 604 km2 map loaded once, then loaded again in the same process (Host after playing), crashing at 70%. The second load is the one with a sidecar to serve. The game runs the alignment pass on many threads (a dozen were in UpdateSubterrains in the dump) and each batched pass that finishes called the done hook, which checked Loaded(), logged, and freed the file with no lock: two finishing close together freed it twice. AddTile threads could also still be decoding from it. Has/ApplyTile now hold g_loadLock shared (ApplyTile across the decode), BeginApply/EndApply take it exclusive, the file is freed once outside it, and EndApply says whether this caller released it: only that one logs "load done". tools/test_terrain_serve.cpp: 200 rounds of eight threads releasing while four decode -- exactly one release, every decode intact or refused. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Synced-from: silver2127/tpf2-multiplayer@363c38c
Merges 11 Windows commit(s) from dev and ports them to the native Linux build. Verification: passed. Agent: codex.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated port of Windows branch
devintolinux-native, run by tpf2-port on the Linux machine. One pull request per branch: each run adds its Windows commits and their port here; this description is rewritten after every run.b755f653a2b755f653a2README.mdtools/linux/build.shRuns
cdfee2a125d99c054979bd0d85f2b026bced4b989180629270926113b78bb755f653a2Windows commits in this run
Rounds
Agent report (this run)
Merged
Integrated the eleven requested Windows commits through
b755f653a2e5067a6ca34e89df64dcabe039a052intoport/dev.Resolved README.md by retaining both native scope/evidence and the upstream
standalone mirror/unified MSI documentation. Windows code paths and history are
preserved. All changes are staged; the merge remains in progress, uncommitted.
The incremental integration and RE evidence are recorded in
docs/linux/PORT.md.Ported (what and how)
linux/tpf2_bigmap.cfgand the nativemissing-key fallback. Explicit 0 and unsupported-userfaultfd fallback remain.
Docs explain upgrade config preservation and kernel-origin SIGBUS mitigation.
Native version identifiers remain unchanged, as requested by the incoming change.
decoding, publication and release. EndApply succeeds for one releaser and frees
detached storage outside the lock. Windows keeps its SRW implementation.
.savin the resolved folder, UTF-8 paths, two-second timestamp tolerance andfail-closed ambiguity/capacity handling. This is offline support, not engine wiring.
time. Fault counts include zero-page and write-protection events.
926113bremain intact;native requests for depths 12/13 still use depth 11 and the 512-tile ceiling.
Upstream depth-13 server results are explicitly distinguished from this port.
Not ported (and why)
Linux signatures and actual SaveGame, LoadGame, GetSavegameInfo, backend forwarding
and AddTile disassembly. SysV SaveGame uses argument 7 for SaveGameId and returns
in rax/edx; the backend virtual slot is +0x20; record+8 directly points to the
vector. Native alignment batching/publication suppression is still absent.
Live terrain ownership, private writable lifetime, worker completion, backend
identity and completed autosave/rotation semantics could not be established:
both lab launches failed before the game started. No speculative hook enabled.
e19eec1):reviewed the Windows policy and native UFFD implementation. Native touched time
sees faults only; resident reads and ordinary unprotected writes are invisible.
Its existing two-second grace and round-robin scan do not implement repeated-fault
retention or oldest-first eviction. Fixed 1024 MiB remains; no unvalidated Linux
equivalent of Windows free-commit accounting is assumed. Live large-save fault,
simulation-rate and constrained-memory recovery measurements were blocked by
startup. Rate diagnostics are ported; the policy change is not.
Earlier larger-root, minimap and other native feature gaps remain unchanged.
This integration is PARTIAL, with a working native build.
Live testing (what you ran, what you observed)
Backed up actor
share/tpf2mpandgame/mods/mp_lockstep_1withcp -ato.before-port-b755f65, and repeated with.before-port-b755f65-finalfor thefinal build. Installed only the built native plugin and native config in the
actor. Used the official launcher from
/home/topsnek/tpf2-multiplayer/becausethis clone does not contain it:
Both time-boxed launches exited immediately with:
bwrap: setting up uid map: Permission denied.No game process, title menu, Vulkan device, save load, gameplay or gdb attachment
was reached. No live correctness/performance claim is made. No input, Steam
operation or save modification occurred. Both actor directories were restored;
recursive comparisons are empty and no TransportFever2 process remains.
meta/live/contains launch output, disassembly, signatures, test logs,restoration comparisons and actor logs/data. Copied actor gameplay logs predate
these attempts and are not new observations.
Tests
tools/linux/build.sh: all seven soldier SDK CTest suites pass; UFFD pagertest ran, not skipped. Config regression checks missing/default/on/off choices.
release each, intact-or-refused decode, repeated begin/reset and invalid
fingerprint handling. Autosave discovery covers UTF-8, zero/one/two candidates,
timestamp boundaries, metadata/directories/symlinks and insufficient capacity.
decodes in that run; no sanitizer finding.
tools/linux/verify_game.py: correct GNU build-id and all 22 guarded sites.on upgrade, multiplayer coexistence, uninstall/save preservation all pass.
git diff --check: clean. No unresolved merge entries remain.An initial new-test compile rejected a lambda default argument referring to a
local array; corrected it and reran the full build successfully.