[Performance] Late-game HEAPMNG Total grows far above Committed memory

[zhanghm18GIT]

Summary

In long, heavy AI/mod games, FAF's periodic memory log can show Heap Total continuing to grow in approximately 31-32 MiB steps while Heap Committed remains much lower.

This consumes scarce virtual address space in the 32-bit process and materially increases the chance of a late-game crash.

FAF logs the values in this order:

Heap: Total / Committed

Source:

https://github.com/FAForever/fa/blob/develop/lua/system/logger.lua

Stress-test environment

The documented runs used a severe practical workload rather than a deterministic vanilla reproduction:

• FAF game version 3836 / fafdevelop;
• Seton's Clutch FAF v0004;
• two private M28-derived Rush AI brains;
• unit cap 1250;
• a large SIM-mod set;
• long observer-mode sessions.

The exact available logs, mod information, executable hashes, and extracted Heap samples are in the attached evidence archive.

Observed behavior

Run	Same-tick queue-length cache	Executable identity	Selected late value
Reference high-growth run	Off	Separate reference build	1.893 GiB / 1.300 GiB
Cache-active counterexample	On	Not recoverable from the log	1.930 GiB / 1.256 GiB
Lowest-growth documented run	On	Allocator candidate 22cb5373...	1.406 GiB / 1.326 GiB

In the lowest-growth run, Heap Total reached 1.406 GiB at session time 00:39:07 and remained at that value through at least 00:48:08, while Committed remained around 1.30-1.33 GiB.

The low-growth run therefore used about 0.524 GiB less Heap Total than the cache-active high-growth run, despite having a slightly higher selected Committed value.

That result is consistent with a large difference in allocator expansion and reuse, rather than merely fewer live objects.

Expected behavior

When HEAPMNG already has suitable free regions, Heap Total should not continue acquiring additional approximately 31-32 MiB regions while Committed remains substantially lower.

The practical goal is to avoid hundreds of MiB of unnecessary reserved address space and preserve headroom for graphics, mappings, DLLs, stacks, and other users of the 32-bit process.

Mitigation that produced the lowest observed growth

The best documented result used two coordinated changes:

1. M28-side pressure reduction
   A private M28-derived build reused only the integer command-queue length for the same unit within the same simulation tick and invalidated it whenever that private M28-derived code changed the unit's commands.

   AI-side record:

   [Performance] Cache same-tick command-queue length in UpdateRecordedOrders maudlin27/M28AI#394

2. HEAPMNG bookkeeping/reuse candidate
   A local executable candidate changed:

   • the page-map allocation from 0x3FF000 to 0x400000 bytes;
   • the right-neighbour coalescing bound from 0xC0000 pages to 0x100000 pages.

   Allocator-side record:

   [Research / Candidate fix] HEAPMNG page-map size and 3-4 GiB coalescing boundary FA-Binary-Patches#163

The cache-active counterexample shows that the AI-side cache alone is not sufficient to explain the full reduction. The available runs are also not a completed controlled 2x2 experiment, so the independent contribution of each component remains unisolated.

Reproduction and validation plan

Use one frozen workload and repeat each cell at least three times:

Cell	Executable	Same-unit/same-tick queue-length cache
A	Baseline	Off
B	Allocator candidate	Off
C	Baseline	On
D	Allocator candidate	On

Hold constant:

• map and map version;
• seed;
• AI count and personality;
• unit cap;
• mod list;
• graphics configuration;
• duration;
• executable SHA-256;
• private-fork source identity.

Record:

• Heap Total and Heap Committed over time;
• process virtual size and private bytes;
• real GetCommandQueue() calls;
• cache hits, misses, and invalidations;
• HEAPMNG growth reservations;
• largest reusable free region before growth;
• high-address free-region coalescing events.

Scope and limitations

• This is a severe stress case, not a deterministic vanilla reproduction.
• The cache alone is not proven to cause the full reduction.
• Neither allocator constant has been isolated independently.
• The mitigation reduces the chance of entering the dangerous high-memory region; it does not make every path above 2 GiB safe.
• Detailed failed experiments and crash diagnostics are retained in the technical archive, not in this game-level issue.

Related records

• HEAPMNG implementation:
  [Research / Candidate fix] HEAPMNG page-map size and 3-4 GiB coalescing boundary FA-Binary-Patches#163
• M28 optimization:
  [Performance] Cache same-tick command-queue length in UpdateRecordedOrders maudlin27/M28AI#394
• Historical 0x0095854F crash index:
  EXCEPTION_ACCESS_VIOLATION at address 0x0095854f #1445

Attachment

The existing evidence archive contains text and CSV material only:

UPLOAD_1_FA_MAIN_EVIDENCE.zip

[zhanghm18GIT]

Allocator-side implementation details:
FAForever/FA-Binary-Patches#163

[zhanghm18GIT]

AI-side implementation and upstream proposal:
maudlin27/M28AI#394

[zhanghm18GIT]

Engine-side implementation PR:
FAForever/FA-Binary-Patches#164