[Meta] Build & sync reliability: root cause of the stall, reproducible build, and the path to release

[melvincarvalho]

Purpose

Tracking issue for the build/sync modernization effort. It consolidates the root cause of the long-standing "won't compile / won't sync past the fork" problems, the fixes (already written as PRs), the release strategy, and live consensus-validation results. It cross-links the cluster of related issues so anyone landing on one of them finds the plan.

TL;DR

• The infamous sync stall at block 450,947 is not a data/network problem — it's a toolchain miscompile. Modern GCC mis-optimizes the LYRA2REv2 / BMW256 hash, so a source-built node computes a wrong PoW hash at the multi-algorithm fork and dead-ends.
• There are also build breakages (Boost 1.74+, Berkeley DB 4.8, blake2 alignment) and an OpenSSL 3.x signature-parsing break (stall at 605,359).
• All of these already have fixes in open PRs: Fix build on Ubuntu 22.04 and 24.04 (GCC 11+, Boost 1.74+) #138, Fix GCC 13+ optimization bug in BMW256 for LYRA2REv2 hashing #142, Fix OpenSSL 3.x signature verification failure #146 (Fix GCC 13+ build error: blake2.h alignment constraints #144 is superseded by Fix build on Ubuntu 22.04 and 24.04 (GCC 11+, Boost 1.74+) #138).
• Plan: pin a single reproducible Docker build on Ubuntu 24.04 LTS, consensus-validate it by full reindex, then release. 26.04 / Boost 1.90 native is a separate later track.

Root cause of the stall (the important part)

Same source, different compiler → different consensus result. Specifically, GCC 13+ at -O2 miscompiles BMW256 (part of the LYRA2REv2 hash chain), producing wrong PoW hashes for LYRA2REv2 blocks. The node then rejects valid blocks at the multi-algorithm activation (block 450,947) and stalls.

Community confirmation: builds work with default flags but break with -Ofast -march=native, which overrides the protective per-file -O1.

⚠️ Build with default optimization. Do NOT use -Ofast or -march=native — they re-break the LYRA2REv2 hash and reintroduce the stall.

The fixes (open PRs)

PR	Fixes	Notes
#138	Ubuntu 22.04/24.04 build (Boost 1.74, BDB, blake2, placeholders)	clean; supersedes #144
#142	GCC-13 BMW256/LYRA2REv2 miscompile (-O1 on the hash files)	the stall fix; field-proven ~1 year
#146	OpenSSL 3.x lax DER signature parsing (stall at 605,359)	field-proven; bespoke parser to be hardened
#144	blake2 alignment	redundant — already in #138; can be closed

Strategy

1. Pin a reproducible build on Ubuntu 24.04 LTS. A settled toolchain whose consensus-affecting bugs are already fixed; the resulting binary also runs on 22.04 / 26.04. Newer toolchains carry higher consensus-miscompile risk for this old code, so we deliberately do not chase the newest OS for the canonical build.
2. Consensus-validate every build by full reindex against the existing chain — building is not the same as correct.
3. Release prebuilt binaries / AppImage so users need not compile — follow-up.
4. 26.04 / Boost 1.90 native — separate experimental track (Boost 1.90 removed copy_option and asio resolver::query; needs source modernization).

Validation status (live)

Reindexing the chain with a clean 24.04 build of #138 + #142 + #146:

• Crossed block 450,947 (LYRA2REv2 activation) clean — the historical stall point — and kept going.
• Zero validation errors so far (currently ~500k and climbing).
• Continuing to tip for full structural confirmation.

Phased plan

• Phase 1 — match the network: reproducible 24.04 build, reindex-validated. (in progress — passing)
• Phase 2 — polish: add a startup PoW known-answer self-test (catches any future miscompile, any algorithm, regardless of compiler); replace Fix OpenSSL 3.x signature verification failure #146's bespoke DER parser with Bitcoin Core's audited ecdsa_signature_parse_der_lax; split/clean the PRs.
• Phase 3 — 26.04 native + tagged release with binaries.

Related issues

• Build: Build fails on Ubuntu 22.04+ (GCC 11+, Boost 1.74) #137, GCC 13+ build failure: blake2.h alignment error #143, Compilation error (std::placeholders) on Ubuntu 20+ #125, Ubuntu 22 compatibility #131, make error on ubuntu 22.04 #123, Cant make on Ubuntu 20.04 #102, unable to compile and run 0.9.7.4 on ubuntu 20.04 #107, Does not compile under Debian 8 / 9 (amd64) #48, cryptonight on 32 bit machines #127, Lyra2v2 segfault on yiimp #67
• Sync stall / consensus: Source-built binaries fail to sync past block 450,947 (Fork #1) #140, GCC 13 optimization bug causes LYRA2REv2 hash failures - fix identified #141, bitmarkd compiled from source doesn't sync after block 450627 #54, OpenSSL 3.x signature verification fails at block 605,359 #145, The network does not appear to fully agree #116, Linux wallet: error: The network does not appear to fully agree! Some miners appear to be experiencing issies #66, inconsistent block emission #124, Total emission calculation #111
• Dependencies: usage of openssl #126, secp256k1 for ECDSA signature validation #132 (secp256k1 — relevant to hardening Fix OpenSSL 3.x signature verification failure #146)
• Related work: Electrum Wallet for Bitmark - Staged Implementation #139 (Electrum wallet — built), PR Fix for incorrect equihash and cryptonight block headers response #133 (block headers), Documentation #130 (docs), RPC commands #128 (rpc)

Checklist

• Review and merge Fix build on Ubuntu 22.04 and 24.04 (GCC 11+, Boost 1.74+) #138, Fix GCC 13+ optimization bug in BMW256 for LYRA2REv2 hashing #142, Fix OpenSSL 3.x signature verification failure #146 (in order); close Fix GCC 13+ build error: blake2.h alignment constraints #144 as superseded
• Land the pinned 24.04 Dockerfile.build + a clear "do not use -Ofast/-march=native" note
• Complete full reindex validation to tip
• Add the PoW self-test (Phase 2)
• Harden the Fix OpenSSL 3.x signature verification failure #146 DER parser (Phase 2)
• Tagged release + prebuilt binaries
• 26.04 / Boost 1.90 native (separate track)

[melvincarvalho]

How we are going to test this (and why it will be slow on purpose)

A note on process, because this is consensus-critical code and the stakes deserve it.

The asymmetry we are optimizing for. A wrong bit in a hash, a difficulty rule, or signature parsing does not "crash" — it silently forks the network, and you cannot cleanly hotfix a chain split. The downside is catastrophic and irreversible; the upside of rushing is a few saved weeks on a chain that has already waited years. So we optimize entirely for being certain, not for being fast.

What we are actually doing (and why that is safer than it sounds). We are not changing consensus rules. The goal is for the code to compute the same answers under modern toolchains that the field-proven binary already computes. That turns "did we break consensus?" into a testable question: "does our build disagree with the live network anywhere?" — and the live chain is the oracle.

Definition of done for any consensus-affecting change

Nothing merges or releases until it clears all of these:

1. One isolated change at a time. No bundling of consensus-affecting changes. Each is reviewed and validated alone.
2. Full reindex equivalence. A clean reindex of the entire chain, matching the network's tip — including a run with checkpoints disabled so signature validation is actually re-exercised (a normal reindex skips signatures below checkpoints).
3. Runtime self-test. A startup proof-of-work known-answer test for all 8 algorithms, so a future compiler miscompile becomes a loud refusal-to-start instead of a silent fork. (This is the permanent guard against exactly the -Ofast/-march=native failure mode.)
4. Differential oracle. We now have three independent implementations that parse Bitmark blocks (the C++ daemon, an ElectrumX deserializer, and a dependency-free JS walker). They must agree; disagreement is a free tripwire.
5. Shadow soak. Run the candidate binary as a non-mining node alongside a known-good node for weeks, comparing tips block-by-block on live data — not just a one-shot resync. Time itself is a test.
6. Reproducibility. Two machines build it and produce binaries that agree on the same chain. The unreproducible legacy binary is retired only once a clean, reproducible one has out-validated it.

Every passing phase compounds confidence on the previous one. We never assume; each finding is re-checked by the next.

Rough timeline (phase-ordered, not date-ordered)

Deliberately no hard deadlines — each phase ships when it is proven, not when it is scheduled.

• Phase 1 — match the network. Reproducible Ubuntu 24.04 build of the existing fixes, validated by full reindex. In progress and currently passing — it has already re-validated the historical stall point (block 450,947 / LYRA2REv2) with zero errors.
• Phase 2 — close the validation gaps. Reindex with checkpoints disabled (validate the OpenSSL 3.x signature path at 605,359); add the proof-of-work self-test; cross-check against the differential oracle.
• Phase 3 — harden. Replace the bespoke signature parser with audited logic where warranted; split and clean the PRs; shadow-soak the candidate.
• Phase 4 — release. Tagged release plus reproducible prebuilt binaries so nobody has to compile. 26.04 / Boost 1.90 native is a separate, later track.

Expect this to play out over weeks, with small, well-tested patches landing incrementally rather than one big change.

If you are blocked right now

You do not have to wait for the release. The fixes exist in PRs #138, #142, #146 — build from those with default compiler flags. Do not use -Ofast or -march=native: they re-introduce the LYRA2REv2 miscompile and the stall returns. If you would rather not compile, say so and we can share a binary.

[melvincarvalho]

✅ experimental branch is up — reproducible build, validated against the chain

We've pushed an experimental branch: the first reproducible, consensus-validated build of Bitmark for modern Linux. It bundles the existing fixes (#138, #142, #146) into a pinned Docker build so anyone can reproduce it — replacing the old unreproducible -dirty binary.

Build it:

git clone -b experimental https://github.com/project-bitmark/bitmark.git
cd bitmark && ./build-docker.sh        # → ./bitmarkd, ./bitmark-cli

What it does / why it's pinned to 24.04:

• One-command containerised build on Ubuntu 24.04 LTS — a settled GCC 13 / Boost 1.83 toolchain whose consensus-affecting miscompiles are already fixed in-tree.
• Avoids the modern-toolchain bugs that changed Bitmark's consensus results before: GCC-13 BMW256/LYRA2REv2 (stall at 450,947) and OpenSSL 3.x DER (stall at 605,359).

Validation (the important part):

• Full reindex genesis → tip (2,399,930), zero errors.
• The reindexed tip hash matches the live network exactly, as do both fork-block hashes — so every block beneath the tip is identical to the network's.

Platforms — good news: builds and runs on Ubuntu 24.04, and the same binary also runs on 26.04 (with the Boost 1.83 runtime libraries supplied alongside via LD_LIBRARY_PATH). A fully static, dependency-free binary is a later release item.

⚠️ Build with default flags. Do NOT use -Ofast / -march=native — they re-introduce the LYRA2REv2 miscompile and the stall returns.

Release ladder. Deliberately slow and paranoid — this is consensus code. We'll move through:

experimental → prealpha → alpha → beta → rc → final

Each rung promotes only when proven, not when scheduled. experimental is where we are now: builds reproducibly and matches the chain. Next rungs add the checkpoints-disabled signature re-validation, the PoW known-answer self-test, the hardened #146 parser, and a static release binary — full plan above in this issue.