Skip to content

BETA.md

Latest commit

 

History

History
292 lines (254 loc) · 23.6 KB

BETA.md

File metadata and controls

292 lines (254 loc) · 23.6 KB
Raw

Android beta (LW-M7-06)

Owner: LW-M7-06. This file is written before the beta starts, as the task's first acceptance line requires. It fixes the entry criteria, the device spread, what is collected, the exit criteria, and holds the go/no-go decision at the end. The entry table records preparation evidence. §6 records tester results as they arrive; §7 holds the owner's stable-release decision (GO, 2026-10-04). The owner changed the original closed-beta channel to a public GitHub prerelease on 2026-09-08; see §3 and the publication record.

Programme ended (2026-10-04). The owner, Manuel Gysin, decided GO for the stable release on 2026-10-04: "we start now, testers are early adaptors". The 14-day device-slot programme below (§2-§4) did not run as written and is replaced by public early-adopter feedback after release, through the same Android bug-report form. The first stable release is Redoubt 157.0-2 (Firefox 157; "157 is our releae we want to go" (verbatim)), a new build from main with the in-app update check compiled in, not Beta 5's binary. §7 records the decision and, for every no-go item, what is evidenced and what was waived. The rest of this file is kept as written: it is the record of what the beta was meant to measure and did not.

Track note (2026-10-02): Betas 1-3 were built from Firefox ESR 153. Android has since moved to Firefox release 157.0 (TRACK.md); "the pinned ESR tarball" below describes those betas.

The beta exists for one reason above all others: the memory cost of site isolation (LW-M5-01) plus isolatedProcess (LW-M5-02) is invisible on a flagship and decisive on a budget phone. Every design choice here is weighted toward finding that out on hardware we do not own.

1. Entry criteria

The beta does not start until every line below is true and the evidence is linked. "Green on the maintainer's machine" counts only where the line says so.

# Criterion Evidence Status
E1 The build carries all landed Android patches (assets/patches/android.txt), compiled from the pinned ESR tarball by make android-apk TARGETS=android Candidate build log, source/input manifest and build-times.txt Met 2026-09-08. make android-apk TARGETS=android completed with R8 on (505 s); the isolated candidate source matches all 248 paths touched by the 50 common/Android patches applied to the pinned tarball. See evidence/lw-m6-08/candidate-build-times.txt and patch-source-comparison.json. Native AARs are the existing three-ABI inputs; the new patch changes Gradle tooling only.
E2 Both ARM ABIs are real Gecko builds; every ABI directory in the universal APK carries its engine Candidate APK entry list and ELF headers for each libxul.so Met 2026-09-08. Inspected all four candidate ZIPs and ELF headers: ARM32, ARM64 and x86_64 are real engines, every native ABI directory is expected, and the universal carries all three. evidence/lw-m6-08/candidate-native-abis.json binds this check to each APK hash.
E3 Smoke suite green on the emulator against the beta payload, except the two strict zero-Remote-Settings checks E12 deliberately leaves red Candidate audit Met 2026-09-08. Exact x86_64 candidate payload: baseline 7/7; search, branding/UI, suggestions OFF→ON→OFF, update privacy, release about:config edit/restart, no-GMS and no-Adjust checks pass. Baseline negative controls and 23 harness regression tests pass. Final runtime evidence. Update checks are compiled out; an enabled update path is untested. The two E12 exceptions remain red.
E4 Runtime pref audit exits 0, with expected-prefs.txt generated from the beta build and its diff reviewed Candidate pref dump, baseline comparison and audit output Met 2026-09-08. Ran the baseline generator against the final candidate: generator, reviewed diff and audit exit 0. All 58 baseline rows are unchanged (SHA-256 2ac8b4f9…). The audit reports zero violations and zero other differences. See final-candidate/pref-baseline-regeneration.json, archived before/generated baselines and pref-audit-final.out in the candidate audit. Of 137 must-lock keys, 20 are covered by this runtime dump; the generated lock/policy gate covers the rest.
E5 board.py --check-fenix-tests exits 0 on a full Fenix suite from the candidate's source Archived JUnit XML, source hashes and subtraction output Met 2026-09-08. Full patched-source run: 598 classes / 5,426 tests; 93 failures = 90 environmental + 3 known-real + 0 unexpected; subtraction exit 0. Final JUnit XML and exact source/candidate linkage are archived in the candidate audit (fenix-gate.txt, fenix-junit-xml.tar.gz, fenix-final-candidate.json).
E6 scripts/android-verify-repro.sh --r8 exits 0 for the candidate inputs and both outputs match the candidate Two independent build logs, hashes and negative control Met 2026-09-08. Corrected version and Glean timestamps; --r8 exits 0 after two independent builds (1,426 s total). All four outputs match each other and the normal candidate byte for byte; the negative control detects its one-byte corruption. Logs and comparison. Same-machine APK assembly only; Gecko/AAR and cross-machine reproducibility remain unverified.
E7 Signed with the release key, v2 + v3, no v1, fingerprint matches SIGNING.md; offline holder signing outside CI, or an explicit dated candidate-specific custody exception; the current runner cannot read the key Signing handoff, final signature/payload verification, dated owner decision Satisfied 2026-09-08 with the owner's beta-only custody exception. All four signed APKs in ~/redoubt-signed/ pass published fingerprint, v2+v3/no-v1 and exact candidate payload checks. Manuel Gysin explicitly approved accepting their pre-QEMU Fedora signing and keeping the key on Fedora outside the VM. The QEMU runner has no host home/key path; the old runner is retired. Historical offline signing is not claimed. Before this first publication, the owner confirmed that no release-key APK had previously been distributed.
E8 A second key holder exists, or the single-holder decision is recorded with a date in SIGNING.md SIGNING.md, “Decision: Redoubt ships single-holder” Met. Manuel Gysin's dated 2026-09-06 decision is recorded in commit aca09eb. This does not satisfy LW-M6-01's separate two-holder criterion.
E9 The parity wording (PARITY.md §5) is signed off PARITY.md §5, commit aca09eb Met. Owner approved the wording verbatim on 2026-09-06. Publication belongs to the public release.
E10 A triage owner and backup are named, or the dated solo-owner decision is recorded, and the Android bug-report form (LW-M7-04) is live TRIAGE.md §0, local form, live repository audit Met 2026-09-08. Following user approval, both form files are on default main at c65d2e4 and all eleven labels are verified live. GitHub’s public preview renders the parsed Android form, field controls, required markers and labels. The owner and solo arrangement remain recorded. Publication evidence. Authenticated submission validation remains the separate, unperformed LW-M7-04 manual check.
E11 First-run network capture uses a resolver that does not sinkhole Mozilla hosts Candidate pcap, DNS control and parsed events Met 2026-09-08. The final candidate was captured with emulator DNS 9.9.9.9; direct controls return public addresses for telemetry, ads and security hosts. The authoritative final-candidate/first-run-capture-final.json records 11 outbound transport events, including three complete security-host SNI names. Raw pcap and DNS controls are retained. These are transport observations, not decrypted requests or complete app-UID attribution.
E12 The Remote Settings allowlist has been decided for Android, and the beta carries that decision settings/android.cfg, evidence/lw-m4-08/RESULT.md, candidate pref dump Met 2026-09-08. The final candidate’s effective librewolf.services.settings.allowedCollections matches all seven approved entries in settings/android.cfg; the eleven local allowedCollectionsFromDump entries are unchanged. See the candidate audit’s final-candidate/prefs-all.json. The two strict zero-traffic checks remain red and retain their original assertions; hostnames alone cannot establish which encrypted collections were requested.

WITHDRAWN 2026-09-13: do not hand the 2026-09-08 candidate to testers. The twelve criteria were satisfied as written, and two defects got through anyway because no criterion asked about either. Both were found by inspecting the signed APK, not the sources:

  1. It ships Firefox branding. The home-screen wordmark is the full-colour Firefox flame with the word "Firefox" beside it; ic_firefox and four more in-app logos are Mozilla's; and eight user-selectable app icons include the classic 2004 Firefox logo. --check-strings passed honestly — the string layer is clean — but it has no view of images and nothing else did either. This is a trademark exposure, not a cosmetic one.
  2. It has no uBlock Origin. LW-M3-07 landed at 23:02 on 2026-09-08; the candidate was built at 16:13 the same day, seven hours earlier. The signed artifact predates the work. uBO is the most visible privacy feature LibreWolf desktop ships, and testers would have reported its absence as the headline.

Neither invalidates the E7 signing evidence: the four hashes are genuine, the custody exception stands, and the verification is reproducible. What is withdrawn is the candidate, not the process that signed it.

A replacement candidate carrying both fixes is being built. The criteria themselves gained nothing from this episode except a demonstration that "all twelve green" describes the checklist, not the product — so scripts/android-brand-check.py now exists, and E3 names it.

Audit reopened 2026-09-08. The previous “eleven of twelve met” summary was not supported for the final unsigned APKs. Their hashes and version codes differed from the old reproducibility and runtime evidence, and the bug form was never live on the default branch. The current audit binds each result to the actual candidate.

Key custody is part of E7, not an optional issue outside these criteria. Signature verification cannot establish where signing occurred or whether the runner can still read the key. The recorded owner exception and measured QEMU boundary address those separately. No release key or passphrase was used by this audit. The original pre-migration signing provenance remains recorded.

Preparing entry does not complete the 14-day beta. Device assignments, tester results, the second-build upgrade and the stable-release GO/NO-GO below remain required at their respective stages.

2. Device spread

The task requires at least one device with 4 GB of RAM or less and at least one running Android 10 or older. That is the floor, not the plan. Target:

slot why who holds it
1. ≤ 3 GB RAM, arm64, Android 10–12 the OOM case the whole beta is weighted toward _______
2. 4 GB RAM, arm64, Android 13+ the common budget phone _______
3. Android 9 or 10, any RAM oldest supported platform behaviour (WebAuthn is lost below 14 with no GMS — see no-gms.patch) _______
4. 32-bit ARM (armeabi-v7a) if any device can be found the ABI with the weakest hardening (hardening-flags.md: no -fstack-clash-protection) _______
5. flagship, arm64, current Android the control: if it fails here it is not a memory problem _______
6. one device on a network with no DNS filtering E11, and the only honest first-run count _______

Record the holder next to each slot and copy the table into TRIAGE.md §0, which asks for exactly this ("who holds which device").

3. Duration and channel

  • Length: 14 days from the first install. The public-launch rotation in TRIAGE.md §5 starts separately when the public download page goes live.
  • Channel: direct signed APKs on GitHub Releases. Since 2026-10-04 each beta is a full release marked Latest (see DISTRIBUTION.md), still titled "Beta"; Betas 1-4 were prereleases. Originally: a public GitHub prerelease. On 2026-09-08, after approving the exact E7 beta candidate, the owner requested adding its APKs to GitHub Releases. That request supersedes the earlier private-link-only plan for this beta. The release is clearly marked as a prerelease; it does not supply the later stable-release GO/NO-GO decision. F-Droid and Accrescent remain separate M6-03/M6-04 work. Published 2026-09-08: android-153.0esr-1-beta.1.
  • Update path test: at least one second beta build is shipped during the window, signed with the same key, and every tester installs it over the first. A beta that never exercises the upgrade path has not tested the one thing the signing key exists for.
  • The in-app update check (update-check.patch) is compiled out of beta builds unless the update-signing key and the endpoint exist by then; the row is absent, which is the store-build shape. If it is compiled in, E3's --check-update-privacy must have run against that exact build.

4. What is collected

Only what a reporter can actually provide (TRIAGE.md §3, item 4), and nothing that identifies a person:

  1. Device model, total RAM, Android version, ABI installed (from Settings > About or adb shell getprop).
  2. Launch: cold start time to first paint of the home screen (stopwatch is fine); whether it launched at all on the first three tries.
  3. Memory: whether the app was killed in the background during ordinary use; adb shell dumpsys meminfo org.redoubtbrowser once with five tabs open, if the tester can run adb. This is the number the beta is for.
  4. WebGL: https://get.webgl.org loads a spinning cube. This is landmine L1's signature and must be checked on every device, every build.
  5. Privacy posture, from the UI: Settings > Search lists DuckDuckGo (default), Startpage, Mojeek, Wikipedia — with icons; "Show search suggestions" is off; Settings > About shows no "Check for updates" row (compiled out) or shows it off (compiled in); about:config is reachable in the release build; privacy locks remain enforced.
  6. Sites that break with RFP/ETP strict, listed by URL; whether stock Firefox for Android breaks them too.
  7. Anything that says "Firefox" or "Mozilla" in the UI, with a screenshot — --check-strings covers the resource table and the deep-linked settings screens, not every dialog.
  8. Battery: subjective only, unless the tester volunteers dumpsys batterystats.

How: one issue per finding through the Android bug-report form (LW-M7-04) with the beta label. The form must be live before the beta starts (E10). If it becomes unavailable during the beta, testers may provide summary messages at day 7 and day 14; this fallback does not waive E10. adb logcat is asked for only on crashes, with the warning that it contains visited URLs.

5. Exit criteria — go / no-go for the stable release

No-go on any one of these:

  • N1. A crash on launch, or an OOM kill within the first minute of ordinary use, on any device in slots 1–3 that is not explained and fixed by a second build inside the window.
  • N2. Any outbound connection in a first-run capture (E11) to a Mozilla telemetry, experiments, ads, or crash-reporting host, or to any Google host from the app itself. Remote Settings sync outside E12's approved Android allowlist is also a no-go. The seven entries in settings/android.cfg are the recorded exception for security updates; log their traffic and verify the effective allowlist. A Mozilla hostname alone does not prove the traffic is approved, and the strict zero-Remote-Settings harness checks remain unchanged.
  • N3. WebGL broken on any device (L1).
  • N4. A search from the toolbar carrying a partner or attribution code, or going to an engine that is not in assets/search-config-v2.json.
  • N5. The second beta build failing to install over the first on any device (signing-key or applicationId mismatch). This one ends the beta on the spot.
  • N6. A privacy pref on must-lock.txt found unlocked or moved on a tester's device (--pref-dump from a tester with adb, diffed against the baseline).

Go requires all of:

  • G1. Every slot 1–3 device completed the 14 days with the app usable for daily browsing, in the tester's own words.
  • G2. Every N-item above checked and recorded as not triggered, with the evidence linked — an unchecked item is a no-go, not a pass.
  • G3. Open beta issues triaged; anything left open is labelled Status Known issue and appears in the release notes.
  • G4. The download page (LW-M7-02) carries the parity wording verbatim and the fingerprint from SIGNING.md.

6. Results log

date build (commit, MOZ_BUILD_DATE) slot finding issue

7. Go / no-go

Recorded by the agent on the owner's instruction of 2026-10-04 (the owner's words, verbatim: "1. we start now, testers are early adaptors 2. 157 is our releae we want to go. 3. like said, testers are early adapotrs. 4. done"). The decision is the owner's; the evidence lines below are the agent's reading of the repository and say plainly where real-device evidence does not exist.

DECISION:     GO   (owner decision, early-adopter release model)
DATE:         2026-10-04
SIGNED BY:    Manuel Gysin (maintainer) -- recorded by the agent on the owner's instruction
BUILD:        Redoubt 157.0-2, commit 27240eb6d0140704f637135c7985c778a9f46e51,
              MOZ_BUILD_DATE 20261005000000 (CI run 37248744119; accepted on the
              emulator 2026-10-05, evidence/lw-m7-01/release-157.0-2/acceptance/
              run-37248744119/)
NO-GO ITEMS:  see below; every item was checked on the emulator only, none on a
              tester device. What a tester device would have added is waived by
              the owner's early-adopter decision, not shown to be absent.

The evidence is for the 157.0-1 builds that preceded 157.0-2: the device acceptance of rc3 (6202ee6d, Beta 4's source, evidence/lw-m7-01/release-157.0/acceptance/) and the regression smoke of the Beta 5 candidate (cdeadd6c, evidence/lw-m7-41/migration/). All on one android-30 x86_64 emulator (no GMS, SwANGLE); arm64 and armeabi-v7a APKs were checked statically only. 157.0-2 itself is not built yet; the release procedure (DISTRIBUTION.md, "Stable release") repeats the smoke on its exact payload.

  • N1 (crash/OOM on slots 1-3) -- not evidenced on real devices; waived. On the emulator: --check-launcher-start PASS on rc3 and on the Beta 5 candidate (fresh profile and restart, no setup-failure dialog after 45 s); no crash in either acceptance. There is no measurement on a low-RAM phone (slot 1, <= 3 GB) or on Android 9/10 (slot 3). The site-isolation + isolatedProcess memory cost this beta existed to measure is therefore unmeasured; the owner accepts that early adopters will report it.

  • N2 (first-run connections) -- not triggered on the emulator. --first-run-capture with DNS 9.9.9.9: rc3 108 events, Beta 5 candidate 124 events (acceptance/rc3/smoke/first-run-capture/, lw-m7-41/migration/smoke/first-run-capture/, host comparisons in acceptance/rc3/first-run-host-comparison.json and lw-m7-41/migration/smoke/first-run-host-comparison.json). No named Mozilla telemetry, experiments, ads or crash-reporting host and no named Google host. Named Mozilla hosts are Remote Settings (firefox.settings.services.mozilla.com, content-signature-2.cdn.mozilla.net, firefox-settings-attachments.cdn.mozilla.net, E12's recorded exception; the two strict zero-Remote-Settings checks are red as designed) and AMO (services.addons.mozilla.org, versioncheck-bg.addons.mozilla.org, uBO's add-on). The other named hosts are uBO list mirrors and publicsuffix.org. Limits: 23 unnamed destinations (22 IPv6, plus the emulator DNS 10.0.2.3) are transport observations, not attributed per host; the effective allowedCollections was last read from a runtime pref dump on the 2026-09-08 candidate (E12), not on 157; no capture on a tester's network.

  • N3 (WebGL) -- not triggered on the emulator; real GPUs not evidenced, waived. Full graphics acceptance 161/161 twice on rc3 (acceptance/rc3/smoke/baseline-smoke/, static-no-gms-no-adjust/) and once on the Beta 5 candidate (lw-m7-41/migration/smoke/baseline-smoke/graphics-acceptance/), after the rc1 WebGL-permissions defect was fixed. The emulator renders through SwANGLE; no physical GPU/driver was tested.

  • N4 (search partner codes / engines) -- not triggered. --check-search PASS on rc3: the query went to noai.duckduckgo.com with no partner parameter; 4 engines, default DuckDuckGo No-AI (acceptance/rc3/smoke/check-search/). --check-no-suggest PASS on rc3 (its negative control red, as it should be) and on the Beta 5 candidate. --check-search was not re-run on the Beta 5 candidate (it changed no Kotlin).

  • N5 (second build over the first) -- not triggered on the emulator; real devices not evidenced. Real adb install -r upgrades kept the profile and signer: Beta 3 -> rc3 (acceptance/rc3/upgrade/; versionCode 2016187942 -> 2016188078, settings, bookmark, DoH and uBO selection kept) and Beta 4 rc3 -> Beta 5 candidate (lw-m7-41/migration/device/s1/, s2/). These used the throwaway test key on both sides, not the release key; the release-key path is covered by Betas 4 and 5 being published with the same key (fingerprint-verified downloads, evidence/lw-m7-01/release-157.0/beta5/downloaded-verification.txt). No tester reported installing one beta over another. 157.0-2's versionCodes must exceed Beta 5's (2016188256-63); the release build pins build_date for that.

  • N6 (must-lock prefs on a tester device) -- emulator only; tester dump waived. rc3 pref audit: generator 0-line diff against the committed baseline, audit 0 violations, 0 notes; 20 of 132 must-lock keys are in the harness dump universe, the other 112 are enforced by the generated locks (acceptance/rc3/pref-audit/). --check-aboutconfig: 52 prefs locked, on rc3 and the Beta 5 candidate. No --pref-dump from a tester's device exists.

  • G1 -- waived by the owner: no slot 1-3 device completed 14 days; early adopters take the testers' place after release.

  • G2 -- per the N-items above: each was checked on the emulator and recorded as not triggered there; the real-device parts of N1, N3, N5 and N6 are not evidenced and are waived by the owner's decision, not passed.

  • G3 -- met trivially: no beta issues were filed, so none is open or needs a Status Known issue label.

  • G4 -- met (checked 2026-10-04 on site/index.html): the parity sentence from PARITY.md §5 appears verbatim, with the link to the PARITY.md table, and the SHA-256 fingerprint 64:14:EB:33:...:28:3B:D0 matches SIGNING.md (also on site/install.html).

The parity limits stand as published: the decision changes how the release is tested, not what PARITY.md says Redoubt does and does not protect.