Devel

[pbuncic]

bits: concurrent-build scheduling, graceful CPU prioritisation, repository providers, recipe/provenance features, and a docs overhaul

Summary

This branch brings bits from serial, single-recipe-repo builds to a parallel, resource-aware build system with pluggable recipe repositories, richer recipe/provenance features, and a rewritten documentation set.

44 commits; ~5k lines added across bits_helpers/ with matching test coverage.

Highlights

1. Concurrent build scheduling and resource awareness (--builders)

• --builders N builds N packages at once through the built-in scheduler, respecting the dependency graph.
• Builder-aware $JOBS: each package's parallel-compile budget is divided across the builders (effective_jobs) so the concurrent jobs together never oversubscribe the machine's CPU/load.
• Memory budgeting: recipes can declare mem_per_job (peak RSS per compile process) and mem_utilisation; the scheduler's ResourceManager admits builds so total memory stays within budget.
• Self-tuning stats: per-package CPU/RSS traces are recorded (--resource-monitoring) and auto-loaded on the next run to schedule with up-to-date estimates.
• Download/build overlap: source checkouts run as scheduler download tasks that overlap with compilation; prefetch is on by default and --parallel-sources fetches multiple sources: URLs concurrently.

2. Graceful CPU prioritisation: --build-nice ladder + straggler watchdog

• --build-nice (on by default for --builders > 1; opt out with --no-build-nice) staggers concurrent builds across OS priority: one runs at top priority (full speed), the rest are progressively backed off, and the freed top slot is taken over as builds finish — so contention degrades gracefully instead of all builds fighting equally. Native builds use nice; --docker/podman builds use the cgroup-equivalent --cpu-shares.
• Straggler-renice watchdog: a long single compile (e.g. a heavy Fortran/C++ translation unit) that was backed off can drag out the end of a build. A watchdog boosts the longest-running straggler back to top priority, one at a time, after --build-nice-boost-after seconds (default 600):
  • Native builds: the straggler's process subtree is reniced toward 0.
  • --docker builds: each build runs in a named container; the watchdog finds the long-running compiler back-end with docker exec … ps and renices it via docker exec --user 0 … renice (run as root inside the container, so it can raise priority without host privilege). If the image lacks ps, it warns once and disables that path — see operational notes.

3. Scheduler robustness

• Fixed an end-of-build hang: a build task raising SystemExit (as dieOnError does) was not caught by the worker's except Exception, killing the worker silently and leaving the job stuck "running" forever so the build never finished and printed no summary. Workers now catch BaseException, and the master loop has a bounded wait + watchdog that fails stuck jobs if all workers exit — guaranteeing the build always finishes with a summary.

4. Build-error interception and diagnostics

• Real exit code propagated: the post-build status check read $PIPESTATUS after the test had clobbered it, so every failure exited 1. It now captures ${PIPESTATUS[0]} first, preserving the real failing code at all three recipe-run sites.
• Error excerpt in failure messages: BUILD FAILED now includes the matched high-signal error lines (compiler/linker/cmake/make/python) plus the log tail, so triage no longer means opening and grepping the full log.

5. Repository providers

• A recipe marked provides_repository: true pulls an entire additional recipe repository from git at dependency-resolution time and adds it to BITS_PATH, enabling modular and nested recipe sets. always_load: true clones it unconditionally at startup; repository_position and --provider-policy control search order (with prepend downgraded unless explicitly granted, to prevent recipe-controlled PATH hijacking). The provider's commit hash folds into every dependent's build hash for reproducibility.
• Bootstrap: bits build PKG with no local recipe dir clones bits-providers (default or --organisation <org>) and the recipe repo it points to. Provider repo version is selectable via the recipe's tag: or an @tag suffix on the providers URL.

6. Recipe authoring and defaults-profile features

• Defaults-profile variables in recipe bodies: %(VAR)s-style placeholders resolved from the active defaults profile, with soft expansion so unknown placeholders don't hard-fail.
• Version pinning: pin a dependency's version from a recipe's requires (name = version, with optional :matcher), in addition to the existing defaults-*.sh overrides: mechanism (which can also set tag/source).
• %(name)s / %(version)s substitution in sources: URLs and patch names.
• Generic build-time PKG_CONFIG_PATH so the build environment mirrors each package's runtime modulefile; shared arch-detection helper; per-default append_arch.

7. Source checkout and patching (workarea)

• Auto-apply patches declared in a recipe's patches: after source preparation (patch --batch to avoid interactive reversal).
• Correct re-extraction: sources are re-extracted when the patch set changes or a previous patch run left a dirty tree / stale sentinel; fixes for strip-level and single-file-archive prefix-depth (--strip-components) bugs.

8. Provenance and integrity

• Manifest schema v3 records the resolved sources, patches, and expanded variables for each package; builds can be replayed with --from-manifest.
• Source checksums: --check/enforce/print/write-checksums plus per-profile/per-recipe checksum policy; store integrity verification of recalled tarballs.

9. Sandbox and miscellaneous robustness

• Don't probe/require podman for local Linux builds; clean, actionable error messages replacing raw crashes; ==> Patching PKG@VERSION progress; download OSError interception; guard against accidental deletes.

10. Documentation

• Rewritten and consolidated docs under docs/: USERGUIDE, REFERENCE, COOKBOOK, WORKFLOWS, ROADMAP; obsolete mkdocs site and stale pages removed. New material covers --builders/--build-nice, the three version-pinning mechanisms, repository providers, and recipe error-handling pitfalls.

New CLI flags (selected)

Flag	Purpose
--builders N	Build N packages concurrently.
--build-nice / --no-build-nice	Staggered-priority ladder for concurrent builds. On by default for --builders > 1.
--build-nice-step N	Nice increment between ladder slots.
--build-nice-boost-after SECONDS	Boost a straggler compile after this long (0 disables; default 600).
--parallel-sources N	Concurrent sources: downloads per checkout.
--prefetch-workers N	Background tarball prefetch threads.
--resource-monitoring / --resources FILE	Record / consume per-package resource stats.
--from-manifest FILE	Replay a build from a manifest.
--provider-policy	Control provider BITS_PATH insertion order.

New recipe / defaults fields (selected)

mem_per_job, mem_utilisation, provides_repository, always_load, repository_position; requires: name = version[:matcher] pins; overrides: may set version/tag/source; %(name)s/%(version)s/profile %(VAR)s expansion in sources, patches, and recipe bodies.

Backward compatibility

• --builders defaults to 1 (serial), preserving prior behaviour; --build-nice only engages for --builders > 1.
• Existing recipes and defaults files are unaffected; all new recipe/defaults fields are optional.

Operational notes

• --docker builds require ps (the procps package) in the build image so the watchdog can find the in-container compile; if absent, it warns once and disables in-container renicing for the run (the build is unaffected). Build images used with bits build --docker should include procps.

Testing

• New/updated unit tests: test_scheduler.py (incl. the SystemExit-hang regression), test_nice_ladder.py (ladder, native renice, docker exec renice, ps-missing fallback), test_memory.py/test_build_stats.py (builder-aware jobs, resource stats), test_repo_provider.py/test_always_on_providers.py/test_provider_staleness.py, test_manifest.py (schema v3), test_workarea.py (patching/re-extraction), test_utilities.py (version pins, variable expansion).
• Full suite passes. The only environment-dependent items: test_sync.py S3 tests require the optional botocore dependency, and test_build.py::test_coverDoBuild is a slow, fully-mocked coverage test independent of these changes.

[akritkbehera]

Is there a way to make sure certain packages are present first without adding it as an dependency? It shouldn't change the hash if it changes.

[akritkbehera]

@pbuncic How can I test this seems like a useful thing to have but unable to use it.

requires:
 - cuda(?cuda)
BASH
And I need to define a variable cuda in defaults? Is that it? 

[pbuncic]

Yes, have a look into stacks.bits

[akritkbehera]

Ah ok ig there is an issue with it this works

package: defaults-release
version: vBits
variables:
  gcc: 'true'
  gcc-compiler: 'true'
---

package: zlib
version: 1.2.3
requires:
 - "gcc:(?gcc)"
 - "gcc-compiler:(?gcc-compiler)"
 ---

For gcc it works perfectly fine but if the name is gcc-compiler. I think '-' is causing the issue? Maybe?

raise source.error("unknown extension ?" + char,
re.error: unknown extension ?l at position 1

[pbuncic]

Yes — names must be C-style identifiers, so a hyphen isn't allowed. Use underscores instead (gcc_compiler, not gcc-compiler).

[pbuncic]

Besides, I am about to merge devel into main, hope it is OK with you.

[akritkbehera]

Before you merge it one change would be

==> Packages will be built in the following order:
     - db6@6.2.32
     - zlib@v%(version)s
     - gmake@4.3
     - libffi@v3.5.2
     - expat@R_2_7_1
     - sqlite@version-3.48.0
     - autotools@1.5
     - bz2lib@1.0.8
     - gdbm@1.26
     - libuuid@2.40
     - xz@v%(version)s
     - Python@v%(version)s
     - cuda@12.9.1

version is not replaced in this display we can maybe just call resolve_spec_data here to replace version. So the print shows it.

[pbuncic]

Sorry, @akritkbehera, I didn't see your comment before merging. I do not see such a problem in my builds. I normally derive the version from the tag, like:

package: ROOT
description: CERN ROOT data analysis framework
version: "%(tag_basename)s"
tag: "v6-38-00"

Where do you set the version in your recipe?

[pbuncic]

Is there a way to make sure certain packages are present first without adding it as an dependency? It shouldn't change the hash if it changes.

This I do not understand, can you clarify?